AI research needs a drastic inversion of its epistemology
“Meaning” does not arise from features, but from motives
This is the twenty-second post in a series on AGI. You can read the previous post here. You can also see a list of all posts here.
(Note: This post is a continuation of the previous two posts — part 1 and part 2 — which discuss why autonomous, i.e. unsupervised, identification using AI is so difficult, and then discuss a possible solution.)
There are an infinite number of “things” around you that you can learn to identify or name. It makes sense to focus on those that are important for achieving your immediate goals. This tendency is readily observable in humans: infants learn the names for things they care about quite quickly (e.g. “ice cream”), sometimes on the basis of a single experience. The same is true of adults; the more invested you are in the topic, the more readily the concepts stick.
Indeed you only start to engage with a domain, and learn the meaning of its relevant concepts, when something about it becomes a problem for you. As you start to experience financial trouble you become intimately acquainted with concepts like budget and compound interest. After your first experience of social humiliation, you start to become aware of your appearance, popularity, etc. The sudden introduction of a new type of concern in your life coincides with the arrival both of a new perspective, as well as new concepts by which you now start to categorize your experiences¹. This suggests that what you learn to identify, and even why and how you do so, is motivated by your underlying goals. Identification can then be framed as the act of searching for and finding the “useful identity” of an experience.
Even standard models of categorization assume that learners selectively attend to features that are diagnostic… In that sense, similarity (in the sense of overlap in features) is the byproduct, not the cause, of category learning. — Oxford Handbook of Thinking and Reasoning
Reversing the order of identification
This leads to a crucial and counter-intuitive observation. In an earlier post we suggested that identifying objects and events is a necessary step for recognizing problems that involve those entities, and ultimately for solving them. E.g. you can only recognize that going over your budget means you are losing money (a problem) if you can first recognize budget. What we’re starting to see is that the problem actually comes first, and only then can identification take place in its context — i.e. you only begin to recognize budget because you find yourself losing money.
As we’ll see in this post, you cannot identify anything, anything at all, without a problem context in which this “identity” is born and lives. Just as it doesn’t make sense to identify things using words outside of problems of communication, identifying entities to achieve a goal doesn’t make sense outside that goal. Every act of identification must be tied to a problem you are addressing. We already discussed a specific example of this reversal in another post, to explain how the mind understands numbers. Now we’re starting to consider it in the general case.
Should a painting of a cookie be identified as a “cookie”? Your answer will change depending on your current goals. You can represent the same entity as a “cookie” or a “painting” depending on what you are trying to do. In fact you can identify anything as anything as long as a motive is present in your mind that is satisfied by the identification — e.g. you can identify a flower as a “miracle”, or a person as “evil” (see footnote 2 regarding the relationship between identification and interpretation). Some religions even identify the “self” and the “universe” as the same thing. The key driver of any identification is the underlying motive that is benefited by it.
The motivating context is the reason why a given act of identification matters. It not only drives the effort and focuses attention, but also determines what the correct way to apply the category is with respect to its goals (see “cookie” and “painting” above). This makes it vitally necessary for addressing ambiguous cases.
Motives resolve ambiguities in identification
One of the biggest challenges of identification, as mentioned in an earlier post, is the problem of resolving ambiguous cases, such as “where does a mountain end and the valley begin?” This riddle and others like it register as problems when you first hear them, because you suddenly find that you are no longer sure how to use the terms correctly³. The objective definition on which you relied is no help. Nor can you look at the mountain itself and its objective features to get an answer; reality can’t resolve this problem for you. It is up to you to define, or redefine, the truth of the matter, so that you can continue to use the terms confidently. This is the first clue that definitions are not automatically determined by statistical correlations of features, but must incorporate personal utility.
The problem of ambiguity is not a secondary difficulty added on top of the basic act of identification. It is a part of learning to identify from the start. As long as the stimuli are very similar in appearance, identification can be done automatically. But as appearances diverge from what you are familiar with, each new case that you identify becomes a case of resolving “ambiguity” — e.g. “should this sight be called a mountain?” This is how motives enter into the picture. In some cases the answer may come easily through surrounding cues; in others a bit more effort, searching, and thinking is required. In some cases there is no clear resolution.
You may have heard that a banana is technically a berry. You may also have thought “but it doesn’t feel like a berry”. This hints at a deeper authority than the scientific one for deciding how something should be categorized. Let’s take a moment then, and instead of trying to find objective definitions of the words “mountain” and “valley” based on features, let’s look inwards at the gut feeling that drives the distinction between them.
The feeling of a concept, such as the feeling of “grandeur” associated with mountains, is usually downplayed as a personal, idiosyncratic reaction to an otherwise objective concept. Most people assume that the feeling is therefore not relevant to the objective definition. Rather you must first understand the object or concept, and only afterwards can you develop personal feelings about it. However, as we just saw, motives are also a critical part of identifying what an object is, and so they must be present from the start. So how can we resolve this circularity?
What if we reverse our interpretation? Imagine instead that this feeling actually points to the motives that originally created the two concepts in your mind, and thus gave the words a foothold in your psyche. Coming at it from this side, you could say that a “mountain” feels like the experience when, from below, you imagine it is difficult to climb up. It ‘rises up’ impressively and presents you with a challenge. A valley feels like an easier downward traversal into an ensconced space, with the accompanying sense of anxiety. So the same part of the landscape may be designated as either of the two depending on which frame of mind you are in. Bringing in the motivational context helps resolve the problem of ambiguity; in fact it is the only way you can resolve it.
Often we categorize some entity in order to help us accomplish some function or goal…. the category representations people develop in laboratory studies depend on use and that use affects later categorization — Oxford Handbook of Thinking and Reasoning
This is not to suggest that there is only one meaning of “mountain” and only one problem context through which it can take root in your mind. The singular nature of the concept of “mountain” is created by the existence of just one English word. You could just as easily divide mountain into multiple concepts like mountain-looking-from-below, mountain-when-at-the-top, etc. All it takes to establish a set of thoughts as a concept is to simply choose to identify them as one; and even here, the reason (the motive) why you choose to do so matters to the result.
This approach shifts the “human” meaning of a word from it’s clinical features into the underlying needs that ‘motivate’ its presence in your mind; the ones that drew it there in the first place. Everyday experience shows that the meaning of a concept or word is fluid and ever-evolving. Some common sense is required to understand and use it effectively. The only way to keep meaning flexible and adaptable is for the mind to want to resolve ambiguities, especially where communication breaks down, and to see in each new moment of identification an opportunity for redefinition. This is why statistical methods of object recognition are often accused of lacking a true understanding of the “meaning” of the things identified. Without the driving motives or goals (e.g. successfully climbing up a mountain) the agent has no grounding by which to resolve edge cases.
In fact, when it comes to words, the reason you use a word is its meaning. We can explain this better by looking at words from a non-literal angle — i.e. as metaphors.
Metaphors show how meaning is created
It is curious that people find it easier and more intuitive to use words in a metaphorical sense than to come up with their clear, feature-based definitions. Most people, for example, would have great difficulty providing an objective definition for the word “space” (physical space). Yet those same people would find it easy to use it in phrases like “the space of solutions to this problem” or “give him some space”. It’s relatively easy to connect space to personal problems and solutions; like arranging objects in it (vis. “the space of solutions”), or allowing you room to move without constraints (vis. “give him some space”). This should give us a clue as to which of the two ways of using words — metaphorical vs literal — is the more natural way the mind organizes its experiences.
Popular theories that try to explain metaphorical thinking rely on comparing common features between the two parts of the metaphor. E.g. when comparing a room with a pigsty, the common feature is messiness. This approach, however, only begs the question: how did the speaker decide which features to focus on and which to ignore? The answer is obvious: it’s the features that mattered to their immediate communication goals.
Consider two metaphorical uses of the word “mountain”. You could say “I have a mountain of work to do”, which focuses on insurmountability. Or you could say “that warrior is a mountain of a man”, focusing on immovability. The choice of which feature of the mountain to focus on is clearly guided by your goals — “mountain of work” implies “I need support or consideration”; “mountain of a man” conveys your confidence in their defensive stature, or perhaps a warning to take him seriously.
It is not just metaphors that show the motivated nature of meaning. Any non-literal use of words, such as metonymy, is also instructive. For example, when discussing an army of soldiers, the phrase “we have a thousand spears in the enemy city” is intended to give confidence to a monarch; whereas “we need long supply lines to feed a thousand stomachs” is a complaint about costs. The part of the soldier that the speaker focuses on is driven by their communicative goals.
A surprisingly large portion of everyday language employs words non-literally, to the point that there is a no clear separation between metaphorical and regular word usage. For example, the phrase “enjoy the fruits of one’s labours” sounds like you are using the word “fruit” outside its literal definition, to mean “enjoy the profits of one’s labours”. But the word “fruit” originally comes from the Latin word for “enjoyment or profit”, and was later used to mean the organic seed-bearing foodstuff in a metaphorical sense. So the phrase “to enjoy the fruits of one’s labours” is reverting back to its original meaning. The English dictionary is awash with masquerading metaphors.
Since metaphors are always linguistic, their use is always socially motivated. They are employed to achieve a social outcome, like convincing someone, or getting help. Consider the word “pain”. If, as suggested in an earlier post, you originally use the term “pain” as a conversational signal for others to help ease some unpleasant experience you are having, then the word is as effective a solution to emotional pain as physical. It even applies outside individual psychology to domains like marketing, where we refer to a customer’s “pain points”. Here the goal is to foster empathy for a customer who would want us to change something about a product.
It is an error to separate what a metaphor is — by focusing on the essence or features of the word in isolation— from the effect that using it has on others. This effect is the connective tissue at the heart of every metaphor. To try to explain why a metaphor was used based on the definition of the word itself, irrespective of the problem it is solving for the user, merely leads to an infinite regress of basing meanings upon meanings, with no ground to start from.
It is not the objective features of “pain” which unite the usage of the word in the three cases above. Rather you identified it as “pain” because of the effects of using the word. The phrase “pain point”, as a metaphor, feels natural because using it is so obviously effective at driving the required change. You can use “pain” in any setting in which you want to have this effect. This is what “the great reversal” is all about⁴.
Letting go of the epistemological crutch
It may feel strange to think in this inverted way. It’s like saying that cat, as I understand it, is not a concrete set of objective features; rather it is a union of my motivated interactions with it, such as “I like to pet it”, “it will move towards me of its own accord”, “it leaves hair that I have to clean up”, etc. This seems backwards, or at least incomplete. But if you start to include more communicative problem-solutions like “I should call it ‘cat’ when adults test me on what it is”, and “it comes when my mother yells Whiskers”, and “cute baby cats come from big cats”, and even “a surprising fact is that cats and cheetahs are like cousins”, you can start to see that objective, declarative definitions are also a type of motivated interaction — they just happen to be based on more collaborative motives, like authority and truthfulness.
And to be clear, reality does play a role — it is not all left to your motives. For one, the motives themselves must have a cause in reality. And they can only pick their solutions from a flow of stimuli that they have no choice in. Everything you like to pet appears “fluffy”, and has a certain set of common visual markers. Also, a concept is usually at the intersection of many motives, as we saw with cat above. So reality plays a role by making the same visual experience elicit all of those problems and their solutions in contiguity (near each other in space and time).
Doubtless many readers are still not convinced. It is hard to let go of objective features as the sole driver of definitions. Our natural tendency is “empirical”; we see an object’s effect on our motives as a consequence of its features. It seems absurd to flip that and say that motives are the cause of the observed features. Reality should still be the ultimate source of truth; anything else feels like the tail is wagging the dog.
And again, this is true from an outside perspective. It is the actual properties of reality that makes some parts of it useful or detrimental to your motives. But this is not how you yourself learn about these properties. You don’t learn about telephones because they were made in a factory, or are made of plastic, or are rectangular — you learn about them because you need to make a call, and you only learn as much about them as will serve your purposes. Even the features of the the telephone you learn about — call button, earpiece — are the product of some need. This explains our human propensity for selective blindness: people tend to only see as much of the world as they need to see.
There is a distinction here between epistemology (how you learn) and ontology (what is real). From the epistemological side, it is not that similar entities afford similar solutions (e.g. telephones let you communicate), it is that we define them as similar entities because they afford similar solutions:
As telephones have evolved drastically over the last 70 years, the same concept has been carried over through each, because each iteration addresses the same motive. Looking at the examples of phones in the image above, there is a core “feeling” of telephone-ness that seems to unite them. That feeling is its utility, specifically its utility for you.
The alternative theory — that identification is built bottom-up from objective features — only gets weaker as the discussion steps outside concrete concepts like telephone and into subjective and abstract ones like game or small. Fortunately, this is where our inverted approach pays out in spades. As we’ll see in the next post, thousands of words that are difficult or impossible to define empirically — e.g. existence, time, self — or which have vague, fluctuating meanings, can now be grounded in the underlying problems they address for the mind. We can give finally AI a way to build such concepts autonomously, from scratch, something that modern AI research has no viable way to accomplish.
Next post: Defining “time”
¹ A person who didn’t have any problems at all would never need to learn anything. It is not so much that “ignorance is bliss”, but rather that “bliss leads to ignorance”.
² Some people will call this “interpretation” not “identification”. But there is no easy way to separate the two; they are on a spectrum. Identification only starts being called interpretation when people start to disagree about it, in the same way that a “fact” starts to become an “opinion” in proportion to popular disagreement. Identification vs interpretation is not an essential property of the mental behaviour itself, only how we perceive them.
³ Note: answering a question as stated is a problem of communication. This should not be confused with the pragmatic side of identification, where “mountain” and “valley” are used as affordances (see the previous post on parallel systems).
⁴ Learning to detect and predict which contexts a word works well in is also the key to generalization, a topic we’ll look at in later posts.
