Experiences can’t contradict each other, only motives can
A detailed critique of the human faculty of contradiction
It is easy these days to find examples, posted online, of popular Large Language Models (LLMs) making self-contradictory assertions, and then immediately reversing their position when questioned on their claims. They apparently rely on prodding or additional information to realize that something is off in an earlier statement. They then act like they knew it was an error all along, and that they just misspoke:
This behaviour is a fairly transparent illusion; the agent is no more convinced after being corrected than before. It is merely incorporating the user’s rebuttal in its next text prediction.
If the model could actively discover self-contradictions on its own, that would go a long way towards reducing hallucinations in natural language-based AIs. Such an agent would always “think” about what it would say before it spoke, and could change its output as it uncovers hidden contradictions.
At first glance, spotting contradictions may seem like an easy task for a computer to learn. A contradiction is simply any set of claims asserting that something is true and simultaneously not true, with the proviso that the two entities are the same “thing”. To say that Alice is young and that Alice not young is a contradiction, assuming that “Alice” is the same entity in both cases (and the same goes for “young”).
Standard software languages make it easy to spot contradictions, since all you need to do is compare two registers containing binary numbers and see if they match — a built-in operation in all computers. For a mind or Neural Network to do the same would also require that the entities being compared are discrete and clear-cut. This is where the problems begin, since human sensory experience is fluid and ambiguous, which means all empirical knowledge is built on an ever-shifting foundation. Your streams of visual or audio input do not, or themselves, indicate any clear boundaries to distinguish the objects and events around you (e.g. imagine a black phone placed on a black table). This is why identifying objects in images and video continues to be a difficult challenge. For a mind to discover contradictions, it must first somehow convert that sensory stream into discrete symbols or concepts and compare them.
Natural language processors (including LLMs) skip over this conversion step by pre-tokenizing their inputs as words. The human eye or ear, unfortunately, does not have the luxury of having its data so nicely prepared for it. The words you hear or say are not discrete, they blend into one another both in terms of their sounds (different accents, homophones) and their meanings (ambiguity of definitions). The same goes for logical AI system like PROLOG which avoid the conversion step by setting up the relevant entities beforehand, and pre-establishing how they will be interpreted within a formal framework. Their very design creates a situation where contradictions automatically reveal themselves.
Both the above systems ignore the process of converting real world experiences to logical symbols as irrelevant to the function they perform. This is an understandable omission. Experience shows how easily and intuitively your brain orders ambiguous stimuli into identifiable entities — e.g. a person you know, or a car. That same intuition can lead you to believe that everyday experience will readily inform an AI of those entities it should include in its logical calculations.
In practice, however, the conversion step creates a nearly insurmountable problem, one that is particularly noticeable in predictive models used by Model-Based Reinforcement Learning (MBRL). MBRL agents create internal models of how the world around them works, then use those to help them decide how to act. Most standard predictive models do this by trying to predict future states of the world based on what the agent did in previous ones.
Since the agent’s sensory inputs (e.g. camera input) only provide unstructured, fluid images, the model’s predictions also take place in an unstructured, continuous space of fine-grained pixels. This in itself doesn’t exclude them from discovering latent structures within this soup of colours. But predictive models are also obliged to shape themselves around each new experience as just another piece of ground truth data, using it to adjust the world-model by changing the expected likelihood of seeing a given state. They are like a sheet hung up and drying in the wind; they mould themselves to every new current — but they never tear. Anomalies like flukes or noise eventually get washed out by a preponderance of opposing data.
In contrast, an anomaly in the sense of a contradiction is not a fluke, it is significant. It represents a tear in the sheet; an immediate problem to be resolved. And the resolution must be binary. One or the other must win; it cannot be washed out as probabilities. This means that predictive models can’t, by definition, contradict themselves since they do not contain any “stubborn” entities that might stand against one another, only states that probabilistically lead to other states. It does not even seem possible for it to have a thought that it deems contradictory. Every experience it has is by definition a “true” one in that the agent experienced it. This is even the case if it were an illusion or magic trick.
Strict formal logic, like that used by programming languages, is able to discover contradictions only because it “holds its ground” and forces the contradictory entities to confront one another, rather than weakening them as probabilistic variables or ambiguous states. This act changes a simple thought into an assertion. An assertion makes an unambiguous claim, and is able to do so specifically in opposition to contradictory claims. Thus contradictions can’t arise in experiences, nor even in predictions, but only in assertions.
Predictive models, which try to match predictions with reality, assume that a prediction (or a belief) is the same thing as an assertion. LLMs similarly conflate the two, because for a language model assertions and predictions are identical in that they are both made of words. But the two must be kept separate. Without the ability to make assertions outside of a predictive world model there is no way to for the AI to separate a contradiction which must be resolved, from a distinction that must simply be learned or refined. To see why, consider the following logical statements:
My head is part of my body
My nose is on my head, and not on my body
and
My car is in the garage
My phone is in the car, and not in the garage
As given, there is no way to know whether either of these pairs of statements is a contradiction, or simply a case of ambiguous terminology, and that context can indicate that the same word is being used in two different ways. No set of statements, even when clearly stated, can by itself conclusively prove a contradiction exists. The above sentences only become contradictory if the agent maintains that having the same letters in the word “body” or “garage” make the two underlying entities equivalent. It must not allow the definitions of the words to be flexible, or to be misunderstood. If it did, it would never be able to confidently discover a contradiction. It would always hesitate and second guess itself. To discover a contradiction the system must strictly equate two “experiences” as being the same — that is, as having the same identity.
Systems that detect contradictions using formal logic abstract that process of identification (i.e. equating two or more terms as being the same) behind a simple comparison of word-strings or symbols: e.g. if both terms use the word “body”, then they are the same. They chuck the more serious challenge of identification over the fence, and assume that identifying the entities across varying experiences in the real world is easily performed, through clustering, association, or some other method.
This assumption seems reasonable. At the level of the senses you inherently respond to similar experiences in a similar way — (within a certain margin of variation), which is why you can regularly predict the word “cup” from the sight of one. But identification just as often requires you to unite together dissimilar experiences under a common response. For example, two pictures of a person at 16 years old and at 80 years old may be identified as the same person. In some contexts both may elicit the same output —e.g. how you act towards it, what linguistic label you give it, what thoughts occur to you, etc.
There is no inherent, logical reason why the mind must group experiences over time under a common identifying response. No two moments in time are exactly the same, so the act of identification always entails equating two experiences or thoughts that are by necessity not equal or identical. It would be perfectly consistent and logical for the brain to treat every moment of its life as a special case.
Consider the so-called law of identity, put simply as A = A. When someone says “this cup = this cup”, what they mean is “this cup, as I think of it in this moment, should produce an equal response to this cup as I may think of it in other moments”. The mechanism is equating across two different moments, and for two different thoughts or experiences. Otherwise nothing useful or meaningful would be done by saying that A = A. Once could call it synthetic, i.e. it produces a novel connection.
Yet it would be entirely reasonable to not equate the two cups, and to say that the experience or thought of a cup at one time is not the same as at another time. Perhaps the cup, or your mind, have both changed in the interim. The law of identity is therefore a mental tendency to select a subset of inputs or thoughts — here, the thought of the cup — out of the total mass and to isolate them from the surrounding context in order to equate them. This is not mandated by any rule or axiom of logic, or even of statistics. The act of identification precedes both logic and statistics, it is their foundation. A brain that couldn’t equate two different things as being the same thing could never perform logic.
So if logic is built on a selective — perhaps even subjective — identification of different experiences and thoughts, then the same selection process would also be a deciding factor for uncovering a contradiction. If someone told you that “Bob is at the store”, and you later see Bob at the zoo, you may find the two experiences contradictory because you identified (equated) the sound of the word “Bob” with the sight of his body, and distinguished the word “store” from the sight of a zoo. This was not logically necessary. It could be a different “Bob”, or perhaps Bob is at the store in the zoo.
So what lead you to identify the sound of the word “Bob” with his appearance? Prediction is one possibility; e.g. you predict that the sight of the person will be followed by the sound of his name. But prediction is not identification. What you infer when you make a prediction is not necessarily identical to the thing you inferred from, since it could just be a property or association — e.g. the fact that all swans you know of are white does not mean a black swan is a contradiction. Identification establishes an “is” relationship; an assumption that what applies to one experience must also apply to the other.
So we are plunged back into the same confusion as when we started, since there is no automatic way to confirm that an inference or association between two experiences is necessarily an identity. The world is too fluid and nuanced for this process to be mechanised into a simple heuristic. The person you saw and the word “Bob” may not always have the same identity — he may change his name. In other cases, an apparent contradiction may not actually be one, but rather represents an exceptional situation — e.g. if Bob was playing the role of “King Lear” in a play. All methods of identification — and consequently all logical inferences — involve some context-specific interpretation, and thus can only be learned through experience.
Still it is fair to assume that your mind must always have some reason to attach the sound of “Bob” to the sight of him. But if the glue by which you stick different experiences together doesn’t come from the experiences themselves, the only remaining option is that it comes from outside. As an analogy, we know that images of food can’t by themselves tell you that they are all pictures of the same thing, food, because the data do not contain the critical thing that connects them all — namely, that they are all eaten. In the same way, the reason you choose to associate dissimilar experiences together under one identity must come prior to, and from outside of, the immediate experiences themselves.
And as with the case of food, which is unified by the fact it satiates hunger, the act of uniting disparate experiences must serve a useful reason or purpose¹. The question is not “are these experiences equivalent?” but rather “how useful is it for me to equate them?” We should come at the question from a different angle. Instead of trying to discover how ambiguous experiences can somehow demonstrate the presence of a contradiction, let’s consider the useful consequences of spotting a contradiction.
As noted above, assertions — not experiences or thoughts — are at the heart of every contradiction; the latter can never be self-contradictory. Even the word “contradiction” means to “speak against”, and not to “think against”². The difference between thoughts and assertions is reflected in the distinction between a surprise and a contradiction. If you merely thought that Bob was at the store, and then saw him at the zoo, you would perhaps be surprised, but you wouldn’t call it a contradiction. It only becomes a contradiction when an assertion is made: e.g. “You told me Bob was at the store. Then I saw something that contradicted it”.
What makes an assertion so special? What distinguishes it from any other type of thought or experience? The answer was also hinted at above, when we said that an assertion is something that stands its ground in the face of opposing assertions. It is resistance that entrenches a thought as an assertion. An assertion is created when you discover a potential problem with naively holding on to a given belief. A problem arises that forces you to take sides, to discard one of the two as ineffective or useless.
This means that the consequences of two assertions have somehow become incompatible. Not because both can’t be “true” — that would create a circular argument, since truth itself depends on assertions — but because you can’t act on both. You may want to speak with Bob, but find you can’t both go to the zoo and to the store. Or perhaps someone asks you where Bob is, or how to get to him, and you find you are unable to answer them. Only when you realise that your ability to plan your actions has been frustrated, do you first twig that a contradiction exists. It is not that contradictory beliefs lead you to fail; rather the failure of your plans first leads you to identify it as a contradiction. Until that point they remain orthogonal to each other. In this sense, facts can never be contradicted, only your plans can.
Contradictions are no different from any other impediment to your goals, except that they happen to only occur in communicative context as assertions. When you originally differentiated the zoo from the store it was because with respect to planning where to go, they would, jointly considered, cause you to fail. Once you were told that the store was in the zoo, and therefore that both could be accomplished, the impediment disappeared, and the contradiction along with it. This is what gives contradictions their discrete nature — they represent a binary outcome, either a failure or a success, with no shades in between.
This is also how assertions can remain stubborn against contradictory ones. You originally grabbed onto either assertion because doing so would help you achieve some goal. This required you to pick a particular interpretation of the statements that was useful to your needs; that is, you must fix its meaning into a specific definition that guides your actions towards your goal:
This is what gives assertions their rigidity, their tenaciousness. Otherwise you could always explain any contradiction away as a misunderstanding. Your mind must choose to enforce an specific interpretation, and willingly hold to it in the face of other options, in order for a contradiction to arise. And the only way to do this is for the interpretation to match your motives.
Formal logical definitions of “contradiction” try to sanitize these motivated roots from their systems, in order to build a reusable, computable framework of logical inference. But in doing this they cut the legs out from under the system, by making it “formal” — as in related to form — and severing its attachment to any reality or content. This has made it brittle and unable to deal with nuance, which repeatedly arise from ambiguities in its content, an inability that lead to the downfall of GOFAI systems in the late 20th century. The only way forward for logical AI models or probabilistic predictors is for them to be attuned to the motives that frame logical inference.
The consequences of this conclusion cannot be overstated. Formal logical systems can never, by themselves, assure you that the terms of a predicate are being correctly identified. On the other side of the wall, sense experience is ambiguous and can never give you clear information about the discrete objects it is supposed to contain:
If both sides are to work together organically, logic must be informed by moment-to-moment, context-specific motives. This melding of logic and motives is uncomfortable for some people, who prefer logic to remain a pure system of symbol manipulation, hard-coded in AI as a kind of inductive bias. But there must always be an act of identification that precedes any logical calculation, and identification, and ultimately contradiction, cannot hold themselves together of their own accord; they must be held together by the broader utility of doing so. Otherwise the agent would find contradictions at every turn, and in every unexpected outcome.
¹ By the same token, to distinguish two things previously thought to be the same, such as separating blue into light blue and cyan, must also be useful.
² It is also noteworthy that the word “logic” comes from the Greek meaning “word” (logos), not from any words for “truth” or fact”.
