The “uncanny” is the experience of a loss of control
You experience the uncanny when your mind can’t get what it needs from the world and others
The “uncanny valley” is a well-known affective phenomenon where simulations of human beings — such as robots or computer-generated characters — that look almost realistic, but are off by some small difference in appearance or behaviour, cause people to feel uneasy. There have been a few proposed explanations for this effect, ranging from disease avoidance to transgressions of human identity, but they all start from the same point: you detect a deviation from how you expect humans to look or behave.
When an agent looks like a human, based on a lifetime of experience, the brain generates a prediction that this appearance will be associated with a particular kind of behavior (e.g. movement kinematics). When the behavior of the agent violates the prediction, an error is generated — Predictive coding and the uncanny valley
There is a fundamental weakness with all such theories, especially those that rely on deviation from expected behaviour: what counts as an uncanny amount of deviation is unclear. The famous graph from Mori’s original paper (below) shows how a linear increase in a robot’s similarity to humans causes a non-linear change in discomfort. But the graph assumes you have an objective and linear way to measure similarity to humans. In reality, we rely on human intuition to know what is more or less similar, and this leads to wildly inconsistent determinations.
There is no clear definition of how much aberration, or of what kind, is acceptable before the creature turns uncanny. The decision is made retrospectively, after you feel the discomfort. For example, you can watch a character in a movie who has green skin or skull protuberances (clearly signs of aberration or disease) without feeling any unease. Such characters still look human — they are played by human actors — though slightly off, and they therefore should fit the criteria of uncanniness; but they don’t. Other differences, like an odd glide of the upper lip, do. If a character with green skin did cause unease there is no doubt that such theories would chalk it up to the aforementioned effect. So it remains an arbitrary choice which deviations are the wrong ones.
Still, you cannot deny that people do subjectively experience the uncanny valley, so it is real in that respect. The question then is: what counts as “different” enough, and what makes something “a little off”? On a gut level, it seems like an easy question. Much of cognitive science is based around the assumption that humans naturally notice when things are different from what they expected. Principles of similarity and difference form the foundation of object recognition, perception, and world modelling:
The act of comparing events, objects, and scenes, and establishing similarities between them is of critical importance for the cognitive processes we depend upon. The utility of similarity for grounding our concepts has been rediscovered in all of the fields comprising cognitive science — Oxford Handbook of Thinking and Reasoning
It is also assumed that when you detect that something as “different”, it grabs your attention, and the novelty gets integrated into your knowledge base:
Orienting is a rapid response to new (never experienced before), unexpected (out of context) or unpredictable stimuli, which essentially functions as a “what‐is‐it” detector. — The brain’s orienting response
Because it seems so straightforward to detect differences in stimuli from an information-processing perspective, such detection has been largely reduced to calculations on sensory data. “Different” is calculated in a standard, data-centric way, and applied uniformly across all examples. The individual’s motivations and preferences play no role in this process. This, of course, makes it difficult to answer the question above: why does a small twitch of the eyes trigger such distaste, but green skin, a birthmark, or the presence of skull protuberances is ignored? Even if a character with green skin smiles in the way that you are used to, the rest of their face is still mostly unusual, and so the majority of the visual data in their smile will not match what you’ve seen before. Despite this, you do not feel uneasy; unless, that is, the smile is off too.
The first and most obvious explanation for this selective discrepancy is that you are not focusing on the entire face and body, but on a specific subset of the total visual stimuli, say around the eyes and mouth, while ignoring skin colour, facial hair, wrinkles, liver spots, birthmarks, etc. This implies that your mind is looking for certain specific things to happen. A subset of the stimuli must progress in an expected way for the mind to feel comfortable. This is why everyday humans you meet can have all kinds of strange haircuts, facial hair, moles, unusual tattoos, etc. without triggering your subconscious disgust. It is therefore not about the magnitude of changes in visual stimuli, but that you expect specific parts of the whole to look and behave in a certain way, and are sensitive to cases when they don’t. The smile must be off in the “wrong way”.
The word “wrong” also suggests that it is not just incorrect or different, but somehow unlikable, against your preferences — hence you feel unease rather than simply being surprised. It goes against what you were hoping for. Admittedly nothing in life ever works out exactly how you want it to. But as we saw with the example above, maybe it doesn’t need to. Only that subset of the experiences that are important need to behave as expected. The rest can vary as it wishes.
So what draws your attention to just those subset of stimuli? And why does a deviation from expectation elicit discomfort, instead of surprise or indifference? This post will argue that those two questions are actually two facets of the same process. We discussed in a previous entry how the motives that drive attention also drive what you perceive; they would therefore determine if an experience is identified as the same or different from another. The reason that current theories have difficulty explaining the uncanny valley is that they separate the act of detecting changes from the discomfort these changes cause; and then focus only on explaining the discomfort. They pay too much attention to why we feel uneasy, and not enough on how we detect that something is wrong or different. This post will unite the two aspects and show that the uncanny valley is part of a larger system that regulates all aspects of attention and perception.
The unfamiliar is cued by the familiar
So how does your mind determine when an experience is “unusual”? Mere novelty of sensory inputs is not enough. Any random pattern of colours, such as you’d see in an abstract painting, is technically “unexpected”, but that is unlikely to draw or keep your attention. An image of an abnormal creature, on the other hand is more recognizable and yet somehow unsettling.
Clearly your attention is not drawn by mere novelty, or the degree of difference from what is habitually seen. Every moment of your life already presents you with ceaseless novelty: unexpected events, random noise, changes in lighting conditions, etc., most of which you ignore completely. When you see an uncanny creature, what makes it “unusual” is not that it is unfamiliar, it is that it is familiar, but not in the way you think it should be. For any entity to be called “unfamiliar” it must, paradoxically, first trigger a familiar reaction; if for no other reason than you must be able to point to it, and say “this entity” is unfamiliar. The word “this” implies you have identified an object in your stream of experiences — a face, a creature, a smile, etc. — and that you now have expectations about it. You cannot notice a deviation from your expectations unless you have a jumping-off point from which to start “expecting” something.
This is why, despite the fact that the majority of human experience is a flow of unpredictable stimuli, most of it escapes being labelled as odd or unfamiliar, since there is no baseline or reference to trigger such a comparison. The random colours in the painting above contain no “thing” to point to, but the creepy image of the animal does. You know something about the latter — that it should be an animal, or a baby, etc. — except that something is incomplete.
Therefore, before you can become aware that a robotic smile is uncanny, you must have first perceived that there is a face making some expression. This initial perception raises a kind of “question” in your mind — what is he thinking, or feeling, or doing? If you now look at the face and it does not give an answer — it does not match any expression you know — the experience will feel unfinished and odd.
To notice anything — whether odd or not — is a two-step process: a demand and a response. An initial trigger sets up a demand, and you feel content only when you find the complementary answer. The period of time between the initial uncertainty and subsequent recognition is what we call “attention”.
So what happens when what you’re looking for doesn’t show up?
Detecting a negative
It has long been known that the mind can only register a match, and not a negative or absence. When presented with any new experience your mind can detect an existing concept it is a part of, but you don’t have a detector for everything it is not. A car is “not” an infinite number of things — it is not a house, it is not a monster, etc. — and your evaluation changes depending on what you are looking for — e.g. this car is not a Hyundai. By the same token, a given smile can be wrong in a million ways, and you cannot store responses for every version of wrongness. So when you detect something uncanny about a smile, you are always detecting a negative, the absence of an expected pattern, not the presence of a new or different feature. Something must be missing.
If the absence of the expected pattern makes you feel uncomfortable, by implication the presence of the correct or expected smile would be a source of relief. It is therefore desirable, if for no other reason than because you avoid experiencing the uncanny. You want to see a natural smile, and so, when you see a face, you look for an expression that matches what you hoped for. The uncanny is not just caused by a difference from expectation, but a difference from a desired expectation. This was always implied in the phrase “it feels wrong” — it suggests an opposition to “what feels right”. You want the creature or face to be the right way, because it matches what you’d normally prefer to see. “Normal” here does not mean “average”, since human faces and behaviour vary a great deal, it means matching desired patterns of behaviour.
There are numerous other examples where events deviate from expectations but do not cause any discomfort, regardless of how strong the expectation was, or how intimately familiar you are with the subject. We are all quite familiar with consumer cars; we see dozens of them every day. It may surprise you, then, to know that TV ads have been using computer-generated cars for decades, and most people have yet to notice that the light doesn’t hit them quite right, or that the colour temperature is usually off. The reason for this is that we frankly don’t care that much about the textures and lighting on vehicles. We may even prefer the glossy sheen of artificial simulations.
If you took a popular painting like the Mona Lisa, and altered it slightly, few people would notice, and even fewer would deem it “odd”. Or if an oak tree in a park had maple leaves instead of oak leaves, not many bystanders would realize it was “unusual”, despite the fact that it would be supremely bizarre. The uncanny is not a simple divergence from familiar expectation, but a divergence from something you care about.
And there are good reasons to be more attentive to humans than to trees, cars, and paintings. Human beings can often pose a danger to you, or alternatively provide great benefits. Your own history since childhood has shown that it is worth checking that the next person you meet will be a benign and not a threat. So when you spot a face or humanoid shape your attention is drawn to it because of a predominating social need, the need to understand what they are doing or thinking. You may not even consciously notice this, but you become momentarily unsettled — it triggers a tension, or “attention” if you prefer.
Most of the time you are quickly placated when they behave in a way that you know is benign. On the other hand, a creepy face or unsettling behaviour doesn’t give you the desired assurance since it doesn’t match any of the accepted, safe patterns. As with horror movie monsters that have no eyes, the lack of a “read” on a sentient entity can be disquieting. Your mind reverts to the discomfort of the negative, of “difference”, of the unexpected, in the absence of what you wanted or were looking for. Like a building whose architecture looks imbalanced, you feel uneasy about the consequences of interacting with a creature that has yet to prove it is safe or benevolent.
Given that social interactions are highly nuanced, there are a plethora of specific reasons you might feel ill at ease. In many cases the uncanny feeling is caused by a distaste or dislike for a creature that fails to satisfy your need for communication — it is being unclear about itself, as if it were not trying hard enough. In others you may be unnerved by their secret, possibly malicious intents. In still other cases it triggers your dislike of hypocrites, whose contrived expressions display fabricated emotions. And in still other cases you may be annoyed at the creator of the robot for trying to pull the wool over your eyes with such an elaborate deception. These all trigger a wariness based on your history of human interactions, and what you have learned counts as safe¹.
The key point is that the discomfort of the uncanny is not elicited as a consequence of seeing something unusual, as most theories suppose. It was already present, and merely temporarily suppressed while you awaited confirmation that the person was benign. It resurfaces when it fails to find that confirmation. The originating primal need, the compelling force of a possible threat is the underlying source of the discomfort, and it precedes the detection that something is off. It only surfaces and becomes conscious when the world fails to deliver what you need.
This approach unites the detection of difference with the discomfort of not having what you want, as one holistic process. A change, or a “difference”, or a negation is identified as such because it becomes a problem for you. Throughout this post I have tried to provide a new definition of “different”, one that more accurately matches how humans notice differences. It is not based on a deviation from what is expected, but rather the absence of what is desired, preferred, or useful.
Most theories of the uncanny only stipulate that the mind only detects a difference from what was expected, not a difference from what was desired, even though the latter is implicit in every theory:
Our results […] can be reconciled with the ‘predictive coding’ framework of neural processing, which is based on minimization of prediction error among the levels of a cortical hierarchy. The key idea in this context is that brain activity will be higher for a stimulus that is not well-predicted or explained by a generative neural model of the external causes for sensory states — Predictive coding and the uncanny valley
Some theories go on to assert that traditionally animated characters like Aladdin or Shrek do not trigger an unpleasant effect since they are far too different from humans to do so. I have to assume that those researchers don’t believe that traditionally drawn characters can ever look uncanny — which, when phrased like that, shows the absurdity of such a proposition; of course they can.
The reason that non-realistic animations don’t usually cause you to feel uneasy is because, despite being significantly different from real humans, the artist can still hit the right psychological notes — those few subsets of stimuli your mind is looking for, to inform you of what the character is thinking². This is the basic idea behind a “caricature”: the artist highlights only those aspects that matter, setting up a cycle of expectation and satisfaction, tension and relief, with the assumption that all else can be ignored.
If instead of trying to explain the uncanny valley based on the degree of difference in sensory inputs, we simply said that “uncanny robots set up the expectation that they are human, but do not then behave in ways we’ve learned are safe or beneficial”, suddenly the whole phenomenon becomes trivial to explain. Even actual human beings who you deem “creepy” can be understood this way.
The uncanny is a loss of a sense of control
As you may realize by now, the uncanny valley is part of a larger system of learning how to respond to the world, and what to expect from it, of which it is only one special case. In its most general application, it can be used to explain and even formalize how the mind organically learns to detect that something is unfamiliar by focusing on an absence, and to do so specifically in the context of what it is looking for. This is a disambiguation problem that has otherwise defied explanation via statistical models.
Many related phenomena, in which something unfamiliar causes you discomfort, and the familiar causes ease, can be explained using this approach. For example, say that you have just accepted a new office job. At first everything about the new office is disquieting. All is unfamiliar, and you struggle to satisfy your many needs, both small and large. As you settle in, you build patterns of successful intentions: you learn how to get to the bathroom, you know the soap dispenser works and how to fix it when it doesn’t, you know how to interact with your colleagues so they don’t get mad at you, you know where to find food, and that it will be affordable. You have built a series of plans for this new set of circumstances that works for you. Deviations from normal routine, such as an announcement of a company reorganization, make you uncomfortable because you don’t have familiar plans for such eventualities.
To understand a space (in other words, to view it as familiar) only requires that you learn those patterns that are useful to your everyday concerns. You will never understand everything about either a place or a person; you need only understand enough to satisfy your needs and suppress your anxieties. As you acquire more reasons to feel anxious — a consequence of harsh experience — you must now try to learn new and different ways to appease those — e.g. who to talk to if your boss loses his temper again.
[We desire] not “to know” but to schematize — to impose upon chaos as much regularity and form as our practical needs require. — Nietzsche, The Will to Power
All thinking is imposing your will on the world. As discussed in a prior post, you think about the set of states you want the world to be in, when you need them. Making the world and other people familiar is yet another example of this same process. You identify a situation as uncomfortable or unexpected when you don’t know a way to impose your will on the progress of events, through thought or action. It feels uncanny not because it is infrequent, rather that feeling is a symptom of a situation for which you have no real or imagined resolution. The uncanny is a loss of control.
¹ Those with social anxiety may maintain this tension longer, and in more situations than others, because they have fewer ways to recognize what makes a person safe, and so strangers are more likely to remain a threat in their minds.
² Realistic animated characters do not have the luxury of exaggerating those aspects of their expressions which would trigger the right reception in the audience; they would look ridiculous. So they must keep expression subtle, and also include a myriad of other realistic features and movements that are not as obvious or easy to simulate, but that we still expect for the illusion to be complete. In so doing the artist’s job becomes an order of magnitude more difficult, which is why they end up with a confused result. You have no such expectations of unrealistic characters, and are satisfied with just a few signals or indicators of intent.
