The self-contradictions in Artificial Super-Intelligence
Claims about ASI outstrip their own assumptions
By Yervant Kulbashian. You can support me on Patreon here.
It is a testimony to the immaturity of our field that the question of what we mean when we talk about intelligence still doesn’t have a satisfying answer. — Chollet, On the Measure of Intelligence
If you’ve been following the trajectory of modern Artificial Intelligence there appears to be every reason to speculate that Artificial Super-Intelligence (ASI) is just around the corner. The term ASI is used to designate any form of synthetic intelligence that exceeds human capabilities in some meaningful and insurmountable way. Although few people claim that examples of ASI are already in circulation, there are many, including those whose credibility on the matter is above reproach, that suspect the writing is on the wall — ASI is coming, perhaps sooner than we’d like. And the outcome may not be to our benefit.
The next stop [after AI], just a short distance farther along the tracks is super-human machine intelligence — Nick Bostrom, Superintelligence
A detailed definition of what a “super-intelligence” would entail is not presently available. Still, given the number of people who have sounded the alarm on the topic it is worth taking a moment to explore where this intuition and consequent foreboding prophecies come from.
The argument for super-intelligence
Intelligence is commonly represented as a natural capability, similar to physical strength. It can therefore be graded on a spectrum: some people are more intelligent than others. And just as there is no theoretical limit on the amount of force a robot could exert, there are apparently no bounds on how far intelligence may extend. Moreover, the last three centuries of human ingenuity have aptly demonstrated that every human and animal skill either can or already has been superseded by our species’ technological innovations — robotic arms work more precisely and with greater power than human limbs, automobiles move faster than any cheetah ever could, etc.
Nor is the human brain itself exempt from technological displacement, as many of our mental capabilities have been replicated and improved upon by computational devices. Mathematical calculations can be done at breathtaking speeds by CPUs, chess and Go have been mastered with ease in a few thousand lines of code, and reams of information can be “memorized” in greater volume and with greater accuracy using dirt-cheap, run-of-the-mill hard drives.
It will be comparable with the industrial revolution. But instead of exceeding people in physical strength, it’s going to exceed people in intellectual ability. — Geoffrey Hinton
The arc of technological progress seems to be bending in one unavoidable direction; the lines of the graph all point up. All it takes is to connect the obvious threads and draw the natural conclusion: if intelligence is indeed a human faculty or skill, is it not inevitable that very soon human technology will replicate and quickly supersede the limitations of our own fallible grey matter?
A local point of view
Let’s take another look at the examples above. In every case, we humans have defined the criteria for success or excellence. Trees and clouds cannot understand the magnificent power of our large excavators, the superiority of Google’s storage services, or the utility of OpenAI’s chatbots. The grandeur of human technology is not a catalogue of “objectively” superior accomplishments, they are superior only to an intelligence that can evaluate them as such. Our proliferation and dominance as a species is something other animals don’t understand since they lack any concept of species dominance. They don’t know we’re superior or more powerful than them; our notion of superiority isn’t even recognizable to them¹.
What a creature evaluates as “superior” is always tied to their form of intelligence — different forms of intelligence necessarily gauge “better” in a different way. An ape, staring at racks of servers, would not appreciate their ingenuity; only humans recognize and value their output. If the noise and lights emanating from the racks annoyed or scared the ape, it might take a rock to them and, having silenced the machine, would feel that it had truly accomplished something of value. The broken machine would be, for it, “better” than the functioning one in every way that it cared about or understood.
Let’s apply that distinction to our own situation with respect to ASI. Say that at some point in the future humans did create a “superior intelligence”, (leaving aside questions of “how” or “what” that was). Once placed in a robot, this robot now stands still, then sits down, and repeats this pattern indefinitely. When asked what it was doing, and why it wasn’t, say, conquering the world, the robot responds saying: “you wouldn’t understand”. Truly, how could you? Why should you ever expect it to behave even remotely as you think it should?
Unlike judgments of relative strength or height, evaluating relative intelligence presents us with a special case. You don’t have to be tall to recognize that someone is taller than you, since it is your intelligence, not your height, that has contrived that designation. But you must be intelligent to gauge relative intelligence. You cannot deduce simply by looking at a piece of software or its outputs that it is super-intelligent, any more than a toddler could distinguish between a calculus textbook and an advanced calculus textbook. You must have first come up with some criteria for identifying a form of intelligence that is greater than that of any human being’s. But any criteria conceived of by an intelligence to test for a superior intelligence could not possibly be asking the right questions; no more than a dog could evaluate your IQ.
In practice, whenever we humans try to gauge intellectual superiority we are forced to do so on our own, local terms. We focus on what we ourselves value — like military or strategic superiority, wealth and social influence, technological innovation, computational speed, or even a large vocabulary. Our popular image of a future ASI is a pastiche of greatest-of-all-time individuals we know or have superficially read about, alongside a distorted and lionized account of their accomplishments, likely beyond their true desserts. The resulting chimera appears, to us, somewhat more intelligent than humans, yet still maintains — and is even judged against — our presumably atavistic world-view, values, and wants. This is why, despite the impressive accomplishments of modern computers to date (e.g. complex dynamic simulations) we continue to exclude computers from the designation of “super-intelligent” precisely because they have not yet accomplished what we (apparently) value: resource domination.
So we end up, as the ape did, projecting our own narrow perspective on something that by definition we don’t understand. Even the values of the hypothetical “paperclip maximizer” — an apocalyptic ASI whose sole aim is to produce unlimited paperclips — echo our own species’ modus operandi: domineering, strategizing, deceptive, exploitative, resource hungry. Only the content is adjusted.
If there were such a thing as a superior intelligence, the only way to actually gauge it is to be as intelligent as the entity in question, and that is the contradiction. All our predictions can currently do is hint at some magic ingredient or suggestive algorithm (recursion, amplification, “strategic superpower”, etc.) which, once added to AI, makes every science-fiction dream or nightmare possible. Yet without first providing a clear definition of what intelligence itself is, any claims about how it can be augmented carry their own refutations with them.
A robust theory of general intelligence, human-like or otherwise, remains elusive. — Artificial General Intelligence
AI researchers have not had a strong record of being able to predict the rate of advances in their own field or the shape that such advances would take. — Bostrom, Superintelligence
That is not to say bizarre computer intelligences can never be created — no one can predict the future. But an ASI agent is not just “very” intelligent, it is incomprehensibly so, to the point that our narrow definitions of intelligence cease to be applicable. Our concept of intelligence and its defining criteria are inventions of our own presumably primitive minds. They are tailored to, and fit well within our own niche of experiences and worldly requirements.
Conversations regarding the future of ASI, despite all their humble preambles to uncertainty, always assume an implicit understanding of the dimensions along which intelligence can be improved; otherwise any talk about ASI would be empty air. Curiously, these dimensions tend to resemble those along which software applications are made more powerful; speed, parallelization, robustness; with the function of intelligence reduced to a form of “data crunching”:
Many cognitive tasks could be performed far more efficiently if the brain’s native support for parallelizable pattern-matching algorithms were complemented by, and integrated with, support for fast sequential processing. […]
hardware is rapidly improving, and the ultimate limits of hardware performance are vastly higher than those of biological computing substrates […]
Just how easy it would be to scale the system by a given factor depends on how much computing power the initial system uses. — Bostrom, Superintelligence
This implies intelligence is analogous to a software routine, and super-intelligence, though less comprehensible to us, is still amenable to such a characterization:
Enhancing the quality of an existing [full brain] emulation involves tweaking algorithms and data structures: essentially a software problem — Bostrom, ibid
In fact, most descriptions of ASI lean heavily on contemporary principles of modern Machine Learning (ML), such as rational goal optimization, rewards, instrumental goals, etc.:
Most, if not all known facets of intelligence can be formulated as goal-driven or, more precisely, as maximizing some utility function. It is, therefore, sufficient to study goal-driven AI. — AIXI
By “intelligence” we here mean something like skill at prediction, planning, and means–ends reasoning in general. — Bostrom, Superintelligence
We also see an overvaluation of our own peculiar cognitive idols and fetishes, including rationality and logic which, as I’ve argued elsewhere, are not pragmatic but a social utilities. Overall the discourse reflects a narrow, ML-focused appraisal of intelligence, one that frequently disregards other human pursuits like artistic expression as a form of simian weakness to be extirpated on the path to greater technological glory². We are somehow convinced that the same principles we currently employ in sub-human AI will hold for ASI, all while admitting that ASI is “incomprehensible”.
Are these the only ways one could characterize intelligence? Let’s, for a moment, consider an alternative.
Intelligence as we know it
At the start of this post we made the assumption that intelligence is indefinitely expansive, like physical strength. That may not be the case. Intelligence could be like the immune system. Although an overly weak immune system gives license to too many diseases, an overly strong one begins to attack the body itself, manifesting as what are known as autoimmune diseases. Like a police force, one that is too timid and one that is too strong are equally deleterious. The ideal is a balanced, homeostatic system, one that can adjust to unexpected situations without overreaching.
Similarly, intelligence may be at its best when targeted, focused, on-topic. In contrast to software, it may prove that bringing in too much contextual knowledge or piling on operations floods or overloads the system, the result resembling a person with severe autism or schizophrenia. An increase in speed or computational capability might provide a minor benefit until a point; any more and the agent’s experience of the world alters so fundamentally that the niche of reality in which the AI lives can no longer be meaningfully compared to ours — and nor can its motives, perspective, or ultimately its intelligence.
Even if we stick to our own parochial definitions of intelligence, any evaluation of super-intelligence must look not just at the type or quantity of internal processing, but also the feedback relationship between the mind and the environmental niche to which the intelligence is adapted. As with an immune system that must adapt to new diseases, an intelligent agent can’t know beforehand what behaviour will be optimal, especially when delving into uncharted intellectual domains. In humans excessive armchair pontification produces only limited returns; a mind must experiment with and learn from its surrounding reality to discover what is useful, that is, to become intelligent. And this process may not admit to being rushed or short-circuited.
Any intelligent agent (as we know it) must continually define and redefine the criteria of its intelligence by creating its own internal measures of success as it learns about the world. Utility — not generativity or prolific output — is the key feature of our definition of intelligence. Utility, in turn, is necessarily intertwined with the agent’s values, those to which its intelligent actions are presumably useful. Discussions about super-intelligence normally isolate values from intelligence and frame “intelligence” as analogous to “muscle strength”: at the service of, and being used by, an unrelated system of values:
Intelligence and motivation are in a sense orthogonal: we can think of them as two axes spanning a graph in which each point represents a logically possible artificial agent. — Bostrom, Superintelligence
But values and intelligence cannot be separated. As is well-known, the biggest difference between people who are considered academically intelligent and those who achieve political power is their values — the former love to explore, read, hear new ideas; the latter love mastery, financial control, the absence of surprises. The values of the engineer, the artist, and the despot are frequently at odds, their goals and aims incompatible, their definitions of “truth” incongruous. The singular trait we call intelligence is actually scattered over a myriad of ad hoc tasks and supporting real-world experimentation, each with their own values and aims. Intelligence is not an inherent power that can be uniformly optimized across a single agent.
Nor can we separate intelligence (as we know) it from the preferences of the society to which an agent’s behaviour is useful. Too often “high intelligence” or “genius” are characterized as inherent, measurable qualities of a given brain isolated from the values of the society to which it is adapted. It’s strange, then, that we frequently encounter cases of “undiscovered geniuses” like Van Gogh, Nietzsche, or Semelweis whose “brilliance” is only appreciated posthumously, as society’s needs, context, and judgments shift. After all, there are no “posthumously discovered” tall or strong people; we tend to be able to measure those immediately.
Geniuses wouldn’t be “geniuses” if no one around cared for their work; on the other hand tall people would still be tall. Our understanding of “great” human intelligence is necessarily intertwined with a surrounding society that judges its products as useful to its aims³. This is equally true for artists as for dictators. We have absolutely no example or precedent for an intelligence that is so self-sufficient it need not incorporate society’s larger aims into its values.
Furtive hand-wringing about such asocial intelligences are more the result of a fear-induced imagination than any known practical reality. They are science-fiction made to appear plausible by hand-waving references to hockey-stick charts, combined with a lack of any in-depth understanding of the nature of intelligence. They rely on the ignorance of the listener — and the speaker — to carry the argument.
No doubt there is, at the root of prognostications about super-intelligence a certain amount of anxiety regarding the dangers it may pose to human existence, and that uncertainty is the driver of exhortations to restraint. We visualize the worst of humanity and multiply it by some indefinite number. This suggests that nervousness about super-intelligence is at root nervousness about super-power, i.e. that an AI will be too powerful to control. That in turn equates intelligence with power, as though they were the same:
If someday we build machine brains that surpass human brains in general intelligence, then this new super intelligence could become very powerful — Bostrom, Superintelligence.
But consider for a moment the most powerful people alive today. Would you argue they are the same group as the most intelligent? One can perhaps glean some insight, and dare I say comfort, from contemplating that question.
¹ If bacteria could talk they would strongly disagree with any self-assessment of our inventions as “superior” — they own the planet, and will continue to own it long after we’re gone.
² Predictions regarding the threat of ASI also assume that the agent will remain as much an inscrutable black box as modern ML models are. For example, the assumption that a conniving ASI may hide its true intentions from us until it has gained a strategic advantage assumes that we can’t see its intentions — since that would include its intention to hide its intentions. Otherwise we would see its intention to conceal its true aims before it was actually able to do so.
³ Perhaps a future super-intelligence would be so far out of the ordinary, and its output so incomprehensible, as to be perpetually labelled a hopeless nut-case.
