How the mind comprehends the strange realities and interactions in video games

Video game interactions teach us about epistemology

Video game worlds, despite being entirely non-physical, often present a superficial appearance of substantial reality — 3D worlds with objects, dynamics, causes and effects, etc. As mere facsimiles of the real world, they inevitably lack many features that their creators simply weren’t able to put into them. Consequently, as players interact with and conceptualize these “partial” realities, epistemological patterns are born that are often in stark contrast to what they would be in the real world.

This makes games a great setting to learn about how fundamental concepts arise in your mind, since you can selectively remove and add aspects of concepts, even for transcendental concepts, in a manner that is not possible in real life; e.g. “physical” objects that vanish completely. In so doing, video games sometimes undermine past theories of mind and of metaphysics, as we realize for the first time that certain assumptions of what a mind necessarily does actually reflect empirical patterns in reality, not features of the mind itself.

In this post we will look in particular at how games create fragmented or partial concepts of space, object, and solidity, as well as a few others, and from these we will see that such concepts are created, piecemeal, through embodied interactions that serve the player’s motives.

Growing and shrinking space

One of the most interesting insights you can gain from video game worlds is how your mind forms a concept of distance. Consider the following:

How do you know how big a virtual world is compared to the real one? Are you playing with action figures in a toy box, or is this a massive planet-sized world?

In real life, this distinction would be easy. Naturally, you might assume that your mind has a built-in physiological intuition of distance that is determined by actual distances, say, in relation to your own body size. You may then use these to make decisions about where to walk, or which of two destinations is preferable. In addition, you can combine distance with other concepts like time, and build a derivative concept of speed, which is equal to the distance you can cover over a certain time.

speed = distance/time

In games, however, you can have no absolute sense of scale, even with the presence of an avatar character, since you can never be sure your character is meant to be human-sized. Often there may not be a visible avatar at all. In abstract games, moreover, there may be no other references for scale, like trees or cars. Yet you still seem to have a concept of distance in games, and might say that a particular destination is “far” when seen on a map. It even feels far.

There is no inherent way to tell how large something is supposed to be. The world above, though clearly microscopic, still feels relatively enormous.

Where does this feeling of distance come from? One straightforward answer involves inverting the equation above, and saying that the time it takes to get somewhere suggests the distance covered.

distance = speed x time

This makes intuitive sense, since the longer it takes to cover a distance, the further the distance feels. But there is a wrinkle in this theory. The time taken is not a simple function of distance. It depends on both your speed of movement, and whether there are impediments in the way.

It may surprise you to discover, then, that distances in games actually feel smaller the faster your avatar goes. You can see this most clearly when you enable “no-clip mode” — a development feature not intended to be used by players — which allows you to speedily move in any direction while passing through objects. As you speed up, the world feels smaller, regardless of what its 3D contents are supposed to represent. Even if it’s a realistic urban centre, it feels like a toy model. The addition of “fast travelling” to locations across the world map also makes landmarks feel closer together, in the same way that the invention of the airplane made our own planet feel small.

The world seems to “shrink” as you speed up. Source.

All this is the opposite of the equation above; an increase in speed decreases the perceived distance, whereas it should have increased it. Ultimately, the equation was misleading, since it assumed you can know your speed, which is impossible if you don’t already know the distance travelled. A more likely explanation is that your mind always assumes you haven’t changed your speed, since it doesn’t sense any changes in your vestibular (inner ear) acceleration. The increase in the motions of objects around you are then attributed to the world itself getting smaller.

This lends credence to the idea that time alone is the deciding factor, rather than speed or visual cues. However, this hypothesis also has problems, not the least of which is that humans don’t have an perfect sense of time. Time seems to fly by when you’re busy and slows when you’re bored or uncomfortable. So the more annoyances or resistance the virtual world musters against you — like bushes, or hills to climb, or bandits and monsters you must stop to fight, or swamps that slow your movement — the farther a journey might feel.

Despite having clear references for size (the human in the racer), the world in Pod Racer feels toy-sized due to both speed and fluidity of motion.

Another problem is that distance isn’t solely about the time taken — if you stay still or dawdle you would not assume the world was enormous. You can only know that travelling took a long time if you know what the appropriate time for that distance should be. Otherwise you would guess that your character was simply very slow. But this is exactly the issue under discussion — how do you even know you’ve moved unless you know that you’ve covered some distance? It must on some level be about the time taken in moving from one scenario to a different one — e.g. from your base to the shop. You know you’ve moved because you now have access to something you didn’t earlier, or there is a change in visible scenery. Your attempt to access that new location, the impediments to getting there, the tediousness of the change, and of course any reference points along the way must all be included in your attempt to define distance.

Distance, then, is not given to you by the world, or through some biological function, it is something you create anew as you explore every novel situation, and every new game world, based on the difficulties you encounter in movement. You can clearly see this act of invention in certain extreme cases like Hyperbolica, which takes place in a non-Euclidean space. Most people have never encountered such spaces before, and so they must learn to traverse it from scratch, with no prior experience to guide them:

Hyperbolica. After playing this game for a while, you start to build an intuitive sense of its warped space and distances.

It seems that the earlier question was actually a trick one — you have no objective way to compare the distances in games with those in the world. The fact that this same concept of distance can still be applied independently to both the real world and to virtual ones suggests that it is a learned abstraction without any psychophysical reality — otherwise you would innately be able to compare the two.

If, as we’ve shown, there is no universal measure of distance, and if you invent it for every new situation, why do you still use the term “far” in both cases as though people should understand what you’re saying? The answer is that “far” does not mean, for your mind at least, a long distance, but rather a trying journey, one where the scenery frequently changes, but you don’t get where you want to be. This is the only factor common to both virtual and physical spaces, and one that others would also understand. “Far”, in a sense, is a complaint. The metaphorical use of the term “far” reflects this: “you’ve come so far” is not about the absolute measure of achievement (e.g. winning the lottery doesn’t count), it means you have overcome many difficulties on your journey.

Purely abstract games have no sense of scale.

This interpretation of distance is a novel contribution to our self-understanding that video games have helped us see. It is only because we were able to separate distance into its several factors (speed, time, difficulty, vestibular acceleration), with each factor varying independently, that we were able to come to this conclusion. Such insights may never have happened in the real world, since these factors cannot be separated from each other in physical reality.

Simulated solidity

Turning on no-clip mode reveals another facet of how you build fundamental concepts; namely, how you learn about solidity. Take, for example, the concept of a wall. As you pass through walls in no-clip mode, they suddenly feel flimsy and insubstantial. Although they still look like walls, they no longer feel like walls. The experience shows that the concept of a wall is not based on its appearance — bricks or stones or concrete — but about whether it hinders you from getting where you want to be. This is why video games can have illusory walls (things that look like walls but which you can pass through) and invisible walls (areas with no visible objects, but which cannot be traversed). Only the latter are actually considered walls; the phrase illusory wall means “not really a wall”.

There are many well-known theories in cognitive science which all suggest that concepts, like wall, are derived from similarities of appearance or by associations with other stimuli. The above example of invisible walls shows that the theory does not hold up in practice. The appearance of bricks or concrete textures is not what you use to designate an object as a wall. Wall is an embodied interaction defined by its relationship to your motives. You are trying to get somewhere, and are prevented. Or perhaps your enemy is trying to come near you and is blocked. Either way — directly or through imagined empathy with an enemy — you identify it as a wall because of this effect on your motives.

Solidity is a more general concept that derives from the same source. However, solidity includes a spate of other motivated interactions too. For example: “can I use this to support myself, to get where I need to go?” In these cases it becomes perceived as a floor rather than a wall. In the real world, all objects hard enough to be walls can also be floors; thus you might assume that there is a common concept behind them called solidity. But in software the concept can be fragmented: there are cases where an object is not a hindrance to horizontal motion, but can still support your character (e.g. a floor with zero thickness). Or it may prevent motion in one direction, but not the opposite direction (a one-way wall). Surprisingly, your mind has no difficulty learning to interact with these unnatural examples.

Glitches like this can allow you to pass in parallel through walls, but not perpendicular to them.

Such cases show how your mind can split apart the various aspects of the concept of solidity, and designate each by your special interactions with it. This makes it possible for your mind to learn about and perceive solidity in a piecemeal manner. Before video games, such a thing would have been inconceivable — they were always united.

Half-objects

There are a plethora of other partial interactions that only happen in video games, and the way we treat them provides clues to how we construct concepts in everyday life. For example, items in video games — bottles, herbs, parchment, etc. — could at first glance be interpreted as analogous to physical objects in the real world. However, such items may have only a few of the features you’d expect from their real-life counterparts. Consider the following list of properties you assume physical objects should have:

1. An object is in exactly one place at any given time
2. It stays where you put it relative to other objects
3. It takes up space if carried on your body
4. It weighs you down if carried on your body
5. It cannot be duplicated (Conservation of Matter)
6. Using an object does not destroy its material substrate or its atoms
7. It can be used in numerous other ways as a solid object, e.g. thrown at someone, used to jam a door, used to weigh down a floating object, etc.

Any one or more of these may be missing as a feature of “objects” in video games, whereas in the real world all of these either appear together, or are not present at all. Your apparently unified concept of a physical object is actually a sum of many individual interactions, and these can be discovered and addressed separately, based on what you need them for. For example, understanding that objects weigh you down is learned quite early in life when you find you can’t run while carrying a heavy toy. That they cannot be duplicated becomes evident when you are forced to share your toys with others. Learning that objects stay where you put them is useful when you need to drop one for the moment, but also to retrieve it again later. These same motives drive you to comprehend the various properties of items in video games.

Infinite carrying capacity turns items into their bare numbers rather than real objects

In some games where a large portion of these features are missing, there is not much “objective” reality left in the objects except that they serve as mathematical constraints — e.g. “how many potions do I have left, and how much life does an enemy’s hit subtract from my health?” The mind ultimately treats such collected items as numbers, or like playing cards, and not the thing they are supposed to represent.

A similar dilution can be observed with in-game weapons. It is a common experience that weapons in games begin to lose their full meaning, as you realize that your gun, which would be fatal in real life, doesn’t do much to hinder apparently flesh-and-blood enemies, nor even slow them down until they are completely destroyed. Because of this simplification, players start to conceive of weapons by the amount of damage they inflict per second (DPS). They become a sort of “number-decrementer” combined with a test of hand-eye coordination, since for the player’s purposes that is all they are. Besides eventually neutralizing a threat they do not possess any other useful interactions you’d get from such weapons. And so they cease to be guns or swords, regardless of how impressive they look.

Fights often get reduced to number games, with none of the meaning of the weapons left in them. The yellow numbers indicate various increases and decreases of player properties.

All unique objects in a game gain their meaning solely by their utility to the player. Every partial concept in a video game gets reduced in your mind to the subset of your needs it satisfies. Imagine a fire in a game that burns enemies around you but not yourself. Such a thing is impossible in real life, which gives real fire its perceived uniformity of effect. Of course it might be useful in real life to be immune to such a damaging element, since you could then use it as a weapon. And so the player perceives such fire not as “fire”, but as a weapon with a large area of effect. Were the fire more unconstrained, and could damage the player, it would take on a more complete essence of “fire”: chaotic, impulsive, dangerous, etc. Once again, what was otherwise a unified concept (fire) gets reduced to a collection of its momentarily useful effects, as fragments of a concept.

Dead landscapes

It can be strange to think that all concepts in your mind get their primary meaning for you only insofar as they are useful to your motives. The impulse to view them as objective entities is quite strong¹. It is generally simpler to project the meaning of concepts into the external objects themselves, and assume that was their source.

This unfortunately leads many game developers to lean on a graphic designer’s representation of the scenery and items. They build the asset and assume it will be interpreted as such by the player, with the full impact of the concept it apparently represents. But assets are not the units of a video game experience, interactions are. The landscape of a game is as wide or as narrow as these interactions. It is a mistake to think that appearances, or even dynamics, signify the meaning of concepts. This mistake is our present inheritance from the empiricist school of philosophy, one which we have yet to dislodge from cognitive science.

I should say that superficial appearance can still guide a player to initially think about the world in a certain way based on their past experiences. Appearances are the setup, the cue, not the delivery. For example, the appearance of guns indicates they are a means to neutralize other characters who would otherwise damage you, even if the guns are not actually full of explosive bullets. Showing open fields and mountain landscapes hints at the possibility of wide exploration. But if you can’t explore or shoot, the situation becomes frustrating since the cue telling you that you should use it to address your goals was deceptive, like sitting on a chair that vanishes from beneath you, or seeing a door you can’t open.

Look at all that beautiful stuff you can’t touch or interact with. From The Order: 1886. Source.

There are many games which feel stifling because they present you with a wide diversity of lush scenery and objects but simply lack the fullness of the concepts represented by the imagery. In recent Resident Evil games, the few, rare objects that you can interact with are painted yellow to stand out from the rest of the visual clutter. As you play, you find that your mind quickly filters out the remainder of the world as meaningless except as an impediment to motion. Like the plexiglass sets and scenery you find in an amusement park, it all becomes undifferentiated “matter” or substance in the purest sense; merely impenetrable stuff, since that is the only way those stimuli influence your interactions and goals.

Everything in Resident Evil 4 not marked yellow is reduced to an obstacle. Source.

In order to properly shape your perspective of them, you must be able to use the various parts of the scenery to achieve some goal. And just as with collectible items, how you use the scenery defines what it will be conceptualized as. For example a bridge is not simply an asset that looks like a stone arch, it gains its meaning as bridge by enabling you to get across a gap you wanted to cross, but otherwise couldn’t. That meaning is reinforced the more desperately you need to cross. A bridge that you can’t cross, or a bridge that is unnecessary since your avatar can fly freely, ceases to have any meaning as a bridge — even though it may still look like one when compared to your memory of other bridges. Such pseudo-bridges are merely borrowing their meaning from an initial reflex association; and they will lose it just as quickly.

A world of affordances

More generally, it is an error to separate game nouns (objects in a game) from game actions (what you do with them), as is often done in game design, since the action and its consequences are themselves what defines the meaning of a noun. A gun is not a gun unless you can use it as a gun; and it is only a gun to the degree that it is effective. Assets that you can’t interact inevitably lose their impact and emotion.

The purpose of this post is to show how your understanding of the real world arises in the same way. Both the world of video games and the real world are conceived, in your mind, through affordances. An “affordance” is a perceived object that is framed around its utility to you — such as a “door handle” being defined by how easy it makes opening a door. This applies not only to items and scenery in a game, but also, as we saw, in relation to space, object, or solidity, which we rarely think of as affordances.

Thinking of concepts with respect to how they influence a player’s motives is also a useful framework for designing games, as it helps avoid cargo cult thinking — that is, confusing appearance for meaning. Appearances may help frame the interpretation of experiences (e.g. you are in outer space, or on a farm) but they will not resonate unless tied to a correlated desire. A setting involving pirate ships and tropical islands looses all its meaning unless the player can freely explore fascinating locations and make their fortune through dastardly deeds.

Makers of video games have the unique opportunity to shape our experiences directly through finely crafting virtual interactions. The very thing that made Roger Ebert believe that games weren’t art, namely their interactive nature, is in fact their fundamental meaning as art. Games have the ability to directly make contact with our epistemology, by forcing us to interact with their features towards a goal they themselves define. The consequences the creators impose on your interactions become the essential units of their artistic expression. There is no doubt that many games can feel either miserable or empowering, claustrophobic or freeing, and not just because of their visual style, but because of the concepts players are lead to create through their motivated interactions with them.

¹ Objective reality is a set of affordances that are useful for academics, scientists, and nitpickers.