Grand Text Auto

Figure 1.1: Authoring data and process.

A few paragraphs ago I said that the possibility of creating new simulated machines, of defining new computational behaviors, is the great opportunity that digital media offers. Seizing this opportunity requires a bit of a shift. It is common to think of the work of authoring, the work of creating media, as the work of writing text, composing images, arranging sound, and so on. But now one must think of authoring new processes as an important element of media creation.

3

In undertaking this shift, it may be helpful to think of the creation of a piece of digital media as being organized like figure 1.1. The work is made up of data and process, with a somewhat fuzzy line between them.¹ The data elements are mostly pre-created media (text, still images, video and animation, sound and music) and the sorts of things that are stored in spreadsheets (lists and tables of information, with varying degrees of structure).

The processes, on the other hand, are the working parts of the simulated machine. Some are dedicated to tasks with simple structures, such as displaying a series of video images on a screen. But many of digital media’s tasks are more complex in structure, requiring processes capable of performing in a range of different ways. Even a simple piece of digital media such as Pong (figure 1.2) has processes that define behaviors much more complex than showing a series of images in quick succession. The processes of Pong define and calculate simple rules of physics (how the ball bounces off the paddles and walls) and simple game rules (who receives each serve, how points are scored, and how winning is achieved) that, when well-tuned, can combine to create a compelling experience of gameplay — even in the face of remarkably primitive graphics.

Figure 1.2: The iconic early video game Pong gives players a simple goal: to use their simulated paddles to knock back a simulated ball — keeping it in play until one player misses, causing the other player to score.

3

Of course, the idea of creating media through the authoring of novel processes is not new. Tristan Tzara’s Dada cut-up technique was presented, in the wake of World War One, as a process for turning a chosen newspaper article into a poem. On a more technological level, the pioneers of early cinema had to develop novel processes (embodied in physical machinery) to capture and display their sets of image data. And, on a longer-term level, the creation of board and card games has always primarily been the development of process definitions, embodied in game rules, that determine how play moves forward.

In important ways the non-computational media processes mentioned above are like the processes of digital media: they are defined previously, but (at least in part) carried out during the time of audience experience. This is true as Tzara pulls a paper scrap from his sack, as the Zoetrope image flickers, as the poker hand goes through another round of betting, and as the image of a Pong ball bounces off the image of a Pong paddle. The processes of digital media are, however, separated from non-computational media processes by their potential numerousness, repetition, and complexity. For example, we might play a game of tennis using the rules of Pong — they’re simpler than the normal rules of tennis. But we wouldn’t want to play Pong as a board game, having to hand-execute all the processes involved even in its (extremely simplified) modeling of physics. It is the computer’s ability to carry out processes of significant magnitude (at least in part during the time of audience experience) that enables digital media that create a wide variety of possible experiences, respond to context, evolve over time, and interact with audiences.

7

Process intensity

Returning to data and process, we might think of Pong and many other early computer games (e.g., Tetris) as being authored almost entirely in terms of processes, rather than data.² An “e-book,” on the other hand, might be just the opposite — a digital media artifact authored almost completely by the arrangement of pre-created text and image data. In an influential 1987 article, game designer and digital media theorist Chris Crawford coined the phrase “process intensity” to describe a work’s balance between process and data (what he called its “crunch per bits ratio”).

Crawford points out that, in early discussions of personal computers, certain genres of software failed despite widespread belief that they would be attractive — specifically, he cites checkbook balancing software and kitchen recipe software. He argues that these genres failed for the same reason that the 1980s computer game hit Dragon’s Lair (which played sequences of canned animation, rather than dynamically drawing graphics to the screen) was a dead end, rather than the first example of a new game genre. In all these cases, the software is designed with low process intensity. In fact, Crawford goes so far as to argue that process intensity “provides us with a useful criterion for evaluating the value of any piece of software.”

In Crawford’s article games other than Dragon’s Lair come out quite positively. He writes, “games in general boast the highest crunch per bit ratios in the computing world.” But Crawford wrote in 1987. Almost two decades later, game designer and theorist Greg Costikyan gave a keynote address at the 2006 ACM SIGGRAPH Sandbox Symposium titled “Designing Games for Process Intensity” — reaching a rather different conclusion. As Costikyan writes in a blog post from the same year:

Today, 80+% of the man-hours (and cost) for a game is in the creation of art assets. In other words, we’ve spent the last three decades focusing on data intensity instead of process intensity.

4

In fact, the shift has been so profound as to call for a rethinking of the very concept of process intensity. The games cited by Crawford — such as Flight Simulator and Crawford’s own game of political struggle, Balance of Power — use much of their processing toward the game’s novel behavior. However, in the time between Crawford’s and Costikyan’s statements the graphics-led data-intensive shift in computer games has not only increased the amount of effort placed in creating static art assets. It has also driven an increasing share of processing toward greatly improved visuals for remarkably stagnant behavior. While this represents an increase in processing, it’s the same increase that could be achieved by taking a kitchen recipe program and adding live 3D extrusion of the typeface, with the letters coated in simulated chrome and glinting with the latest lighting effects. Executing these computationally expensive graphical effects would send the recipe program’s process intensity through the roof ... while running completely counter to Crawford’s ideas.

2

This kind of distinction — between processing used for graphics and processing used for behavior — is not only of interest to game developers. It is also a distinction understood by players. For example, as Jesper Juul (2005) and others have pointed out, it is not uncommon for players of PC games to choose a lower level of graphical rendering (e.g., in order to increase the responsiveness of the interface or reduce the visual weight of elements not important to the gameplay). Players who choose to lower levels of graphical processing are not considered to be playing significantly differently from players who choose higher levels. On the other hand, some games also allow players to vary the level of artificial intelligence processing employed by the system. This changes the game’s behavior by, for example, making computer-controlled opponents easier to defeat (e.g., in computer chess or a first-person shooter). Players view this type of change, a change in behavior-oriented processing, as a much more significant change to gameplay.

10

Players have also “voted with their feet” in favor of behavioral processing. While many games pursue increasingly photorealistic graphical rendering, Will Wright and his team at Maxis designed The Sims around low-end graphics and comparatively complex behavioral landscapes. The game’s publisher, Electronic Arts, at first resisted the title — in part because its process-intensive design had created innovative, unproven gameplay focused on managing the lives of simulated suburban characters. But when The Sims was released it became the best-selling PC game of all time. It accomplished this in part by reaching a significantly wider audience than the “hard core” (stereotypically, young males) toward whom most computer games seem to cater. However, despite the success of The Sims and the fact that game company executives regularly express the desire to reach wider demographics, innovation at the level of behavior-oriented processes is still largely resisted within the game industries, viewed as a risky alternative to the tried-and-true approach of combining flashier graphics with the same gameplay behaviors as previous data-intensive hits.

2

This book’s focus is on what systems do — what they enact, how they behave — rather than what the surface output looks like. This could be characterized as an interest in “behavioral process intensity” of the sort practiced by digital media designers like Wright (which is probably what Crawford meant from the outset). As is likely already apparent, this will bring a significant amount of “artificial intelligence” into the discussion.

Expressive AI

The problem with artificial intelligence (or “AI”) is that, in trying to capture the structure of the world or the way reasoning works, it always captures someone’s idea of how things are, rather than any transcendental truth. Of course, this isn’t a problem in all contexts, but it is when trying to understand human intelligence (the overlap of AI and cognitive science) or when trying to create a software system that acts intelligently in a real-world context (most other uses of AI). This, in part, is why the most prominent AI efforts of recent years have been statistically-driven approaches to very focused problems (e.g., Google’s search results, Amazon’s recommendation system) rather than hand-authored approaches to large problems (e.g., general-purpose reasoning).

However, when it comes to media, the goals are no longer general-purpose. Rather, the authoring of media is precisely the presentation of “someone’s idea” of something. For fiction, it’s someone’s idea of people, of stories, of language, of what it means to be alive.

Given this, if we look at the history of artificial intelligence from the perspective of media, we see something other than a sad collection of failed attempts at objectivity and universality. Rather, we see a rich set of tools for expressing and making operational particular authorial visions. This is the shift marked by Michael Mateas (an AI researcher, artist, and game developer) in naming his practice “Expressive AI.”³ As Mateas puts it:

Expressive AI views a system as a performance of the author’s ideas. The system is both a messenger for and a message from the author. (Mateas, 2002, 63)

Of course, from the point of view of digital media (rather than AI) Mateas is saying something rather everyday. For example, Ted Nelson, in a 1970 article later reprinted in his seminal book Computer Lib / Dream Machines (1974), described “hyper-media” computational systems that would embody and perform authorial ideas — more than three decades before Mateas. Similarly, designers of computer games clearly author processes to embody and perform their ideas for audience experience. But both hypermedia and computer game designers have been content, largely, with data-intensive approaches, while AI has traditionally developed process-intensive solutions. And it is Mateas’s approach of combining AI’s process intensity with the authorial perspective of digital media and games that has allowed him to co-author groundbreaking digital fictions such as Terminal Time (perhaps the only successful story generation system yet created) and Façade (the first true interactive drama) — both of which will be discussed further in coming pages.

For this book’s purposes, of course, the important issue is not whether any particular technique arises from, or connects to, traditions in AI. Rather, it is the potential for using process-intensive techniques to express authorial perspectives through behavior. This brings me to one of the two meanings for “expressive processing” in this book: a broadening of Mateas’s term, beyond AI and into the processing that enables digital media in general.

Notes

¹Though the concepts of “data” and “process” seem clear enough as ideas, in practice any element of a system may be a mixture between the two. For example, the text handled by a web application is generally thought of as data. However, this textual data is often a mixture of plain text and markup language tags (from an early version of HTML or an XML-defined markup language). These markup language tags, in turn, may define or invoke processes, either on the server or in the web browsers of the site’s audience. Luckily, this sort of intermingling (and more complex cases, as when a process is used to generate data that might as easily have been stored in the system initially) does little to diminish the basic usefulness of the concepts.

²It is perhaps worth clarifying that my argument here is not that authoring digital media requires authoring both data and processes. The data and process elements of a work of digital media may be newly-authored, selected from found sources (e.g., found footage is still data and the vendor-supplied behaviors in an authoring tool such as Flash are still processes), or even largely undefined at the time of authoring (and instead defined by processes executed at the time of audience experience). In any case, they will rest on a foundation of process and data that make up the platform(s) on which the work operates.

³I pick out Mateas because of his particular interest in fiction and games. But similar shifts have been undertaken by a number of other prominent young researchers with AI backgrounds, such as Phoebe Sengers and Warren Sack.

Continue reading...