The real-time display idea is FANTASTIC. Consider: simply by providing players the opportunity to compose their own thoughts into words is an experience that a command-interface cannot offer. With a real-time display, you retain the precision of a command-interface while allowing players the vital immersive experience of composing their own thoughts into their own words.

That way, whatever qualms the player may have with the system’s interpretive capabilities will be directed towards the *system*, rather than the fellow characters in the drama. To put it bluntly, players will be calling the computer stupid, rather than Grace and Trip. That’s tremendously important — maintaining the illusion that *if only the parser had understood*, surely Grace and Trip would have responded as rich, complex human characters.

Natural language: I really like the idea of a real-time display showing how the words will be interpreted before the player speaks them. This avoids the main problem of natural-language input: you don’t know what the response will be, until it is too late.

Dialog menus: I agree that the dialog menu strategy is often used in a very limited way. There aren’t many options, and the NPC always responds in the same way to the same input. But these are not limitations with the dialog menu strategy *in principle*, just a problem with (most) instances of the strategy. Convicting the strategy of limitedness is like saying that socialism is in principle unworkable because it didn’t work in Soviet Russia. (There is, in other words, a difference between *didn’t work* and *couldn’t work*).

You say “A major [drawback with context-dependent menus] is that the player is limited to speaking only what the game designers have offered, typically only 3 or 4 choices at any one moment”.

There are two separate points here – first, that the player’s choice is limited to what the game understands, and, second, that the number of choices is very small. The first is true but unproblematic – we actively *want* the player to only be able to choose things which the system understands! The second is problematic, but isn’t an essential aspect of dialog menu choices. We can have a dialog-menu system with 50 or more interactions at any moment, as long as we group them with parent menus and sub-menus.

You describe the dialog-menu system – “Multiple-choice lists leads to subsequent multiple-choice lists, with the overall “conversation” having a branching-tree structure”. Again, dialog-menu systems *typically* have a branching-tree structure (where the choice of the current action is the sole determinant of the set of next available options), but they don’t have to. We *could* have a multiple-choice dialog system in which the set of choices is determined by, e.g, taking the actions available in the most probable states in a set of concurrent Hierarchical Markov Models (i.e. use state-estimation in POMDPs to drive the set of available choices)…

Your other main argument, that natural language input allows multiple-discourse acts, also applies to only *some* instances of the dialog menu strategy. A dialog menu choice can be tagged as simultaneously satisfying multiple discourse acts.

I think the dialog menu strategy has a lot of unexplored potential…

I guess I was a little surprised to see the numbers you report, as I’m one of those under the impression that the true misunderstanding error fraction was more than 20%. But like you say, from the player’s perspective it’s difficult to determine if the unexpected response is due to misunderstanding or one of the other explanations.

What I think would be more useful to report, however, is the *kind* of responses that fall into each category. After playing Facade a few times, I found that the most efficient communication occurred with very short (one or two word) responses — those situations requiring a yes or no response, or a simple decision between different options. If the “satisfying” response fraction was 33%, what proportion of those fell under the category of a simple one- or two-word response?

Likewise, of the 20% of “unsatisfying” responses, how many of those were misunderstood because they were longer phrases, or phrases that used more sophisticated “natural language”?

I think that’s a more accurate measure of a NLU system’s ability to interpret. If you’re succeeding mostly as simple responses that’s one thing, but if you’re really succeeding at some more complicated phrases, then that’s important to note as well. Perhaps a breakdown of % satisfying responses by number of words in the player’s input?

I also think it’s important to note that some of the errors that are probably classified as #2 (NPCs do understand, but are limited in response) may be related to the issues around typing in real-time, and failure to input a response in time for an appropriate reaction. I found on a few occasions that I didn’t type my response fast enough, and by the time I hit return they had already moved on to something else, resulting in a confused response.

This is at least an important consideration if, as you later note, you would like to engage non-gamers. Many of those are not going to be good typists, resulting in typos, the need for backspacing to correct, and just generally slow response time.

What we’re trying to do with Vespers3D is just an extension of the command-line interface to a graphical format. The content issue is a beast, as you know, but I think we’re finding that just having an audible and visual response to an IF-like command adds a pretty nice level of immersion over and above straight text IF. But it really heightens the desire for a true NLU interface.

- What do non-gamers want from games?
- Transparency in the behavior of, and interface to, NPCs
- HCI goals for NPCs
- The aesthetics of the interface to NPCs

Here I’ll clear up a few miscellaneous points.

Gilbert, you asked if NLU is just a facade for discourse acts under the hood, why bother with NLU and its drawbacks. Well, think about discourse acts as interpretations of the meaning of the words spoken. Extracting the meaning out of the player’s “raw” dialog will always be necessary in order to reason about them; even we humans do that, subconsciously. Instead we should debate how detailed or nuanced the interpretations need to be. Also, you suggested that the player’s text is always interpreted the same way in any context; that isn’t so. In fact, differing interpretations of the same text (e.g., “I think you’re funny”) in different contexts is one of the primary features of our approach to NLU (more on that in the transparency thread).

Breslin, I agree that it is a pretty thin claim to allow the player to merely be able to “speak” anything they want, without a guarantee that they are understood. Such a claim implies that the player will be understood; but in this discussion, we are actually addressing how well we are actually doing that.

That is one of the best arguments in there for NLU. It might even be worth further emphasizing that it’s not possible to issue any arbitrary combination of discourse acts with an NLU interface. What combinations you can use follow from the logical encoding of the language. It might even be conceivable that translating the game to, say, German would therefore also change the gameplay possibilities.

Ian,

You might also consider the aesthetic justifications for previewing discourse acts. If a player becomes frustrated with their inability to communicate to the machine causing them to lose their suspension of disbelief, then they have aesthetic grounds for complaint. True, having an intentionally imperfect NLU interface seems like a worthwhile aesthetic goal, (ie. mimicking human miscommunications) but excusing current technological shortcomings with an aesthetic argument seems a little disingenuous here.

This brings me to a question that relates somewhat to the comment of mine from nearly three years ago Andrew linked to above. One of the wonderful things about your position on NLU is that it is an aesthetic one. That is to say, you are making an argument for an interactive experience that uniquely embraces NLU rather than showing it off as a tech demo or using it for instrumental reasons.

In this post, however, I’m starting to see a conflation of aesthetic and market justifications. The question of feedback, such as “previewing” discourse acts before commiting them, is an example of the logical end of such justification. This is understandable as you guys try to figure out how to make this work more commercially viable than Façade. But it also strikes me as a somewhat dangerous concern, in that the NLU could become mired in an HCI-style argument about intuitiveness and transparency rather than aesthetics. I might be reaching, but it’s possible that one of the primary appeals of NLU in your work is in fact its aesthetic immoderation rather than its expressive intuitiveness. Back in 2004 I suggested that NLU in Façade is more about a creative signature than it is about usability. I wonder if emphasizing that position rather than dampening it might be a wiser creative move — and even possibly a wiser commercial move, although it is a riskier one.

> Richard is equating the term “speak to” as the same as “be understood”, where as I am
> considering “speak” or “type” to be the same as “utter”, separate from “be understood”.

No, I don’t think so. Richard is saying that if “speak” means “utter” as entirely apart from “be understood,” then “Façade attempts to allow the player to speak anything they like to the characters” is a pretty thin claim.

Gilbert Bernstein writes:

> Other than simultaneous input and combinatoric restriction of actions, it doesn’t seem
> clear that stripping away the Phase 1 parsing wouldn’t be an equally good approach.

That’s a big “other than.” It’s like you’re saying “other than the benefits of NLU, NLU doesn’t seem to have any benefits.” The most interesting part of andrew’s post for me is the fact that Phase 1 parsing allows multiple discourse acts with a single expression. I’ve never heard that argument in favor of NLU, and it sounds dead on. I’m a proponent of command-line interaction, but andrew’s post turned a light on for me.

> It seems to me that NLU interfaces (and interfaces in general) should be justified from
> the machine side as well as the player side.

There’s a pithy little game programmer’s motto, I learned from the guy who wrote Frogger: “you don’t have to solve the problem, so long as it looks like you did.” That said, I hold that the primary enjoyment of AI in games is learning the AI mechanism and how to mess with it. So for my money, “believable agents” is a fun trick, but the final goal is making a mechanism that’s fun to mess with. (If, on the other hand, “believable agents” were the final goal, then the results-oriented motto holds: it wouldn’t matter how many cheats were involved in achieving believability.)

But I entirely agree with everything else you said. I have much the same point of view.

Allowing players to express themselves would be a really good thing, but if the player realizes that the NLU interface is really just a facade for the discourse acts, (even if they don’t understand that technically) then the natural language processing might just become an obstacle they have to wrestle to issue discourse acts. The interface might reduce to a glorified version of an IF interpreter. If, say, typing “I think you’re funny” makes someone like you in pretty much any situation, you accrue a lot of disbelief in the computer’s ability to understand the player. If you instead expose a “compliment” button or action to the player explicitly, the repitition is built into the interaction, and so there’s no disbelief that needs suspending. Similarly enumerating options helps prevent the disillusionment of not being able to do whatever you want.

It seems to me that NLU interfaces (and interfaces in general) should be justified from the machine side as well as the player side. If the machine’s just going to remove these input nuances, (confusing and disappointing us in the process) is it really worth adding them in the first place? Maybe we should be looking for ways to make these nuances useful to the machine?

Secondly, were there specific design patterns you guys formed around problems 2a and 2c? When the NPC understands you, but can’t or doesn’t want to respond (and not for lacking content) – in terms of conveying that to the player, I mean. Specifically, it always seemed to me that giving NPCs a clear agenda to get information out of the player & communicating that agenda simplifies/reduces 2c. For example, let’s say the NPC is an interrogator and you’re a prisoner. Any attempt you make to derail the conversation and the interrogator will punish you to bring you back in line with their questioning. Or, let’s say it’s an undesirable guest at a party that’s hitting on you; they’re going to keep bringing the conversation back to that. Are there other types of solutions to those problems you’ve run into?

The dialog/interpretation feedback to the player as they type sounds really interesting though – definitely something worth experimenting with. I’d imagine the effect on immersion would be less than you think (the fact that the player was getting feedback at this skill of communicating with NPCs would have a much larger, opposite effect).

…the bulk of the players the games would ultimately appeal to — the largely untapped market of TV and movie lovers who dislike current games — would forgive the interface’s shortcomings to get a chance to finally play games that interest them.

I would have doubts about this point without some solid play-testing to back it up. Why is this new audience going to be more forgiving of technical problems than an audience which is already used to wrestling with the input systems of various digital games and systems? It may draw in a new audience, certainly, but it seems risky to assume that the appeal of a new genre is going to be strong enough to overcome usability issues. When your targeted audience are those who currently enjoy passive drama, couldn’t the usability and accessibility of the game be an even higher concern? Your games are turning the audience into the actor, and failure of the game’s comprehension may feel like a failure of the player in this new and unfamiliar role.

Has your previous play-testing actually shown otherwise? I’d gladly be proven wrong, I’m just concerned that you’re working with dangerous assumptions.

Of course, it may be that reducing the ‘misunderstanding’ rate down to 10% would be enough for anyone, gamer or non-gamer, to forgive and move on easily. Human-to-human conversation is robust in it’s own way, although it’s probably more forgiving of signal errors (“Sorry, what was that? I didn’t hear you.”) than semantic shortcomings. (Nobody asks, “What did that word mean?” outside of a very safe social context in my experience!) Perhaps the best way to avoid frustration is to find ways to steer conversations away from repeated semantic misunderstandings, to allow for the illusion that any confusion was a signal error? Maybe that illusion isn’t going to hold up in a typed input interface anyway, I don’t know. Slightly random idea which may have to wait for voice input.