The speech interfaces for interactive drama remains an interesting challenge. There are at least three reasons speech recognition is not feasible (at least in Façade): a) recognition is difficult in uncontrolled noisy environments, b) Façade limits the user to 8 words and there is no way to effectively truncate the user during the middle of a speech act, and c) Façade allows unconstrained language where most speech recognition software rely on limited vocabularies. To maintain some illusion of conversation, we must employee wizard methods. The question is how do we structure that interaction? As Michael suggests, we could give the player a hand held device with two buttons, one to enter the text when it finally appears and another to clear the text if the characters have already moved on to a different topic. Our earlier study points out that “natural” interfaces may not be ultimately what the player needs to have proper hooks into the game interaction. So I’m not against adding buttons. Timing is key. In our studies of desktop Façade, we noticed many players typed in words only to erase them later. This happened a lot. The explicit act of “entering” the text does not have an equivalent with the delayed speech interface.

I am actually investigating another approach. What if we tear out the NLP completely and task our wizard with selecting high-level discourse acts? So, instead of typing “Hi Trip, good to see you”, the wizard would select the built in Façade construct called “GREET”. Façade has about 30 such discourse acts with parameters. So it will force the wizard to be an expert with the interface and to know the story well, but it’s possible that this approach could lower the chance of communication breakdowns. It will in most cases be faster than typing, and it will likely eliminate some of the NLU breakdowns (the type 1 breakdowns mentioned above by Andrew). We have implemented this as a separate page in the wizard interface and it’s currently an option available to the docents at the Beall, so we will see. My initial observation is that some time delay still exists and that the player still needs some sort of feedback as to when the system “hears” their statement. I will report on what I find later.

This approach certainly does diminish the AI appeal of the piece to some extent. We tore out the natural language parser, but most of the AI engine remains untouched. In my view, we are still presented with an opportunity to riff on a player’s expressive input. Only now, we are using an intermediary–the wizard–who not only interprets the meaning of the player’s speech act, but can also read into the player’s gestures and emotions. If we did, in fact, enable a wider range of expressive possibilities for the player, how would this change the design of the behaviors for Trip and Grace?

We’re very aware that the wizard interface introduces another layer of noise and delay in the interaction, and that the fact the constraints imposed by Façade’s NLU are not made explicit in the (wizarded) voice interface is particularly problematic. However, it’s interesting to note that if we’d somehow magically had a robust speech recognition solution, this would not have solved the problem. Speech recognizers don’t recognize words one by one (since they’re using language models that contain statistics of word co-occurrences to compute the maximally probable textual representation of a speech signal), so the “display words as they are recognized and show when you hit a boundary” interface wouldn’t work. We talked about creating a speech recognition interface in which, after your speech has been “recognized” (in our case, by the wizard), the text would appear in a buffer at the bottom of the screen, with a truncation indication if it hit the buffer boundary; the player would then explicitly commit the recognized text, the speech equivalent of hitting the enter key in the text interface. But we decided not to implement this (at least for the first interface approach), as this requires giving the player an additional interface element to operate (such as a clicker they hold in their hand, or the introduction of meta speech commands which are not heard by the characters). This certainly begins interfering with the “naturalness” of the speech interface; it becomes something you have to start training on (like typing). All this is to say that speech doesn’t magically make interface issues go away; if anything, it makes the interface design issues more pressing, because a speech interface is supposed to be all about doing away with interface (you know, “natural”, “transparent”).

Our CHI 2007 paper did a detailed comparison study of player’s reactions to traditional desktop Façade, a wizarded speech recognition version of desktop Façade (just like the original, except you talk to it instead of typing to it) and AR Façade. The big result is that increased sense of presence does not necessarily yield increased engagement. In the HCI community, there is a long held belief that the more present you can make someone feel in an experience (and this is tied to notions of naturalness and transparency in the interface), the better (meaning more effective, more engaging) the experience will be. In our study we found this is not that case, and that this was not an effect of immature technology (if the underlying technologies were perfect, you’d still have the effect). It’s no surprise to media artists that mediation has positive value, and that you should explicitly design your mediation instead of wishing it away. But within the discourse of HCI, this has been rarely discussed.

We also have an AAMAS 2007 paper, using the same AI traces and interviews as for the CHI study, on the effectiveness of Facade’s NLP. Our approach was to note places in the retrospective protocols where player’s noted breakdowns in the conversation, and to correlate this with what was happening in the AI. There’s not a simple one-to-one relationship between failures in AI processing and perceived breakdowns in the experience. Sometimes NLP has a failure (like a failure to understand an utterance) and the player notices nothing unusual in the conversation, and sometimes NLP does exactly what it’s supposed to do, and the player is confused and feels like they are experiencing a bug (so this is a conversation design bug, rather than an NLP failure). We wanted to understand this relationship in detail. The paper reports some interesting results.

I also just want to point out that as a member of the AR Façade team and a GVU faculty member during the time the project was done, I was fully involved in the development of AR Façade. However, Steven Dow, a Ph.D. student at Tech, and someone who will be on the job market soon (you should hire him), was responsible for bringing the project to fruition. He served as the unofficial project manager, bringing all the pieces together, making sure that all the technology, experience design and physical design needed to pull off this project melded together into a coherent unity.