018 - Multimodal Interaction

Summary:

Adler et al describe a system where users draw sketches and interact verbally with the computer. The user can just talk and explain what they are drawing, or the computer will ask questions when unsure. The paper describes a Wizard of Oz type user study with an "experimenter" person sitting in pretending to be the computer. The experimenter asked the user questions to clarify the user's intent. He had his own screen to view the drawing, but avoided eye contact with the user. The study showed that user's tend to over-clarify their answers to simple questions, and they also pause their speech to finish their drawing.

Discussion:

Really some very interesting results. I can't help but think of the old starship computer on Star Trek when I think of talking and pressing buttons. Sadly, these results are still inconclusive in my mind, as the study WAS a Wizard of Oz and NOT a real computer. There is a HUGE transition between a person sitting across the table and a computer talking to you, both with the programming and with the attitude of the user. All in all, it's a very interesting idea, but a bit obscure. The paper was all about a user study, but further testing needs to be done for anything solid.