Skip to Main content Skip to Navigation
Conference papers

Managing Consistency in Wizard of Oz Studies: A Challenge of Prototyping Natural Language Interactions

Stephan Schlögl 1, 2 Gavin Doherty Saturnino Luz 
1 MM - Multimédia
LTCI - Laboratoire Traitement et Communication de l'Information
Abstract : Wizard of OZ (WOZ) is a prototyping method that uses a human `wizard' to mimic the functions of a prospective system. Although the WOZ technique is widely used in HCI design, surprisingly little research has been done on how wizard performance and consistency may effect the findings of a WOZ trial. In this paper we present a meta-analysis of three WOZ studies that shows how wizard behavior can vary significantly, potentially influencing the user experience of test-participants. In text-based interactions, for example, wizards may be required to send several consecutive utterances before receiving any feedback. In order to deliver a consistent experience they then need to give test participants enough time for reading and processing an utterance before sending the next utterance. In this study we examined this aspect in some more detail and found that these time-windows can vary significantly within as well as between studies. By contrast, a similar experimental setup simulating speech-based system output did not suffer from this type of inconsistency. Our analysis highlights one of the challenges involved in using the WOZ method for designing novel interactive systems (both stationary and mobile), and suggests that additional support for wizards is needed in order to improve consistency.
Complete list of metadata
Contributor : TelecomParis HAL Connect in order to contact the contributor
Submitted on : Saturday, September 14, 2019 - 6:44:30 PM
Last modification on : Tuesday, October 19, 2021 - 11:15:24 AM


  • HAL Id : hal-02288377, version 1



Stephan Schlögl, Gavin Doherty, Saturnino Luz. Managing Consistency in Wizard of Oz Studies: A Challenge of Prototyping Natural Language Interactions. MobileHCI Workshop on Prototyping to Support the Interaction Designing in Mobile Application Development (PID-MAD 2013), Aug 2013, Munich, Germany. ⟨hal-02288377⟩



Record views