The "Wizard of Oz Method" is used to effectively collect the real feelings and behaviors of users in the process of using the product, so how does this method apply to the user experience, and what are its application scenarios and research methods?
The Wizard of Oz is a reference to the famous fairy tale The Wizard of Oz, in which Dorothy and her friends first encounter a large, powerful-looking wizard, only to find out that it is nothing more than an ordinary person operating behind a curtain. The purpose of this method in UX design is to effectively collect the real feelings and behaviors of users in the process of using the product, and the users are not aware of the existence of human operations behind the scenes.
1. What is the Wizard of Oz Law?
The Wizard of Oz Method in User Experience (UX) design is a research and testing approach that focuses on convincing users that they are interacting with a fully automated system, when in reality, some or all of that system is being operated by humans behind the scenes.
The Wizard of Oz approach helps teams test designs driven by complex technologies at a lower cost. Instead of building the technology itself, designers can simulate the responses that the technology might provide by having a person "play" the role of the system.
The Wizard of Oz method is often used in moderated usability tests. Similar to traditional moderated usability testing, conducting a Wizard of Oz study requires a coordinator and a target user. In addition, someone needs to play the role of a "wizard" who is responsible for selecting or generating responses to the interface. This approach is somewhat similar to testing a paper prototype (which may require someone to act as a computer), but in the Wizard of Oz, the design can be digitized and the user is not visible to the person who provides the user with a systematic response.
2. Application scenarios
The Wizard of Oz method is particularly useful for testing new types of interfaces built on advanced techniques when they can't be effectively tested with prototypes with static content alone. The types of interfaces involved include:
- Conversational user interface (UI) similar to chatbots
- Uses learning algorithms to provide users with an interface for recommending content
- An interface that enables real-time retrieval of information and display the results to the user
Scenarios covered include:
- Early prototype testing: When the product prototype is still in the early stages and some features have not yet been developed, the Wizard of Oz method can be used to simulate these features and observe how users react to these envisioned features and use them.
- Voice and conversational interfaces: When designing voice or conversational interfaces, such as smart assistants, this approach to simulate conversations can test the natural flow and comprehension of a user's conversation without writing complex code.
- Complex interaction simulation: For some interactions that are technically difficult to implement or require high-cost development, the "Wizard of Oz" method can be used to evaluate whether the interaction design really meets the needs of users and avoid unnecessary resource investment.
For example, the researchers used the Wizard of Oz method in the following research projects:
In order to improve the user experience of a tech retailer's website customer service bot, the team explored how to provide useful personalized service suggestions based on the user's device through the Wizard of Oz method. In the course of talking to users, the designers update the interface prototype in Figma in real time.
In designing a new voice assistant project, the Wizard of Oz method was used to understand how users would communicate with the new voice assistant. A Bluetooth speaker is placed in a cardboard model, and when the user communicates with the model, a voice assistant reply generated by a text-to-speech application is transmitted to the user through the Bluetooth speaker.
3. The research steps of the Wizard of Oz method
Conducting a Wizard of Oz study requires the cooperation of multiple parties. Please refer to these five key steps.
Step 1: Define the test objectives: Define what aspects of the product you want to test with this approach, whether it's functionality, cloud interface, or overall user experience. (Create a prototype)
To conduct your research, you'll need to prepare a new design model for user interaction. This may take the form of:
- Prototypes in some design software such as Figma
- Written code prototypes
- Existing technology as an agent for new functionality (e.g., leveraging an existing messaging platform to simulate a new chatbot)
If you're planning to use design software (like Figma), you'll need to find a way to make model updates quickly. For example, you can prepare components in advance for elements of the model that need to be modified during the study. This way, the operator does not have to go through the hassle of finding and modifying specific parts of the design little by little.
(If you're using Figma, its multiple editing features make it easy to update the same UI element in multiple components at once.) ）
Step 2: Operation: Design and build a behind-the-scenes operating system that simulates real-world functionality. This system can be as simple as manual manual sending of a message to simulate an automated response from the software. (Determine how the operator will reply)
- Fixed Responses: The actor picks the answer from a fixed list of responses.
- Impromptu reply: The operator creates a reply in real time according to the situation in progress.
- Hybrid responses: Actors can either choose from a minimalist list of replies or create new replies if necessary.
The fixation method is useful when the system may give only a few answers.
Improvisational methods are more useful for interfaces that provide smooth user interaction, such as conversational UIs, where it can be difficult to anticipate and provide preset responses.
The hybrid approach provides maximum flexibility; some responses are predetermined, but if no suitable one is available, a new response can be created.
Step 3: Pick Participants: Select a target group of users as test participants. (Development of research protocol)
The research protocol is a detailed overview of the research objectives and how the study will be conducted. It will guide the specific behavior of the operator and facilitator during the study session.
In addition to all the elements that would normally be included in a test plan (such as the tasks you would give to the user), the Wizard of Oz study should include:
- An overview of the division of roles including who is the coordinator of the session and who is the operator
- Questions from the facilitator to the user at the beginning of the session that may affect the operator's feedback during the session (if applicable)
- Operator-controlled design elements and control methods (e.g., decision trees, step-by-step instructions, or screenshots that the operator can refer to)
If a fixed or hybrid response is selected, the actor can choose from the response (for example, you can select "Loading" when the actor needs extra time...... Please wait", or you can select "Under Construction" if the user's interaction exceeds expectations)
Guidelines to follow if the operator is temporarily creating new system feedback in the middle of the session (e.g., intonation suggestions if the operator needs to act as a chatbot)
Step 4: Test and collect data: Engage users with operating the system or product in simulated use cases and collect their behavior, feedback, and suggestions. (Selection and training of operators)
Operators need to be aware of:
Product concept and design: The operator should be clear about the original design intention of the product and how it operates. Ideally, they should also be aware of the technical limitations that may exist so that they can prevent answers that are not actually possible with the system.
Responses they will provide: The operator may have been involved in the development of the response when performing fixed or mixed method testing.
If the operator needs to change elements in the prototype, they should also be familiar with the prototyping software or programming language used.
Often, a designer or developer might act as an operator. Before the start of the study, the operator should be fully prepared so that the test can proceed smoothly. Possible preparations include giving the operator a preview of the research process, practicing answering or revising the prototype, or even inviting the operator to participate in a rehearsal test (see step 5 for details).
Step 5: Analyze and iterate: Analyze user feedback and behavior, and adjust and optimize product design based on this information. (Rehearsal testing conducted)
Since the process involves multiple steps, running a test run ensures that each part works as planned and that the operator can react quickly. You can invite friends, colleagues, or actual users to participate in the test. A trial run can give the operator a chance to practice before the real study, and it can also help you avoid time-consuming technical issues during the real study.
In the process of doing rehearsal testing, you may find that you need to add some new responses that you didn't expect beforehand. Such findings can help you adjust and refine your research protocol before the study begins.
Is it necessary to reveal the identity of the operator?
The presence of the operator is not usually disclosed to the study participants to ensure that true behavioral data is collected during the session. However, sometimes participants may guess that they are providing a response, especially if the design fidelity is low or the participant has extensive technical knowledge. In this case, the method is closer to role-playing.
Don't worry if a participant guesses or asks if the response was human-generated. Just remind them to act as if they were interacting with a real system.
In user research sessions involving deception, a briefing at the end of the session is required to ensure that participants are aware of accurate information about the study and can decide whether they are willing to withdraw or not.
There is no need to disclose that the system response was human-generated unless the participant would be harmed by not understanding the information.
Fourth, the origin of the method
The Wizard of Oz method was first documented by Don Norman and Allen Munro in 1973 and used to test automated airport computer terminals for travel assistants. The name of this method was coined by researcher Jeff Kelley in 1983 in a paper on natural language interfaces at Johns Hopkins University. These fundamental studies explore the interaction of users with natural language interfaces in the early stages of the technology's development.
5. Advantages and disadvantages
Merit:
The Wizard of Oz approach reduces investment risk in complex technologies such as generative AI by providing an initial understanding of the anticipation, practicality, and operability of a product early in the context of a company's large investment.
- Prove your concept quickly and at low cost.
- Provide real-world user communication and interaction data to help understand user needs.
- Easy to implement and modify.
The Wizard of Oz method is often used when developing a minimum viable product (MVP). A well-known example of an MVP is the online shoe store Zappos, which has also applied this approach. Before investing in infrastructure such as warehouse, inventory and service automation, founder Nick Swemen first validated the company's business model by executing orders himself. After a user places an order on his website, Wiseman personally buys the shoes and sends them to the customer as a way to test the market.
Shortcoming:
- Behind the scenes is required, which may affect the accuracy of the test results due to the negligence of the operator.
- Users may feel deceived after learning the truth.
- The scale of the tests is usually small and may not be suitable for large-scale data collection.
Overall, the Wizard of Oz method is an effective tool in UX research, especially for exploring and validating new design concepts and ways of interacting. However, how to design and implement such tests requires careful consideration to ensure that valuable user feedback is gathered.
Columnist
Kids, everybody is a product manager columnist. Pay attention to e-commerce and social trends, and be good at prototyping, demand mining, etc.
This article was originally published by Everyone is a Product Manager and is prohibited from reprinting without permission
The title image is from Unsplash and is licensed under CC0
The views in this article only represent the author's own, everyone is a product manager, and the platform only provides information storage space services.