Design:
This week we built a Wizard of Oz prototype in teams of 3. Wizard of Oz prototyping is creating a prototype that seems to work automatically from the user’s perspective but in reality, it’s a person controlling the prototype’s responses on the other end. This allows designers to test a concept on the user without needing to build a high fidelity, expensive prototype and focuses the test on the general idea, giving the designer insight on what to change before moving forward to high fidelity prototypes.
We had Miles as our test facilitator since he had available roommates to participate as our test users, April as our “chatbot” responding to the users’ questions in real time, and Jeremy as our note taker, documenting the questions, responses, and most importantly, the user’s reactions.
Prototype:
For our set up, we simply had Miles save April’s number as “TestBot” and gave the phone to each user to text and test.
To start, we had Miles preface the testing by explaining to the users that we were simply testing our prototype and that the tasks were in no way a reflection of their ability. Miles also added that there may be a delay in the response time because we’re not the most experienced coders. In reality, this was a preemptive excuse to provide April with time to respond in case the users asked anything complicated that she needed to research before responding.
We had each user perform 7 tasks:
- Activate chatbot by sending it a message
- Ask the chatbot a simple question
- Ask the chatbot a complex question
- Send the chatbot some gibberish
- Tell the chatbot a story
- Ask the chatbot a question about the story
- Check on the status of a flight
User 1 is represented in the white background texts and user 2 in the black background texts.
Task 1: Activate chatbot by sending it a message
Both of our users sent our chatbot a simple greeting with a “Hello” or “Hey”. They both appreciated how simple and straightforward it was to activate it. They also enjoyed the warm response from the chatbot.
Task 2: Ask the chatbot a simple question
Next, we asked our users to ask the chatbot a simple question. User 1 asked “what’s the date”, user 2 asked “what’s your name?”. These were both easy questions for April to answer so the response time was short and both users found the answers to be pretty much what they expected. Our second user appreciated that the chatbot gave him the option to change its name.
Task 3: Ask the chatbot a complex question
The questions our users asked in this step were slightly more complicated. One user asked about the history of the Byzantine Empire whereas the other asked about the projected weather. When they asked the chatbot slightly more complex or specific questions, it took April a few more seconds to reply than other questions which made the users feel a little concerned about the “coding” of our chatbot. April had said that when the first user asked his question, she panicked and scrambled to find an article to send back to the user. After April did answer the questions, they were both amused to see that a chatbot answered their questions correctly.
Task 4: Send the chatbot some gibberish
Both our users quickly mashed their keys and sent in their messages of nonsense. They were both very impressed and found it entertaining that the chatbot sent a meme expressing its frustration in trying to understand their messages. Behind the scenes, April stated she wanted the chatbot to seem more modern and have a sense of humor to engage the user so she simply googled “what are you saying meme” and used the first or second one she found. Both users received the same response.
Task 5: Tell the chatbot a story
This task allowed our testers to express some more creativity as well. User 1 referred to the story of the ugly duckling and user 2 told a short story of him growing up. For the former story, our user was impressed that the chatbot was able to identify the story and confirm it back to him. User 2 was amused that the chatbot responded again with a little bit of humor: “thank you for that information… I think.”
Task 6: Ask the chatbot a question about the story
Next, we asked our users to ask the chatbot a question about the story they just told. User 1 found April’s response entertaining because it was so blunt and not afraid to call the duckling ugly. From April’s perspective, her thought process was that user 1 gave enough information and the story was common enough where an advanced Artificial Intelligence program would probably be able to find information on the story within its database and answer the question “why did the other duckling bully him?”. For user 2’s story, April decided that since it was a personal story, the bot should probably respond with slight confusion. When user 2 asked a question about their story, April decided that the most desirable response a user would probably like to see is that the bot understands human concerns for privacy.
Task 7: Check on the status of a flight
For this task, each user asked the chatbot to check the flight status of a specific flight number we provided them with. April on the other end had made up a random departure and arrival time and destination. Similar to the previous tasks, our users were mildly concerned about processing and response speeds. However, both testers did not explore the various flight facts and further and did not ask any more questions about it. Each tester received the same flight information. April later explained that it took especially long because this question required a longer answer and she needed to make sure it was grammatically correct. She also mentioned that she was trying to think how long it would take to fly from New York to Seattle because she was worried the users would question the flight time.
Finally, we revealed to the users that the chatbot was really April on the other end.
Analysis:
Overall, the users really enjoyed the personality of the chatbot. Through the use of memes and humor as well as conversational messages, the chatbot gave a small sense of humanity to our users. On the other hand, speed in processing and response times were frequently on the users’ mind after sending their messages. The users had both mentioned that the chatbot would feel more personal if it responded with questions and made the chat more of a back and forth conversation.
If we were to do this again, we would try to give the users more specific tasks and have pre-written responses ready so that we can minimize the response time since that seemed the most concerning for users. It would also be nice for each of the team members to switch roles just to experience what the other team members go through during the testing phase.
Going further, it would be interesting to test responses that have more built in questions to create more of a back and forth conversation and see what feedback we get from users. To implement feedback questions would have a fine line between useful and resulting in the user having to respond to questions they didn’t need in the first place.
