Chatbot vs Traditional Form

A Research on the change of user experience in the Process of Data Collection

• Have you ever felt so frustrated doing complicated form entry?

• Have you ever felt so annoyed by how “cold” the forms are?

• Have you ever felt so bored reading all the information in the form that may or may not apply to your case?

• Have you ever wished customers can ever become more willingly on providing more usable data?

Project Over View

The emergence and pervasiveness of the Internet provides opportunities for new types of interaction between users and services providers. One such technology is Chatbot: a computer program that simulates human conversation enabled by the Internet. Traditionally, many people might get bored during the process of filling Forms and might provide meaningless information or even drop off the whole site. Even some web browsers which have auto - fill functions, they cannot help when there is new data requested, such as patients’ current health situations. This problem is important because the vapidity will stop users from completing the Form, which is typically the first step of the service. Related works in this domain, however, are limited to draw a confident conclusion when facing the dilemma. Our group focused on increasing user experience and data collection efficiency by replacing the Forms with Chatbot.

Our team built two registration tasks for patient’s networking platform with both Form and Chatbot. After comparing two prototypes with the same question set, we collected user feedback derived from lab experiments and quantified data using the System Usability Scale (SUS) and other scales. Majority of the participants in this research study (12 out of 16) preferred Chatbot. The Chatbot platform received an average score rating of 66.9 out of 100.

Traditional Form:

Chatbot:

Hypothesis and Background

Our group identified user pain points and prioritized them through user empathy map and affinity diagram at the touch point of user registration. We found that user always get annoyed by how many fields in the data collecting form they have to fill up in order to sign up for a Healthcare Application. We assumed Chatbot could be an effective solution to relief users from from this fretful process.

The observation that conversational interfaces affect and thereby have implications on internet products is not breaking news. Besides, providing better user experience in general, utilization of Chatbot in data collecting process is a great use case.

To test our hypothesis, our team decided to design two prototypes (one traditional Form and one Chatbot) and a task-based one-on-one between subject lab experiment in the context of SomeoneLikesYou, which is an online patient community. The task given to participants was the registration process of the platform. For further analytics, we collected both quantitative and qualitative data through the lab experiment.

SWOT Analysis

Current market situation of both approaches

Research Personas

Based on customer segmentation, we created two different personas representing two different types of user needs. Different user personas present different preferences during the data collection process. We have found that users with rigid demand users have less expectations on the UX of the website. They care more about the functionality. However, users with more elastic demand or have less need from the website will be more likely to be influenced by the UX of a site.

Participants

We recruited 18 participants from diverse demographics

Based on our developed personas, we recruited two groups of users: Users with Rigid Demand and Users with Elastic Demand. We scheduled 18 participants in total, 7 of them are living with serious diseases; 6 of them are suffering from minor illness (where participants consider themselves suffering from not so serious symptoms); 5 of them have no major health concerns. We labeled and categorized them into separate user groups.

Prototypes

Interact with only one prototype

Each participant can only interact with one of the two prototypes in the experiment. otherwise their awareness of the website from the previous experience will affect the accuracy of their second experiment.

In the Chatbot, we created short cuts in user flow. For example, if you already set the gender as a male, the Chatbot will not ask you about pregnancy. This is also one of the most important competitive advantage of Chatbot.

User Experiment

We measured our experiment on two dimensions: Subjective and Objective. We collected the metrics through pre-test questionnaire and post-test questionnaire. We collected observational such as comments, expressions and emotions during lab experiment. Our team assigned 18 participants respectively and asked them to complete the registration process.

Our team developed unanimous criteria to conduct usability test, collect subjective and objective metrics. We retrieved and designed the question set based on the registration process of UMass Medical School and the other online patient community platform (SomeoneLikesMe). The procedure of lad experiment is shown below:

Pilot Experiment

Experiment first, then critique

We asked a UX expert to do hands-on experiment with us. Overall, he enjoyed our Chatbot prototype and gave compliments toward research. He gave us several valuable advices for our improvement as well.

“The purpose for your Chatbot was not clarified at the beginning. Participants might be offended because you asked for great amount of personal private information which might not be considered necessary being a part of the registration process. “

We improved it by clarifying our purpose and that we do not need participants’ information before the experiment started.

Another things was that he got confused between choosing and inputting and did both actions on one question. However, for all questions, we meant that participants only need to do one between selecting or inputting an answer. As a result, we add an additional clarification at the beginning of experiment to reduce the confusion that might be caused by us.

Over all, during the experiment, our expert participant Chatbot prototype gave compliments toward our research. His advices were very helpful too.

User Quotes

Think out loud during the process

We encouraged participants to speak out their thoughts during the process (Think-out-loud Protocol). In addition, we asked for the participants’ opinion after the complete the task. We noted the experiment key words in their comments here.

Data Analysis

Number speaks for the truth too

We conducted statistical analysis on the collected measurements. The result manifest itself primarily in two ways first, the subjective core rate by testers. Second, the objective question answer rate counted in the two prototypes.

Surprising Facts

We were not looking for the result that we have already assumed

· Data Speaks

The Chatbot received only a slightly higher satisfaction rate and SUS score than form while the Chatbot received almost all the positive feedbacks that re have received.

· Task itself matters more

Participants reported longer perceived time than actual completion time on both prototypes. We assumed that the reason result is inherent in the length of the question sets.

· Problem - Solution - new problem - new solution loop

There are the same amount of users in both prototypes that vote for wanting to switch to the other prototypes after or experiment. We assumed that the reason behind this could be that users who just experienced form could not or unwilling to imagine the advantage of Chatbot only by exploring one screenshot of our chatbot prototype.