A Usability Test of Web-based User Assistance
By Matthew Ellison
Contents
Click a link below to jump to a particular section; click any "CONTENTS" image following a section heading to jump back here.
Introduction 
Recently, WinWriters used the web-based registration process for its annual Online Help Conference as a case study to test several models of user assistance for the Web. We conducted a series of usability tests to:
- Find out how what sort of questions and problems users have as they complete a web-based transaction
- Discover how users respond to various forms of user assistance available within a web-based task
- Refine the registration process for future conferences
The tests provided some clear conclusions as to user preferences. In this article, we describe the tests performed and summarize the findings.
2002 WinWriters Conference Registration Process
In the past, the registration process consisted of completing a single web-based form. Anyone registering more than one attendee for the conference had to fill out the form for each person. The form contained Help in the form of popup browser windows available from all the fields on the form.
For this test, we split the single form into four separate pages, each representing a separate stage of the registration process. We chose to split the original form in this way to make the registration process more representative of a typical process for buying items or services on the Internet. Mirroring a typical process enabled us to draw conclusions that were more widely applicable to other commercial web sites.
Registrants select the appropriate registrant type (with associated pricing) and any optional seminars on the first page (Step 1), and supply contact details on the second page (Step 2). Figure 1 shows the form for Step 2 of the registration process.
Figure 1: Step 2 of the registration process.
What We Tested
We created five versions of the registration process, and used the test to compare the users' success with each version. One version had no user assistance; the others had the same user assistance content presented in four different ways:
- No user assistance (version 1 — see Figure 1)
- Field-level context-sensitive Help, presented in a frame to the right of the form (version 2 — see Figure 2b)
- Page-level context-sensitive Help, presented in a separate browser window (version 3 — see Figure 3)
- Field-level context-sensitive Help, presented in a small separate browser window (version 4 — see Figure 4)
- Help built into the registration form (version 5 — see Figure 5)
We based each variation of the user assistance on a model for Help that is currently used within commercial applications on today's Internet. We were interested in observing how users responded to each of these different ways of accessing and viewing Help. The rest of this article provides more detailed information about each of the Help models.
Field-level Context-sensitive Help in a Frame
We created this version of the registration pages as a frameset, with two side-by-side frames. The user works on the registration process in the left-hand frame. Every field contains a Help link, with the user assistance appearing in the right-hand frame. When each page first appears, the information in the right-hand frame provides overall guidance on that particular step of the registration process, as shown in Figure 2a.
Figure 2a: Page-level Help in frame.
When the user clicks on any of the Help links within the form, the instructions for that particular form field appear in the right-hand frame, as shown in Figure 2b.
Figure 2b: Field-level Help in frame.
Page-level Context-sensitive Help in a Separate Browser Window
This version contains a single Help link for each step of the registration process. Clicking on the link displays a Help topic in a separate browser window (see Figure 3). This Help topic contains both an overview of the step, and all the field-level content for that step.
Figure 3: Page-level context-sensitive Help in separate window.
The "Show More Help" link opens a navigation pane within the Help window, and enables the user to navigate to other topics using a table of contents.
Field-level Context-sensitive Help in a Separate Browser Window
This version of the site is closest in behavior to the actual registration page used for the 2002 WinWriters Conference. Each form field has a link to field-level Help. Instead of using the word "Help", the link text consists of the name of the field. A small separate browser window displays the field-level user assistance, as shown in Figure 4.
Figure 4: Field-level context-sensitive Help in separate window.
Help Built into the Registration Form
The final version of the site has no separate area for user assistance. Instead, we include within the form the same field-level information as in the other models. Each step of the registration process also starts with the page-level Help included within the other models. The result is much longer registration pages in this model compared to the other versions. Figure 5 shows the form for Step 1 of the registration process.
Figure 5: Help built into UI.
The Usability Test
Number of Participants
Sixteen people participated in the test, and each was asked to complete the same fairly complex, but realistic, registration task (described in detail below). Two participants each used the two field-level Help versions of the registration process; four participants each used the other three versions.
The participants were not given a choice about which version of the registration process they used—we assigned these by rotation, in order to even out the number of participants using each version. However, we only asked two participants to use versions 1 and 4, because we felt that tests on these two versions had the least information to reveal.
Instructions to Participants
The participants were asked at the start of the test to speak aloud their thoughts as they worked through the task. The test moderator did not provide any instruction during the test unless the participant became totally stuck. In this case, the moderator provided a verbal hint on how to proceed. The moderator also occasionally asked questions to solicit more information about the participants' decision-making process.
None of the participants were told that the test was focusing on user assistance. The moderator simply told the participants that they were testing the online conference registration process. The reason for this was to avoid predisposing any of the participants to select the Help option. We wanted to see under which circumstances participants chose to select Help.
Record Keeping
The moderator made detailed notes during the test, which included the following information:
- All actions taken by the participant, including errors
- Details of any hesitations or pauses by the participant
- Whether or not each step of the task was completed successfully
- All comments spoken by the participant
- All questions asked by the moderator
- All hints provided by the moderator
Participant Profiles
The test participants were all computer literate and were familiar with using the Internet. None of the participants were professional online Help authors, although three had specialized knowledge of Help and documentation issues. The remainder had no connection to online Help development at all. Thus, we were satisfied that the tests showed how an administrator tasked with registering authors for the conference would perform.
Details of the Test
Each test participant had to register three fictional attendees from the same company for the conference. The fictional attendees had different requirements, objectives, and status, which required the test participants to select different options as they registered each person. The instructions for the test were given to each participant in the form of a printed sheet, which is included in the Appendix.
Out expectation in devising the test was that participants would require "domain knowledge" Help, rather than procedural Help. In other words, we assumed that common reasons for needing Help would be questions such as "Does Kylie have to be studying full-time to qualify for the Student discount?" and "What is this thing called the LaunchPad Certificate Program?" We did not anticipate many questions along the lines of "How do I fill in this form field?" or "How do I move to the next step in the process?" With this in mind, most of the content of the user assistance we provided was aimed at answering "domain knowledge" questions.
What We Learned from the Tests
With the relatively small number of test participants, it was not appropriate to do any statistical analysis on the data collected. However, many of the observations made by the moderator provided interesting insights into the way people complete web-based tasks, and even with this the small sample size, some interesting behavior patterns emerged.
The Importance of Up-front Orientation Information
An orientation page preceded the registration form within versions 4 (field-level Help in a separate browser window) and 5 (Help built into the UI). The orientation page, shown in Figure 6, provided users of these versions with the important fact that the registration process has to be completed for each member of the group of three attendees in turn—in other words, it's not possible to register the entire group in one pass through the registration process. This fact can also be inferred from the design of the forms as you work through the registration process, but we were interested in seeing how important it was for users to have this information before starting the task.
Figure 6: Orientation page.
Our conclusion from the tests was that the advance orientation was critical to success with the registration process. Those participants presented with the above orientation page read though it carefully, and as a result performed significantly better during the registration process. The participants who were not presented with this screen did not seem concerned that they were lacking the orientation information that they needed—they were quite happy to dive straight into the registration process, often with the misconception that they would be able to register all three attendees simultaneously. This misconception caused many errors and setbacks, and even resulted in some participants abandoning the task altogether.
Here are comments, recorded during the test, from two participants who were not presented with the orientation information:
"What's throwing me here is that I'm registering a group, but the seminars seem to be for individuals. It seems clear that I should choose the group, and deal with the seminars later on."
[This comment was made on Step 1 of the registration process, which requires you to select the registrant type (e.g., a member of a group) and then select the seminars desired by the registrant.]
"That's interesting—I've told them I'm registering 2-5 people, but they only have one name... I'm worried about sending the registration now—they only have one name."
[This comment was made on Step 4 of the registration process, which requires you to submit the registration details for the first member of the group of three registrants]
By contrast, here is an extract from a conversation between the moderator and a participant who had just read the orientation information and was completing Step 1 of the registration process:
Moderator: "Happy?"
Participant: "Yes—it's good that it has all this information."
Moderator: "What are you trying to do?"
Participant: "I'm trying to register Ann Kirkman"
And here is a comment from another participant that was presented with the orientation information:
"So, I've registered one person—I have only identified that person yet... I have to do each person individually. It did tell me on page one that I had to do the whole process for everyone."
The Positioning of Embedded Help in Relation to the Task Flow
We noticed that participants using version 2 (Help presented in a frame) generally ignored the guidance information in the right-hand frame, unless they caused the content of the right-hand frame to change through their own action.
For example, when presented with the following orientation page, all four participants clicked almost immediately on the button to move to Step 1. They did not take the time to read the orientation information in the right-hand frame, which we have already established is critical to the successful completion of the task.
Figure 7: Orientation information in right-hand frame.
One of the reasons for this behavior could be that the Help information is outside of the task flow. Users would naturally start reading the page at the main heading of "The Registration Process" and continue reading downward until they reach the button for moving to Step 1.
By contrast, when participants clicked one of the field-level Help links and caused the guidance for that field to replace the existing content of the right-hand frame, then they did read the information. There was actually a slight, but appreciable, delay before they realized that the contents of the Help frame had changed, and then shifted their focus to the right. Figure 8 shows an example of one of the Help topics that participants displayed and then read.
Figure 8: Help topic in right-hand frame when selected by a user.
Lack of Expectation of the Help Providing Domain Information
As mentioned, the Help text provided information relating to the domain, such as the conditions for discounted rates, the target audience for certificate programs, and the details of specific seminars. However, participants clearly didn't expect the Help to provide this kind of information. When they needed domain information, they would typically scratch their heads and puzzle over the task for several minutes before finally resorting to clicking on the Help link (sometimes only after prompting by the moderator).
Participants Learned To Use the Help
Participants became gradually more successful as they progressed through the task. Although only 7 out of the 16 participants registered the first attendee accurately, all 11 of the participants (who managed to progress that far) registered the third attendee accurately.
There seemed to be a combination of two reasons for this. First, even though participants were reluctant to select the Help initially, once they discovered that it provided the domain information needed, they increased their use of the Help from that point onward.
Here is an extract from a conversation between the moderator and one of the participants early in the test:
Moderator: "How sure are you that Kylie qualifies for the Student discount?"
Participant: [On considering Help link]: "I hate going to Help!"
And here is a comment from the same participant later on in the test:
"He's interested in the other certificate program. Let me go check the Help again." [Suddenly a Help devotee!]
The second reason for participants' improved success as they progressed through the task was that they learned from their own mistakes and gradually inferred knowledge of the registration process through familiarity with the forms. This kind of "on the job" learning is acceptable for systems in which initial mistakes are not critical. However, in the case of the WinWriters Conference registration process, accuracy is just as important for the first registrant as for the third. We need to remember that it's important for user assistance to enable users to be successful even when they are still familiarizing themselves with the forms.
Misinterpretation of Text Links
As mentioned earlier, version 4 of the registration process (field-level Help in a separate browser window) used the name of the field as the text for the link to the field-level Help. This caused confusion for one of the participants who incorrectly thought that clicking on the underline text would cause that option to be selected in the form.
Unpredictable Sequence
Participants did not always follow a predictable sequence through the task. There were cases of the following behavior:
- Completing the fields on a form in a seemingly random order, or even (in one case) starting form the bottom of the form and working upward
- Back-tracking one, two, or even three pages through the overall process
- Selecting an option before reading the full description of the option
Participants were frustrated if they returned to a form that they had already completed, only to find that the data had disappeared and that they needed to enter it again.
Link to Navigation System
As mentioned previously, the Help for version 3 of the registration process (page-level Help in a separate browser window) contains a link with the wording "Show more Help". This link, shown in Figure 3, displays a navigation pane that enabled other topics to be selected. Figure 9 shows the Help window after this link has been clicked.
Figure 9: Help navigation pane.
None of the four participants that used this version of the Help chose to click on the "Show more Help" link without prompting from the moderator. This remained the case even when they were looking for information that was not contained in the topic currently on view.
Comparison of Different Versions
Our observations indicated that participants were (surprisingly) no more disposed to selecting Help links positioned beside the individual fields of the form than they were to selecting a single Help link at the top of the page. The critical factor that determined whether participants would select the Help or not seemed to be the participants' degree of confidence that the Help would provide the information they needed. In the early stages of the test, most participants were reluctant to select the Help because they did not believe it would contain the domain information required. However, once they became aware of the nature of the Help content, they were prepared to select the Help option, even if it meant scrolling back up the page to find it.
Overall, the participants using version 5 of the registration process (Help built into the UI) were the most successful in registering all three attendees accurately. (Accuracy was measured by how closely the registration data submitted by the participant corresponded to the requirements for each attendee, as specified within the instructions.) However, they were also among the slowest of those to complete the task because they were forced to read more text, and the forms were longer, requiring more up and down scrolling.
The version with which participants had least success was the one with no user assistance (version 1). Both the participants using this version gave up and abandoned the task before completing it.
Conclusions
The participants performed significantly better when were informed at the start that they had to register each attendee separately. From this, we conclude that it's important to present certain key information up-front, before users start on a task. It's not good enough to rely on users discovering or inferring this information as they progress through the task.
One of the key observations from the test was the critical nature of domain information. Without it, participants struggled to complete the task, or even abandoned it completely. We therefore conclude that domain information (key concepts, definitions, rules, conventions, prices, etc.) is a vital element of user assistance for web applications, and is potentially more useful to users than procedural information.
Unfortunately, participants did not appear to expect to find this domain information within an option labeled "Help". This suggests using a different wording for links to user assistance. One option is to use key words and phrases of existing text within the UI as links to the Help, as in version 4 of the registration process. Another solution is to embed the user assistance within the UI. The success that participants had with version 5 of the registration process shows that this can be an effective technique, especially in a situation where users are unfamiliar with a web application but need to complete a task accurately the first time.
Our tests showed that users become more successful as they familiarized themselves with the task, and they also became more willing to use the Help when they realized it contained useful information. This leads us to conclude that an embedded Help system has less advantage in cases when it's acceptable to learn through trial and error. For users who are familiar with a specific web application, embedded Help could be counter-productive because it tends to clutter the UI with information that is unnecessary for experienced users.
Appendix — Instructions Given to Test Participants
The Task: Registering for the 2002 WinWriters Conference
You have been given the task of registering three writers from your company's Documentation Department for the 2002 WinWriters Conference.
The writers have told you that there may be a discount available for group registrations. You will be using a company Purchase Order, number 1234.
Your company is familiar with the conference, having sent writers in two previous years. Your writers have been prompted to register for this year's conference by receiving a brochure and also by receiving an email notification.
The details for the three writers are as follows:
Ann Kirkman:
Email: ann_k@abccompany.com; Phone extension: 3221
- Manager of the other two writers, and the contact person for the group for all future correspondence about the conference.
- Anne attended the 2001 WinWriters Conference in Santa Clara.
- Ann would like to attend the all-day seminar on Sunday, but only if it covers the steps for planning and preparing for a site visit.
Kylie Parker:
Email: kylie_p@abccompany.com; Does not yet have a phone extension
- Kylie has recently joined the company from college. Since she is still following a part-time evening class, Kylie thinks that she may qualify for a special student rate at the Conference.
- She feels that she needs as much guidance as possible throughout the Conference, and wants to participate in the certificate scheme for novice authors.
- Kylie does not plan to attend any Sunday seminars.
Bill Chambers:
Email: bill_c@abccompany.com; Phone extension: 4879
- Bill is an experienced Help author who is interested in furthering his career.
- He has a special interest in Information Design and would like to attend as many of the sessions in this interest area as possible. He has heard that he can participate in a certificate scheme that recognizes this specialty.
- Bill would like to attend the Sunday seminar on creating better PDF, and also the seminar on Dreamweaver (but only if it is specifically targeted at an advanced level).
Company Details:
ABC Company, Inc., Large Gray Building, 1st Avenue, Seattle, WA 98199, Phone: (206) 352 72AA
Copyright © WinWriters. All Rights Reserved.
Joe Welinske: jw@winwriters.com
Last modified on
|
| 
