Case Study: AnswerWorks and the AutoCAD 2000 Help System

By Jeffrey Allen

The AutoCAD Technical Publications department at Autodesk evaluated the customer experience of using AnswerWorks with the AutoCAD online documentation and found it to be emphatically positive.

AnswerWorks was added to the current AutoCAD Help system (no content changes were involved) and the AnswerWorks-enhanced Help was distributed to a group of customer volunteers via the AutoCAD Beta Program. An email survey was conducted to measure their reaction and collect their comments, as well as to collect data to guide certain design decisions. This case study describes the test and reports on the results of the survey.

How the AutoCAD Help System Was Modified

The Help system that ships with AutoCAD 2000 consists of 36 WinHelp files. From the user’s perspective these files form two Help systems: User Help, comprising five of these files; and Developer Help, consisting of six files. The remaining files are supplemental driver and support files. To get some quick feedback on the effectiveness and usability of an "AnswerWorked" Help system, we decided to provide this functionality for the User Help only.

We used the AnswerWorks Wizard to add AnswerWorks to the five User Help files. No other modifications were made to the existing source files for those Help projects, and no synonym file was used. The AnswerWorks Wizard creates a LIM (index) file for each HLP file, and took about an hour to create the LIM files for all five HLP files, which total about 8 MB. We had to recompile the main HLP file so it included references to the AnswerWorks DLL, and also updated the CNT file so the AnswerWorks tab (we called it "Query") would display in the Help Topics dialog.

We provided a bare-bones interface in the Query tab for this test: a simple text entry control plus a panel to display a list of topics with relevance ratings. We arbitrarily limited the list to the top 15 topics.

Distribution and Installation

The updated Help system files were bundled into a self-extracting EXE and posted to a secure location on the AutoCAD Beta Programs web site. Visitors to the site who were current users of the English language version of AutoCAD 2000 were invited to download the files once they had completed a brief enrollment process.

Although the package was very simple, sometimes the environments it entered were not. Some customers had difficulty getting the new Help system installed and working properly, usually due to the existence of multiple versions of AutoCAD, or multiple AutoCAD-based products. Although there were no cases in which the problems were related to AnswerWorks, some of the negative survey responses are attributable to installation problems rather than the performance of the Help system.

It should be noted that we were trying to get a quick assessment of customer response to the basic functionality offered by AnswerWorks. Our customer sampling technique (self selection) and the design of the survey questions were not rigorous, but we have had great success using this methodology on features in development to perform a "reality check" on our expectations. In addition to feedback on the basic capabilities of the system, the survey allowed us to collect input on specific implementation details, including the beginnings of a synonym list for our domain content.

Summary of Results

In a nutshell, the customer response to AnswerWorks in the AutoCAD Help system was emphatically positive. Overall, 78% of survey respondents said that the enhanced Query was a useful addition to online Help, and 51% indicated that it would become their primary search method as opposed to the Table of Contents, Index, or Find (full-text search). In our experience with similar tests on software features these figures are unusually high, especially considering the "out-of-the-box" implementation and the fact that they are probably conservative (some users were unable to complete the installation, but still responded to the survey and were not filtered out).

Future Directions

In addition to strong validation of customer enthusiasm for AnswerWorks provided by the test, feedback in the survey is also helping us refine our ideas for a more robust AnswerWorks implementation—although we are impressed by how much success we achieved with so little effort.

We are converting the AutoCAD Help system from WinHelp to HTML Help. Our experience with HTML Help on AutoCAD LT has given us some specific notions about what can be effective for content organization and presentation in HTML Help, and we’re confident that AnswerWorks will fit in extremely well.

Because our documentation set is relatively large and falls naturally into discrete volumes with distinct audiences, we intend to provide a method for users to constrain their searches to specific collections if they wish. AnswerWorks easily supports this capability.

Customer expectations around the number of topic hits to display are undergoing rapid modification via increased interaction with web searching, but we’re finding that customers welcome shorter lists when the accuracy is high. We expect to reduce the default from the value we used in the test, but to provide a one-click method for expanding the list to display all hits.

For some time now we have supported the ability of users to update their Help content via web download. We can continue to refine and improve the AnswerWorks search results by incorporating a feedback mechanism directly into the interface that collects and transmits query strings back to us along with optional comments for those occasions when results are unexpected.

Survey Data

• In total, 246 customers enrolled for the test and downloaded the files.
• The survey was delivered to these 246 customers via email on March 2, 2000.
• These results reflect 103 responses (42%) as of March 6, 2000.

1. How would you rate the Query’s ability to help you find answers in AutoCAD’s online Help?

2. For what percentage of your questions did the Query return an acceptable solution?

3. If you could control the number of topics that a query displays, what would be a good number?

4. Overall, did you find Query a useful addition to online Help?

5. Now that you’ve used the Query tab, rank the FIRST search method you feel you would use most often to find information in AutoCAD’s online Help. (Questions 6 and 7 asked customers to rank SECOND and THIRD choices. The results of these three questions are compiled below for clarity).

8. What could we do or change to make Query more useful? (Note that this question was directed at the Query implementation as well as the AnswerWorks technology. Representative examples of the 64 responses received are given below; responses specific to AutoCAD content have been omitted.)

• "Add functionality for desktop programs."
• "Add it to the other Help files like the Active X, VBA files,...."
• "At this time, I haven’t seen anything negative about Query. I’m not sure what could be changed to make it more useful. None of our users have had a problem with it or suggested improvements."
• "Categorize the results into subject matter. This could be by which reference it is contained in or by similar subjects."
• "Continue to add to the database so that it finds answers to more obscure questions."
• "Highlight or bold the key word in the resulting information."
• "I like the 'ask a question' approach. The irritating facet of most Help systems for me, however, is having to wade through too many hits when I search for information. Most seem to search all the Help text for occurrences of the word for which I am searching, which returns a bloated list from which I must choose the one containing the information I want. I would like to be able to limit the returned list of hits to articles pertaining directly to the keyword, not articles containing casual references to that keyword."
• "I think that overall it could be a very useful tool; however, the difficulty is in translating common terminology into CAD lingo."
• "I thought it was fine as is."
• "Improve query so that it doesn’t return so many misleading results. I got a lot of totally off-the-wall suggestions mixed in with the correct answers."
• "Include online resources in returns where available."
• "Integrate it more closely with the Help system."
• "It’s quite an improvement already; don’t see anything for now, but will be looking."
• "Keep it current; allow for user input."
• "Make it start with a tool button."
• "Make this option the first to come up on the online Help window."
• "My language is Russian; when this feature will work on my language I will use it more frequently."
• "Nothing, it works good."
• "Possibly ask defining questions if the query is ambiguous, i.e, does the question you asked relate more to set of topics 'a' or topics 'b.' This would help refine the search when you did not know what you were really searching for."
• "Search inside results."
• "Seemed pretty good as is."
• "Seems to work great as you have it."
• "The basic format is good!!"
• "The times that I used it I was very satisfied at its current state."
• "When selecting the topic could the actual topic be displayed to the side of the dialog box? Sort of like scrolling through previews to see if the topic contains the information desired."

9. Which of your search words/phrases turned up unexpected results? (This question was specifically designed to collect input for developing a synonym list. Responses have been omitted as they are not of general interest.)

10. Other comments. (Express your likes and dislikes. Comment on usability, interface, results, etc.)

• "As I’ve previously stated, I like the query approach to getting information, but I am daunted by the glut of references presented to me. Looking at 15 possible sources of information makes me want to stumble through the problem in real time, or seek out a printed reference where information is usually more concisely organized."
• "Excellent addition—useful for beginner or experienced users—wish it would have been available when I went from R14 to A2K."
• "Focus should return to the 'Request' box for faster use."
• "Good beginning; would like to see this be a growing, ever changing, resource."
• "How can I tweak the '%' numbers, i.e., show me 'XXX' number of results and not a preset value."
• "I am impressed with the results to the questions that I asked. The answer was usually one of the first three or four displays."
• "I like the answers which are returned; they seem to be reasonable."
• "I like the format and layout of the query tools."
• "I like the interface, but did not really see a drastic improvement on search."
• "In all I was impressed with the Query tab and would suggest it be adopted as standard Help."
• "It seemed like, too often it came up with ACR rather than something that is usable. A new user, typing a query, obviously doesn’t understand the task at hand. I think the concept should come up more frequently than an ACR. I must say that it works better than Microsoft’s answer thing."
• "Like I said, most queries got the right answer, but were mixed in with quite a few misleading or wrong suggestions. I like the overall concept, just need to sharpen the query results"
• "Like it; no negatives."
• "Liked it very well and thanks for sharing it."
• "My initial response to this tool in ADT2.0 is quite good. I feel that it will be very useful, and hopefully, as a CAD support person, also should make things easier, in that people should not have to always come to me with simple questions. I actually think this is one of the most effective features to get people to actually use the Help system, which I am continually trying to get others to do. I think one of the things that needs to happen is to group similar subjects such as 'Xref' and 'External Reference' into the same category so that it produces the same results. And thanks for not implementing some goofy character that dances around on my screen waiting for me to ask a question (i.e., MS Answer Wizard or Dell’s 'Ask Dudley.' Bottom line—I want it on our AutoCAD products!!"
• "Not much different than searching index, but maybe more user friendly."
• "Overall, satisfaction."
• "Overall, I like it. I think it will meet the expectations of anyone who is used to the pros and cons of using an Internet Search Engine."
• "Precision of the results is good, plus the limited number of returns are not overwhelming."
• "So far as I can tell, this is a great addition to the AutoCAD product. I can’t see any drawbacks with it as it appears to be quick and accurate when you are looking for answers."
• "The interface is too similar to the Index tab; should be different somehow."
• "This is a very good addition. I spent less time looking for solutions than I usually do. I usually cannot find what I want about 20% of the time because I am trying to find arcane stuff unless I do a find and sort through the list a few times with different words. This came up with the right thing easily. I like the normal syntax approach."
• "Usability and interface seem well done."
• "Usability was very good as were the results. I liked it very much."
• "Very good addition to the Help command. The interface is good. I would add the feature of searching a selection set of online Help files instead of all of them."
• "Very nice feature. Similar to web searches. The ranking is helpful."
• "Very useful; good idea."
• "Worked well, and led me to answers faster than I would have using other methods."

Jeffrey Allen is the manager of AutoCAD Technical Publications at Autodesk.