from Evaluation of Human-Computer Interaction Whilst Using the World Wide Web
by Nicolas J. Blaza · The Buckinghamshire College, a college of Brunel University · May 1996
6.1 Usability Survey
Because the Web is considered to be the interface that will bring the masses to the Internet (Gates, 1995), it is vital that Web browsers are easy to use by a broad range of people.
What the author has attempted to do throughout this project is to evaluate the theory behind usability concepts, and that of designing good interfaces, and then relate this information to Web browsers; this has been done by producing a usability survey.
In chapter 4 the author discussed the various methods of evaluating user performance. Ben Shneiderman, the guru on usability surveys, writes that there are a number of different methods of assessing if a software product is acceptable to the user it is being aimed at, and one approach that can be taken is a written method, viewing user performance and attitudes.
6.2 The Questionnaire for User Interface Satisfaction (QUIS)
The Questionnaire for User Interface Satisfaction (QUIS) was developed by Ben Shneiderman and Dr. Kent Norman. It has proved useful in demonstrating the benefits of improvements to, amongst other things, the redesigning of an online public-access library catalogue. A copy of this questionnaire is in Appendix A. This type of evaluation was undertaken because it is cheap and relatively easy to implement.
To evaluate Shneiderman's theory on User Interfaces, and to see if it has been adhered to in the designing of a Web application, a questionnaire was designed to assess the usability of them.
Three different groups of students from The Buckinghamshire College, a college of Brunel University were interviewed and asked to participate in a short evaluation test using the product Netscape 2.01.
As outlined in chapter 1, group one were 6 students from the design department, first year BA (Hons) Textile Design and Surface Decoration. The second group were 6 students from the Social Science department on either the Sociology and Criminology course or the Sociology and Psychology course; these students were second years. The final group of 9 students were from the BSc Business Information Technology course, third year.
These three groups of people had been chosen to represent three different user groups that may want to use the Web. Group one are considered novices to the W.W.W. and in some cases, as the results show, are novices to using computers. Group two have some understanding of computers — these are the average, or occasional users — and group three are the frequent users, some of whom spend at least an hour a day on the W.W.W.
6.3 Designing the Questionnaire
Using Shneiderman and Norman's questionnaire as a starting point, the author then had to decide what else the survey should try and discover; for example, in order to evaluate Shneiderman's five factors of design it would be necessary to have some sort of task for the user to do, and so the task "locate the IBM Web page" was added into the questionnaire.
The QUIS questionnaire was designed to be given to respondents and then they go away and fill it in. Shneiderman suggests a time period of around 45 minutes is sufficient to complete it, but because of time restraints and the need for as many results as possible the author questioned the respondent face to face and wrote the reactions down as they happened.
A trial run of the questionnaire also proved to be very useful. This was done in week five, with three members of group one. In designing the prototype questionnaire the author had overlooked the fact that some of the users may be unfamiliar with the Windows environment. For example, some of group one had only a basic knowledge of the Apple Macintosh working environment, which although similar to Windows does have some fundamental differences — for example the expanding of window sizes and opening applications in Windows is slightly different. Further questions were added to overcome this problem.
It was decided that although there were three different groups of people involved in the survey, one questionnaire should be sufficient enough to get the information required. The author had to explain some of the questions to the respondents (question 6.7 — some respondents did not know what a search engine was); some of group three found the basic questions tedious.
The author wanted to try and implement some of the ideas mentioned in the Sun Microsystems case study, for example evaluate how user-friendly certain Web pages were; did users understand what was a "hypertext/URL" link and what was plain text? The author also wanted to discover how effective the use of extra icon menus (at the top of the application) had on all three user groups. Was there a noticeable change in how much these icons were used as opposed to the traditional Windows drop-down menu bars, and was it true that the more familiar a user is, the more they use the shortcut keyboard commands as opposed to using the mouse?
The question was asked, "Do you know what the icon functions are?" and the respondent had to reply with either the answer or a guess at their function.
It was also appropriate to explain what the Web was to some interviewees, as they had only very basic knowledge of it. For this the author used the idea of an encyclopaedia full of information, but instead of reading it in a linear fashion the user can jump to different sections, and those sections will have "invisible links" to other pages that are relevant to the subject.
To prompt the user to comment on Web page design, the author showed the respondent various Web pages which are referenced in Appendix G.
6.4 Results of the Survey
Each respondent took about 5 minutes to respond to all questions. The survey was done over a period of three weeks.
It was decided by the author that for the purpose of this report it would be unnecessary to list and comment on every single question in the survey. Instead, trends have been found and commented on, as well as the user's reactions to the system. A full list of the results is in Appendix F.
11 respondents were male and 10 were female. All six respondents from group 1 were female, and the other 4 were from group 2. The age range was from 19 to 27, with an average age of 23. The author would have preferred a much greater age span to evaluate children's responses and the use of the Web in the workplace.
Because of College resources, and the time taken to evaluate all respondents, the type of hardware differed. Sometimes the author used 486 8 meg machines (12 respondents), at other times 386 4 meg machines (7 respondents), and twice the Pentium 75 8 meg machines were used. This invalidates question 7.1, which asks the user to comment on the speed of the system, as some machines obviously run a lot slower than others. The 386 machines were unable to show all graphics in the proper format, which hindered the respondent's evaluation of design and layout issues.
All respondents used Netscape version 2.01. Ten respondents had not used a Web browser before; only one respondent from group one had used it, and only one respondent from group two. All of group three had used Netscape, at least 4 hours a week or more, and one respondent claimed to use it over 10 hours a week.
18 respondents had used other computer systems before, mainly the Apple Macintosh and UNIX systems.
Question 2.2a highlighted the fact that most people had used a file manager (2 hadn't), although all had used a word processor, even if it was an "electronic typewriter". The results of question 2.2b are very useful for designers when considering the type of input device the average user prefers. The mouse is still very popular with users (43% of respondents chose it), but 11 respondents liked the idea of a touch-screen method of accessing the Web. If this idea were to be instigated into a computer system, the question of inputting text arises — how would it be done, by a separate keyboard, or by an on-screen virtual keyboard?
Overall the respondents were neutral about Netscape as a package. Group 1 were impressed at the design of icons and the layout of Web pages, and group three liked the use of lots of hypertext links.
The author showed various Web sites to group 1 and asked for their opinions on the layout. Their response was for a need to have a company logo on the top of the document to understand who it is by. The search for the IBM page proved to be difficult in some cases — typing IBM into a search engine brings up a mass of possible Web sites, which the author concluded needs to be filtered on the search engine. There needs to be some kind of selection criteria to know which Web sites are corporate, industry, research or personal Web sites; it can be frustrating trying to research and find one specific article.
The author did not spot any of the proficient users using the shortcut keyboard commands, which are designed to save time for expert users. This, the author concludes, is because the users are not as proficient as they thought, or they are more happy using the mouse.
6.5 Conclusions
The author would have liked to have spent more time on the actual questioning of users; this small-scale test highlights some of the problems software designers must face when dealing with the general public. In the ideal world it would have been useful to interview people from different age groups and of different backgrounds, as opposed to a very limited set of people.
The author expected to see substantial differences in how the application was perceived by different people; with 21 applicants there was not a lot of room for big differences.
Another factor that made the results similar was that in group 2, the type of user the author had portrayed them to be (occasional user to the Web) was not true, and they were in the class of "unfamiliar user". It would have been more useful to try and find people from the Computing department who had only used the Web a few times.
Group 1 grasped the idea of the Web very quickly, although some people did say, off the record, that they probably wouldn't use it again.
The major concern from all respondents was the system speed of the Web — pages would take far too long to load, leaving the user impatient, and occasionally the graphics would not load.
© 1996 Nicolas J. Blaza. Reconstructed online edition, 2026.
Best viewed in Netscape Navigator 2.0 or higher at 800×600 resolution.
