JCMC 4 (1) September 1998

Message Board

---

Table of Contents

• Abstract
  
• Introduction
  
• Method
  
• Results
  
• Conclusions and Suggestions
  
• Acknowledgments
  
• References
  
• About the Authors
  

---

In this study several aspects of the usability of an online newspaper are examined. More specifically, the effects of reading-manipulation techniques such as scrolling and using hyperlinks on finding information at different locations are studied. Subjects participated in two sessions with a one-week interval. In each session subjects received a number of searching tasks consisting of finding information at different locations in the newspaper. Speed and accuracy were measured, and afterwards subjects received a recognition task. In general subjects were highly satisfied with the online newspaper. Their performance on the searching tasks was very adequate, even on deeper levels. It took, logically, more time to locate the information by scrolling down or by using a hyperlink to go to a next level than when the hyperlink leading to the desired information was immediately available on screen. The recognition performance was also worse. Locating information after scrolling and after using a hyperlink took approximately the same amount of time, and the recognition performance was about equal. However, an interaction effect was also found between reading-manipulation technique and hypertextual level. In particular, finding information for which scrolling down on a deeper hypertextual level was necessary took extra time and probably extra cognitive resources, leading to a lower recognition performance. It is concluded that it is probably better, if possible, to avoid presenting information on deeper hypertextual levels for which scrolling is necessary.

  Introduction

In a research project together with a Dutch publisher, 'Perscombinatie', we evaluated different aspects of the usability of a prototype of an online newspaper de Volkskrant on Internet, the World Wide Web supplement to the Dutch daily newspaper de Volkskrant. We will only present a small part of that project here: we will examine how readers find information in an online newspaper. More specifically, we studied the effects of scrolling and using hyperlinks as reading-manipulation techniques on finding information at different locations in this hypertextual system.

Reading the information available in an online newspaper that is constructed as a hypertext requires that users navigate through the provided information, and manipulate the available navigating means, such as scrolling and using hyperlinks (e.g. Piolat, Roussey & Thunin, 1997). As opposed to a traditional linear document, a user now has numerous options to wander through the provided information. A well-known problem for users with hypertext systems is 'getting lost in hyperspace' (Dillon, 1994; Ebersole, 1997; Edwards & Hardman, 1989; McAleese, 1989; McKnight, 1996). For this reason the structure of information, the available orientation clues, overview possibilities, reading-manipulation techniques, and so on, are very important, as shown in studies by McKnight, Dillon and Richardson (1990), Britt, Rouet and Perfetti (1996), Dee-Lucas (1996), Dieberger (1997) and Nimwegen, Pouw and Van Oostendorp (in press).

An aspect of practical utility concerns the issue of how much information should be contained in a node in the hypertext structure (Kreitzberg & Shneiderman, 1988; Kreitzberg, 1989; Nielsen, 1990, 1995). Nodes are units of information; they hold chunks of content and they are connected by hyperlinks. The size of nodes, i.e., the number of screen pages belonging to that node, may vary. Hypertext designers, editors and writers are continuously confronted with the decision as to whether to add information to a node, and to present more information than one screen page belonging to that node, or to split that node into many smaller nodes, and consequently to create a deeper hierarchy of nodes. This last option seems to be the solution Kreitzberg and Shneiderman (1988) recommend. Instead of discussing a topic more extensively, one could merely mention the subsidiary topics and make these topics the subject of their own sections (or 'nodes'). They mention "this shields the reader from unnecessary details, but provides a path to them when the reader deems it relevant" (p. 4). Unfortunately they don't present empirical data to support this claim. Kreitzberg (1989) also suggests that in order to reduce information overload subordinate information should be removed from the initial presentation and accessed by means of links. To phrase it briefly, Kreitzberg (1989) advocates the principle of "just enough up-front, details on demand" (p. 461), i.e. accessible by means of links. A negative consequence of relating information by using hyperlinks could be that the feeling of context diminishes. One gets abruptly confronted with new, rather isolated chunks of information, while at the same time the old context disappears from the screen. The question we want to address empirically is, what are the consequences for the usability (efficiency and effectiveness) of the system of expanding a node (but having to present information under the screen border) as opposed to splitting a node in more, smaller nodes accessible by links. Please note that this question is relatively independent of the specific interface we used, and also relatively independent of the specific online newspaper we used. All hypertexts have to deal with this choice.

 It is useful to give a brief impression of the interface used in the context of the Internet browser Netscape 2.0. We used this version of Netscape because at that time (end of 1996) most non-professional users had installed Version 2 on their PCs at home, so they were familiar with this version. In the prototype we examined the pages of the online newspaper are depicted as white paper sheets against a gray background. In Figure 1 we have presented a copy of the opening page.

Most manipulation and navigation functions are located outside these white paper sheets or near the borders of it. This conforms to one of the 22 recommendations for good screen design distinguished by Grabinger and Osman-Jouchoux (1996): "Keep navigation controls in a separate area". When entering the online newspaper, left of the page is a vertical row of 9 buttons representing the 9 main sections of the paper, such as the Front Page, Media, Traveling, Service, and so on. Clicking on one of these nine buttons changes its color from blue to white/gray, and opens the first screen page of information in that section, and remains white/gray as long as you are in that section. As usual in a hypertext system in Internet environment all links are underlined, in this prototype in blue. Clicking on these links leads to a new 'node' or page of information, and so on. On the right side of a page in this prototype, a Netscape scroll bar is present. For more details of the appearance and interface of the online newspaper we refer to http://www.volkskrant.nl. The site has been on the net for approximately two years now, and the main characteristics that we describe here have remained the same. Recently Fransen (1998) has described in greater detail how the site was developed. A starting-point of the project was to allow the editorial staff to focus on the content of the online newspaper and to maintain a growing site with minimal technical support.

As mentioned above, an important aspect of navigating through a hypertext is the structure of the information. We will distinguish a so-called 'horizontal' dimension and a 'vertical' dimension concerning the structure of hypertexts. Several levels of nodes or information units can be distinguished concerning the horizontal dimension. A top level represents a brief description of the main articles (level 0), and navigating along the hot spots or hyperlinks the deeper embedded levels can be activated (See Figure 2a).

In the prototype a horizontal structure of 5 levels (level 0 - 4) deep was present. Besides this dimension, it happens that text information or links are presented 'below' the screen border, thus, to read that information a user has to scroll down to get that information visible on screen (Figure 2b, the vertical dimension). This can happen on each horizontal level. In this prototype there were instances where three or more screens (to a maximum of 5) had to be scrolled down by using the scroll bar to show the lower part of that node on screen. We call this dimension the vertical dimension.

Our main research question is thus: what is the influence of having to use the vertical dimension (that is, reading-manipulation by scrolling) versus the horizontal dimension (reading-manipulation by using a hyperlink) on finding desired information that is available at different locations in the hypertext structure? Regarding scrolling we might expect to find that information is harder to locate when scrolling is needed than when the required information is immediately available on screen. In the same way we might expect that information is harder to locate when more hyperlinks have to be "visited" compared to having to visit fewer hyperlinks. Besides these two hypotheses it is worthwhile to examine the effect of scrolling compared to using hyperlinks. This comparison addresses the question whether scrolling is more detrimental to finding information than having to visit hyperlinks. This effect should be expected if the previously mentioned claims of Kreitzberg and Shneiderman (1988) and Kreitzberg (1989) are correct.

The usability of these aspects of the hypertext system was primarily evaluated by measuring the speed and accuracy of locating information, and by measuring the recognition of information encountered during reading the hypertext. Of course, the logic of the recognition measure is that minimizing cognitive overload will enhance comprehension and memory (Thüring, Hanneman & Haake, 1995). Consequently we should expect a better recognition of information present in the hypertext when it is located at an easy-to-find location.

Subjects: Twenty students (10 male and 10 female) with varying experience in using PCs, MS-Windows, and the Internet participated in the experiment. Preceding the experiment subjects' experience with hypertext was determined by means of a six-item questionnaire. The questionnaire sought to measure the frequency with which subjects used certain operating systems, hypertext applications, and the Internet. If subjects frequently surf the Internet, they are already quite familiar with a mouse, scrolling and using links in hierarchical hypertext structures. An example of a question is "How often are you visiting the Internet?" Response could range from 1 (never) to 5 (very often). On the basis of the percentile scores, subjects were classified as inexperienced (n=6), moderately experienced (n=8) and highly experienced (n=6).

The pre-existing attitude of subjects to reading news electronically was determined by means of an open question: "What is your attitude to electronic newspapers on Internet?". Answers were assigned to one of three categories: negative attitude to reading the news electronically, neutral attitude, and positive attitude.

At the end of the session subjects received a questionnaire with 5 qualitative, open-ended questions and 9 rating scales focused on satisfaction. Examples of the qualitative questions include "What is your opinion about the colors used?", "Did you like the layout and fonts?" and "Was it clear how to use all functions?". The rating scales were designed to measure satisfaction of the subjects with the system. For instance, representative questions were "Would you use this online newspaper more frequently if you had access to the Internet?" with a scale 0 (definitely not) to 4 (yes, certainly) and "Try to give your judgment of usability'" with a scale from 0 (very unfriendly) to 4 (very friendly).

The independent variables computer experience (low, moderate, high) and pre-existing attitude (positive, negative) were used as between-subjects variables in analyses of variance, and scrolling and using hyperlink(s) were treated as within-subjects variables.

Materials: The documents were newspaper articles about all kinds of topics (politics, travelling, sports, and so on). The prototype as briefly described above was used.

Equipment: The online newspaper was presented in the context of Netscape 2.0, on a 486-66 PC with 8 MB RAM.

Procedure: Subjects participated in two sessions, with a one-week interval between sessions. Prior to the experiment, the computer experience of subjects and pre-existing attitudes towards reading news electronically were assessed by means of a number of questions. In each session, after a brief exploration of the newspaper, subjects received 12 search tasks consisting of finding certain specific information. They were instructed that it was the purpose of the task to find a requested article as fast and accurately as possible. For example, subjects were assigned a task such as finding the article "Asia" present somewhere in the main section Traveling. In the first and second session subjects received 12 tasks, randomly sequenced. For each level of the four examined we had 3 search tasks. Task performance was recorded with a video camera. For each task speed and accuracy were measured.

Next, subjects received a recognition task. In this task they had to recognize 12 titles of articles in the hypertext. Response alternatives were "Yes", "No", and "Don't know". In order to reduce the tendency to respond yes, 12 non-existing titles were also presented. We here report only the correct recognition of presented information. At each location 3 items were presented. Since there are 4 locations, the correct recognition scores may thus vary from 0 to 3 per location. The final measure subjects completed was the satisfaction questionnaire.

As described above we gathered audio-video tapes of the subjects while they read the online newspaper and performed the tasks. Analysis of these data enabled us to detect a number of detailed shortcomings of the prototype interface which we communicated to our private partner. With respect to navigation, subjects regularly complained about the scrolling reading technique. Regarding the articles in the newspaper itself, readers missed dates on it, and they also complained that they had no idea of their length.

Next, we mention very briefly some general outcomes, mainly based on the questionnaires and rating scale data. The results, on the basis of the open-ended questions, showed that the subjects in general were highly satisfied with the prototype. Some of the features this online newspaper offers, like the possibility to consult the archives (though one will have to pay for this), and a kind of 'customised news-delivery' (not implemented yet) were highly appreciated. However, there were also some minor points to improve, such as the size of the pictures, and the meaning of red buttons in the upper-right corner of the page (indicating search, table of contents and help functions). Another minor point perceived by the subjects was that there was a rather small amount of printed text on a page; however, this has since changed.

Further, pre-existing attitudes of subjects to electronically reading news had no significant (p>.05) influence on performance (neither on search time nor on recognition), but pre-existing attitudes had a significant influence on satisfaction, measured by the rating scales, in working the first time with the system (F (2,18)= 7.49, p<.05). Subjects with a negative attitude were less satisfied than those with a positive attitude. However, the second time this difference disappeared, and satisfaction became also quite high for subjects with an originally negative attitude. There were at this time no significant (p>.05) effects remaining of pre-existing attitude on satisfaction.

Now we focus on the specific aspect of this study that we want to discuss more extensively in this paper: the effects of location of information in a hypertext system, and the influence of type of navigation to find this information. The performance of subjects on various search tasks was very adequate; even on the deeper levels subjects could find the requested information. Approximately 95% of the search tasks were solved correctly. Further, subjects with little computer experience performed almost as well as subjects with extensive computer experience. There were no significant differences in percentage correct nor in search time (p>.05), though the difference in search time between inexperienced users and the other categories was somewhat (but not significantly) greater during the first session than during the second session. Only at the deeper levels we did find significant differences between inexperienced users and moderately and highly experienced users during the first and second session (F(2,18)=5.89, p<.05; F(2, 18)= 6.68, p<.05).

The first specific question examined was how much time users need to locate information in the hypertext system with the two kinds of manipulation, and what is the influence of location? More specifically, what is the influence of using different numbers of hypertext links (that is, using the horizontal dimension), what is the influence of scrolling (the vertical dimension), and what is the influence of their interaction on search time? What was measured here is the time subjects needed from the start till the moment they found the link needed to lead to the requested information. For this analysis, the location of information relevant to the search tasks was determined, and we analyzed the data for the horizontal dimension on level 1 (that is, to reach the desired information one had to click once on a hyperlink) and level 2 (that is, one had to go deeper and click on two hyperlinks to see the requested information on screen). We did not analyze level 0 because the only thing subjects had to do here was to select the main section in which they wanted to search, which was a very easy task. Furthermore, this information (at level 0) was always immediately available on screen. In Figure 3 we have schematically presented the paths subjects had to follow. The upper part represents the paths subjects had to follow when the needed link was immediately available on screen (vertical dimension level 0) but on one or two hyperlinks had to be clicked.

Concerning the vertical dimension the link leading to the desired information could be directly available on screen (vertical dimension level 0) or outside the first screen border and one had to scroll down to reach the desired link (level 1, that is, outside the first screen, and within 60 lines of a section). The last situation is represented in the lower part of Figure 3. So we registered how much time it took to find information after one hyperlink as opposed to two hyperlinks (horizontal dimension level 1 versus 2). Further, the needed link could be directly on screen as opposed to outside the screen border (vertical level 0 versus level 1). These four combinations were examined. It is possible to distinguish more than these four locations on both dimensions. However, due to practical constraints we analyzed only these four.

A first general outcome was that the search times on all levels were significantly (p<.05) shorter in the second session than in the first session. In Figure 4a we have presented the mean (correct) searching times (in seconds) in the four conditions. Analysis of variance showed main effects of horizontal dimension (F(1,18)=120.50, p<.001) and vertical dimension (F(1,18)=250.55, p<.001): Reaching the desired information after clicking one hyperlink (level 1) logically costs less time than finding the information after clicking two hyperlinks (level 2), and also finding a link when the information is directly available on screen (level 0) takes less time compared to finding a link after scrolling down (level 1). These effects are as can be expected. Furthermore, it appeared that reaching the desired information after scrolling takes approximately as much time as finding information by using hyperlinks. But there was also a significant interaction effect of horizontal and vertical dimensions (F(1,18)=16.03, p<.001). In particular, reaching the requested information after scrolling down (vertical level 1) at the second horizontal level (level 2) takes much extra time. In both sessions the same pattern was found.

Figure 4a: Average time needed to find a hyperlink per horizontal and vertical level, after selecting a main section on the front page (level 0).

Figure 4b: Average recognition of links per horizontal and vertical level.

The second question examined was: what is the recognition performance of subjects for topics of varying location? That is, the location of topics or hyperlinks varied, and could be as indicated above, at horizontal level 1 or 2, and at vertical level 0 or 1. The question examined is whether recognition is influenced by the horizontal or vertical level or their interaction.

In Figure 4b we have presented the mean recognition performance for information previously presented. The scores in each condition may range from zero to three. Analysis of variance showed significant main effects of horizontal dimension (F(1,18)=11.66, p<.05) and vertical dimension (F(1,18)=3.75, p<.07): Links located on the horizontal level 1 are better recognized than links on the second horizontal level, and links reached within the first screen border (level 0) are somewhat better recognized than links reached outside the screen border (level 1). These effects could be as expected because this information appears first and the most frequent, but more important was the significant interaction effect, (F(1,18)=8.78, p<.05): in particular, information reached outside the screen border and after clicking two hyperlinks is badly recognized. Also here the same pattern was found in both sessions.

An interesting result, important for practical purposes, was that subjects who initially had a negative attitude get used to, and are satisfied with working with a system like this one and to processing the news information very quickly. Only during the first session did they showed lower satisfaction. By the second session the difference disappeared, suggesting that less experienced users got used to the system quickly and became almost as fast and adept as experienced users during the second session. This indicates that the online newspaper had an interface that was apparently very easy to learn. Please note that these data are tentative and need to be replicated on a larger scale.

We will focus here on the results with respect to the influence of reading-manipulation technique on locating information and subjects' memory for this information. As expected, finding information by scrolling down or by using a hyperlink to go to a next level takes more time than having the hyperlink leading to the desired information immediately available on screen. The results showed that finding information by scrolling down the screen border and by using a hyperlink took approximately the same amount of time, and recall was about equal. However, finding information for which scrolling down on a deeper hypertextual level is necessary takes extra time and probably extra cognitive resources, leading to a lower recognition performance. So if the choice would be as simple as: must we lengthen a page so that it is scrollable, or must we put the information on a next page, then it wouldn't matter much. However, if one puts the information in a place that has to be reached by scrolling and paging, it takes far more time, and the recall is inferior. It seems that taking more than one step has a dramatic influence on search time and recognition of items. If one wishes users to remember information, it shouldn't be presented in a deep hypertextual location outside the screen border. It should be noted that memory plays an important role in reading conventional papers: the behavior readers often display is first to skim the whole context to see what's there, and then to read more extensively what they decided to read. But one has to remember where to find it!

On the basis of these results we may conclude that expanding a node (and having to scroll) is not worse than splitting a node (and having to use a hyperlink). This result deviates from the claim put forward by Kreitzberg and Shneiderman (1988) and Kreitzberg (1989), mentioned in the introduction, "Just enough up-front, details on demand". It is possible that this discrepancy can be explained by the materials used. We used texts or discourse with a less clear segregation in logical units. This issue deserves more study. Our study, however, also shows that especially for details on deeper levels scrolling should be avoided.

Figure 5: Moving text to a next deeper level (upper part) or to a level above (lower part).

We want to emphasize again that our conclusions are tentative. This study was actually a combination of a field study and laboratory experiment: the prototype we had to work with (in particular, the supply of articles), did not allow us to examine all combinations of levels on the two dimensions. More careful laboratory experiments in which the horizontal levels and vertical levels are systematically varied, keeping the information constant, are needed to support our findings. Were our findings to be confirmed, further research has to make clear whether in that case (in order to avoid scrolling on deeper levels) it is better to include a next deeper level, that is, to create an extra node (see Figure 5, the upper part), or to extend the level above it, that is, to enlarge the node on the direct superordinate level (see Figure 5, lower part).

We would like to close with some suggestions based on this study. First, under the present conditions it could be better to avoid presenting much information outside the screen border on deeper levels in the hypertext hierarchy. On higher hypertextual levels presenting information outside the border is less problematic. Knowing this could be of practical utility for designers of electronic newspapers. Second, page designers should indicate clearly the end of a section of information, so that it is immediately visible for users that there is no more information in that section below the screen border. One reason for the fact that it sometimes took extra time to find the requested information was that subjects didn't realize that information was available below the screen border. One solution is to make it more clear with visual cues that a section is finished (or not). Third, we recommend that page designers indicate clearly how much information in a section has already been read and how much information is still left, for instance by emphasizing the position of the pointer of the scroll bar or by using some kind of index on the screen page or right next to it (see also Shneiderman, 1997). A number of subjects, particularly in-experienced users, didn't completely use the information that is in principle provided by the position of the pointer of the scroll bar. Often they didn't realize that the pointer position corresponds to how much is done or still left. Our suggestion is thus to emphasize the relative position.

We would like to thank Bert Arnold, Andrew Dillon, Monique Fransen and Jenny Preece for useful comments on an earlier version of this article.

Britt, M. A., Rouet, J. F., & Perfetti, C. A. (1996). Using hypertext to study and reason about historical evidence. In J. F. Rouet, J. J. Levonen, A. Dillon & R. J. Spiro (Eds.), Hypertext and cognition. Mahwah, NJ: Erlbaum.

Dee-Lucas, D. (1996). Effects of overview structure on study strategies and text representations for instructional hypertext. In J. F. Rouet, J. J. Levonen, A. Dillon, & R. J. Spiro (Eds.), Hypertext and cognition. Mahwah, NJ: Erlbaum.

Dieberger, A. (1997). Supporting social navigation on the World Wide Web. International Journal of Human-Computer Studies, 46, 805-825.

Dillon, A. (1994). Designing usable electronic text: Ergonomic aspects of human information usage. London: Taylor & Francis.

Ebersole, S. (1997). Cognitive issues in the design and deployment of interactive hypermedia: Implications for authoring WWW sites. Interpersonal Computing and Technology: An Electronic Journal for the 21st Century, 5, 1-2, 19-36.

Edwards, D., & Hardman, L. (1989). 'Lost in hyperspace': Cognitive mapping and navigation in a hypertext environment. In R. McAleese (Ed.), Hypertext: Theory into practice. Oxford: Intellect.

Fransen, M. (1998). Electronic newspaper case study: Flexible user interface design for easier web site maintenance. Trends in Communication. Special Issue on Interface Technology: Enhancing the Quality of Life, 2 (1).

Grabinger, S., & Osman-Jouchoux, R. (1996). Designing screens for learning. In H. van Oostendorp & S. de Mul (Eds.), Cognitive aspects of electronic text processing. Norwood, NJ: Ablex.

Kreitzberg, C. B. (1989). Designing the electronic book: Human psychology and information structures for hypermedia. In G. Salvendy & M. J. Smith (Eds.), Designing and using human-computer interfaces and knowledge based systems. Amsterdam: Elsevier Science Publishers.

Kreitzberg, C. B., & Shneiderman, B. (1988). Restructuring knowledge for an electronic encyclopedia. Proceedings International Ergonomics Association 10th Congress. Sydney, Australia.

McAleese, R. (1989). Navigation and browsing in hypertext. In R. McAleese (Ed.), Hypertext: Theory into practice. Oxford: Intellect.

McKnight, C. (1996). What makes a good hypertext? In H. van Oostendorp & S. de Mul (Eds.), Cognitive aspects of electronic text processing. Norwood, NJ: Ablex.

McKnight, C., Dillon, A., & Richardson, J. (1990). A comparison of linear and hypertext formats in information retrieval. In R. McAleese & C. Green (Eds.), Hypertext: State of the art. Oxford: Intellect.

Nielsen, J. (1990). Hypertext and hypermedia. San Diego, CA: Academic Press.

Nielsen, J. (1995). Hypertext and multimedia: The Internet and beyond. New York: Academic Press.

Piolat, A., Roussey, J-Y., & Thunin, O. (1997). Effects of screen presentation on text reading and revising. International Journal of Human-Computer Studies, 47, 565-589.

Shneiderman. B. (1997). Designing information-abundant web sites: Issues and recommendations. International Journal of Human-Computer Studies, 47, 5-29.

Thüring, M., Hanneman, J. & Haake, J.M. (1995). Hypermedia and cognition: Designing for comprehension. Communications of the ACM, 38 (8), 57-66.

van Nimwegen, C., Pouw, M., & van Oostendorp, H. (in press). The influence of structure and reading-manipulation on usability of hypertexts. Interacting with Computers. Special Issue on 'Designing for Human Needs and Capabilities."

Herre van Oostendorp is lecturer at Utrecht University in the Department of Media and Communication, and the Department of Psychonomics. He earned a degree in Experimental Psychology. He received his Ph.D. from Amsterdam University on "Regulation Processes during Text Processing". His current research is directed to the description of factors and processes that can play a role in adjusting the accuracy of mental representations in connection with new information. Further research interests lie in the area of aspects of easy-to-use multimedia systems, such as online newspapers.
Address: Dept. of Psychonomics, Utrecht University, Heidelberglaan 2, 3584 CS Utrecht, The Netherlands. Email: H.vanOostendorp@fss.uu.nl

Christof van Nimwegen studied Cognitive Ergonomics, Department of Psychonomics at Utrecht University. He specialized in communication and interactive media. He is now with Capi Lux Amsterdam, Department of Image Management. He is responsible for the development of user interfaces of multimedia products.
Address: Capi Lux, Dept. of Image Management, Basisweg 42, 1043 AP Amsterdam, The Netherlands. Email: Nimwegen@capi.nl