Hyperlinks to resources available on the Web are commonly used in footnotes or citation of references in scholarly online journals. A review of three well-established online journals revealed that nearly half of such links are broken, raising questions regarding the archival integrity of the published material. Some practical recommendations are advanced for online journal editors and authors based on a consideration of the role of hyperlinked references.
As more academic and technical journals move to disseminate their contents over the Internet, either exclusively or concurrently with conventional, printed versions, the distinctive features of online publications have become well assimilated into the practice of scholarly writing (Cesarone, 1999). In particular, hyperlinks to related material available online at other Web sites are commonly used in footnotes or citation of references in the literature. All the major style manuals have expanded their guidelines to include the citation of resources on the Internet (e.g., Greenhill & Fletcher, 2003; Lester, 1997; Walker & Taylor, 1998). However, as with any Web-based material, hyperlinks in scholarly online publication can become invalid (Markwell & Brooks, 2002), especially when the citing authors have no control over changes on the linked sites. Entire Websites may become defunct, or specific pages removed resulting in "broken" links typically indicated by "Page not found" or "Page cannot be displayed" error messages. In other cases, the URL for the hyperlink may be active but no longer pointing to the intended content. This occurs, for example, with links to material released under such rubrics as "Current Issue" or "Latest News." While the fleeting nature of such links is fairly self-evident, and citing authors should be wary, the longevity of other seemingly stable links can still be unpredictable. How serious is the problem of broken hyperlinks in online scholarly publications? The results of a review of several well-established online journals are revealing.
A Review of Three Online Journals
The review was conducted in March 2004 of three online journals: The Journal of Computer-Mediated Communication, First Monday, and The Journal of Interactive Media in Education. These journals were selected because of their established presence and accessibility on the Internet. All have been in existence since at least 1996, and all are peer-reviewed, open access, exclusively online journals. The review was comprehensive, covering all articles published in the first and third cases, and was based on a random sample in the second.
For each article, all external hyperlinks, or links pointing outside the article, were analyzed. Two kinds of external hyperlinks were identified. The first are "citation links" used in bibliography or reference sections, e.g.,
Ho, J. (2001). APEC Multilingual International Trade Project: Methodology and case reports on needs assessment. Journal of Computer Mediated Communication 9 (1) [online] http://jcmc.indiana.edu/vol9/issue1/ho.html
The second are "resource links" pointing to other external material providing further information about the content in the text or footnotes, e.g.,
The first prototype service based on this concept is TigerTalk[sm], which is offered free on the Web site http://www.cyber-tigers.com.
Internal links within an article, e.g., those pointing to graphical images or tabulated data stored separately, or to other articles in the same journal, were excluded. Each link was visited to verify its validity. In case of uncertainty, for example as to whether failure was due to temporary networking or server problems, the link was re-checked until a conclusion was reached. The numbers of valid and broken links of each kind were recorded. The results are summarized below.
The Journal of Computer-Mediated Communication (http://jcmc.indiana.edu) has been published quarterly on the Web since June 1995. It follows the American Psychological Association Manual of Style for academic citations. In addition, authors are explicitly encouraged to link their references to online sources, where available. Volumes 1-5 (1995-2000) contain 130 articles (including editorials), 86 of which (66%) had at least one external hyperlink. There were a total of 491 citation links in the "References" or "Bibliography" section, 299 of which (61%) were broken. In addition, 291 resource links were used within the text or footnotes of these articles, with 179 (62%) being broken. Volumes 6-8 (2000-2003) contain 74 articles, 60 of which (81%) used external hyperlinks. Of 364 citation links, 129 (35%) were broken. Of 157 resource links in text or footnotes, 28 (18%) were broken.
First Monday (http://www.firstmonday.org) publishes articles on the first Monday of each month on all aspects of the Internet, including commentary on trends and standards, technical issues, political and social implications of the Internet, and educational uses. It specifies its own citation and reference guidelines, including for articles on the World Wide Web. Since May 1996, First Monday has published 504 peer-reviewed papers by 609 different authors in 93 issues. Using the "Random Monday" feature on its Website, a sample of 30 articles, dating from April 1997 to October 2003, was selected. Fifteen articles were from July 1997 to July 2000; all (100%) used external hyperlinks. These articles contain 97 citation links, of which 52 (53.6%) are bad, and 76 resource links, of which 44 (57.9%) are bad. The other 15 articles were from August 2000 to October 2003; all but one (93%) includes external links. The newer articles contain 97 citation links, of which 29 (29.9%) are bad, and 102 resource links, of which 43 (42.2%) are bad.
The Journal of Interactive Media in Education (http://www-jime.open.ac.uk/) is targeted at researchers and practitioners interested in educational technology. Its citation and reference style guidelines are similar to those of the American Psychological Association. No special instructions are given regarding online resources, although authors are encouraged to include "an interactive component, or supplementary materials of some other form." Of the 14 articles published from December 1996 through December 1998, all but one (93%) used external hyperlinks. Of 30 citation links, 19 (63%) were broken. Of 137 resource links in text or footnotes, 81 (59%) were broken. Of the 14 articles published from March 1999 through July 2002,1 all but two (86%) used external hyperlinks. Of 39 hyperlinked citations, 18 (46%) were broken. Of 38 resource links in text or footnotes, 15 (39%) were broken.
This review shows an overall increase in the use of hyperlinks in the more recent volumes, as well as a higher "mortality rate" of hyperlinks in the earlier volumes. Both trends are expected, as the hypertext format continues to gain in popularity, and older links are more likely to go bad. The statistics are summarized in Table 1.
| Journal | JCMC | First Monday | JIME | ||||
| Years | 95-00 | 00-03 | 97-00 (sample) |
00-03 (sample) |
96-98 | 99-02 | |
| # articles | 130 | 74 | 15 | 15 | 14 | 14 | |
| % with links | 66% | 81% | 100% | 93% | 93% | 86% | |
| # citation links | 491 | 364 | 97 | 97 | 30 | 39 | |
| % broken | 61% | 35% | 54% | 30% | 63% | 46% | |
| # resource links | 291 | 157 | 76 | 102 | 137 | 38 | |
| % broken | 62% | 18% | 58% | 42% | 59% | 39% | |
| Total # links | 782 | 521 | 173 | 199 | 167 | 77 | 1919 |
| % broken | 61% | 30% | 55% | 36% | 60% | 43% | 49% |
Of all the hyperlinks reviewed, close to half were broken. This is clearly significant in terms of the archival integrity of the published material. Therefore, a closer examination of the role of hyperlinked references is in order.
The Purpose of Hyperlinked References
The general purpose of an academic citation is well established (see, e.g., Dartmouth Committee, 1998). To include online references, we propose the following threefold characterization:
- Credibility: By referring to previous contributions, the author establishes a sound foundation for his or her own work.
- Credit: By giving credit where it is due, the author acknowledges the contributions of previous work.
- Connection: By giving a precise reference, the author helps others locate and connect to the cited work.
In conventional, printed media, the Credibility and Credit purposes are pillars of the scholarly tradition and are served by proper identification of the cited work in specific formats set by venerable style manuals. In print media, the Connection purpose is rather incidental. There is usually no attempt to show the reader how or where to obtain the material, or what the cost might be to make such a connection. If it is a book, the reader may have to find it in a library, purchase it from a bookstore, or order a copy directly from the publisher if it is still in print. If it is a journal article, the reader needs to be an individual subscriber or have access to a library with a subscription. In any case, the burden of location is on the reader seeking the connection.
Web-based publication has not only transplanted printed pages to Web pages, the hypertext format has also transformed the Connection purpose of citations and references. By using a hyperlink, the author is pointing the reader directly to another Web-based resource. As any active link on the Web is freely accessible, this gives the impression that the reader can connect to the cited material readily and at no cost. This expectation is reinforced by standards of precision in academic citations. A broken link leading nowhere is clearly disappointing. Even a live one opening up to a subscriber log-in page can be frustrating. And a nebulous link to the homepage of a newspaper, magazine, or publisher, when the citation is to a specific article, is downright frivolous. So even though the Connection purpose of academic citations does not conventionally imply direct connection, the presence of any kind of broken or misleading links in an online publication can make the work appear sloppy, because of the very nature and function of hyperlinks.
As a caveat emptor, a common practice is to include in the full citation the date on which the hyperlink was actually accessed and, presumably, verified to be active and accurate. This is sufficient to serve the Credibility and Credit purposes, while disclaiming any guarantee of the continuing validity of the hyperlink for the Connection purpose. However, to properly achieve this goal, a hyperlink should be removed—de-activated to be exact—as soon as it is discovered to have gone bad, while leaving the URL intact for the first two purposes. In some online journals, editors may reserve the right to remove or update broken hyperlinks. The following is an example that appeared in the Journal of Early Childhood Research and Practice (http://ecrp.uiuc.edu/v4n1/saluja.html):
National Resource Center for Health and Safety in Child Care. (2000). Individual states' child care licensure regulations [Online]. Available: http://nrc.uchsc.edu/states.html#TOP
[Editor's note (9-18-03): This url has changed: http://nrc.uchsc.edu/STATES/states.htm] [2002, February 12]
Considerable effort and progress is currently being made towards the adoption of more persistent URLs and digital object identifiers (see, e.g., www.doi.org; www.purl.org; www.w3.org). The scholarly community can and should play a leading role in such efforts. However, no amount of standardization and protocol can eliminate broken links in the absence of good practice. After all, as pointed out by Tim Berners-Lee (1998), "URI's don't change: people change them. There are no reasons at all in theory for people to change URIs (or stop maintaining documents), but millions of reasons in practice."
In light of the above, we advance several practical recommendations for online journal editors and authors:
- Journals should adopt precision standards for hyperlinked citations. Authors should provide a link only if the cited material is directly available, and should avoid nebulous and generic links, e.g., to the homepage of a publisher.
- Online journal websites should provide a simple mechanism for authors and readers to report broken hyperlinks. This could be in the form of a button beside each link that when clicked generates an e-mail message to the appropriate editorial address.
- Once a broken hyperlink is reported, it should be de-activated so that it no longer implies that it will serve the Connection purpose of academic citation. The unlinked URL should be left intact to serve the Credibility and Credit purposes.
These recommendations constitute realistic compromises: de-activating a hyperlink once it has been reported to be bad is a one-time action, as opposed to the alternative of keeping up with all eventual updates, which is impractical.
In short, if broken links are a disservice to the readership of online journals—and the problem is clearly evident from our review results—editorial policies need to be very specific on the issue in order to maximize the added value of the Internet as a medium for disseminating scholarly work.
Helpful suggestions regarding this article were provided by the editors and anonymous reviewers.
- The 2003 JIME articles were in a special issue that made use of a different format from earlier issues; that issue was not included in the analysis.
Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, 18 (2), 159-174.
Berners-Lee, T. (1998). Style Guide for Online Hypertext. Retrieved February 15, 2005 from http://www.w3.org/Provider/Style/
Cesarone, B. (1999). Writing for electronic journals. Early Childhood Research & Practice, 1 (1). Retrieved February 15, 2005 from http://ecrp.uiuc.edu/v1n1/cesarone.html
Dartmouth College Committee on Sources. (1998). Sources: Their Use and Acknowledgement. Retrieved February 19, 2005 from http://www.dartmouth.edu/~sources/index.html
Greenhill, A., & Fletcher G., Eds. (2003). Electronic References & Scholarly Citations of Internet Sources. Retrieved February 15, 2005 from http://www.spaceless.com/WWWVL/
Lester, J. (1997). Citing Cyberspace. New York: Addison, Wesley, Longman. [Available online.] Retrieved February 15, 2005 from http://www.apsu.edu/~lesterj/cyber.htm
Markwell, J., & Brooks, D. (2002). Broken links: The ephemeral nature of educational WWW hyperlinks. Journal of Science Education and Technology, 11, 105-108.
Walker, J., & Taylor, T. (1998). The Columbia Guide to Online Style. Columbia University Press.
is a professor of information and decision sciences in the College of Business Administration at the University of Illinois at Chicago. His recent work focuses on parallel natural language generation in multilingual online communication, and topological analysis of online auction markets. For more information, see http://www.uic.edu/~jimho/
Address: m/c 294, 601 South Morgan, Chicago, IL 60607, USA