JCMC 6 (3) APRIL 2001
Collab-U CMC Play E-Commerce Symposium Net Law InfoSpaces Usenet
NetStudy VEs VOs O-Journ HigherEd Conversation Cyberspace Web Commerce VisualCMC
Vol. 6 No. 1 Vol. 6 No. 2
Survival of the Fittest Online: A Longitudinal Study of Health-Related Web Sites
Sally J. McMillan
University of Tennessee, USA
- Literature Review
- Quantitative Findings
- Qualitative Findings
- Features and Technology
- Organizational Issues
- External Issues
- Most Important Reasons for Success or Failure
- Conclusions and Discussion
- Suggestions for Researchers
- About the Author
AbstractLongitudinal studies of Web sites can not only trace the history of specific sites but can also provide some guidelines that might help start-up Web sites improve their chances for survival in the ever-changing Web landscape. This article reports on a longitudinal study of health-related Web sites that found about 73% of those sites survived for three years. E-mail surveys gathered quantitative and qualitative insights from survivors and non-survivors. Generally, organizations that invested more resources (money, people, and time) in their Web sites in 1997 were more likely to have surviving sites three years later. Technological expertise, as measured by the level of feature-richness at sites, had relatively little impact on survival. Sites created by individuals were least likely to survive while education and government sites were most likely to survive. Respondents indicated that organizational commitment to Web-based communication was an important factor in long-term survival of sites. Sites with more visitors in 1997 were more likely to survive than those with fewer visitors, but many respondents pointed out the importance of targeting rather than sheer numbers in measuring the Web site audience.
IntroductionThe World Wide Web (Web) is dynamic. The Internet Software Consortium (2000) reported that about 1.3 million Internet hosts (domain names such as ix.netcom.com) existed in January of 1993. By January 2000, more than 72.4 million hosts were found. But despite this exponential growth, not all Web sites have survived.
Koehler (1999) reported that in a one-year period from December 1996 through December 1997, 25.3% of a randomly selected sample of Web sites ceased to exist. In an ongoing study of those sites, Koehler (2000) found that overall survival rate had dropped to 40% by April 2000. However, among military (.mil), government (.gov), and non-profit sites (.org) overall survival was at 80% in April 2000.
But, other than domain suffix, what other factors might help to predict the permanence of Web sites? What is the relative importance of resources, technology and Web site features, organizational factors, and external opportunities and threats to Web site survival? Longitudinal studies are the best way to answer questions such as these. This article addresses the importance of longitudinal studies and provides data from one study of health-related Web sites that provides insight into how Web sites can avoid becoming endangered species.
This study is important because it provides information about both the why and the how of conducting longitudinal studies of Web sites. It is important because it provides a profile of organizations that have been able to make the Web a long-term part of their organizational communication strategy. Finally, it also provides some important information specific to health communication. First, it profiles a tool that can be used for dissemination of public health information. Second, the study provides some insights into the impact that commercialization may be having on the dissemination of health information on the Web.
Literature ReviewThe Web first became widely available to the public in 1993 with the dissemination of the first Web browser, Mosaic. Zakon (1999) reported that in the subsequent six years, the Web grew from 130 separate Web servers (each of which may have multiple sites by using different domains or port numbers) to more than 7 million servers. In addition to growth in the number of potential Web sites, the actual amount of information available at many Web sites has also grown over time. For example, Koehler (1999) found that the average number of total bytes at a random sample of Web sites grew from about 3.5 million to 6.3 million in one year. Individual Web sites seem to be almost doubling in size each year.
Yet, as Feldman (1997) pointed out, growth of the Internet does not necessarily mean that content is permanently available. She defined the Internet as a “world frustrating and confusing in which materials that have some lasting intrinsic value are treated as disposable” (p. 52). Germain (2000) echoed this frustration in a study of Web sites that were identified in articles published in major academic journals. Over time, she found a significant decrease in the availability of those Web sites.
The University of Waterloo Index (2000) reported that since 1997 the URLs (uniform resource locator) or addresses of Web sites have begun to stabilize. As of May 2, 2000 the Waterloo study found 71.5% stability among its sample of professional societies. The researchers suggested that organizations that have their own domain name server (DNS) are more stable than are those that operate at a second- or third-level domain. So, for example, the URL http://myorg.com would be more stable than http://aol.com/~username. Among professional societies, communication societies were found to be most stable with a stability index of 88.5% and classical studies societies were least stable with a stability index of 25.0%.
Longitudinal studies are an ideal way to study growth, decline, and change in computer-mediated communication environments. Longitudinal studies have examined CMC-related topics such as acceptance and evaluation of CMC by college students and other users (Hiltz & Johnson, 1989; McMurdo & Meadows, 1996; Reis & Tymchyshyn, 1992), the impact of CMC on trends in public opinion toward privacy (Katz & Tassone, 1990), introduction of voice mail in organizational settings (d'Ambra & Rice, 1994), computer-aided analysis of documents such as annual reports (Kabanoff, 1996), trends in advertisements for technology-related jobs (McMillan, Sheehan, Heinemann, & Frazer, 2000; North & Worth, 1998), and use of electronic data interchange systems (Crum, Johnson, & Allen, 1997). Koehler has reported on a longitudinal study of Web site survival that uses automated systems to check for continued existence of Web sites (Koehler, 1999, 2000). He and a colleague (Koehler and Wall, 2000) recently reported that after sites have survived for about 2.5 years, they are subsequently less likely to cease to exist.
What none of the above-cited studies provides is detailed insight into why some Web sites are more stable than others are. And, among those that do survive, what factors may contribute to their stability?
Health-related Web sites offer an ideal venue for examining these issues of Web site permanence and change for five reasons. First, messages about health and health-related subjects have played a central role in the early development of many media types. For example, patent medicines were among the first products advertised in newspapers (Jones, 1996) and makers of health-related products were among the first sponsors of radio programming (Barnouw, 1966). Second, health-related sites are among the fastest growing topic areas on the Web (Fisher, 1996). Third, researchers have found that health-related Web sites are used heavily (Pingree et al., 1996). Fourth, health sites encompass a wide range of content types including public health information (Dorman, 2000), doctor patient communication tools (Grandinetti, 2000; Harris & Campbell, 2000), and support groups for patients suffering from diseases such as depression (Christensen, 2000). Finally, the literature on health communication is recognizing the importance of longitudinal studies in understanding not only health trends but use of health communication tools as well (Dick & Gabler, 1995; Gray & Brookmeyer, 2000).
The literature suggested four factors that might have an impact on site longevity: resources, Web site features and technology, organizational issues, and external issues. Among the resource issues most frequently identified in the literature are money, staff, and time (Domanski, 2000; Woodard, 2000). However, as Moschella (1999) pointed out, organizations that are resource-rich are not necessarily more capable of fully utilizing the Web. Some of these large organizations have failed to recognize the unique opportunities the Web offers and have either under-invested in Web-based communication or have invested unwisely. But economic observers have suggested that any big organization that fails to grapple with issues related to the World Wide Web is putting its future at risk ("Leaders: Dotty about dot.Commerce?," 2000). Despite the unwillingness of some organizations to invest resources in the Web, others have invested – often with the objective of using the Web as a way to maximize resources. But many organizations were too optimistic about short-term cost savings and abandoned the Web when they did not see quick return on investment (Hoffman, 1998). Other organizations have shut down their Web-based businesses because they have run out of money, often after over-spending on advertising, or because they failed to build a successful business model (Angwin, 2000; Evans, 2000). In addition to dedicating people and money resources, organizations must also recognize that building and maintaining content at Web sites takes time. Ozok and Salvendy (2000) suggested that organizations must recognize that taking the time for updating sites is critical. Sites that are updated frequently are expected to be more likely to survive than those that are not.
A second factor that may influence Web-site longevity is the technological capability of the site creators. Such technological capability is often reflected in the features of the Web site. For example, features such as chat rooms, search engines, personalized content, interactivity, and electronic payment options might be critical features that are required for the long-term success of some types of sites (Angwin, 2000; Domanski, 2000; Ha & James, 1998; Hoffman, 1998; Massey & Levy, 1999; McMillan, 2000; Walker, 2000). The features offered at a Web site should both utilize the technologies available and also be designed carefully to provide the content that visitors most desire (McMillan, 2000; Woodard, 2000).
Organizational issues are a third factor that might impact on site survival. Koehler (1999) found that some organizational types (e.g. government and non-profit groups) seem to be more likely to survive than others. Cunningham (2000) pointed out the importance of organizational commitment to current technology. Woodard (2000) noted the importance of leadership and vision among Web sites that survive. Other key organizational factors that seem to be important for Web site survival include understanding customers, providing good customer service, identifying niche markets, managing growth, and developing a vision for the future (Domanski, 2000; Evans, 2000; Hoffman, 1998; Moschella, 1999).
Finally, external issues may also impact on Web site stability. Both competitors and site visitors are external to the Web site, but both of these factors can have significant impact on site survival. The ability of a site to attract and retain qualified visitors is a key element of Web site success. Sites that fail to focus on the needs of visitors are less likely to survive than are customer-focused sites (Domanski, 2000; Evans, 2000; Hoffman, 1998; Walker, 2000; Woodard, 2000). Additionally, Domanski (2000) suggested sites that track usage of visitors might be more likely to survive than those that don’t. Similarly, sites that are built without a clear sense of what competitive sites offer are also often doomed to failure because they don’t offer unique or significant benefits (Cunningham, 2000; Domanski, 2000; Woodard, 2000).
A longitudinal study of health-related Web sites was conducted to address the question of why some Web sites are more stable than others. The longitudinal study reported in this article began in early 1997 with a random sample of health-related Web sites. The first time-point involved both an e-mail survey and a content analysis of all sites that responded to the survey. The second time-point in the longitudinal study came three years later in 2000. The researcher began the follow-up study by checking the URLs for all sites that participated in the 1997 study to identify survivors and non-survivors. Both quantitative and qualitative e-mail surveys were sent to survivors and the qualitative survey was also sent to non-survivors.
The sampling frame for the original study was all sites that are listed in the Yahoo! directory of health-related topics. Yahoo! is a topically arranged electronic directory of Internet content. Yahoo! does not sequentially number sites within its categories. Therefore, a strategy was needed for identifying how many sites were in the universe, randomly selecting sites from the sample, and determining how to apply random numbers to specific sites. The size of the universe was determined by adding numbers that Yahoo! provided for each health-related category on the “opening page” (or index) of the Yahoo! category listings. In January of 1997, 80 unique categories were listed in the health sections of Yahoo! The numbers following each of these categories were added to obtain the universe size of 14,974.
Using Excel software, a list of 1,050 unique random numbers was drawn. These numbers were applied by starting at the top of any given online page of Yahoo! listings and working through to the bottom of that page. For example, the first item on the first page of the health category was a topic listing for alternative medicine. When that listing was selected, another screen appeared which included additional sub-categories (e.g., institutes, magazines, etc.) as well as direct links to sites. Sub-categories were followed until they resulted in a direct site listing. The number of items in the category was used to help track assignment of random numbers. For example, the index indicated the alternative medicine category included 306 sites, so the number 307 was used at the start of the next category (conferences). Thus, if any slippage occurred in counting the sites, it would be corrected at the start of each category.
Within Yahoo! there is some cross listing of sites but in most cases this is not a problem because each site is assigned to a primary category.Within the Yahoo! list, other references to that site are followed by an @ symbol which indicates that the primary listing is in another category. Thus, no Yahoo! listings followed by the @ symbol were counted in determining the universe size.
Several factors led to reduction of the valid sample size. First, some sites had to be eliminated because they duplicated either the URL or the e-mail address of another site. Most of these duplicates were for companies that developed Web sites that have multiple sections focusing on different diseases (e.g., asthma, allergies). In each case, the first site with the duplicate URL and/or e-mail address was kept and others were discarded. Second, some of the URLs listed by Yahoo! were not functioning. Third, some of the sites did not have an e-mail address. Finally, in some cases an e-mail address was found but the mail was undeliverable.
After removal of duplicates, non-functioning Web addresses, and bad e-mail addresses from the original sample of 1,050 sites, the valid sample size was 834 Web sites. Managers of these sites were surveyed (see Appendix 1). Surveys were sent and received via e-mail. A reminder message was sent one week after the initial message. A total of 395 completed surveys were returned resulting in a response rate of 47.5%. Content analysis was conducted on all 395 of these sites.
For the follow-up study, the researcher examined all of the original URLs to determine if they site still existed. If sites were not found on the first check, the researcher checked twice more on different days to see if the site was back after a temporary hiatus. A total of 281 sites (71.1% of the original 395 sites) were found. For each site that was still operating, the researcher sent an e-mail message indicating a follow-up study was planned and soliciting the site manager’s participation. The e-mail address that was used for the original study was also used for this pre-notification message. The researcher wanted, if at all possible, to survey the same individual who had responded to the original survey.If the original e-mail address was not functioning, the researcher searched the site for another e-mail address and re-sent the pre-notification. In some cases the pre-notified individual indicated that he/she was no longer involved in the site and gave an alternate e-mail address that the researcher then used. No usable e-mail was found for three surviving sites, reducing the valid sample size to 278.
An e-mail survey (see Appendix 2) was sent two days after the pre-notification. A reminder message was sent one week after the initial message. Of the 278 functioning sites that had usable e-mail addresses, 110 responded resulting in a response rate of 39.6%. This survey, which was primarily quantitative in design, duplicated many of the questions that had been asked on the 1997 survey (see Appendix 1). The primary purpose of this follow-up survey was to determine how sites had changed over the three-year period.
Another, more qualitative, e-mail survey was also conducted in 2000. This survey (see Appendix 3) contained open-ended questions that sought more depth of information about why sites had failed or continued to survive and was sent to both survivors and non-survivors. Of the 114 presumably non-functioning sites that received this survey, seven responded by indicating that their sites were still functioning but had moved to new addresses. This raised the total number of surviving sites from 281 to 288.E-mail could not be delivered to managers of 67 (58.8%) of the 114 non-functioning sites.Six managers of non-surviving sites responded to this survey.
The 2000 qualitative survey was also sent to 88 individuals at functioning sites that had been identified as particularly interested in issues of longevity because they requested a summary of findings from the quantitative survey sent in 2000. A total of 30 responses were received from individuals whose sites did survive from 1997-2000. Table 1 summarizes sample size and response rates for each of the surveys conducted in 1997 and 2000.
Sample Size and Response Rate
Original Sample Size
Size after removal of duplicates, non-functioning sites, bad-email
Total Responses (response rate)
Note: Respondents to the qualitative survey provided new URLs for seven surviving sites bringing the total number of surviving sites to 288 (survival rate of 72.9%)
Finally, all of the sites that were still operating in 2000 were downloaded for content analysis. However, because the original sites found in 1997 had not been downloaded, direct comparisons between content were somewhat limited. Therefore, this article focuses primarily on findings from the survey data.
The question of why some sites survive while others do not is addressed first on the basis of quantitative data from the two-stage study. Of the 395 sites in the 1997 study, 107 (27.1%) were no longer functioning in 2000. Thus, the overall survival rate was 72.9%. Of the 288 sites that were functioning, 231 were still using the same online address and 57 had moved to a new URL. Most (87.7%) of those that had moved provided an online pointer to the new URL. But to explore the key research question of why some sites survive while others do not, more information is needed both about survivors and non-survivors. Quantitative data from the two-stage study is first examined to seek insight into survival rate in terms of the four topics identified in the literature: resources, Web site features and technology, organizational issues, and external issues. Following this quantitative analysis, results from the open-ended questions are considered for further qualitative insight into key issues.
Three key resource issues were considered: cost, personnel, and time. In pre-tests of the first study, the researcher found respondents unwilling to provide exact cost for maintaining content at the site. Therefore, respondents were provided with cost categories. Table 2 summarizes survival in 2000 for each of the cost categories. Sites that invested the least were least likely to survive. As investment in maintaining the content of the site went up, likelihood of survival also went up – to a point. Sites that spent over $50,000 on content creation and maintenance (the highest cost category) had the second-highest failure rate. This may suggest that these high-cost sites burned through investment money before they proved valuable to organizations.
Survival in 2000 and Costs for Maintaining Content in 1997
Numbers in parentheses are percentages for the cell within the row.
X 2 (4, N = 364) = 15.52, p < .01
In the 2000 study, respondents were not provided with cost categories but were asked to estimate total cost for creating and maintaining content. Pre-testing of this survey indicated that these “known responders” might be more likely to provide specific dollar amounts. The average annual cost of creating and maintaining content reported by the respondents to the 2000 study was $8,760 and the total number of sites spending less than $1,000 dropped from 46.5% of the sample to 41.8%. This tends to suggest that survivors have begun to invest more in their sites. But for most of the sites in this sample, investments were still modest. The highest cost reported in the second survey was $200,000 annually.
To further understand how these sites are funded and changes in funding sources over time, respondents to both the 1997 and 2000 surveys were asked to indicate what percentage of the cost of creating content comes from one or more of the five types of funding sources. Table 3 summarizes average percentages of funding reported from each source in 1997 and 2000. Two significant changes were found. In 2000, organizations were more likely to identify the maintenance of content at their sites as a cost of doing business and less likely to report funding from advertising revenue. These Web sites may be “commercial” in the sense that they provide a venue for organizational voices to be heard, but the influence of traditional advertising does not seem to be strong in this sample of health-related sites.
Average Percentage of Revenue from Specified Funding Sources
Paired Sample T-test
(N = 103; df = 102)
Cost of Doing Business
p < .05
p < .01
Subscriptions/ Pay-per Use Fees
p > .05
p > .05
In both 1997 and 2000, many respondents indicated that content was funded completely from internal sources as a cost of doing business (25.6% in 1997, 35.7% in 2000) or from volunteer efforts (33.7% in 1997, 32.1% in 2000). These two funding sources dominate in both samples. Four sites indicated that they have begun to get revenue from e-commerce. At this handful of sites that generate e-commerce revenue, percentage of funding from transactions ranges from 12.5% to 100% with an average of 48%.
The second resource issue examined in the quantitative data was personnel. Sites that survived had an average of 3.03 people working on the site in 1997 as compared with an average of 1.93 people working on sites that did not survive. This difference was significant t (392, N = 394) = 2.09, p < .05. Overall, the average number of people working on surviving sites increased slightly to 3.3 in 2000.
In the 1997 survey, respondents were asked to indicate not only how many people were responsible for creating content at their Web sites, but also what types of job responsibilities those individuals held. Seven types of jobs were identified: health professionals, communicators, marketers, technology experts, graphic designers, administrators, and volunteers. In cross tabulation of these job types with survival rates, only one significant relationship was found. Sites that employed one or more technology experts were more likely to survive (81.3%) than were those that did not have such technical expertise (66.4%). This relationship was statistically significant ?2 (1, N = 395) = 10.83, p < .001.
Time as a resource issue was examined from the perspective of how frequently sites were updated. More frequent updates would tend to suggest that site managers devote more time to the site. Both the 1997 and 2000 surveys asked about how frequently sites were updated. One interesting fact is that no response was received in 2000 from any of the five sites that had indicated in 1997 that they never updated the site. Two of those sites were no longer functioning and e-mail to another of the sites bounced. This suggests a high mortality rate among sites that are never updated.Table 4 summarizes frequency of updating in 1997 and survival in 2000. Sites that updated at least daily were most likely to survive. Sites that updated quarterly or yearly were least likely to survive.
Survival in 2000 and Frequency of Updating Content in 1997
At least once a year, but less than quarterly
At least quarterly, but less than once a month
At least monthly, but less than weekly
At least weekly, but less than daily
At least once daily
Numbers in parentheses are percentages for the cell within the row.
X 2 (4, N = 379) = 12.11, p < .05
Features and Technology
The types of features included in a Web site can often reflect the technological expertise of site creators. In the 1997 study, content analysis included coding for features that might enhance the interactivity of the site (feedback forms, chat rooms, newsgroups, bulletin boards, online service/support) as well as some other features that might indicate that the site creator had some sophistication in technological development of the site (search engines, menus for navigation, systems for in-site and out-of-site linking, and a table of contents or site map). Among the three trained coders who examined the sites, average intercoder reliability for these items was .87. Differences among coders most often occurred because one coder failed to identify a feature. Discrepancies in coding were easily resolved.
Chi square tests were run in 2000 to determine relationships between these features and site survival. Significant differences were found between survivors and non-survivors for only one of these features: the presence of a table of contents and/or site map χ2 (1, N = 384) = 3.81, p < .05. Sites that had this relatively simple technology tool in 1997 were more likely to survive in 2000 (76.3%) than those that did not (66.3%). None of the technologically more advanced features (e.g. chat rooms, search engines) had a significant relationship to survival.
Are some kinds of organizations more likely to build sustainable sites than others? This question can be examined based on self-defined organizational type. In the 1997 survey, site managers were asked to indicate which of the following best described their organization: educational organization, government agency, non-profit organization, for-profit company, individually operated site, and other. Only 15 sites were identified as government agency sites. These sites shared many characteristics with education sites (many of which represented government-funded colleges and universities). Thus, to reduce the number of cells with small expected counts, government and education sites were combined for analysis. The few (six) sites that were identified as some “other” organizational type were also eliminated from analysis. In addition to examining survival, location of surviving sites was also examined. As revealed in the literature, sites that are at the same domain name after three years, might be more stable than those that have moved. Table 5 summarizes relationships between Web site survival and self-defined organizational type.
Survival and Self-Defined Organizational Type
(Overall survival 82.7%)
(Overall survival 75.5%)
Overall survival 67.5%)
(Overall Survival 64.4%)
Numbers in parentheses are percentages for the cell within the row.
X 2 (6, N = 389) = 14.4, p < .05
Sites from the 1997 study that were non-functioning in 2000 were most likely to be operated by individuals. By contrast, sites operated by educational and government institutions in 1997 were most likely to be still functioning in 2000. However, education and government sites were the most likely to have moved to a new URL. Several managers of these government and education sites indicated that their sites had been moved because of changes in internal computer system structures that did not take into account the need for stable URL addresses. The sites least likely to move to a new location were those operated by for-profit companies.
The primary external issue addressed in the quantitative survey was number of site visitors. The 1997 survey asked respondents to indicate if they tracked usage at their sites. One might expect sites that monitor their audiences to be more likely to survive than are those that pay less attention to how their site is used. About two thirds (67.3%) of the original sites did track usage. However, there was no significant relationship between such tracking and site survival X 2 (1, N = 391) = .922, p > .05.
Among sites that did track their audience, audience size was a predictor of survival. A t test revealed that sites that survived had an average of 3,650 visitors per week in 1997 while sites that did not survive had an average of only 783 visitors per week in 1997. This difference was significant t (249, N = 251) = 1.44, p <.05. Among survivors, the average weekly audience size grew dramatically to 12,100 in 2000.
While the quantitative survey questions and content analysis data enabled quantifiable measures of some potential reasons for site survival or failure, these measures may overlook issues of importance to site managers. Therefore, managers of both surviving and non-surviving sites were asked open-ended questions about the role of resources, Web site features and technology, organizational issues, and external issues in the fate of their sites. While responses generally supported the quantitative findings, they also provided some additional insights. Respondents were guaranteed anonymity. Thus, when quotations are used, the identity of the respondent is not provided.
Resource issues were identified by managers of surviving sites as being the most important reasons for their survival. Seventeen of the 30 site managers who responded to the qualitative survey reported that resources played a positive role in their survival. One respondent wrote:
Resources have become significant. In fact, we believe that our Web site is vitally important to our progress and have increased the amount of time, money and staff we allocate to this form of communication…. Our original Web site was created by a volunteer and did not cost us anything other than the costs of an ISP, on-line credit card capability, etc. But it has been successful enough to warrant a significant allocation of funds, staff and time.
Other respondents also wrote about the importance of funding to keep their sites going. However, time was also a critical factor. For example, one wrote: “Without the time that my secretary and I have been able to spend updating the site, it would have quickly become irrelevant and we would have given up on it. We have made it a priority to update and add to its content.” Among those who felt that resource issues were not important many indicated that, over time, resources had been committed to the Web site and few, if any, new resources were needed.
Among non-survivors, only one of the six respondents indicated that resources were an important reason for the demise of their sites. The respondent wrote: “About a year ago I decided to shut it down. I was out of time.” However, for most managers of failed sites, resources were not a key issue. For example, one wrote: “The site was easily maintained by myself and there were not specific issues such as resources, money, time, etc. that led to its closure.”
Features and Technology
Technology issues were identified by managers of surviving sites as being the second-most important reason for survival. Fourteen of 30 site managers who responded to the qualitative survey reported that technology played a positive role in their survival. One wrote: “Our site depends on the reliability of our hardware, software, and programming. The presentation of our site is only as good as the worst of these.” Several others noted that staying current with technology was a challenge, but by doing so they were able to stay competitive. Among respondents who indicated that technology was NOT very important to survival, many noted that only a small investment in technology was needed to maintain the core features of their sites. For example one wrote: “We don’t get into really fancy stuff on our site – most of it is pretty basic HTML code – so we don’t need to invest a lot in hardware or software.”
No managers of non-surviving sites reported technology as being a reason for their demise. None elaborated on this point. They all simply wrote that technology was not important. One went so far as to say that technological issues were “NOT important at all.”
The quantitative study examined organizational issues primarily from the perspective of organization type. However, responses to open-ended questions provided some additional insights into organizational issues. Twelve of the 30 respondents indicated that organizational factors were important. For example, one respondent wrote: “We look at our Web site as ‘changing the way we do business’ since we will offer member services online.” Another wrote: “Management support is key. Without my boss’s support and guidance, the site would not have been a high enough priority to keep it going.” Others reported on specific organizational changes that had been made to find an internal “home” for the Web site and make sure that it was supported.
Among surviving sites that indicated organizational issues were NOT important, many wrote that the site was run by a small organization in which coordination was easy. One respondent commented:
Organizational issues are not important to us. We are a small company, but everyone in the company realizes the importance of the Web site. I don’t think company size really matters as long as the decision-makers are committed to the company’s Web strategy, and in 2000 I would think it would be difficult for any company not to realize how powerful a sales tool the Web is.
It is interesting that the respondent began by pointing out that organizational issues weren’t important, but then went on to explain that organizational commitment to the Web is a critical factor for survival.
Among sites that did not survive, only one of the six respondents to the qualitative survey indicated that organizational issues were a factor in the demise of the site. The respondent wrote: “There was a change of management and the new manager was not interested in supporting the site. If a brand manager is interested then fine. If not, then funding is placed elsewhere.”
Twelve of the 30 site managers who responded to the qualitative survey reported that external issues played a positive role in their survival. Several focused on the issue of attracting visitors. One wrote: “Our Web site is relatively busy with visitors from around the world…. We depend on generating contacts to increase our membership and we feel the Internet is doing that for us.” Others noted that while their audience was small it was targeted and the Web site was a good vehicle for reaching the audience. Some respondents also commented on other external issue such as competition. One wrote: “I think that having established my site in mid-1995, way before all the mongo health sites, has done a lot for keeping me high in the search engines.”
Among respondents who indicated external factors were NOT important to their survival, many noted that they held a specialty or niche position on the Web that neither depended on large numbers of visitors nor faced serious competition. One wrote that external factors were “not really an issue; however, the number of visitors to our site is high because we provide up-to-date content on issues important to our members and non-member readers.” Another wrote: “I have no competition so to speak. Now there have been a few copycats who set up their sites after visiting mine.” One emphasized the low relevance of external issues by writing: “It [the Web site] would continue to operate regardless of external factors.”
Despite the low importance that survivors placed on external factors, non-survivors were likely to see external factors as being the primary reason for their failure. Four of the six respondents attributed their sites’ demise primarily to external factors. One wrote that the site failed because members of the target audience (doctors working in a specialty area) were unwilling to support the site – they read the newsletter but would not contribute to site upkeep. Another reported that a site was developed as a “demonstration” in 1994. It was never intended to be maintained. But, it became popular and taking it down after the demonstration period proved difficult. Another wrote that the site ceased to exist when the company that was responsible for its creation went out of business and another reported that the site was eliminated when the responsibility for the association that it represented was moved to another city.
Most Important Reasons for Success or Failure
Respondents were asked to indicate what they believed to be the most important reason for their site’s survival or demise. Respondents were allowed to give multiple responses. Some responses were unique to a specific situation, but three types of responses seemed to dominate replies provided by those whose sites had survived: public service, promotion and marketing, and resources.
Thirteen of the 30 site managers whose sites had survived from 1997-2000 indicated that one of the primary reasons their site had continued to exist was because it was providing some kind of public service. One respondent wrote: “It is perhaps the best means we have for communicating the nature and content of our research to the academic community – thus it is an essential component of our laboratory.” Another wrote: “Because I’m bipolar and the best way to heal is with other bipolars who find a home at my site.” Another wrote. “The main reason for my site still operating is because of the poor quality of treatment, and poor to no communication between patients and their doctors, and my willingness to offer insight into patients’ problems.” The manager of a site focused on panic disorders echoed this sentiment:
Because people from all walks of life are looking for answers. I give them hope without any charge. I hope they find the answers or at the very least a friend or two to help them on their way. I’ve been told hundreds of times that people like my site because I don’t have anything to sell or expect money for my experiences. I’m just a normal person who has panic attacks. No mumbo jumbo words. Just what’s from the heart.
The second-most frequently mentioned reason for survival was issues related to promoting and/or marketing the organization and its good or services. Promotion and marketing issues were identified as important by 10 managers of surviving sites. One respondent wrote: “It is an important marketing tool for the laboratory and for the department.” Other similar comments included: “To fulfill the department’s responsibility to present itself to the outside community and to promote its educational facilities.” The manager of a more commercial site wrote: “It’s free. Any orders it generates are gravy.”
Finally, six of the managers of surviving sites identified resource issues as a key reason for their continued survival. One respondent wrote: “Availability of resources, especially my time, would be the most important reason.” Others also wrote about the importance of time commitment: “Because I spend the time maintaining and updating it to ensure that it does continue to operate.” But others focused more on money. For example, one wrote: “Because we are funded entirely by research monies and the site generates no income, funding for the site is minimal.” Another wrote: “We have a server that hosts our domain at no charge…. There are no costs to keep the information available to the public.
Among sites that did not survive, managers uniformly identified a single reason for their site’s failure. As noted earlier, external factors were most frequently mentioned followed by resources and organizational issues. None of these respondents identified technology issues as being the most important reason for failure.
Conclusions and Discussion
The longitudinal analysis of health-related Web sites reported in this article provides insights into the kinds of sites that were most likely to survive from 1997 to 2000. Both the qualitative and quantitative data suggest that survivors invest money, personnel, and time in their sites. In particular, survivors are likely to have invested in people with technological expertise. There is also evidence that many of the organizations and individuals who are responsible for these sites have shifted their expectations of the sites. They are less likely to seek outside funding and more likely to support the site with organizational resources in 2000 than they were in 1997. This emphasis on sustained internal support is further evidenced by the fact that the most expensive sites identified in 1997 had a high failure rates. Another key characteristic of surviving sites is that they are more likely than their failed counterparts to update content regularly. But among the small sample of failed sites, resources were seldom listed as a reason for failure. As one respondent noted, Web sites can actually be maintained with relatively little investment.
The data tend to support the idea that simple sites can survive as well as their more technologically advanced (and often more resource-intensive) counterparts. Managers of surviving sites tended to see technology as a relatively important factor while non-survivors placed none of the blame for their demise on technology. Only one significant relationship was found between a technological feature and survival. That feature, a table of contents/site map, requires little technical expertise to create. Future studies should do more to trace the technological development of sites to determine if and how survivors are advancing technologically.
Organizational commitment was a key factor that emerged from the qualitative surveys. Both survivors and non-survivors wrote about the importance of management support of the Web site. Sites run by individuals were the most likely to fail. This suggests that these freelancers on the digital frontier may not have had enough support, technology, or resources to survive as the Web began to mature. By contrast, sites connected with education and government organizations were most likely to survive. The Internet began in government and education. There is a strong organizational commitment to the technology in those public and research organizations. But there also seems to be an organizational tradition of “tinkering” with the technology that leads to instability in URLs. New servers are added, internal departments are moved to new systems; about one fifth of all government and education sites moved during the three year period.
Non-profit organizations also seem to have a strong commitment to Web technology. As several respondents noted, the Web offers an ideal low-cost tool for distributing information. For-profit companies were slightly less likely to survive, but those that did were most likely to be found at the original URL. These profit-oriented companies probably bought their own domain names and have been able to retain them even as technology changed and they moved from one server to another.
Sites that had more visitors in 1997 were more likely than low-traffic sites to survive. And among surviving sites, site traffic has grown dramatically. However, despite these positive external forces, survivors were not likely to identify external factors (such as site visitors or competition) as key reasons for the survival. Survivors were more likely to see internal resource issues (cost, people, and time) as keys to survival. They also indicated that a primary reason for their success was that their sites provided public service and/or marketing information that was in demand.
However, external factors were most often cited by non-survivors as causes of failure. This may be tied to attribution theory in which individuals are more likely to take credit for positive outcomes and shift blame for negative ones. Or, it may be that factors beyond an individual’s control are indeed the most likely to shut down a site. For example, if a site were dependent on support from external sources, it could be difficult to survive if those sources failed to fund the site.
In summary, this study suggest some early trends that may help managers of health-related Web sites identify survival skills: make an organizational commitment to support the site with money, people, and time; invest in technological expertise; recognize the importance of organizational support (rather than trying to run a free-lance site as an individual or seek outside funding from sources such as advertising); and try to limit the number of external factors that could impact on the site.
Suggestions for Researchers
This study provides an example of both how to do longitudinal studies of Web sites and of what can be learned from studies that examine Web sites over time. One key point for researchers is the importance of using more than one method of finding Web sites in follow-up studies. In this study, simply looking for the original URL would have suggested a survival rate of only 58.5%. Even following links from old sites would have under-reported survival rate. By also sending an e-mail message to managers of seemingly defunct sites, overall survival rate was increased by an additional 2%. Future studies may consider going a step further and using search engines to try to locate sites that may have moved. However, site managers should be cautioned. Most users will not take measures like these to try to find an old site. Buying a domain name, as for-profit companies often do, seems to be one way to ensure that site visitors will be able to make return visits even when technology changes and servers are moved.
This study also provides one example of how to draw a random sample of Web sites. While every effort was made to ensure random assignment of Web sites, the method reported here is not perfect. A key concern is the fact that Yahoo! does not include all health-related sites in its health categories. Using Yahoo! as a source may also bias results because the site exercises quality controls that may exclude some “fringe” sites. Also, the English version of Yahoo!, which was used for this study, tends to be dominated by USA-based and English-language sites. Other methods for identifying a universe from which to sample need to be explored. Furthermore, it should be recognized that a sample of health-related sites that was representative of that category in 1997 is not representative three years later. A second sample, drawn at a later time, could reveal additional insights into changes in the health category of Web sites.
The response rate to the initial 1997 e-mail survey was 47.5%. This response rate is in line with response rates from other survey methods. But in subsequent surveys, response rates dropped. Only about 40% of survivors responded to the follow-up quantitative survey; response rate to the qualitative survey among managers of non-surviving sites was only 12.8%. In some ways, these numbers are actually encouraging. Being able to find and get responses from almost 13% of individuals who are no longer “in business” might be difficult if one were studying traditional communication tools. Still, the overall effect of these diminishing returns is that the N for analysis becomes increasingly smaller, making it more difficult for the researcher to have confidence in the generalizability of findings. Future longitudinal studies should consider using larger numbers in the initial sample to help reduce these problems.
A major challenge for future longitudinal research of Web sites will be to assess changes in the rich multi-media environments of cyberspace. Tools are now available for easily downloading Web sites for future analysis. But how should that future analysis be conducted? Should computerized tools be prepared to automate the comparison of sites as they existed in two or more points of time? Or should human coders be used to look for nuances of similarity and difference that a machine-based comparison may not detect? Or, is a combined human/automated process required?
In summary, Web site permanence was relatively strong. Koehler’s 1999 study suggested that Web site mortality rates might be as high as 25% per year. But, at least among this sample, it seems that sites surviving past the one-year point might go on to experience a longer life – of three years or even more for Web sites that were already well established in 1997. Koehler and Wall’s (2000) ongoing longitudinal study supports this finding. But within this permanence is a great deal of change. The challenge for researchers is to develop meaningful tools and techniques for measuring survival and change. The challenge for site managers is to manage change in such a way as to adapt and survive in the evolving environment of computer-mediated communication.
AppendicesAppendix 1 - 1997 Survey
(1) When did your site (URL for site inserted here) begin operation (month and year)? _____________
(2) Which of the following BEST describes the organization responsible for your Internet site:
_____ for-profit company
_____ not-for-profit organization
_____ government organization
_____ educational institution
_____ other (please describe)
(3) How many people are responsible for creating, updating, or in any other ways maintaining the content of your site? _____
(4) Briefly describe the people who create, update, or in other ways maintain the content of your site. Do NOT provide names. DO provide job titles and/or a general description (e.g., reporters, marketing communication staff, volunteers, dedicated individuals, etc.)
(5) Do you measure the number of visitors to your site? _____ If not, skip to question 8.
(6) Briefly describe the method you use for measuring the number of visitors to your site:
(7) Approximately how many different persons access your site during an average week. _____
(8) About how often is the content in your site updated? Place an X in front of the option below that best describes your updating procedures:
_____ from 1 to 3 times each year
_____ from 4 to 11 times each year
_____ from 1 to 3 times each month
_____ from 1 to 6 times each week
_____ other (describe)
(9) Briefly indicate the purpose of your site. If you have a mission statement, you may include it here.
Questions 10 through 16 ask for your opinions about cyberspace. Please rate how strongly you agree with each statement. Use a scale of 1 = strongly agree; 2 = somewhat agree; 3 = neither agree nor disagree; 4 = somewhat disagree, 5 = strongly disagree.
(10) _____ The concept of authorship should not exist in cyberspace.
(11) _____ Information that can be found on the Internet should be free and available to everyone.
(12) _____ Copyright laws should be enforced in cyberspace.
(13) _____ Good content creators should receive some kind of financial reward for the information that they post in cyberspace.
(14) _____ More security is needed to prevent unauthorized access to information in cyberspace
(15) _____ One should not have to pay for information one can find for free in cyberspace.
(16) _____ The primary role of information at our Internet site is to facilitate transactions (e.g., sales, donations, signing up new members, etc.)
Questions 17 through 20 ask about links to and from your site.
(17) Do you ever pay to have links to your Internet site from other Internet sites (an example would be paying for "banner ads" in search engine sites)? _____ If not, skip to question 19.
(18) On average, how much have you paid per month for each link to your Internet site from other Internet sites?
Does your Internet site include links to (place an X in front of all that apply):
(19) _____ Other pages in your own site
(20) _____ Sites created by other individuals and/or organizations
Questions 21 through 34 ask about how the CONTENT of your site is funded. In answering these questions, DO include the costs for purchasing copyrighted information, paying freelance writers, paying salaries of people who are responsible for updating and managing the content of the site and so on. Do NOT include the cost of buying and maintaining computer hardware and software.
Listed below are some of the ways that the CONTENT of Internet sites is funded. Some sites are funded in more than one way. Please indicate what percent of your funding comes from each source. If you get no funding from a source, enter 0. The numbers in lines 21 through 27 should equal 100 percent.
(21) _____ advertising
(22) _____ pay-per use fees
(23) _____ subscription fees for accessing the site
(24) _____ cost of doing business for the sponsoring organization
(25) _____ sponsorship fees/donations/grants (from outside organizations/individuals)
(26) _____ volunteer efforts (content created by people who do not get paid for their time and efforts)
(27) _____ other (describe)
If you entered 0 on question 25, skip to question 34. If you entered a number greater than 0, indicate which of the following types of sponsors/contributors fund your site. Mark with an X all that apply.
(28) _____ for-profit companies
(29) _____ not-for-profit organizations
(30) _____ government organizations
(31) _____ educational institutions
(32) _____ individuals
(33) _____ other (please describe)
(34) Approximately how much is spent annually on the CONTENT of your Internet site. If the content of your site is created, in part or in full, by volunteer and/or staff efforts, convert the time volunteers and/or staff members spend in creating content for your site into an approximate dollar amount.
_____ Less than $1,000
_____ More than $250,000
(35) Do you want a summary of my findings? _____
Appendix 2 - 2000 Quantitative Survey
(1) Which of the following BEST describes the organization responsible for your Internet site:
_____ for-profit company
_____ not-for-profit company
_____ government organization
_____ educational institution
_____ other (please describe)
(2) How many people are responsible for creating, updating, or in any other ways maintaining the content of your site? __________
(3) Briefly, describe the people who create, update, or in other ways maintain the content of your site. Focus not on individual names but rather on job titles and/or a general description.
(4) Do you measure the number of visitors to your site? _____ If not, skip to question 7.
(5) Briefly describe the method you use for measuring the number of visitors to your site.
(6) Approximately how many different persons access your site during an average week?
(7) About how often is the content in your site updated? Place an X in front of the option below that best describes your updating procedures:
_____ from 1 to 3 times each year
_____ from 4 to 11 times each year
_____ from 1 to 3 times each month
_____ from 1 to 6 times each week
_____ other (describe)
(8) Listed below are some of the ways that the CONTENT of Internet sites is funded. In answering this question, DO include the costs for purchasing copyrighted information, paying freelance writers, paying salaries of people who are responsible for updating and managing the content of the site and so on. Do NOT include the cost of buying and maintaining computer hardware and software.
Some sites are funded in more than one way. Please indicate what percent of your funding comes from each source. If you get no funding from a source, enter 0. The numbers should total 100 percent.
_____ cost of doing business for the organization that maintains the site
_____ advertising and/or sponsorships from outside organizations
_____ subscription or other pay-per-use fees
_____ volunteers (content created by people who do not get paid for their time and efforts)
(9) Approximately how many US dollars are spent annually on the CONTENT of your Web site?
For questions 10 through 16 use the following scale to rate the site that you manage: 1=strongly disagree, 2=disagree, 3=somewhat disagree, 4=neither agree nor disagree 5=somewhat agree, 6=agree, 7=strongly agree.
(10) _____ This site is designed primarily to persuade rather than inform.
(11) _____ Visitors to this site have a great deal of control over their visiting experience.
(12) _____ This site facilitates two-way communication.
(13) _____ Visitors to this site get a sense that they are in a “place” in cyberspace.
(14) _____ Visitors to this site need to take an active role to fully experience the site.
(15) _____ This site is interactive.
(16) _____ This site allows visitors to communicate at times that are most convenient for them.
(17) Briefly describe the greatest opportunities for your site in the past three years.
(18) Briefly describe the greatest challenges for your site in the past three years.
(19) Do you want a summary of the findings of this study? _____
Appendix 3 - 2000 Qualitative Survey
(1) Is your site still operating? If so, skip to question 8. If not, go to question 2.
(2) How important were resource issues (e.g. time, money, people, etc.) to the fact that your site is no longer operating? Briefly explain specific resource issues if relevant.
(3) How important were technology issues (e.g. hardware, software, programming, etc.) to the fact that your site is no longer operating? Briefly explain specific technology issues if relevant.
(4) How important were organizational issues (e.g. management support, restructuring of departments, etc.) to the fact that your site is no longer operating. Briefly explain specific organizational issues if relevant.
(5) How important were external issues (e.g. too many competitors, not enough visitors, etc.) to the fact that your site is no longer operating. Briefly explain specific external factors if relevant.
(6) What is the main reason that your site is no longer operating?
(7) Would you like a summary of my findings? This is the last question for you if your site is not operating. After you answer this question, you may skip the remaining questions. Thank you for your participation.
(8) Please provide the correct URL for the Web site that was previously located at the address shown at the start of this message.
(9) How important are resource issues (e.g. time, money, people, etc.) to the fact that your site is still operating? Briefly explain specific resource issues if relevant.
(10) How important are technology issues (e.g. hardware, software, programming, etc.) to the fact that your site is still operating? Briefly explain specific technology issues if relevant.
(11) How important are organizational issues (e.g. management support, structure of departments, etc.) to the fact that your site is still operating. Briefly explain specific organizational issues if relevant.
(12) How important are external issues (e.g. competition, number of visitors, etc.) to the fact that your site is still operating. Briefly explain specific external factors if relevant.
(13) What is the main reason that your site is still operating?
(14) Would you like a summary of my findings? This is the last question. Thank you for your participation.
Angwin, J. (2000, June 7). A true-crime site finds web survival can be murder -- APBnews.Com lays off staff and runs out of money. The Wall Street Journal Europe, pp. 28.
Barnouw, E. (1966). A tower in babel (Vol. 1). New York: Oxford University Press.
Christensen, H. (2000). Sites for depression on the web: A comparison of consumer, professional and commercial sites. Australian and New Zealand Journal of Public Health, 24 (4), 396-400.
Crum, M. R., Johnson, D. A., & Allen, B. J. (1997). A longitudinal assessment of motor carrier-shipper relationship trends, 1990 vs. 1996. Transportation Journal of Advertising, 37 (1), 15-28.
Cunningham, R. (2000). Old before its time: HIPPA and e-health policy. Health Affairs, 19 (6), 231-238.
d'Ambra, J., & Rice, R. E. (1994). Multimedia approaches for the study of computer-mediated communication, equivocality, and media selection. IEEE Transactions on Professional Communication, 37 (4), 231-239.
Dick, R. S., & Gabler, J. M. (1995, February). Still searching for the holy grail. Health Management Technology, 30-35, 80.
Domanski, B. (2000). E-commerce, e-business, e-everything! Enterprise Systems Journal, 15 (10), 44-47.
Dorman, S. M. (2000). Finding online funding for school health program. The Journal of School Health, 70 (2), 72-73.
Evans, M. (2000, May 25). Boo.com's death offers clues to on-line success. The Globe and Mail, pp. 5.
Feldman, S. E. (1997, October). It was here a minute ago: Archiving the net. SEARCHER: The Magazine for Database Professionals, 52-64.
Fisher, L. M. (1996, June 24). Health online: Doctor is in, and his disk is full. The New York Times, pp. C1.
Germain, C. A. (2000). URLs: Uniform resource locators or unreliable resource locators. College & Research Libraries, 61 (4), 359-365.
Grandinetti, D. A. (2000). Help patients surf the net safely. RN, 63 (8), 51-54.
Gray, S. M., & Brookmeyer, R. (2000). Multidimensional longitudinal data: Estimating a treatment effect from continuous, discrete, or time-to-event response variables. Journal of The American Statistical Association, 95 (450), 396-406.
Ha, L., & James, L. (1998). Interactivity re-examined: A baseline analysis of early business web sites. Journal of Broadcasting & Electronic Media, 42 (4), 457-474.
Harris, K. D., & Campbell, J. D. (2000). Internet by proxy: How rural physicians use the Internet. Social Science Computer Review, 18 (4), 502-507.
Hiltz, S. R., & Johnson, K. (1989). Measuring acceptance of computer-mediated communication systems. Journal of the American Society for Information Science, 40 (6), 386-398.
Hoffman, T. (1998, August 31). Financial web sites provide little real value. Computerworld, 32, 33-37.
Internet Software Consortium. (2000). Internet domain survey. Available: http://isc.org/ds/WWW-200001/report.html [2000, January].
Jones, C. (1996). The great chain of buying: Medical advertisement, the bourgeois public sphere, and the origins of the French revolution. American Historical Review, 101 (1), 13-40.
Kabanoff, B. (1996). Computers can read as well as count: How computer-aided text analysis can benefit organizational research. Journal of Organizational Behavior, 3, 1-21.
Katz, J. E., & Tassone, A. R. (1990). Public opinion trends: Privacy and information technology. Public Opinion Quarterly, 54, 125-143.
Koehler, W. C. (1999). An analysis of web pages and web site constancy and permanence. Journal of the American Society of Information Science, 50 (2), 162-180.
Koehler, W. C. (2000). The bibliographic management of Web documents in digitial and hybrid libraries. In J. Borbinha & T. Baker (Eds.), Research and Advanced Technology for Digital Libraries. 4th Europeran Conference, ECDL 2000, Lisbon, Portugal (1923 ed., pp. 407-412). Berlin: Springer.
Koehler, W. C., & Wall, J. (2000). Web page constancy and permanence: "Stability" after a while. Paper presented at the Association of Internet Researchers, Lawrence, KS. Available online at: http://www.ou.edu/cas/slis/ethics/Kansas/stability.pdf
Leaders: Dotty about dot.Commerce?(2000, February 26). The Economist, 354, 24-26.
Massey, B. L., & Levy, M. R. (1999). Interactivity, online journalism, and English-language web newspapers in Asia. Journalism & Mass Communication Quarterly, 76 (1), 138-151.
McMillan, S. J. (2000). Interactivity is in the eye of the beholder: Function, perception, involvement, and attitude toward the web site. In M. A. Shaver (Ed.), Proceedings of the 2000 conference of the American academy of advertising (pp. 71-78). East Lansing, MI: Michigan State University.
McMillan, S. J., Sheehan, K. B., Heinemann, B., & Frazer, C. (2000, August). What the real world really wants: An analysis of advertising employment ads. Paper presented at the Association of Education in Journalism and Mass Communication Conference, Phoenix, AZ.
McMurdo, G., & Meadows, A. J. (1996). Acceptance and use of computer-mediated communication by information students. Journal of Information Science, 22 (5), 335-348.
Moschella, D. (1999, September 6). Why the web's sleeping giants keep snoozing. Computerworld, 33, 33-34.
North, A. B., & Worth, W. E. (1998). Trends in advertised SCANS competencies: Technology, interpersonal, and basic communication job skills, 1992-1996. Journal of Employment Counseling, 35, 195-205.
Ozok, A. A., & Salvendy, G. (2000). Measuring consistency of web page design and its effect on performance and satisfaction. Ergonomics, 43 (4), 443-460.
Pingree, S., Hawkins, R. P., Gustafson, D. H., Boberg, E., Bricker, E., Wise, M., Berhe, H., & Hsu, E. (1996). Will the disadvantaged tide the information highway? Hopeful answers from a computer-based health crisis system. Journal of Broadcasting & Electronic Media, 40 (3), 331-353.
Reis, J., & Tymchyshyn, P. (1992). A longitudinal evaluation of computer-assisted instruction on contraception for college students. Adolescence, 27 (108), 803-811.
University of Waterloo, Scholarly Sciences Project (1999). URL stability index for the scholarly science project. Available: http://www.lib.uwaterloo.ca/society/URL_stability_index.html [2000, August 21].
Walker, D. M. (2000, February 29). B2B systems 'must offer more'. The Age, pp. 5.
Woodard, C. (2000, March). Web survival guidelines for small magazines. Folio: The Magazine for Magazine Management, 29, 81-86.
Zakon, R. H. (1999). Hobbe's Internet timeline v 4.2. Available: http://info.isoc.org/guest/zakon/internet/history/hit.html [2000, January 27].
About the AuthorSally J. McMillan is an assistant professor in the Advertising Department at the University of Tennessee-Knoxville. Her research focuses on exploring the concept of interactivity, online research methods, definitions and history of new media, and impacts of communication technology on organizations and society. Her research has been published in journals such as Journalism and Mass Communication Quarterly, Journal of Computer Mediated Communication, and New Media and Society. Prior to earning her Ph.D. at the University of Oregon, McMillan spent 15 years working in industry with almost half of those years in executive-level marketing and management roles for computer technology firms.
Address: Department of Advertising, 476 Communications Building, Knoxville, TN 37996-0343 Phone: 865-974-5097 Fax: 865-974-2826.
©Copyright 2001 Journal of Computer-Mediated Communication