JCMC 6 (2) JANUARY 2001
Message Board
Collab-U  CMC Play  E-Commerce Symposium Net Law  InfoSpaces Usenet
  NetStudy VEs VOs O-Journ HigherEd Conversation Cyberspace Web Commerce VisualCMC


E-mail Survey Response Rates: A Review

Kim Sheehan

School of Journalism and Communication
University of Oregon

Abstract

Electronic mail (e-mail) has been used to distribute surveys and collect data from online users for almost fifteen years. However, some have suggested that the use of e-mail is becoming obsolete. This study analyzes response rates to e-mail surveys undertaken since 1986 and examines five influences to response rates: the year the study was undertaken, the number of questions in the survey, the number of pre-notification contacts, the number of follow-up contacts and survey topic salience. Response rates to e-mail surveys have significantly decreased since 1986. Correlation and regression analyses suggest that year that the survey was undertaken and number of follow-up contacts had the most influence on response rates. A discussion of other influences and future research into this area is provided.

Introduction

In 1986, results from the first e-mail survey were published in Public Opinion Quarterly (Kiesler & Sproull, 1986). Since the publication of these results, we have seen phenomenal growth in the Internet in terms of the number of users, the number of sites online, and access availability worldwide. Fifteen years after Kiesler and Sproull's work, the Journal of Computer Mediated Communication published an article discussing techniques for using e-mail to survey Internet users in the United States (Sheehan & Hoy, 1999). At that time, the use of e-mail for data collection via surveys appeared to hold a promising future. But has this promise been fulfilled?

This study will assess the state of the science of e-mail survey methodology by examining the body of published research using e-mail for data collection. Specifically, we will address the issue of response rates, and whether techniques adopted from postal mail to e-mail have positively affected response rates.

Literature Review

Many studies have touted the promise of e-mail surveys for research (see, for example, Schaefer & Dillman, 1998; Sheehan & Hoy, 1999; Weible & Wallace, 1998). Given that currently about half of Americans have access to the Internet, as well as another 23% of Europeans and 37% of Asians (NUA, 2000b), generating a critical mass of persons to sample and send e-mail to can be fairly easily achieved. For many of these users, e-mail is the most popular application to use online (NUA, 2000b). Researchers have identified numerous benefits of e-mail over postal mail for surveys, particularly in regards to speed and cost efficiency.

E-mail surveys have demonstrated superiority over postal surveys in terms of response speed and cost efficiency. Sheehan and McMillan (1999) estimated that, in studies where both mail and e-mail were used to deliver surveys, mail surveys took 11.8 days to return and e-mail surveys were returned in 7.6 days. E-mail provides an easier and more immediate means of response (Flaherty, et al., 1998). The cost benefits of e-mail have also been highlighted by researchers, with the cost of an e-mail survey estimated to be between 5% and 20% of a paper survey (Sheehan & Hoy, 1999; Weible & Wallace, 1998). The cost savings are derived primarily from the reduction and/or elimination of paper and mailing costs in an e-mail survey. Watt (1999) provided evidence that the costs of e-mail and WWW surveys decrease significantly as the sample sizes increase.

Other benefits to e-mail, while not directly quantifiable, have been identified. Some e-mail software allows for precise tracking of e-mailed surveys. At a minimum, the researcher can know the number of undeliverable e-mail s as well as what time the e-mail survey was opened, replied to and deleted. This can improve sampling procedures (Paolo, et al., 2000). E-mail can also provided heightened response quality. People tend to provide longer open-ended responses to e-mail than to other types of surveys (Paolo, et al., 2000), and responses to e-mail surveys tend to be more candid than responses to mail or phone surveys (Bachman, Elfrink & Venzana, 1999).

Given these positive benefits of e-mail surveys, it is important not to overlook one important element: the response rate. Analysis of studies that have used both mail and e-mail for surveys find that e-mail has not consistently outperformed postal mail: some e-mail surveys did better than mail surveys when it comes to response rates, some did worse, and some were statistically in a dead heat (e.g. Bachman, Elfrink & Vazzana, 1996, 1999; Kiesler & Sproull, 1986; Opperman, 1995; Schaefer & Dillman, 1998).

Moreover, in the past decade, the popular press has reported that response rates are declining for all types and manner of surveys (Bickart & Schmittlein, 1999). Groves, Cialdini and Couper (1992) reported that the US population is being oversurveyed: the growth in the amount of survey research being undertaken has resulted in an increase in the number of requests to individuals to complete surveys. This may lower response rates, since individuals' overall attitudes toward the survey industry may be unfavorable, and the aura of 'uniqueness' to the participation in the survey process diminishes. Low response rates are a concern for researchers, since answers from survey respondents may differ substantially from those of nonrespondents, resulting in a biased estimate of the characteristics of the population (Bean & Roszkowski, 1995).

A related problem with e-mail surveys is the difficulty of obtaining a sample frame in which every subject in the population has a known chance of being selected for participation (Dillman, 2000). Issues such as 'churn' (when users change their Internet Service Provider and their e-mail address) and the holding of multiple e-mail addresses by a single individual have consequences for under-representation (Bradley, 1999). Extremes in variation of user and equipment capability can also effect sampling (Bradley, 1999). Because of this, it has been argued that a survey on the WWW is a more useful methodology, since this type of survey can offer anyone with WWW access the opportunity to respond to the survey (Dillman, 2000). But before anyone can assess whether e-mail surveying is obsolete, an evaluation of trends in response rates in light of the base of knowledge of survey techniques should be performed.

Response Rate Influences

There are numerous potential influences on response rates in both postal and e-mail surveys, including survey length, respondent contacts, design issues, research affiliation and compensation.

As this study is attempting to draw inferences about influences from already published studies, it will focus its discussion on those variables found consistently in e-mail surveys: survey length, pre-notification and survey follow-ups. A panel that evaluated each survey's topic and population assessed an additional variable, issue salience. Other variables have not been discussed in the e-mail survey literature as a way to influence response rates. This is likely due to the technical limitations of the Internet. First, there are multiple e-mail reading programs and it has been difficult, if not impossible, to achieve a consistent look in the design of an e-mail survey. Therefore, it would be difficult to isolate how design issues have effected response rate. In the same vein, it is technically impossible to provide monetary compensation at the same time that a survey is being taken, as it would be with mail surveys. Compensation could only be provided after the study. Very few studies report any type of compensation provided (beyond a report of the study's results) for participation in the study. And while research affiliation has been shown to have a positive effect on survey response rates, very few studies outline whether they were sent using an '.edu' suffix or identified research affiliation in the subject header of the e-mail.

Survey Length

Research results are mixed on the influence of survey length on response rate in both postal and e-mail surveys. While several studies have shown that survey length did not influence response (Brown, 1965; Bruvold & Comer, 1988; Mason, et al., 1961), other studies show that survey length has influenced response rate. The length of the survey was seen to have a negative influence on mail survey response rates in that the longer the survey, the more likely it is that the response rate will be lower (Herberlien & Baumgartner, 1978; Steele, Schwendig & Kilpatrick, 1992; Yammarino, Skinner & Childers, 1991). Recent studies have also indicated that samples in business-oriented studies were more sensitive than consumers to survey length (Jobber & Saunders, 1993) and that survey length was one of the main reasons for business persons' non-response (Tomasokovic-Devey, et al., 1994).

Conversely, Eichner and Habermehl (1981) found that longer surveys had somewhat higher response rates than shorter surveys. This suggests that survey length alone may not be enough to predict response rates. Other elements, in particular respondent contacts and topic salience, may also have an influence on response rates (Bean & Roszkowski, 1995).

Respondent Pre-notification

There is conflicting evidence regarding the influence of pre-notification on survey response rates. Fox, Crask and Kim (1988), Haggett and Mitchell (1994) and Kanuk and Berenson (1975) all found prenotification leads to increases in response rates for postal mail surveys. Heberlein and Baumgartner (1978), in contrast, found little or no effect on response rate with pre-notification, while Jobber and Sanderson (1983) reported a decrease in response rates to mail surveys with pre-notification. For both mail and e-mail surveys, pre-notification has been seen to increase response speed (Murphy, Daley & Dalenberg, 1991; Sheehan & McMillan, 1999; Taylor & Lynn, 1998). Mehta and Sivadas (1995) suggested that pre-notification for e-mail surveys is imperative, and the practice of sending unsolicited e-mail surveys is unacceptable. A pre-notification message may also be considered unsolicited e-mail, and problematic to many consumers. However, if it is of sufficiently short length and allows for 'opting in' to the survey respondent pool, as a practice it may be more palatable to potential respondents.

Follow-up Contacts

Post-notification, or follow-up contact, has been seen to have positive effects on response rates to postal mail (Comer & Kelly, 1982; Jobber, 1986; Murphy, et al., 1990, 1991; Yammarino, Skinner & Childers 1991). Kanuk and Berenson (1975) suggested that follow-ups in postal mail studies can increase response rates from eight to forty-eight percent, while Sheehan and Hoy (1997) found that a reminder message in an e-mail survey increased response by 25%.

Researchers have also assessed how many follow-up contacts are optimal and what content should be provided in the follow-up. Multiple follow-ups have been seen to yield higher response rates than one-time reminders (Heberlein & Baumgartner, 1978), although there were differences between contact methods. The optimal number of follow-ups has been discussed from a cost-effectiveness perspective, where the researcher must balance the cost of incremental contacts to the number of replies received. No differences have been seen in different reminder interval times (Claycomb, Porter, & Martin, 2000), and there does not appear to be a guideline as to the frequency and interval timing of contacts. Futrell and Lamb (1981) caution that unless a replacement survey is included in the follow-up contact, response rate would not increase.

Issue Salience

Salience of an issue to the sampled population has been found to have a strong positive correlation with response rate for postal, e-mail and WWW-based surveys (Sheehan & McMillan, 1999; Watt, 1999). Salience has been defined as the association of importance and/or timeliness with a specific topic (Martin, 1994). Salience, like beauty, is in the eye of the beholder, as a single topic may have higher salience to some sample populations than others. For example, a survey on homeowner taxes would likely be more salient to a population of homeowners than a population of college students.

Heberlein and Baumgartner (1978) reported that issue salience had more influence on mail survey response rates than other factors such as respondent contact and monetary incentives. Other researchers (Martin, 1994; Roberson & Sundstrom, 1990) also reported topic salience as a key influencer for mail surveys. Bean and Roszkowski (1995) suggested salience has more influence on response rate than survey length. They noted that "if a person attaches little interest or importance to the particular content of a survey, then it will not matter if the survey form is short; the person still is unlikely to respond (p. 25)."

Research Questions

The current study proposes the following research questions: RQ1: Have response rates to e-mail surveys increased, declined, or remained constant over the past 15 years?

RQ2: In addition, are e-mail survey response rates affected by the number of questions asked, the number of respondent contacts, and/or the salience of the issue?

Method

Published studies utilizing e-mail surveys were located using eight online databases that encompassed a wide range of academic topics. These databases included Academic Search Elite, Expanded Academic Index, ArticleFirst, Lexis-Nexis, Psychlit, Sociological Abstracts, ABI-Inform, and ERIC. These studies appeared in journals devoted to marketing, sociology, communication, organizational behavior, education, statistics and health. The references of each study were also examined to find any other studies not in the database system that may have utilized e-mail for data collection. The thirty-one studies used for this analysis are presented in Appendix 1.

The traditional way to investigate the significance of the effect of specific variables is via a meta-analysis. However, given the relative youth of the e-mail survey method, this technique is not possible at this time. Meta-analysis necessitates multiple studies that utilize a group that controls a particular influence and a group that tests the influence in order to measure the magnitude of the effect. In the body of literature that utilizes e-mail surveys to collect data, only a small number used the control/test group method, and these studies primarily studied the effect of mail compared to e-mail. In place of a traditional meta-analysis, then, this study will use correlation and regression analysis to examine the influence of the five variables of interest on response rates (year of study, survey length, pre-notifications, follow-ups, and topic salience).

Each of the 31 studies was analyzed to collect the five variables of interest. The issue date of the journal was used to represent the year when the study was undertaken. The journal issue date was selected since few studies reported the year that the e-mail survey was used to collect data. The studies themselves provided information on the number of questions in the survey, the number of pre-survey notifications, and the number of post-survey notifications. (Six studies did not provide information regarding pre- and/or post-survey notification. Three studies did not report the number of questions in the survey.)

To determine issue salience, a panel of academic researchers was provided a list of each survey's sample universe and survey topic, and asked to assess the issue salience for each survey topic (i.e. what is the salience of binge drinking to university students?). The panel members rated issue salience on a score of 1 (not at all salient) to 5 (highly salient). Five studies that did not include both sample universe and survey topic were not evaluated. Mean topic salience scores were calculated. In addition, intercoder reliability was calculated for all studies using Landis and Koch's (1977) formula; intercoder reliability was .82.

Results

The first research question asked whether response rates to e-mail surveys over the past fifteen years have increased, declined, or remained constant. Table 1 provides a summary of the mean response rates to e-mail surveys by year. This table provides an interesting overview of the history of e-mail surveys. The first e-mail surveys were performed in 1986, when e-mail was basically used as an inter-office messaging tool. In 1994, the World Wide Web became open to commercial traffic, and the Internet began its explosive growth both in the United States and worldwide. This growth may account for the increase in surveys published in 1995. As e-mail survey techniques began to be diffused to the academic population, the number of researchers trying this technique grew, as evidenced by the increase in e-mail surveys in 1998 and 1999.

While the number of studies that use e-mail to collect data has been increasing over the past fifteen years, the average response rate to the surveys appears to be decreasing (Table 1). On average, the 31 studies report a mean response rate of 36.83%. The 1995/6 period showed seven studies using e-mail surveys with an average response rate of about 46%. The 1998/9 period, in contrast, showed thirteen studies using e-mail surveys with an average response rate of about 31%.

Table 1. e-mail Survey Response Rates By Year

Year Number of studies Mean response rate Standard Deviation
1986 2 61.5 19.09
1991 1 41.0  
1992 2 72.0 5.65
1994 1 19.0  
1995 6 43.16 22.13
1996 1 52  
1997 3 21.6 11.78
1998 7 35.97 16.58
1999 6 27.5 10.65
2000 2 24.0  

F=2.0, P=.079

The second research question asked whether the number of questions asked, the number of respondent contacts, and/or the salience of the issue affected response rates. Table 2 presents a descriptive summary of these variables in the thirty-one surveys. The mean number of questions asked in the surveys was 42.3, ranging from a low of five questions to a high of 94 questions. Fewer than 20% of the surveys reported that a pre-notification message was sent. However, almost two-thirds of the surveys reported that at least one follow-up message was used. Issue salience ranged from 3 (somewhat salient) to 5 (highly salient).

Table 2. Description of e-mail Survey Characteristics

Variable

Number

Percentage

Number of questions (mean=42.3)

 

 

-5 questions

1

3.2

-9 questions

2

6.5

-12 questions

1

3.2

-18 questions

1

3.2

-21 questions

1

3.2

-24 questions

1

3.2

-28 questions

2

6.5

-29 questions

1

3.2

-30 questions

1

3.2

-31 questions

1

3.2

-32 questions

1

3.2

-35 questions

1

3.2

-36 questions

3

9.7

-45 questions

1

3.2

-46 questions

1

3.2

-55 questions

1

3.2

-60 questions

1

3.2

-62 questions

1

3.2

-65 questions

1

3.2

-66 questions

1

3.2

-70 questions

1

3.2

-72 questions

1

3.2

-93 questions

1

3.2

-94 questions

1

3.2

-information not available

3

9.7
Number of pre-notifications

 

 

-0 messages

19

61.4

-1 message

6

19.3

-information not available

6

19.3
Number of post-notifications

 

 

-0 messages

 9

29.0

-1 messages

14

45.2

-2 messages

3

 9.7

-3 messages

1

 3.2

-information not available

5

16.1
Salience level*

 

 

3

7

22.6

4

8

25.8

4.5

7

22.6

5

4

12.9

-information not available

5

16.1
.

*1=not at all salient, 5=highly salient

To analyze the influence each of the variables had on response rate, both correlation and multiple regression analyses were performed. Means, standard deviations, and Pearson correlations appear in Table 3. The correlation analysis revealed only one predictor variable that was significantly related to response rates: year (r=-.53, p=.005). The negative direction of this correlation suggests that response rates have indeed decreased since the inception of e-mail surveys. It also appears that the year the survey was performed correlated with the number of pre-notifications. The positive direction of this correlation suggests that researchers are heeding the advice of Mehta and Sivadas (1995), who stated that pre-notification for e-mail surveys is mandatory.

Table 3. Influence On Response Rates: Correlations

Variable

M

SD

1

2

3

4

5

1. Response rate

36.83

20.23

 

 

 

 

 

2. Year

1996

3.57

-.53*

 

 

 

 

3. Number of questions

42.3

23.62

.25

.12

 

 

 

4. Pre-notifications

0.19

0.40

-.16

.41*

.18

 

 

5. Post-notification

0.77

0.76

-.17

-.01

.35

.14

 

6. Salience

4.09

0.73

.18

.05

.14

-.06

.35

*p<.05

Using multiple regression, response rates were then regressed on the linear combination of year, number of questions, number of prenotifications, number of post notifications, and salience. The equation containing these five variables accounted for 49% of the variance in response rate, F (5, 25)= 3.283, p=.0295.

Beta weights (standardized multiple regression coefficients) and uniqueness indices were then reviewed to assess the relative importance of the five variables in the prediction of response rate. The uniqueness index for a given predictor is the percentage of variance in the criterion accounted for by that predictor, beyond the variance accounted for by the other predictor variables. Beta weights and uniqueness indices are presented in Table 4.

The table shows that only two influences, 'year' and 'number of follow-up notices', displayed significant beta weights. The beta weight for 'year' was in a negative direction and had the largest beta weight at -.54 (p=.002). This suggests that the year when the survey was undertaken has the largest influence on the response rate. The beta weight for 'number of follow-up notices' was somewhat smaller at .46 (p=.036) and was in a positive direction. This suggests that follow-ups may increase response rates to some extent.

The results for the uniqueness indices are interesting in that two variables displayed significant indices and only one of these, 'year', also had a significant beta weight. 'Year' accounted for approximately 24% of the variance, and number of questions for 17% of the variance. The other three variables accounted for the remaining 8% of the variance. The significant uniqueness index for number of questions could be due to the fact that even though the correlation of number of questions with response rate was not significant, it was the second highest of all five predictor variables

Table 4. Beta Weights and Uniqueness Indices for Five Influencers

Predictor

Beta

Weight

T

Uniqueness

Index

F

Year

-.54

-2.7**

.24

11.5*

Number of questions

-.03

-0.17

.17

8*

Pre-notification

-.20

-1.01

.01

.05

Follow-up

 

.46

2.27*

.03

1.5

Salience

.22

1.24

.04

2.0

*p<.05 **p<.005

The inconsistencies between the correlations, beta weights, and uniqueness indices may be an indication of the very weak influence that any controllable variable has on response rates. Only the 'year' variable shows consistent influence, and this is disturbing since researchers have no real control over this variable, that is, we cannot turn back the clock and undertake research in order to increase response rates.

Discussion

This study has examined influences on response rates to e-mail surveys over the past fifteen years. One surprising overall finding that has not yet been discussed is that few e-mail surveys have been undertaken during this intensive growth period of the Internet. This study attempted to identify as many e-mail surveys done for academic purposes as possible, and only 31 surveys could be identified that contained sufficient data to perform this analysis. This minimal adoption of e-mail surveying to date, combined with falling response rates, may indicate a less than promising future for e-mail surveys.

The results of this study suggest that e-mail survey response rates have been following the pattern of survey response rates overall in the United States. The strongest predictor of response rate was the year in which the survey was published, which was used as systematic way to represent when the survey was undertaken. This early level of high response may be due to the novelty aspect of using e-mail to respond to surveys. This novel period is likely to have passed. Thus, as time progresses, it seems likely that response rates to e-mail surveys will continue to decrease. There are several reasons that might explain this phenomenon. First, it is possible that as the Internet diffused into the population, the types of populations studied with e-mail surveys has changed. Researchers' achievement of higher response rates in the early days of e-mail surveying has been attributed to a group cohesiveness effect that is sometimes inherent to e-mail sampling frames (Kiesler & Sproull, 1986). Dillman (2000) suggested that early e-mail surveys were limited to survey populations with high rates of computer use, such as universities, business, larger organizations, and purchasers of computer equipment. The persons in these populations may have had a high degree of interest in the technology, resulting in higher response rates to e-mail surveys.

The increase in surveying in the United States may be another part of the explanation of lower response rates, along with the increase in unsolicited e-mail to Internet users and the ill will that this may generate among potential respondents. This is problematic for researchers planning to use e-mail surveys, since it is likely that some type of unsolicited respondent contact will be necessary when using random sampling techniques. Studies show that some Internet users receive more than 39 unsolicited e-mails per day at the workplace alone (NUA, 2000a). This information overload causes individuals to develop ways for dealing with e-mail, which includes using filtering software or developing heuristics such as deleting all unsolicited e-mail without opening it. For those who are not suffering from information overload, there are other reasons why they may not chose to read unsolicited e-mail. The threat of viruses delivered from unsolicited e-mail (such as the Melissa virus) may discourage Internet users from reading unsolicited e-mail.

Given these issues, it appears to be time to look at other factors that could be utilized to increase e-mail survey response rates. The challenge will be to work within the existing constraints of the e-mail system, and the first step will be finding a way to get e-mail recipients to open and read the e-mail rather than immediately deleting it. Exploring different type of recruitment methods, such as via postal mail or telephone contacts, may suggest a particular way that enhances response rates (Watt, 1999).

Previous research has identified university affiliation as a positive influence on postal mail survey response rates. New ways to create university affiliation in the e-mail context must be considered. While many e-mail messages have an implied university affiliation via the '.edu' suffix, this may not be an effective technique for several reasons. Potential respondents may not understand the suffix is connected with an educational institution, they may think the sender is a college student playing a joke on them, or they may think the address was 'spoofed', that is, created with a program that can create false e-mail addresses. Many educational institutions currently have telephone research survey labs established, and these institutions should find ways to establish e-mail survey research as a complement to their current activities. Additionally, it may be worthwhile to investigate soliciting potential participants via random digit telephone dialing and following this contact with an e-mail survey. While this may not be as cost-effective as an all-e-mail methodology, it may be beneficial in that it could ameliorate problems with unsolicited e-mail and possibly increase participation among online users.

Once techniques are found to increase the initial readings of e-mail surveys, further study of design issues and their influences on e-mail surveys must be undertaken. Past studies using e-mail have argued that the multiplicity of e-mail programs limits the ability to use colors, imaging, formatting and other actions to enhance the look of the survey and thus increase response rates. However, as software becomes more standardized, it would appear feasible to examine whether design techniques can be developed and applied to use with multiple email clients. Creating an easy-to-respond-to format should be considered as a design element that needs testing among the Internet population. Would an e-mail with 'radio buttons' to respond to be easier for respondents to use (as opposed to a 'type in your answer' text format) and thus have a positive influence on response rate? Would a link to a WWW site, accessible with an individualized password to allow for response tracking, be easier than the other two methods? Do we need to provide multiple options for response technique to increase response rates? These questions can all be studied scientifically in a controlled situation that can help develop our understanding of e-mail survey design techniques.

The next issue is that of compensation. Numerous studies have investigated the influence of compensation on mail survey response rates, yet most e-mail surveys rarely offer compensation beyond providing results to respondents at a future date. Several explanations for this observation exist. Technically speaking, it has been difficult to provide any type of immediate monetary compensation to a potential respondent. Therefore, the receipt of the incentive long after the survey is completed may be an insufficient to increase response. Second, while other types of non-monetary incentives have been proposed (such as access to proprietary online information or free software), no research to date has examined respondents' perceived value of such non-monetary incentives or their propensity to participate in research offering these incentives.

Recently, though, advances in technology have allowed respondents to receive certain types of incentives immediately after completion of a survey. Market-research and public opinion polls partner with Internet Service Provides (such as AOL) to offer 'points' for participating in research studies: the points can be redeemed at a future date for merchandise or reduced service fees. Watt (1999) suggested that there is evidence from online research firms that the possibility of the respondent's receiving a large award (for example, in the form of a sweepstakes prize), is more likely to increase response rates than is the certainty of a small incentive. Cook, Heath and Thompson (2000), however, reported that "the use of incentives in Web surveys seemed to be associated with more homogeneous and lower response rates. This paradox may have occurred because persons implementing disproportionately long or tedious surveys may have recognized the necessity of providing substantial rewards for survey completions (p. 827)."

Partnering with a single ISP and funding such a compensation program may be an option for academic researchers if the sampling method to obtain names from an individual ISP can be reliable and if the ISP population mirrors the Internet population as a whole. However, if academic researchers have to pay for the privilege of working with for-profit ISPs, the cost benefits of e-mail surveying may be eliminated. Further research into the types of compensation that appeal to online users should be conducted to identify the ultimate way to incentivize online users to participate in e-mail surveys.

As researchers, we must address how much of a problem the issue of non-response is. Do people who do not respond to e-mail surveys significantly differ from those who do? This is an issue that faces the survey industry as a whole, not just e-mail surveys. We must attempt to assess whether non-respondents differ significantly from respondents. This could be done via telephone or mail follow-up studies with non-respondents, or with studies that assess the likelihood that individuals will respond to different modes of surveys. It is time to quantify the concern that many express over these low response rates.

Given the concerns over response rates, a final question that must be addressed is: should we continue to even try to evaluate e-mail surveys as a viable data collection method or should we focus our attention on increasing reliability of surveys on the WWW? Although WWW-based surveying raises a host of new methodological issues, Dillman (2000) argues that "no other method of collecting survey data...offers so much potential for so little cost (p. 400)." There are many reasons for this. With a WWW-based survey, concerns with types of user and equipment capabilities are minimized (Bradley, 1999). Watt (1999) points out that the growing standardization of WWW browsers and communication software protocols as well as the widespread adoption of Java-language provide optimum tools for researchers to conduct successful online research. He also argues that lower-cost data collection via the WWW (in terms of time required for both collection and analysis of data) will result in larger samples with more statistical power and more useful results (Watt, 1999). Bradley (1999) similarly argues that utilizing a technique called 'saturation sampling', which attempts to survey all identifiable targets, overcomes any lack of reliable sampling frames.

The future may not lie in the WWW alone. Yun and Trumbo (2000) found that using multiple modes for survey delivery (postal mail with survey and URL for WWW site, and e-mail with both survey and link to WWW site) delivered a high response rate to a randomly selected sample of science writers. A third of responses to their survey were electronic (i.e. via e-mail or WWW site), and the researchers reported few observable differences between the modes. They cautioned that there was likely to be a high level of salience among their population; therefore, further study of multi-mode survey techniques is recommended. In a recent meta-analysis of WWW-based surveys, Cook, Heath and Thompson (2000) found that influences including both topic salience and pre-notifications can have a positive effect on response rates.

This study has examined e-mail survey response rates and determined not only that response rates are declining but also that some of the techniques applicable to mail surveys to increase response rates do not seem to affect response rates to e-mail surveys significantly. It has also identified methodological techniques that may stem the decline and work to make e-mail surveys viable for academic researchers today and in the future. We should continue to study e-mail as both a stand-alone technique or in conjunction with other electronic and traditional formats to best harness the power of this technology for research purposes. .

References

Anderson, S., & Harris, J. (1995). Educators' use of electronic networks. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, California.

Anderson, S. E., & Gansneder, B. M. (1995). Using electronic mail surveys and computer monitored data for studying computer mediated communication systems. Social Science Computer Review, 13 (1), 33-46.

Bachmann, D., Elfrink, J. & Vazzana, G. (1996). Tracking the progress of e-mail versus snail mail. Marketing Research, 8 (2), 31-35.

Bachmann, D., Elfrink, J. & Vazzana, G. (1999). E-mail and snail mail face off in rematch. Marketing Research, 11 (4), 11-15.

Bean, A. G.. & Roszkowski, M. J. (1995). The long and short of it. Marketing Research, 7 (1), 20-26.

Bickart, B., & Schmittlein, D. (1999). The distribution of survey contact and participation in the United States: constructing a survey-based estimate. Journal of Marketing Research, Spring, 286-294.

Bradley, N. (1999). Sampling for Internet surveys. An examination of respondent selection for Internet research. Journal of the Market Research Society, 41 (4), 387-395.

Brown, M. (1965). Use of a postcard query in mail surveys. Public Opinion Quarterly, Winter, 635-637.

Bruvold, N. T., & Comer, J. M. (1988). A model for estimating the response rate to a mailed survey. Journal of Business Research, 16 (2), 101-116.

Claycomb, C., Porter S. S., & Martin, C. L. (2000). Riding the wave: response rates and the effects of time intervals between successive mail survey follow-up efforts. Journal of Business Research, 48 (2), 157-162.

Comer, J., & Kelly, J. (1982). Follow-up techniques, the effect of method and source appeal. American Marketing Association Educators Conference Proceedings, Chicago.

Comley, P. (1997). The use of the Internet as a data collection tool. Paper presented at the ESOMAR Annual Conference, Edinburgh, September 1997.

Cook, C., Heath, F., & Thompson, R. (2000). A meta-analysis of response rates in Web- or Internet-based surveys. Educational & Psychological Measurement, 60 (6), 821-36.

Couper, M. P., Blair J., & Triplett, T. (1997). A comparison of mail and e-mail for a survey of employees in federal statistical agencies. Paper presented at the American Association for Public opinion Research, Norfolk, VA.

Dillman, D. (2000). Mail and Internet surveys: The tailored design method (2nd Edition). New York: John Wiley and Sons.

Dommeyer, C. J.. & Moriarity, E. (1999). Comparing two forms of an e-mail survey: Embedded vs. attached. International Journal of Market Research, 42 (1), 39-45.

Eicherner, K., & Habermehl, W. (1981). Predicting the response rates to mailed questionnaires (comment on Herberlien & Baumgartner). American Sociological Review, 46, 1-3.

Flaherty, T. B., Honeycutt, E. D., Jr., & Powers, D. (1998). Exploring text-based electronic mail surveys as means of primary data collection. The 1998 Academy of Marketing Science National Conference Proceed