JCMC 7 (3) April 2002
Message Board
Collab-U  CMC Play  E-Commerce Symposium Net Law  InfoSpaces Usenet
  NetStudy VEs VOs O-Journ HigherEd Conversation Cyberspace Web Commerce
Vol. 6 No. 1    Vol. 6 No. 2    Vol. 6 No. 3    Vol. 6 No. 4    Vol. 7 No. 1    Vol. 7 No. 2    Vol. 7 No. 3

Design of Web Survey Questionnaires: Three Basic Experiments

Katja Lozar Manfreda
University of Ljubljana

Zenel Batagelj
CATI Center

Vasja Vehovar
University of Ljubljana

Abstract

Despite increased use of Web surveys, relatively little is known about standards for designing Web questionnaires. Since there is no help from an interviewer for the respondent taking a Web survey, the design of self-administered Web questionnaires is even more important in order to achieve high data quality. Question wording, form and graphic layout of the questionnaire are particularly important. This paper presents some basic experiments to address these issues: one vs. multiple-page design, use of logotypes, and survey topic. The research was performed within the national RIS – Research on Internet in Slovenia - project (http://www.ris.org) in extensive testing since 1996.

Introduction

Considerable research has been documented on the methodological issues of Web surveys in the years since 1995 when they became widely applied. The majority of publicly available research, collected on the Web Survey Methodology page – WebSM site (http://WebSM.org), shows over 700 bibliographical units (at the beginning of 2002). Of course, owing to the predominantly commercial nature of Web survey practice, perhaps a greater number of research activities exist, but may not be available to the public. In addition, the existing bibliography is relatively small compared to extensive Web survey practice. Thousands of Web surveys are launched monthly and millions of users participate daily in this specific human-to-computer interaction. In addition, numerous commercial Web survey software packages (more than 50 are listed on the WebSM site) are available and the majority of research agencies already incorporate Web survey mode into their regular offerings to clients.

The reality seems to be rapidly approaching the prediction (Baker, 1998) that on-line self-administered surveys will become the next major step in the evolution of computer-assisted survey information collection (CASIC), the process which started with computer-assisted telephone interviewing (CATI) three decades ago (Couper & Nicholls II, 1998), and was followed by computer- assisted personal interviewing (CAPI). Marketing practitioners also share this view, claiming that the Web is becoming a replacement technology for the telephone survey mode (Black, 1998; Cleland, 1996; Hollis, 1999; Jones Thompson, 1999), just as the telephone survey mode replaced personal interviews in the 1970s.

Of course, we have to face the fact that many drawbacks of Web surveys prevent their yielding valid and reliable results. Most obvious and visible are the problems of non-observation, e.g. non-coverage, sampling and unit non-response (Groves, 1989). When studying characteristics of the general population, non-coverage and unit non-response are the most severe problems. Perhaps the most proper approach to minimizing coverage and non-response errors is the use of telephone pre-recruited panels of the general population (Couper, 2000). However, high costs and unit non-response at all stages of such a survey process cumulate in relatively high overall non-response rates in this case (Krotki, 2001). This represents a serious limitation to the approach. On the other hand, when the population of Internet users is the target, the lack of quality sampling frames is the main problem. Only for special populations with high Internet coverage rates and quality sampling frames can the sampling and non-coverage problems be eliminated. In this case, the next remaining problem is non-response error. Studies comparing response rates of Web surveys to other survey modes have shown that Web surveys usually obtain lower unit response rates (Lozar Manfreda, 2001). Of course, this is in part due to the lack of an adequate set of methodological principles for obtaining high response rates. Such principles have been already developed for other survey modes (for example, Dillman, 1978, 2000), but not yet for Web surveys.

Despite the problems described above, one can observe an exponential expansion of Web surveys, especially in marketing research. A 1999 survey of almost 200 U.S. marketing research professionals showed that two thirds of marketing research firms have already collected data online. They also have positive attitudes towards this mode, and conduct online research regularly (Gupta, 1999). Another study has shown that in the year 2000, about 5% of all market research conducted in Western Europe and the U.S. was conducted via the Internet. This figure was expected to rise to about 8% in the year 2001, and to reach 50% by 2004 (Poynter, 2001). The increased usage of Web surveys is particularly stimulated by their low cost, but also by the speed and quality of computer-collected data. Complex methods of panel sample selection and sophisticated post-survey adjustments (Terhanian & Black, 1999) additionally improve the position of the Web survey mode, particularly because non-observation problems are also becoming increasingly difficult even with other survey modes (Groves & Couper, 1998). Of course, increased Internet usage is also advantageous for the growing practice of Web surveys. It seems that the basic weakness of Web surveys targeted to the general population--coverage and sampling--are slowly being overcome. With the increase in Internet usage, the Internet users are becoming more and more similar to the general population (Pastore, 2001). In addition, with their increased numbers, it is getting easier to contact them for participation in Web surveys. The increased use of incentives additionally contributes to the attractiveness of this activity. In the future we may even expect the establishment of an elaborate computer-supervised environment that will attract and reward potential respondents. The need for fast feedback information in the fields of marketing, management and government will also benefit from the ease and convenience of the Web survey mode.

Human-computer communication via Web-assisted survey questionnaires is thus becoming a daily activity for a large segment of Internet users. In Slovenia, for example, where 28% of the total population use the Internet monthly as of March 2002, a fifth of Internet users have already participated in at least one Web survey, according to a probability telephone survey in 2001 (RIS, 1996-2002). Within this framework, the interactive aspects of Web-survey-responding communication are of growing importance. In addition to the above-mentioned issues of non-observation, the design and usability of the Web questionnaire are becoming central for the successful performance of the Web survey industry.

Two aspects seem to be particularly important as regards the design of the Web survey questionnaire: reducing the measurement error and the non-response error affected by the design of the Web survey questionnaire. However, despite the increased importance of these aspects, little is known about the basic standards for designing Web questionnaires. This is perhaps not surprising for the vast majority of nonprofessional Web surveys. A simple Web questionnaire with a few questions can actually be launched by anyone with some knowledge of Internet technology, especially with the help of several cost free software packages (for example, e-Experiment v2.1, Rostock Survey Tool, WWW Survey Assistant, Zoomerang, to mention just a few; source: WebSM site1). Therefore the prevailing presence of non-professional Web surveys (Onyshkevych & McIndoe, 1999), such as ‘question of the day’ polls on media Web sites, or questionnaires put online by individuals on their home pages, is no surprise. Obviously, we cannot expect that Webmasters and site owners would undertake any study of Web questionnaire design issues. It is somewhat more surprising, however, that even in the area of professional surveys there is relatively little agreement about the basic design features. It is true that the comprehensive work of Dillman (2000) discusses many important aspects, but to date little experimentation has been performed to confirm the principles that arise as an extension of paper questionnaires (Dillman, 1978).

In this paper, we address some basic features of Web survey design that were the subject of research performed within the RIS (Research on Internet in Slovenia) project (http://www.ris.org). Since 1996 this project has addressed some of the most important components of Web survey design in extensive experiments. The results have been presented at several conferences; however, they have not yet been presented systematically in any publication. The general absence of systematic scholarly presentation is perhaps also a general characteristic of this specific field, in large part arising from the commercial nature of much of the research. It is thus only recently that the first classification of Web surveys has been published in a scientific journal (Couper, 2000). Similarly, textbooks only recently began to discuss systematization of the non-response process (Vehovar et al., 2002) and the principles of designing Web questionnaires (Couper et al., 2001; Dillman, 2000).

As we have already mentioned, this paper discusses issues of Web questionnaire design in two directions: first, the impact on measurement error, and secondly, the impact on non-response error. However, with respect to non-response error, we do not discuss here design issues related to unit non-response (Vehovar et al., 2002). We restrict ourselves to the Web survey process from the moment the respondent starts to answer the Web questionnaire. With respect to non-response, we limit the discussion to partial and item non-response.

Measurement Error and Web Survey Questionnaires

Measurement errors in surveys are deviations of the respondents’ answers from their true value on the measure (Groves, 1989). In general, they result from inaccurate responses that stem from poor question wording or questionnaire design, poor interviewing, survey mode effects and/or some other aspect of the respondents’ behavior. Here, we limit the discussion to Web survey questionnaire design as one of the sources of measurement errors in Web surveys.

In general, Web surveys may produce larger measurement errors than other survey modes, owing to several factors. Web questionnaires are often designed by people with no training in survey methodological (Couper, 2000, p. 465), which results in bad questionnaire design. In addition, Internet users tend to read more quickly, they are more impatient and more discriminating than off-line readers (Internet Rogator, 1998). They may scan written material on the site with their fingers on the mouse ready to click on through to the next thing (Bauman et al., 2000). These issues which would be considered of minor importance in other survey modes may be very significant in Web surveys.

There are two main sources of measurement errors in Web surveying that stem from the Web questionnaire itself. The first involves the wording of the questions or the flow of the questionnaire, both of which may have an effect on the quality of respondents’ answers. The other is the question(naire) form, i.e., the visual layout of the questionnaire, of particular importance in self-administered surveys.

With respect to the wording of the questions, there are no specific recommendations for Web questionnaires in comparison to other modes, as long we adhere to general standards for the correct formulation of questions in survey research. There does not appear to have been much research conducted on the subject of specific question wording in Web surveys. However, as we have already noted, the problem arises because Web questionnaires are often designed by people inexpert in survey methodology (Couper, 2000, p. 465). Gräf (2002, p. 74) lists several examples of the most common mistakes in Web questionnaire wording which are the consequence of their implementation by people who are not survey methodology experts. For example, thematic and chronological references are not clearly stated; questions contain more than one thematic reference; expressions and phrases which are unknown to respondents are used; in closed questions, answer categories do not meet the demands of classification (completeness, exclusivity and clarity); unsuitable answer categories (something that does not exist or is not possible) are offered.

Flow and design of the Web questionnaire, on the other hand, have been researched somewhat more often and their impact on measurement errors observed (e.g., Bowker, 1999; Couper et al., 2001; Dillman, 2000; Dillman & Bowker, 2001; Dillman et al., 1998; Gräf, 2002). There have been suggestions (e.g., by Dillman, 2000) that each question should be presented in a conventional format similar to that normally used in self-administered paper questionnaires. On the other hand, researchers (e.g., Couper, 2000, p. 476) also advocate that the Web is a very special medium with special design options, visual features and required respondent actions, all of which require special handling of the questionnaire.

There are three main issues related to the visual design of Web questionnaires:

In the empirical part of this paper we present research under the auspices of RIS Web surveys that deals with some of these issues.

Non-response Error and Web Survey Questionnaires

Non-response error arises because measurement is not performed on all units from the survey sample (Groves, 1989). In Web surveys it may occur at any stage of the Web survey process: during the invitation to the survey, survey review, questionnaire answering and/or transmission of responses (Vehovar et al., 2002). It depends on the respondent’s characteristics, social and technological environment and survey design features including the design of the Web questionnaire. The impact of these factors on unit non-response (i.e., on complete non-response of a unit) is discussed in more detail in Vehovar et al. (2002) and Lozar Manfreda (2001). Here, we are mainly interested in the type of non-response that occurs during questionnaire completion itself (and not the whole Web survey process), and thus is strongly dependent on the design of the questionnaire itself. We therefore discuss partial and item non-response. Partial non-response is measured with the percentage of respondents who prematurely abandon the Web questionnaire among those who began to answer it. Item non-response is measured by the proportion of unanswered items among all items included in the Web questionnaire. For these reasons we concentrate particularly on questionnaire design features which can influence respondent participation during the filling out of the questionnaire but have limited impact on the initial decision to participate in a Web survey.

In relation to non-response error in Web surveys we also discuss the satisfaction of respondents with the Web questionnaire. Respondent satisfaction influences the decision to participate in surveys and therefore survey non-response (whether unit, partial or item non-response). It is thus important that the design of the Web questionnaire provides as much fun and satisfaction to the respondents as possible in order to increase their interest. This would help in convincing respondents to answer all survey questions and retain them until the end of the questionnaire, therefore minimizing item and partial non-response. In addition, it would help in convincing respondents to participate in other Web surveys, therefore minimizing unit non-response. This is particularly important owing to the growing reliance on panel Web surveys (Black, 1998; Comley, 1996; Kottler, 1997a, 1997b; Nadilo, 1999; Spaeth, 1999). Web survey respondents are hard and expensive to recruit; therefore once hooked, their cooperation is extremely precious. There are several measures aimed at maintaining the interest of Internet users in Web surveys, such as incentives, simple questionnaires, constant contact with participants, providing a sense of community, giving the opportunity to email back and keeping promises (Bunofsky, 1999). In addition, we believe that respondents need to receive some additional personal satisfaction from answering the Web questionnaire. Namely, some Web survey participants may see a Web survey as a form of entertainment or as an opportunity to gain new knowledge. For example, in a survey regarding food and beverage Web sites (White, 1996), many respondents spent as much as an hour completing Web site evaluations, paid on-line connection fees for their time and even offered to complete additional evaluations. Several respondents’ comments suggested that they saw the survey as an enjoyable game (“I’ll play some more”). Others described it as an “Excuse to surf the Net” or a valuable resource for discovering new Internet sites of interest to them (White, 1996).

In the literature, several features of Web questionnaire design have been discussed with regard to their direct impact on non-response through partial and item non-response and their indirect impact on non-response through respondent satisfaction. For example, one- versus multiple-page design has been discussed by Aoki & Elasmar (2000), Clayton & Werking (1998), Couper et al. (2000), Dillman (2000), Dillman & Bowker (2001), Dillman et al. (1999), Farmer (1998), Fuchs (2001), Kottler (1997b), Norman et al. (2001), Spain (1998), and Zukerberg et al. (1999). Advanced graphics in general has been discussed by Dillman et al. (1998) and Gräf (2002). Use of a progress indicator as a specific issue in advanced graphics discussions has been addressed by Couper et al. (2000) and Dillman ( 2000). Advances in computer-assisted survey data collection in Web surveys have been discussed by Andrews & Feinberg (1999), Brennan et al. (1999), Dillman (2000), Elder & Incalcatera (2000a), Nichols &d Sedivi (1998), and Zukerberg et al. (1999). Further, presentation and types of questions in Web surveys were addressed by Aoki & Elasmar (2000), Bowker & Dillman (2000), Couper (2001), Dillman (2000), Gräf (2002), Knapp & Heidingsfelder (2001), Kwak & Radler (1999), Sturgeon & Winter (1999), and Willke et al. (1999). Some of these features have also been tested with experiments in the RIS Web surveys, and are presented below.

Results of the Basic Experiments

 General Description of Data

The experiments on the design of Web questionnaires described in this paper are based on data from three national Web surveys, conducted as a part of the project RIS (Research on Internet in Slovenia, http://www.ris.org) at the Faculty of Social Sciences, University of Ljubljana. These surveys were widely advertised with banner ads on frequently visited Slovenian Web sites, through posts on IRC channels, with logon messages of two public ISPs, and through articles and posts in traditional media. In addition, email invitations were sent to Internet users whose email addresses were published in the public email directory.

The first survey, the RIS 1996, was conducted in April 1996 and lasted for 30 days. The average time to questionnaire completion was 7 minutes. The second survey, the RIS 1998, was conducted from June to August 1998 and lasted for 66 days. The average length of time spent completing the questionnaire was 22 minutes. The latest survey, the RIS 2001 survey, was conducted from June to September 2001, and on average it took 19 minutes to complete the questionnaire.

The RIS 1996 survey attracted 2034 respondents, the RIS 1998 survey 6522 respondents, and the RIS 2001, 14333 respondents. Owing to the undefined sample frame and non-probability sampling, unit response rates cannot be calculated precisely here. Nevertheless, with additional post-Web telephone surveys among Internet users in Slovenia we estimated that for all three surveys, almost 10% of active Internet users participated in the Web survey.

The RIS 1996 survey was relatively short and included questions on Internet use and social-demographic questions. An experiment regarding one- vs. multiple-page design was implemented and is further explained below.

The RIS 1998 and the RIS 2001 surveys were longer. In addition to the basic block of questions related to Internet usage and social-demographics, questions about Web site visits and satisfaction with the survey (at the end of the questionnaire) were added for all respondents. In addition, further modules with different topics were added to randomly selected sub-samples of respondents. Some of these modules were compulsory and some were optional. Detailed explanation is provided in the section below in which the experiment regarding survey topics is described.

Another experiment was implemented in the RIS 2001 survey: an experiment with the use of logotypes when measuring Web site visits.

For all these experiments respondents were randomly assigned to different experimental groups. For this purpose a special CGI script2 was used to assign respondents to experimental groups. On the basis of a sequential ID number assigned to each respondent when accessing the questionnaire, respondents were transferred to the appropriate version of the Web questionnaire.

The impact of the described design features on measurement and non-response error is measured. Although some of these experiments were performed some time ago, not much additional experimental research regarding the design of Web questionnaires has been published in the literature. Therefore we believe that the results of these basic experiments contribute significantly to knowledge about designing Web questionnaires in such a way as to collect more reliable and valid survey data.

One-page versus multiple-page design

One of the central issues of the Web survey mode is the question on one- vs. multiple-page design. Both designs have their advantages and disadvantages (Clayton & Werking, 1998; Dillman, 2000; Farmer, 1998; Kottler, 1997b; Spain, 1998) and no final conclusions have been reached as regards the most suitable design.

One-page design is characteristic of static Web surveys using plain HTML forms. These were the first Web surveys to emerge in the first half of the 1990s and are often still used for short simple surveys. In this case survey questions are ordered one after another on a single HTML page. Respondents are able to view the whole questionnaire and usually no alterations in the pattern of questions depending on the respondent's answer are involved. There is no interaction with the respondent during questionnaire completion and the questionnaire is identical for each respondent. Data are sent to the server using CGI scripts or email. Such questionnaires are in essence just electronic versions of paper questionnaires; they should, therefore, be short, simple and without complex skip patterns.

Nevertheless, in some cases, checking up on missing answers can be performed even with this design. More capable CGI scripts can check if unanswered questions appear in the dataset transferred to the server. In this case they can return the questionnaire page to the respondent’s browser. However, this is burdensome for the respondent, since he/she is presented with the whole questionnaire again and has to locate the parts that have not been completed.

Multiple-page design is characteristic of server-side interactive Web surveys enabling automatic skipping and conditional branching, validation of responses, random question/item order, randomization of questionnaire distribution to participants, adaptive questionnaires (assigning questionnaire items based on earlier answers of a participant), control for item non-response, quota controls for accessing the questionnaire, time measuring, etcetera – in fact all the features of computer-assisted survey data collection required by sophisticated questionnaires. They are also called computer-assisted Web interviewing or CAWI by some (e.g., Bickham & Moore, 1999; Kottler, 1997b; Sturgeon & Winter, 1999; Whitlock, 1997). The features of computer-assisted survey data collection are usually handled using CGI scripts on the server side at the beginning and at the end of each HTML page (e.g., server-side interactive Web surveys). When one of these features is used for a particular survey question, such a question should appear on a new HTML page because of the required use of CGI script at the server side. Therefore, the Web questionnaire appears on several HTML pages. Each page, i.e. separate HTML form, is sent to the server using CGI scripts, and immediately validated. Depending on how it is answered, the next appropriate question or set of questions is presented. In the most extreme case, there is only one survey question on each page and the questionnaire therefore consists of as many pages as there are survey questions.

Recently introduced client-side interactive Web surveys can again use one-page design. With the introduction of Java, ActiveX and Javascript, features of computer-assisted survey data collection can be done on the client’s (respondent's) side, without interacting with the Web server. The client (respondent's computer) along with a Web page, downloads and executes a computer program in Java or ActiveX, which may still use the HTML format as a shell. Web pages are active without interaction with the server. However, in practice this solution may also be slow, since the whole questionnaire (program) has to be downloaded before the data collection actually starts. This is especially problematic when there is slow Internet access on the respondent's side (i.e., dial-up access). In addition, the compatibility of browsers should be taken into account when programming these questionnaires, since only later versions of Web browsers support these new programming features and they should be additionally configured to support Java-applets.

The use of static Web surveys with one-page design, or interactive Web surveys, whether server-side with multiple-page design or client-side with one-page design, may have several effects on the quality of survey data, including measurement error, item non-response and partial non-response.

Implications for Measurement Error

Dillman and Bowker (2001) argue that presenting one question per page results in lack of context. If a respondent is able to see only one question at a time, when his/her concentration is interrupted, he/she has to figure out how to back up and view a previous question in order to answer the current one. In general, one question per page makes people lose the 'big picture,' an effect demonstrated for interviewers in interviewer-administered surveys (Fuchs, 2001).

The order effect can also be enhanced, since people are not able to see the entire survey before answering (Couper et al., 2000). On the other hand, placing several items on a single page, especially where several pages are nonetheless used, is likely to lead respondents to view the items as related entities, thereby increasing the correlations among them (Couper et al., 2000).

Experiments in Web questionnaire design have confirmed the above hypothesis. Reips (2002, p. 102) showed that placing one or several questions per page leads to different answers owing to the cognitive contexts. Similarly, Couper et al. (2000) showed that placing several items per page increases correlation among them.

Implications for Non-response Error

Item non-response may be greater when one page is used, since no control for item non-response can be implemented (unless client-side interactive software is used). In addition, scrolling is generally hard to use as a navigational tool and people prefer other paging functions, such as “Page Up” and “Page Down” keys (Norman et al., 2001, p. 40). On the other hand, this design more closely follows the design of many established computer applications like word processors (Zukerberg et al., 1999) and paper questionnaires; respondents navigate freely through the questionnaire, have a better sense of orientation (Dillman & Bowker, 2001), and evaluate the length of the questionnaire, all of which can result in lower partial non-response (drop-out) rates.

Multiple-page designs enable controls for item non-response; therefore, theoretically, item non-response should be lower. On the other hand, the danger of abandoning the questionnaire is potentially higher with multiple-page design, especially if an extreme form of this design with one question per page is used. This can occur for several reasons:

Nevertheless, multiple page design is closer to other Internet applications and may become even more intuitive as Internet usage increases (Zukerberg et al., 1999).

Description of the Experiment with One- vs. Multiple-page Design and Hypotheses

In the RIS 1996 Web survey, an experiment aimed at measuring differences in the effect of the one- vs. multiple-page design was performed. One randomly assigned experimental group (n = 644) answered the questionnaire as one long scrolling page, while the other group (n = 672) answered the questionnaire with each question (or block of questions if there were multiple items with the same scale) on its own page and the next page appeared only when the previous one was finished3.

The time needed to complete the questionnaire in one- and multiple-page designs was measured. Since every single page had to be downloaded from the server and answers to every survey question uploaded separately to the server, we expected multiple-page design to take significantly more completion time.

Because of separate downloading of each single page, we expected respondents in the multiple-page design group to abandon the questionnaire prematurely more often, and therefore to have larger partial non-response.

No control for item non-response was used in any of the designs. The difference in item non-response can therefore be attributed exclusively to the difference in the layout and not the difference in the use of sophisticated features of computer-assisted surveying. When no controls for item non-response are built in, we expect item non-response to be larger in one-page design. Owing to the scrolling that is needed in this case, we believe it is more likely that respondents will skip (either mistakenly or on purpose) individual survey questions.

Results

We confirmed the first hypothesis about the questionnaire completion time. Questionnaire completion time for the multiple-page design was 30% longer. On average, 466 seconds were needed to complete the multiple-page questionnaire, in comparison to 368 seconds needed for the one-page questionnaire (the difference is statistically significant at p < 0.0005). This result is consistent with the results of Couper et al. (2000) and Fuchs (2001). A different result, on the other hand, was reported by Norman et al. (2001, p. 43) and Zukerberg et al. (1999), i.e., no difference in questionnaire completion time. However, this may be due to the laboratory conditions of their experiments. We can conclude that practice shows that multiple-page design slows down questionnaire completion in comparison to one-page design.

As regards the hypothesis on partial non-response, we cannot confirm it. There was no evidence of difference in partial non-response in the two designs: 14.6% of respondents prematurely abandoned the questionnaire when one-page design was used and 16.5% of respondents did not go on to the final part of the questionnaire when a multiple-page design was used. The difference is not statistically significant (p = 0.362).

We can again confirm the third hypothesis on item non-response. One-page design resulted in higher item non-response (see Table 1): it looks like respondents more often omitted particular questions if they had all the questions on a single HTML page. This difference is particularly noticeable for questions in the form of a ‘grid’ (questions in table form with items presented as the row headings and scales on which these are rated as column headings). The questionnaire included two ‘grids’, for both of which the number of skipped items was larger in one-page design (40.3% vs. 35.0%, t-test, p = 0.013 for the first ‘grid’; and 40.2% vs. 35.3%, t-test, p = 0.021 for the second ‘grid’). People more often omitted questions in tables when they were placed on one long HTML page together with other questions than when they were placed on a separate HTML page.

Another experiment regarding the number of questions/items per page, however slightly different, was performed by Couper et al. (1999). They compared a grid question on one page (multiple-item-per-screen version) with separately posed items on individual pages (single-item-per-screen version). The multiple-item-per-screen version took significantly less time to complete and resulted in less item-missing data. In contrast to this result, in our case, the one-page design with multiple questions per page produced higher item non-response. However, the two experiments are not strictly comparable, since in the Couper et al. (1999) experiment the two designs differed not only in the number of items per page, but also in the layout of the questions: table vs. no table.

Unfortunately, the described RIS 1996 experiment did not include measures of respondent satisfaction conditioned by the design. An indirect indicator of their satisfaction can be their willingness to provide their email addresses for future Web surveys. The percentage of respondents revealing their email addresses turned out not to differ significantly by type of design (68% for one-page and 70% for multiple-page design, p = 0.300), so we cannot conclude that respondents were more satisfied with one design rather than another.

However, respondents’ comments at the end of the RIS 1998 Web questionnaire where only the multiple-page design with one question per page was used can be very informative for discussing the satisfaction of respondents with the design of the questionnaire. At the end of the questionnaire respondents were asked to comment on the survey and their comments were coded in order to identify eventual problems they had with completing the questionnaire. Twenty-two percent of respondents answered this open-ended question and commented on the survey. As we have already said, completion of the questionnaire took on average 22 minutes and some of the most frequent comments were those about downloading taking too much time (7% of comments), wishing to have more questions on one page (4%), not being able to estimate how much of the questionnaire they had already completed (2%) and expressing of the wish to complete the questionnaire off-line (2%). All these comments arise from the fact that each question (or set of question-items with the same scale) was posed on a separate page, and indicate their dissatisfaction with the design. In addition, these comments were given only by that subset of respondents who reached the final part of the questionnaire (63% of all who started to answer the questionnaire); those abandoning the questionnaire prematurely would probably have added similar comments.

An experimental study measuring the satisfaction of respondents when one- vs. multiple-page design was used (Zukerberg et al., 1999) reached somewhat different conclusions. They did not find any significant differences in respondent overall satisfaction whether one- or multiple-page design was used. Although the questionnaire was also long (on average more than 30 minutes) a factor which would suggest lower satisfaction for the multiple-page design, this was not the case. In comparison to the RIS 1998 survey, the contrast in the results may be explained by the laboratory environment in the Zukerberg et al. (1999) study. In addition, in their questionnaire a menu bar was used in the multiple-page design, indicating the respondent’s location in the questionnaire.

Inconsistency in reported results regarding response and respondent satisfaction indicates that additional testing is needed. As for now, researchers (Dillman, 2000; Kottler, 1997b) generally suggest constructing the questionnaire so that respondents scroll from question to question, and a new page appears only when some control of answers or skip is needed. Otherwise, they suggest using multiple-page design if order effects are of major concern, and/or telephone and Web survey results are being combined. A hybrid approach was also suggested (Aoki & Elasmar, 2000). In this case a single page is used and the illusion of multiple pages is created through page skip icons labeled “Next” appearing at the bottom of each section. There is a blank screen between sections so that respondents cannot see any text beyond the screen currently being answered. This is supposed to minimize the time involved with multiple-page layout (more downloading) and also problems associated with scrolling in one-page design. However, a comparison with other designs has not yet been made to assess the effectiveness of this approach.

Advanced Graphics: Use of Logotypes

The possibility of using the advanced graphics technology supported by current Web browsers is one of the most often cited advantages of Web surveys. It can be used to illustrate survey questions, to decrease respondents' burden if it is designed to ease navigation through the questionnaire, and/or to improve respondents’ motivation and satisfaction. The latter can be achieved through generating a valuable felling of having a “good time” or “fun” while answering the Web questionnaire.

However, the trade-off with respect to the influence of advanced graphics on participation must be taken into account because of technological limitations. For example, browser incompatibility may prevent some respondents from accessing or from seeing (clearly) some parts of the questionnaire. In addition, download times are longer and respondents may lose patience while waiting to see the questionnaire, or become distracted. In addition, advanced graphics used to illustrate survey questions may also bias respondents' answers if not used properly.

An experimental study of two questionnaire designs - fancy and plain - (Dillman et al., 1998) already provides an important warning about extensive use of graphics, since respondents to the fancy version completed fewer pages and fewer write-in boxes, were more likely to drop out, spent more time working on the questionnaire, and were more likely to have to return to the questionnaire in order to complete it.

Description of the Experiment with Logotypes

In the RIS 2001 Web survey an experiment with the use of advanced graphics was performed. The graphics were used to illustrate survey questions and to increase respondents’ interest in the questions. Advanced graphics, i.e. logotypes in the form of small pictures in this case, were used on one of the questionnaire pages. The page was used when measuring the knowledge and frequency of visits to Web sites. Two randomly selected experimental groups of respondents (both of approximate size n = 750) were administered a part of the questionnaire with 16 questions about visits to Web sites. These questions were placed in two grids, the first 7 items with Slovenian Web sites and the second 9 items with foreign Web sites. The frequency of visits was measured on a scale from 1 to 7 with categories: “don’t know,” “already heard of,” “visited once,” “visiting occasionally,” “visiting monthly,” “visiting weekly or more often,” and “visiting daily." These questions appeared on the third or fourth (depending on some previous answers) questionnaire page. The first randomly selected experimental group was administered a version of the questionnaire with a page where site logotypes were presented together with their names. The second randomly selected experimental group was administered a page where only the names of the sites were written (see Figure 1).

Logotypes were expected to influence non-response and measurement error. The positive effect of the use of logotypes was expected in terms of smaller item non-response. Logotypes were supposed to make the questionnaire look more visually attractive and more interesting. This should motivate respondents to make s stronger commitment to answering all the questions. In addition, logotypes were supposed to illustrate survey questions, i.e. questions about visits to Web sites. We expected respondents to give more knowledgeable answers to the survey questions when a visual aid was presented.

On the other hand, a negative effect of the use of extensive graphics was also expected. Because of the use of logotypes, slower data transfer was expected resulting in a greater number of respondents abandoning the survey if answering the questions with logotypes (therefore larger partial non-response). This was expected to be even more problematic in cases where respondents used a slower Internet connection. In addition, it was expected to be more problematic for those respondents who had to pay for Internet connection time.

Results

Premature abandonment of the questionnaire (therefore partial non-response) was measured with the use of information from log files that are kept when Web questionnaires are used. Information from such log files has already been successfully used to study participation through different stages of the Web survey process (for example, Bosnjak et al., 2001; Jeavons, 1999). In our case we used the information from log files to find out where (at which point in the questionnaire) respondents quit the survey. Since the questionnaire was split into several HTML pages, responses from each page were separately transferred to the server. The last record for each respondent shows the last HTML page that was answered, e.g. the last time the 'Submit' button was hit. After that 'Submit' a next page was also open; however no 'Submit' button was hit on that next page.

For our two experimental groups of respondents, we checked how many times the page before the page with questions on Web sites visits appeared as the last answered page. We assumed that one of the reasons for quitting at that particular page was the next page with questions on Web sites that was also open, but not answered. There are of course other possible reasons for quitting at that point that do not necessarily have to do with a particular question. For example, other possible reasons might inlude a general attitude formed after the questions previously completed, the length of the questionnaire, technical problems or some external factor (e.g., a telephone call). However, since these other reasons are equally distributed in both experimental groups, we assume the difference in the experimental groups stemmed from the difference in the presentations of questions.

The results confirm the hypothesis. The use of logotypes resulted in a significantly higher number of respondents abandoning the survey. in the first experimental group, 5.2% of all respondents quit the survey at the page with questions on Web site visits where the logotypes were used. On the other hand, 3.4% of all respondents in the second experimental group quit the survey at the page with questions on Web site visits without the logotypes (the independent samples t-test shows the difference as statistically significant at p < 0.0005).

We can also confirm that the costs of Internet access influence respondents' reactions to advanced graphics. The difference between the two experimental groups was actually significant for those who had to pay for Internet access but not for those who had free access (either they had a free Internet provider or they were answering the questionnaire from school, library or work where the costs were not their personal costs)4 (see Table 2). We can also partially confirm that technical equipment influenced their reactions to advanced graphics. The difference between the two experimental groups is actually significant only for those using the slowest Internet connection, i.e. an ordinary modem. Of those using ordinary modems. 3.1% abandoned the survey when answering the questions with the logotypes, while only 1.0% of those using ordinary modems abandoned the survey when answering the questions without the logotypes (p = 0.099). For the faster types of Internet connection the difference between the two experimental groups is not statistically significant, although abandonment of the questionnaire at these questions is in general more likely.

While the above result shows the negative consequences of the use of logotypes, their effect as a visual stimulus increasing motivation and commitment was positive. First, item non-response was lower when logotypes were used. Item non-response was measured as the average percentage of unanswered items among all items in a particular grid, i.e. the grid with Slovenian Web sites (7 items) and the grid with foreign Web sites (9 items). For both grids there is a statistically significant difference in item non-response. Item non-response for the grid with Slovenian Web sites was 2.6% when logotypes were used and 4.1% when logotypes were not used (the difference being statistically significant at p = 0.001). Similarly, for the grid with foreign Web sites, item non-response was 4.0% with logotypes and 5.4% without logotypes (the difference being statistically significant at p < 0.0005) (see Table 3).

Measurement error due to the (non)use of logotypes was measured by comparison of substantive answers of the two experimental groups. An analysis across 16 items, across Web sites, was performed. First, answers of the two experimental groups for each item separately were compared using the chi-square test. A statistically insignificant chi-square statistic is an indicator of similar answers for the two experimental groups; this is, therefore, an indicator of no measurement error introduced by the (non)use of logotypes. On the other hand, a statistically significant chi-square statistic is an indicator of different answers for the two experimental groups; this is, therefore, an indicator of measurement error introduced by the (non)use of logotypes. In our case, a statistically significant chi-square statistic appeared for 8 out of 16 items at p < 0.05 and one additional item at p < 0.1, therefore for over half of the items. However, we could not find any pattern to explain at which Web sites the difference occurred more often. It occurred equally as often for well known and frequently visited as for less known and less frequently visited Web sites (Pearson correlation coefficient for Chi-square statistic and average frequency of visits on a scale from 1 to 7 was r = -0.299, not statistically significant, p = 0.243).

Secondly, substantive answers of the two experimental groups were compared by calculating differences in answers. For each item (each Web site) a difference for seven possible answers was calculated. Then, an average difference across all 16 items for seven possible answers was calculated. This data is presented in Figure 2. We can see that, on average, despite significant chi-square statistics for over half of the items, the absolute differences in estimates are rather small. The only more noticeable difference was the category “visiting occasionally." Those who were not presented with logotypes were more inclined to choose the answer “visiting occasionally” than those with logotypes. The answer “visiting occasionally” can be seen as a middle point on the scale from 1 to 7. It appears that when respondents were presented with the Web site along with its logotype, they were somewhat more sure whether they knew it or not and how often they visited it, and therefore they did not choose the middle point.



Figure 2. Impact of logotypes on responses regarding the frequency of Web site visits (RIS 2001).

Results of this experiment closely match the results of a similar experiment (but with other Web sites) that we first performed in the RIS 1998 Web survey (Vehovar et al., 2000). In the 1998 experiment we similarly found that partial non-response was larger with the use of logotypes; however, item non-response was smaller and there were slight substantive differences in results. Also in that case the percentage of “don’t know” answers was smaller and the percentage of “visit occasionally” answers was larger when no logotypes were used. When respondents saw the logotype they were therefore more often inclined to admit that they did not know the site, while on the other hand, when they were not confronted with the logotype, they more often answered that they visit it occasionally.

We can conclude that graphics in the form of logotypes illustrating survey questions positively affect measurement error. When logotypes are present, respondents somewhat less often choose the middle point and more often other answers. Logotypes as a motivating factor also positively influence non-response error: item non-response is smaller when logotypes are used. It looks as if respondents are more committed to answering all the questions when this visual aid is used. Unfortunately, fear of graphics discouraging respondents from continuing to answer the questionnaire also proved to be correct, just as in the Dillman et al. (1998) study.

Survey Topic

Survey topic salience is defined as the association of importance and/or timeliness with a specific topic (Martin, 1994). A single topic may have different salience for different sample populations or for different individuals within the same population. For traditional survey modes, it appears that a respondent's decision to participate in a survey also depends on the survey topic, its salience and the respondent’s knowledge of and involvement in the topic (Couper, 1997; Groves & Couper, 1998; Heberlein & Baumgartner, 1978; Jansen, 1985; Kojetin et al., 1993; Martin, 1994; Roberson & Sundstrom, 1990). Similar findings are also characteristic of some studies involving email and Web surveys. For email surveys Sheehan and McMillan (1999) found a positive correlation between survey topic salience and response rate. For intercept Web surveys5, Comley (2000, p. 331) claims that some of the lowest response rates occurred when surveys were targeted at non-customers, and/or were asking about general issues. The highest response rates, on the other hand, were obtained for surveys where respondents had a strong relationship with the brand or felt favorable towards it and/or when surveys were asking relevant questions, such as comments about the Web site.

Description of the Experiment with Survey Topic

An experiment examining the influence of survey topic was performed within the RIS 1998 Web survey. After a basic set of questions about Internet use, demography and Web sites, respondents were randomly assigned one of ten additional modules of questions with different topics. After answering, 60% of respondents had the chance to select one or more additional modules from the other nine and three additional ones (in all they could choose among twelve modules). The remaining 40% of respondents received a final few questions about satisfaction with the survey (see Figure 3).



Figure 3. Flow of the RIS 1998 Web questionnaire.

With this experiment we wanted to test how the survey topic influences partial non-response and related respondent satisfaction. Because respondents were intensive Internet users and the survey was advertised as a survey on Internet use in Slovenia, we expected partial non-response to be smaller for those survey topics that were more related to Internet and information-communication technologies. This hypothesis follows the arguments that intensive users with strong attitudes towards Internet issues are also more likely to participate in Web surveys, especially those relating to Internet usage or information-communication technology in general (Elder, 1999; Enander & Sajti, 1999; Findlater & Kottler, 1998), owing to a positive correlation between survey topic salience and response rates, as explained above.

In addition, we expected partial non-response to be smaller for those survey topics that promise fun and amusement. As has already been said, some Web survey participants may see a Web survey as a form of entertainment or as an opportunity to gain new knowledge (White, 1996). A recent study (Lozar Manfreda, 2001, p. 218) actually found that respondents who use the Internet for fun are more likely to reveal their email addresses for research purposes than those respondents who use the Internet for business.

In relation to this we expected modules (survey topics) that were supposed to be fun and amusing to be more likely to increase respondent satisfaction with the survey.

The experiment allowed us to study not only the influence of the survey topic, but also the impact of the flow of the questionnaire. More precisely, we could study the effect on respondent satisfaction when certain modules were presented as compulsory or optional. We expected optional modules to increase respondent satisfaction. With optional modules the respondents’ movement through the questionnaire was not completely dictated by the researcher. Respondents themselves had an option to decide which modules they could answer. Because of this we expected those who had optional modules to be more satisfied with the survey than those not having this option.

Results

Influence on partial non-response. For each module (topic) we measured (1) the percentage of respondents choosing a particular module among those who had the chance to choose, and (2) the percentage of respondents abandoning the survey when answering questions from a particular module which was assigned as compulsory. The results are presented in Table 4.

Respondents most often chose modules with questions on “Computer infrastructure” (12.4% of all who had the chance to choose), “Attitudes toward erotica and pornography” (12.2%), “Leisure-time activities” (10.2%) and “Internet software” (9.7%). As hypothesized, these were topics of interest to intensive computer and Internet users and topics that were supposed to be ‘funny’ and ‘amusing.’ However, with respect to the module on “Computer infrastructure” the respondents also abandoned the survey prematurely if this module was compulsory (34% of those answering the questions from this module assigned as compulsory abandoned the survey at one of these questions). In addition, they most often abandoned the survey at compulsory modules on “Attitudes toward the Internet," on “Internet, state and politics” and on “Internet use."

When measuring the impact of survey topic on (non)response it is actually impossible precisely to distinguish topic itself from other characteristics of modules, such as difficulty of the questions, sensitivity of the questions, and length of the module. In our specific case, modules did not differ in topic only, but also in these other characteristics. This can explain the larger number of respondents abandoning the survey during the modules on “Attitudes toward the Internet” and “Internet use” that were supposed to be interesting for respondents. These two modules were among the longest modules and some questions included were rather difficult, requiring more cognitive effort from respondents.

Influence on respondent satisfaction. Respondent satisfaction was measured with a specific survey question on the last questionnaire page. Respondents who reached the final part of the questionnaires were assigned a question about their overall judgment of the survey using a scale from 1 (not appropriate at all) to 5 (very appropriate). Lower scores were taken as indicators of respondent dissatisfaction while higher scores were taken as indicators of respondent satisfaction.

In addition, on the last questionnaire page respondents were also asked to give comments about the survey in an open-ended question. These comments were coded by distinguishing among positive and negative comments related to different aspects of the survey. Comments regarding survey topics as an indicator of respondent satisfaction are of interest here.

Several indicators of respondent satisfaction regarding the survey topic and presentation of modules as compulsory or optional are presented in Table 5:

An interesting result is that average judgment of the survey overall was higher for those respondents who had the chance to choose additional modules after the compulsory one. If they were assigned only a compulsory module, the lowest satisfaction was characteristic of those answering the question on “Attitudes toward the Internet,” “Advertising on the Internet,” and “the Internet, state and politics."

Some comments about the survey at the end of the questionnaire were also directed to the topic of the questionnaire. Negative comments on the topic related especially to the module “Internet, state and politics” (27.1% of those who answered this module and reached the final part of the questionnaire commented negatively on these questions), and the module on “Medicine, pharmacy and the Internet” (22.3%). Negative comments also were made about the modules “Internet and traditional media” and “Advertising on the Internet." Here we must stress that this was an open-ended question and the last in the survey; therefore those abandoning the survey prematurely did not have a chance to answer it. We suppose that there would have been more negative comments if those abandoning the survey had had the chance to answer it.

Discussion, Limitations and Further Research

In scientific surveys we need a valid and reliable instrument to measure characteristics of the sampled units. However, with the emerging Web survey mode, the whole survey measurement process is increasingly being conducted through human-computer interaction. Interaction with the computer is thus becoming the main interface for capturing survey data. Of course, the computer questionnaire itself is also sensitive to design characteristics. The reality we want to capture with survey measurements thus increasingly relies on computer questionnaire specifics and on the corresponding interaction with the respondent.

In this paper three basic Web survey design features were studied and their impact on response and satisfaction was discussed. Below are the main conclusions:

1. Partial non-response was influenced by the use of logotypes and the survey topic, but not by the number of questions per page.

2. Item non-response was influenced by the use of logotypes and multiple-page design.

3. Use of logotypes also influenced the measurement error. Respondents less often chose the middle point on a scale from 1 to 7 (“visit occasionally”) when logotypes were used compared to questions when no logotypes were used.

4. Respondent satisfaction is an important factor that may influence response rates in future surveys. The survey topic and the optional nature of additional modules had particularly strong impact.

Based on these results some suggestions for future implementation of Web surveys can be made. The use of graphics to illustrate survey questions and/or to motivate respondents is desirable; however, an optimum balance between positive and negative effects is needed. Optional modules are also desirable as they increase respondent satisfaction and therefore motivation for participation in future Web surveys. The questionnaire should be constructed so that respondents scroll from question to question, and a new page appears only when some control of answers or skip is needed. One question per page (i.e., extreme multiple-page design) is desirable only when order effects are of major concern, and /or telephone and Web survey results are being combined.

The above experiments were all performed on three surveys of intensive Internet users who self-selected themselves for participation in the survey. In some cases, results might have been different if less active users had been surveyed. Nevertheless, we believe that results are very informative for any type of Web surveys (Couper, 2000).

Another limitation of the study is the fact that our experiments were conducted in Slovenia within national surveys. Nevertheless, we believe that results are in larger part generalizable to other countries. The Internet usage in Slovenia does not significantly lag behind other more developed countries. With 28% of the total population monthly using the Internet as of March 2002, Slovenia is in this apsect close to many developed countries. Of course, there is a possiblity that cultural specifics might influence some of the respondents' reactions to the experimental conditions presented. It is to be hoped that future experiments in other countries will look at the possible impact of cultural specifics on the quality of data in Web surveys.

There are many other questionnaire design features which have not been sufficiently studied and which deserve greater attention in future studies:

  1. Progress indicator. Respondents' comments at the end of the RIS 1998 Web questionnaire show that they missed a form of progress indicator of how many questions remained. What kind of indicator should be used: text or graphic? Will such an indicator have a positive or negative impact on partial non-response?


  2. Instructions. There are several possibilities for implementing instructions on the questionnaire: detailed instructions provided for the first few questions, hyperlinks to special help screens or two versions of the questionnaire, one for less experienced and one for more experienced respondents. These modes may differentially affect the data quality and their impact should be studied.


  3. Forced reminders and checking controls. In theory these should lead to a higher quality of data. However, they might be annoying for respondents or their implementation may demand the newest versions of browsers not used by all respondents.


For traditional survey modes extensive literature on the quality of survey data exists. Procedures for conducting surveys that would generate quality data have been developed over years of experimental research and meta-analytical studies (for example, de Leeuw, 1992; Dillman, 1978). A similar set of procedures including all aspects of survey design has not yet been developed for Web surveys, although there exist some attempts in this direction (for example, the Tailored Design Method by Dillman, 2000). More often, studies report only on the testing of some individual aspect of Web survey design and its impact on data quality. A few papers report on some meta-analytical approaches including several Web surveys (Cook et al., 2000; Knapp & Heidingsfelder, 2001; Lozar Manfreda, 2001; MacElroy, 2000). However, the number of such studies is still rather limited.

With this paper we provide a certain insight into the increasing body of knowledge regarding data quality in Web surveys. With the description of three basic experiments on the design of the Web questionnaire we have illustrated some of the problems and possible solutions in order to decrease non-response and measurement error.

Footnotes

1. http://www.WebSM.org/companiesoft.html.

2. CGI (Common Gateway Interface) scripts are simple programs that enable interaction between HTML pages and a Web server and are used to send the data to the server.

3. In 1996 a significant segment of Internet users still used textual Web browsers. Here, only results of the experiment for respondents using graphic Web browsers are reported, since reference to textual Web browsers is no longer relevant. For the results of the experiment for respondents with textual Web browsers, see Vehovar & Batagelj (1996).

4. Some of the first survey questions were questions on how (what type of connection), through which Internet Service Provider and from where respondents access the Internet while answering the questionnaire. Answers to these questions were used to distinguish between respondents who had free access and those who had to pay for it while answering our questionnaire.

5. In an intercept survey visitors of a Web site are invited to answer a questionnaire on the Web. The invitation appears in a pop-up window (therefore these are also called ‘pop-up’ surveys; Comley, 2000) or on a banner ad. In the first case a new browser window pops up at the top of the existing window. It already includes the questionnaire or only the invitation to participate in a survey.

References

Andrews, S., & Feinberg, S. (1999). Developing and implementing effective Web-based surveys. STC’s 46th annual conference proceedings. Cincinnati, Ohio: Sawtooth Technologies. Retrieved from the World Wide Web: http://www.stc-va.org/proceedings/ConfProceed/1999/PDFs/046.pdf (November 21, 2001).

Aoki, K., & Elasmar, M. (2000, My). Opportunities and challenges of a Web survey: A field experiment. Paper presented at the 55th annual conference of American association for public opinion research. Portland, Oregon, May 18-21, 2000.

Baker, R. P. (1998). The CASIC future. In M. P. Couper, R. P. Baker, J. Bethlehem, C. Z. F. Clark, J. Martin, W. L. Nicholls II, & J. M. O’Reilly (Eds.). Computer assisted survey information collection (pp. 583-604). New York, NY: John Wiley & Sons.

Bauman, S., Jobity, N., Airey, J., & Atak, H. (2000, May). Invites, intros and incentives: Lessons from a Web survey. Paper presented at the 55th annual conference of American Association for Public Opinion Research. Portland, Oregon, May 18-21, 2000. Retrieved from the World Wide Web: http://www.wintellitech.com (November 21, 2001).

Bickham, C., & Moore, C. (1999). Combining telephone, mail and the Internet for effective global business to business data capture and management — A case study. Paper presented at the ARF's Week of Workshops, Online Research Workshop. New York, NY, October 4-7 1999.

Black, G. S. (1998, May). Internet surveys – A replacement technology. Paper presented at the 52nd Annual Conference of the American Association for Public Opinion Research. May 1998, St. Louis, Missouri.

Bosnjak, M., Tuten, T. L., & Bandilla, W. (2001). Participation in Web surveys - A typology. ZUMA Nachrichten, 48, 7-17.

Bowker, D. (1999). Constructing the client-computer interface: Guidelines for design and implementation of Web-based surveys. Summary report 99-15. Pullman, WA: Social and Economic Sciences Research Center, Washington State University.

Bowker, D., & Dillman, D. A. (2000, May). An experimental evaluation of left and right oriented screens for Web questionnaires. Paper presented at the 55th annual conference of American Association for Public Opinion Research. Portland, Oregon, May 18-21, 2000. Retrieved from the World Wide Web: http://survey.sesrc.wsu.edu/dillman/papers/AAPORpaper00.pdf (November 11, 2001).

Brennan, M., Rae, N., & Parackal, M. (1999). Survey-based experimental research via the Web: Some observations. Marketing Bulletin, 10, 83-92. Retrieved from the World Wide Web: http://marketing-bulletin.massey.ac.nz/article10/article9.asp (November 26, 2001).

Bunofsky, T. L. (1999, October). Developing and nurturing an online panel. Paper presented at the ARF's Week of Workshops, Online Research Workshop. New York, NY, October 4-7 1999.

Clayton, R. L., & Werking, G. S. (1998). Business surveys of the future: The World Wide Web as a data collection methodology. In M. P. Couper, R. P. Baker, J. Bethlehem, C. Z. F. Clark, J. Martin, W. L. Nicholls II, & J. M. O’Reilly (Eds.). Computer assisted survey information collection (pp. 543-562). New York, NY: John Wiley & Sons.

Cleland, K. (1996). On-line research costs about half that of traditional methods. Advertising Age’s Business Marketing, 81 (4), B8-9.

Comley, P. (1996, November). The use of Internet as a data collection method. Paper presented at ESOMAR conference. Edinburg, November, 1996. Retrieved from the World Wide Web: http://www.virtualsurveys.com/papers/email.htm (November 21, 2001).

Comley, P. (2000). Pop-up surveys. What works, what doesn’t work and what will work in the future. Proceedings of the ESOMAR worldwide Internet conference Net Effects 3. Publication series - Volume 237. Amsterdam, NL: ESOMAR. Retrieved from the World Wide Web: http://www.virtualsurveys.com/papers/popup_paper.htm (November 21, 2001).

Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in Web- or Internet-based surveys. Educational and Psychological Measurement, 60 (6), 821-836.

Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464-494.

Couper, M. P., & Nicholls II, W. L. (1998). The history and development of computer assisted survey information collection methods. In M. P. Couper, R. P. Baker, J. Bethlehem, C. Z. F. Clark, J. Martin, W. L. Nicholls II, & J. M. O’Reilly (Eds.), Computer assisted survey information collection (pp. 1-21). New York, NY: John Wiley & Sons.

Couper, M. P., Traugott, M., & Lamias, M. (1999). Effective survey administration on the Web. Paper presented at the 1999 MAPOR conference, Chicago, Illinois.

Couper, M. P. (1997). Survey introductions and data quality. Public Opinion Quarterly, 61 (2), 317-338.

Couper, M. P. (2001, September). Measurement error in Web surveys. Invited lecture at Methodology and Statistics International conference. Ljubljana, Slovenia, September 17-19, 2001.

Couper, M. P., Traugott, M., & Lamias, M. (2000, October). Experiments on the design of Web surveys. Paper presented at the Fifth International Conference on Social Science Methodology, Cologne, Germany, October 3-6, 2000.

Couper, M. P., Traugott, M., & Lamias, M. (2001). Web survey design and administration. Public Opinion Quarterly, 65(2), 230-253.

de Leeuw, E. D. (1992). Data quality in mail, telephone and face-to-face surveys. Amsterdam, NL: TT-Publikaties.

Dillman, D. A. (1978). Mail and telephone surveys. The total design method. New York, NY: John Wiley & Sons.

Dillman, D. A. (2000). Tailored design of mail and other self-administered surveys. New York, NY: Wiley-Interscience.

Dillman, D. A., & Bowker, D. K. (2001). The Web questionnaire challenge to survey methodologists. In U.-D. Reips, & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 159-178). Lengerich: Pabst Science Publishers.

Dillman, D. A., Carley-Baxter, L., & Jackson, A. (1999). Skip pattern compliance in three test forms: A theoretical and empirical evaluation. SESRC technical report 99-01, Pullman, WA: Washington state University.

Dillman, D. A., Tortora, R. D., & Bowker, D. (1998). Influence of plain vs. fancy design on response rates for Web surveys. The 1998 Proceedings of Section on Survey Research Methods. Dallas, Texas: American Statistical Association. Retrieved from the World Wide Web: http://survey.sesrc.wsu.edu/dillman/papers/asa98ppr.pdf (November 26, 2001).

Elder, A. (1999). Power buyers or cautious shoppers? Segmenting the E-commerce market. Proceedings of the ESOMAR Worldwide Internet Conference Net Effects (pp. 15-34). Amsterdam, NL: ESOMAR.

Elder, A., & Incalcatera, T. (2000). Pushing the envelope. Moving a major syndicated study to the Web. Proceedings of the ESOMAR Worldwide Internet Conference Net Effects 3. Publication Series - Volume 237. Amsterdam, NL: ESOMAR.

Enander, J., & Sajti, A. (1999). Online survey of online customers. Value-added market research through data collection on the Internet. Proceedings of the ESOMAR Worldwide Internet Conference Net Effects (pp. 35-51). Amsterdam, NL: ESOMAR.

Farmer, T. (1998). Using the Internet for primary research data collection. Market Research Library. Retrieved from the World Wide Web: http://www.researchinfo.com/library/infotek/index.shtml (April 28, 1999).

Findlater, A., & Kottler, R. E. (1998). Web interviewing. Validating the application of Web interviewing using a comparative study on the telephone. The Book of Papers from the Worldwide Internet Seminar 1998. Amsterdam, NL: ESOMAR.

Fuchs, M. (2001, May). Screen design in a Web survey. Paper presented at the 55th Annual Conference of the American Association for Public Opinion Research. Montreal, Quebec, May 17-20, 2001.

Gräf, L. (2002). Assessing Internet questionnaires: The online pretest lab. In B. Batinic, U.-D. Reips, M. Bosnjak. & A. Werner (Eds.), Online social sciences (pp. 73-93). Seattle: Hogrefe & Huber (Forthcoming).

Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. New York, NY: John Wiley & Sons.

Groves, R. M. (1989). Survey errors and survey costs. New York, NY: John Wiley & Sons.

Gupta, S. (1999). Online research: Results from the first Acorn-IIR survey of online research. Retrieved from the World Wide Web: http://www.iir.acornis.com/survey (January 5, 2001).

Heberlein, T. A., & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, 447-462.

Hollis, N. S. (1999). Can a picture save 1,000 words? Augmenting telephone tracking with online ad recognition. ARF's Online research day - Towards validation (pp. 41-49). New York, NY: Advertising Research Foundation.

Internet Rogator (1998). Internet Rogator help with surveying. Retrieved from the World Wide Web: http://www.internet-rogator.com/htm/help.htm (November 21, 2001).

Jansen, J. H. (1985). Effect of questionnaire layout and size and issue-involvement on response rates in mail surveys. Perceptual and Motor Skills, 61, 139-142.

Jeavons, A. (1999). Ethology and the Web. Observing respondent behaviour in Web surveys. Proceedings of the ESOMAR worldwide Internet Conference Net Effects (pp. 201-213). Amsterdam, NL: ESOMAR. Retrieved from the World Wide Web: http://w3.one.net/~andrewje/ethology.html (November 21, 2001).

Jones Thompson, M. (1999). When market research turns into marketing. The Industry Standard, August 23rd. Retrieved from the World Wide Web: http://www.thestandard.com/article/0,1902,5995,00.html (March 30, 2002).

Knapp, F., & Heidingsfelder, M. (2001). Drop-out analysis: Effects of the survey design. In U.-D. Reips, & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 221-230). Lengerich: Pabst Science Publishers.

Kojetin, B. A., Borgida, E., & Snyder, M. (1993). Survey topic involvement and nonresponse bias. 1993 Proceeding of the Section on Survey Research Methods. Volume II (pp. 838-843). Alexandria: American Statistical Association.

Kottler, R. E. (1997a, October). Exploiting the research potential of the World Wide Web. Paper presented at Research '97, London, October, 1997. Retrieved from the World Wide Web: http://www.spss.com/spssmr/web_bureau/knowledge/r97a.htm (November 21, 2001).

Kottler, R. E. (1997b). Search no further. Yahoo!'s audience analysis project - A case study. Quirk's Marketing Research Review, June/July. Retrieved from the World Wide Web: http://www.spss.com/spssmr/web_bureau/knowledge/yahooqq.htm (November 21, 2001).

Krotki, K. P. (2001, August). Web-based surveys for a nationally representative sample. Paper presented at Joint Statistical Meeting 2001, Atlanta, Georgia, 5-9 August 2001.

Kwak, N., & Radler, B. T. (1999, November). A comparison between mail and Web-based surveys: Response pattern, data quality, and characteristics of respondents. Paper presented at 1999 MAPOR Conference, Chicago, Illinois, 19-20 November 1999.

Lozar Manfreda, K. (2001). Web survey errors. Unpublished Ph.D. dissertation. Ljubljana: Faculty of Social Sciences, University of Ljubljana.

MacElroy, B. (2000). Measuring response rates in online surveys. Quirk's Marketing Research Review, April. Retrieved from the World Wide Web: http://www.quirks.com (November 21, 2001).

Martin, C. L. (1994). The impact of topic interest on mail survey response behaviour. Journal of the Market Research Society, 36(4), 327-338.

Nadilo, R. (1999). Online research: The methodology for the next millennium. ARF's online research day - Towards validation (pp. 50-51). New York, NY: Advertising Research Foundation.

Nichols, E., & Sedivi, B. (1998). Economic data collection via the Web: A Census Bureau case study. 1998 Proceedings of the Section on Survey Research Methods (pp. 366-371). Alexandria: American Statistical Association.

Norman, K. L., Friedman, Z., Norman, K., & Stevenson, R. (2001). Navigational issues in the design of online self-administered questionnaires. Behaviour and Information Technology, 20 (1), 37-45.

Onyshkevych, V., & McIndoe, D. (1999, May). Internet technology: Gaining commercial advantage. Paper presented at the 54th Annual Conference of the American Association for Public Opinion Research. St. Petersburg, Florida, 13-16 May, 1999.

Pastore, M. (2001). Online consumers now the average consumer. The cyberAtlas newsletter, July 2001. Retrieved from the World Wide Web: http://cyberatlas.internet.com/big_picture/demographics/article/0,,5901_800201,00.html (November 21, 2001).

Poynter, R. (2001). A guide to best practice in online quantitative research. In A. Westlake et al. (Eds.). The challenge of the Internet. London, UK: Association for Survey Computing.

Reips, U.-D. (2002). Context effects in Web-surveys. In B. Batinic, U.-D. Reips, M. Bosnjak. & A. Werner (Eds.), Online social sciences (pp. 95-104). Seattle: Hogrefe & Huber (Forthcoming).

RIS (1996-2002). Research on Internet in Slovenia. Faculty of Social Sciences, University of Ljubljana. Retrieved from the World Wide Web: http://www.ris.org (March 30, 2002).

Roberson, M. T., & Sundstrom, E. (1990). Survey design, return rates, and response favorableness in an employee attitude survey. Journal of Applied Psychology, 75, 354-357.

Sheehan, K. B., and McMillan, S. J. (1999). Response variation in e-mail surveys: An exploration. Journal of Advertising Research, 39(4), 45-54.

Spaeth, J. (1999). At the dawn of a new day. ARF's online research day - Towards validation (pp. 5-7). New York, NY: Advertising Research Foundation.

Spain, S. W. (1998). Top-10 Web survey issues and how to address them. Market Research Library. Retrieved from the World Wide Web: http://www.researchinfo.com/library/top_10_web.shtml (April 28, 1999).

Sturgeon, K., & Winter, S. (1999). International marketing on the World Wide Web. New opportunities for research: What works, what does not and what is next. Proceedings of the ESOMAR Worldwide Internet Conference Net Effects (pp. 191-200). Amsterdam, NL: ESOMAR.

Terhanian, G. & Black, G. S. (1999). Understanding the online population: Lessons from the Harris Poll and the Harris Poll Online. ARF's online research day - Towards validation (pp. 28-33). New York, NY: Advertising research foundation.

Vehovar, V., & Batagelj, Z. (1996 ). The methodological issues in WWW surveys. Paper presented at CASIC ’96, San Antonio, Texas. Retrieved from the World Wide Web: http://www.ris.org/casic96/ (March 30, 2002).

Vehovar, V., Batagelj, Z., Lozar Manfreda, K., & Zaletel, M. (2002). Nonresponse in Web surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 229-242). New York,NY: John Wiley & Sons.

Vehovar, V., Lozar Manfreda, K., & Batagelj, Z. (2000). Design issues in WWW surveys. 2000 Proceedings of the Section on Survey Research Methods (pp. 983-988), Alexandria: American Statistical Association.

White, G. K. (1996). An on-line survey of food and beverage consumers on the Internet: An evaluation of the survey methodology. Marketing Research On-line, 1, 39-59. Retrieved from the World Wide Web: http://mro.massey.ac.nz/online3.pdf (November 11, 2001).

Whitlock, C. (1997, September). CAWI - Computer assisted Web interviewing. Tautology or innovation? Paper presented at a one-day conference ‘The interview, the Internet, the impact’, organized by ASC, September 30, 1997, London.

Willke, J., Adams, C. O., & Girnius, Z. (1999). Internet testing. A landmark study of the differences between mall intercept and on-line interviewing in the United States. Proceedings of the ESOMAR Worldwide Internet Conference Net Effects (pp. 145-157). Amsterdam, NL: ESOMAR.

Zukerberg, A., Nichols, E., & Tedesco, H. (1999, May). Designing surveys for the next millennium: Internet questionnaire design issues. Paper presented at the 54th Annual Conference of the American Association for Public Opinion Research, St. Petersburg, Florida, 13-16 May, 1999.

About the Authors

Katja Lozar Manfreda (Ph.D., University of Ljubljana) is a postdoctoral researcher at the Faculty of Social Sciences, University of Ljubljana. Her research interests include survey methodology, especially methodology of Web surveys, and particularly nonresponse in Web surveys. She maintains a Web site on WWW survey methodology (http://www.websm.org) which houses a large collection of papers and resources on Web survey data collection.
Address: CATI Center, Trzaska 2, 1000 Ljubljana, Slovenia. Phone: +386 1 241 0072

Zenel Batagelj is a Ph.D. student at the Faculty of Social Sciences, University of Ljubljana, and research director at a private research company CATI Center, which is a leading market research company for telephone and Web surveys in Slovenia, and also the partner of Gallup International. He developed his own software for integrated computer support for all modes of surveying. His research interests include methodology of Web surveys, particularly design issues, as well as survey measurement of information-communication technologies.
Address: CATI Center, Tr~aaka 2, 1000 Ljubljana, Slovenia. Phone: +386 61 241 0072.

Vasja Vehovar is a professor at the Faculty of Social Sciences, University of Ljubljana. He was educated at the University of Ljubljana (Ph.D.), the University of Essex (M.A.), and he was a Fullbright scholar at the University of Michigan, Institute for Social Research. His research interests include survey methodology, particularly survey sampling and Web survey methodology. He also conducts substantial research on the Internet and e-commerce. Since 1996 he has been principal investigator of the national research on Internet in Slovenia (www.ris.org). He published in Journal of American Statistical Association, Journal of Official Statistics, as well as chapters in the books of major publishers (Wiley, Greenwood Publishing Group).
Address: Faculty of Social Sciences, University of Ljubljana, Kardeljeva ploa ad 5, 1000 Ljubljana, Slovenia. Phone: +386 1 5805 297 Fax: +386 1 5805 100.

©Copyright 2002 Journal of Computer-Mediated Communication