Abstract

While much research has shown that asynchronous learning networks (ALNs) can produce learning equivalent to face-to-face (FTF) classrooms, there has been little research that explicitly explores similarities and differences between the learning processes that occur in ALN and FTF activities. This study used a content analytic framework (derived primarily from previous work of Anderson, Archer, Garrison, and Rourke) to analyze transcripts from eight case study discussions, four FTF and four ALN. While previous authors developed a model that studies cognitive, social, and teaching processes in ALN discussions, the current scheme also considers characteristics of the discourse process. The findings provide evidence that ALNs generate high levels of cognitive activity, at least equal to, and in some cases superior to, the cognitive processes in the FTF classroom. The findings also suggest that students assume some aspects of the teacher's role in ALNs, and that student-to-student interactions contain a greater proportion of high-level cognitive indicators than do student-to-teacher interactions.

Introduction

Many researchers have argued that asynchronous learning networks (ALNs) provide an excellent vehicle for discussion-based learning activities (Benbunan-Fich & Hiltz, 1999). Some of the reasons cited for this assertion are increased reflection time, more democratic participation, benefits attributable to writing, etc. (Benbunan-Fich & Hiltz, 1999; Harasim, 1990). While much research has shown that ALNs can produce learning equivalent to or better than face-to-face (FTF) classrooms (e.g., Alavi, 1994; Alavi, Yoo, & Vogel, 1997; Benbunan-Fich & Hiltz, 1999; Hiltz, 1994), there has been little empirical research that explicitly and rigorously explores the similarities and differences between the learning processes that occur in ALN and FTF activities. There is a lack of rigorously obtained data that shows in detail how FTF and ALN discussions are conducted, and how they achieve their respective effects. One form of discussion-based learning is the case study method of instruction. While research suggests that students working in asynchronous networked environments can produce high quality case study solutions (e.g., Benbunan-Fich & Hiltz, 1999), a similar problem exists—we have little detailed understanding of how case study discussions work in each mode. This paper addresses these needs by presenting the results of a study that rigorously compared case study discussions in both FTF and ALN modes.

The study had three objectives:

1. Descriptive. To provide a rich, detailed, descriptive comparison of actual case study discussions in both FTF and ALN modes.
2. Methodological. To expand our understanding of several content analysis approaches for analyzing ALN discussions.
3. Pedagogical. To explore methods for improving the conduct of case study discussions in ALN mode.

To achieve these objectives, we observed eight case study discussions: four FTF, four ALN. The study was designed to reduce and control for systematic order effects, group composition effects, and effects due to differences between the two cases used as discussion stimuli. The same instructor conducted all eight discussions.

We used a content analytic framework derived primarily from the previous work of Anderson, Archer, Garrison, and Rourke (see Anderson, Rourke, Archer, & Garrison, 2001; Garrison, Anderson, & Archer, 2000, 2001; Rourke, Anderson, Garrison, & Archer, 1999). These authors have been developing analysis tools intended to be efficient, valid, reliable, and practical for the use of researchers and teachers alike. They are designed to evaluate the learning process of individuals collaborating using ALN (Garrison et al., 2000). Building on the work of scholars focusing on social interdependence, critical thinking, and constructivist learning (Hass, 1996; Hiltz & Turoff, 1993; Newman, Johnson, Cochrane, & Webb, 1996; Olson, 1994; White, 1993), Garrison et al. (2000) presented a model that studies cognitive, social, and teaching processes. Based on the work of Aviv (2000), our scheme also considers characteristics of the discourse process.

This paper reports primarily on objectives 1 and 2 above. We report on the third objective in a separate paper (Heckman & Annabi, 2004). In the next section, we provide background for the study and describe our research framework. The section after that describes the content analytic method used. The two final sections present results, and discuss the implications of our findings.

Background

The Case Study Method of Instruction

The case study method is used in many settings in which professionals are trained, including management, medicine, law, and education. It is a collaborative and constructivist learning technique because students are expected to consolidate their learning by teaching one another (Ragan, 1996). The purpose of case study discussions is to allow each student to help other students in the class gain a new perspective on case events.

Research on collaborative learning in technology-mediated contexts suggests that online collaborative learning can produce results comparable to or better than those in face-to-face environments (Alavi, 1994; Alavi et al., 1997; Benbunan-Fich & Hiltz, 1999; Hiltz, 1994). But while ALN results can be equivalent or better, ALN processes may be significantly different than in FTF situations. For example, Curtis and Lawson (2001) found that while there were substantial instances of collaboration in the ALN activities they observed, the nature of these collaborative behaviors was different than in FTF collaborative learning. Heckman et al. (2000) found that the number of roles is reduced and the role structure simplified when technology is the primary means of group interaction. These studies suggest that the critical success factors for asynchronous collaborative learning may be different than in FTF environments. For this reason, it is important to understand in detail how the dynamics of FTF and ALN case study discussions are similar or different.

Content Analysis of ALN Discussions

Prior to 1992, ALNs were commonly studied using surveys, interviews, empirical experimentation, participant observation, and case study methodologies (Mason, 1992). Mason (1992) pointed out that researchers had ignored the learning indicators available in the content of transcripts of ALN discussions and attempted to identify the skills and abilities ALN participants demonstrated in the discussions as indicators of learning. Henri (1992) also recognized the richness of the ALN transcript in indicators of learning processes. She argued that in order to assess the use of computer-mediated communication (CMC) in education, a detailed content analysis framework was needed.

Henri (1992) built a model containing three levels: the first addressing the product of learning, the second and third addressing the process of learning. There are five dimensions to this framework: participative, social, interactive, cognitive, and metacognitive. This framework served as the launching pad for the use of content analysis to analyze ALN content by many other researchers. Scholars like Hass (1996), Hiltz and Turoff (1993), Newman et al. (1996), Olson (1994), and White (1993) built on Henri's framework and ideas to improve our understanding of ALNs.

In recent years, Anderson, Archer, Garrison and Rourke (Anderson et al., 2001; Garrison et al., 2000, 2001; Rourke et al., 1999) have developed an analysis tool designed to evaluate the learning process of individuals collaborating using ALNs that is intended to be efficient, valid, reliable, and practical for the use of researchers and teachers alike. Building on the work of scholars focusing on social interdependence, critical thinking, and constructivist learning theories (Garrison et al., 2000; Hass, 1996; Hiltz &Turoff, 1993; Newman et al., 1996; Olson, 1994; White, 1993), they presented a model that studies cognitive, social, and teaching processes (or presence, as they refer to their dimensions). Garrison et al. (2000) argue that from the presence of overlapping social, teaching, and cognitive processes, a community of collaborative inquiry emerges. Their framework identifies the intellectual content of messages (cognitive presence), the instructional role (teaching presence), as well as the interaction among the members (social presence). Aviv (2000) also developed a framework to analyze the content of messages and the nature of interactions that builds on work of scholars focusing on critical thinking, constructivist learning and social interdependence theories (e.g., Henri, 1992; Hiltz & Turoff, 1993; Johnson & Johnson, 1981; Newman et al., 1996). Aviv's framework identifies three processes present in ALN discussions: social process, response process and reasoning process.

Research Framework

We use the research framework presented below to think about the processes that occur during the conduct of a learning activity such as a case study discussion. It describes this activity in terms of antecedent variables, structuring processes influencing learning outcomes, and various other outcomes. Some of the antecedent variables that might influence any learning activity include, but are not limited to, the technology used, the structure of the group, students' prior experience, and the teacher's design of the task. These antecedents may influence a set of four interdependent structuring processes that occur while the activity is taking place. They include social, teaching, cognitive, and discourse processes, and it is the digital traces of these processes that we observe when we analyze transcripts of ALN or FTF discussions. The structuring processes lead to a variety of outcomes which may include product quality, other indicators of performance (e.g., grades), others' opinions, satisfaction, and learning.

We developed a content analytic framework to study the structuring processes, as presented in Figure 1. We attempted to create a unified framework based on previous work that could be used to analyze the content of both ALN and FTF discussions. The framework we developed is based on the schemes used by Anderson, Archer, Garrison, and Rourke (see Anderson et al., 2001; Aviv, 2000; Garrison et al., 2000, 2001; Rourke et al., 1999), which represent the most current integration of past work on critical thinking, constructivist learning, and social interdependence theories. Preliminary data suggested the addition of certain themes that previous research did not account for (as presented in more detail in the following sections describing the framework). Thus, the final framework is an integration of the works of Anderson, Archer, Garrison and Rourke (Anderson et al., 2001; Aviv, 2000; Garrison et al., 2000, 2001; Rourke et al., 1999). It uses four interdependent process dimensions to characterize the learning process: social, teaching, cognitive, and discourse. The coding scheme is progressively elaborated in three hierarchical levels, as described below.

First Level: Four Structuring Processes

The presence of cognitive, social, and teaching activities, shaped by a particular style of discourse, is what creates the learning community. Both Aviv (2000) and Garrison et al. (2000) consider these elements to be overlapping or interdependent. We present the discourse process as a separate category merely for analytical purposes. The discourse process is at the heart of social, cognitive, and teaching processes shaping a community of inquiry. While Garrison et al. (2000) refer to their three categories as presences, we adopt the terminology used by Aviv (2000), processes, to refer to the dynamic and interdependent nature of these behaviors. While "presence" may be a more commonly used term, we prefer "process," since our focus is to describe the dynamic actions which generate cognitive, social, and teaching presence.

Cognitive Process

The cognitive presence (Garrison et al., 2000, 2001) and reasoning process (Aviv, 2000) are similar in both frameworks in that they identify activities used to construct meaning and achieve learning. In their frameworks, Garrison et al. (2000, 2001) and Aviv (2000) attempt to "assess the systematic progression of thinking over time (Garrison et al., 2001, p. 11). Garrison et al. (2000, 2001) arrange their indicators based on the practical inquiry model which includes four phases: triggering, exploration, integration, and resolution. Similarly, Aviv arranges the indicators of cognitive process in a hierarchal fashion to suggest an increasing level of skill and complexity, a practice we adopted in our framework. The cognitive element is perhaps the closest structuring process to an outcome indicator for educational purposes, since critical thinking and the achievement of higher levels of learning are assumed to be the goals of education. The other processes may be considered to be processes that support this goal (Aviv, 2000; Garrison et al., 2000, 2001).

Social Process

The social process aims to capture interpersonal characteristics and group cohesiveness (Aviv, 2000; Garrison et al., 2000). Social interdependence theory of cooperative learning (developed by Johnson & Johnson, 1981) suggests that cooperative learning promotes higher-level reasoning (Aviv, 2000). This is especially important in the asynchronous computer-mediated setting since the technology can hide personal characteristics that are usually more readily apparent in the FTF setting. Aviv (2000) employs broad and general categories for the social process, indicating whether or not there is a social response present or not. Rourke et al. (1999) define specific indicators to measure the characteristics of the social interaction.

Teaching Process

In the analytic scheme of Garrison et al. (2000), this element identifies aspects of both the design of the learning experience, as well as its delivery and facilitation (a detailed description of this element is found in Anderson et al., 2001). Either instructors or learners can carry out these activities; however, instructors usually perform the role. The instructional design and organization element of their scheme was not adopted in this framework for two reasons. First, we controlled the design process by holding it constant across observations. Second, since our framework focuses on the structuring processes that occur during the educational activity, we consider the design one of many possible antecedents that can affect these structuring processes.

Discourse Process

Aviv (2000) introduced the response process as his third element in the framework. This element is to measure the content-relevant communication between learners and instructors. This is a response to the social interdependence theory of collaborative learning that suggests that responses from learners to learners differ than those responses from learners to instructors. The theory suggests that responses of learners to learners are at the highest level, from a cognitive perspective, preceded by responses of learner to instructor and the last in the non-responsive utterance (Aviv, 2000). Again, this element supports the cognitive process. Because we expanded this process to include other attributes of discourse, we refer to it as the discourse process in our framework.

Second Level: Sub-Categories

Table 1 (below) shows the second level sub-categories that further group indicators within each of the four major processes.

Aviv (2000) includes a social process in his model, but does not identify specific indicators of social process. For that reason, we chose to adapt Rourke et al. (1999) without modification. Rourke et al. (1999) identified the three categories of affective response, cohesive response, and interactive response.

Our framework combines the cognitive process indicators developed by Garrison et al. (2000, 2001) and those of Aviv (2000). While two of the second level categories we adapt are those of Garrison et al. (2000), we do include some of the basic third level indicators that were developed by Aviv (2000) in addition to some indicators of our own that emerged from the data. The cognitive process includes three categories on the second level. The first two, exploration and integration, were developed by Garrison et al. (2000), while the third, analysis, was introduced by us. These three categories represent different levels of learning. Consistent with Aviv's approach, the second level categories are ordered to illustrate the hierarchal nature of the levels of learning each indicator measures. The category we named analysis was introduced to fill the gap between exploration and integration. Analysis is the stage where frameworks are applied and problems are identified systematically. We include the triggering event (Garrison et al., 2001) in the exploratory (Garrison et al., 2001) stage. Since the main stimulus (the case study) is presented by the teacher, we consider the triggering events to be part of the exploratory stage where the learners explore the details of the case and identify the root problems. We introduce the analysis to account for students' use of frameworks to evaluate the ideas generated in the exploratory stage. We combine resolution (Garrison et al., 2001) under the integration (Garrison et al., 2001) stage as the end result of this stage.

The direct instruction and the facilitating discourse sub-categories of the teaching process were adopted in our framework with no modifications from (Anderson et al., 2001).

Discourse process includes two second-level categories, the response process developed by Aviv (2000), and discourse characteristics that we introduced. Response process measures the interactions occurring by identifying the speakers and targets of utterances. The discourse process, on the other hand, was developed to highlight several linguistic attributes of the discourse. We coded the discourse process simply. However, this process might be expanded in future analyses to include other aspects of discourse.

Third Level (Specific Indicators)

The third level includes the specific indicators in each second level category described above. These indicators or codes were applied directly to the data in the transcripts. As indicated above, most third level indicators were adopted directly from previous work (Anderson et al., 2001; Aviv, 2000; Garrison et al., 2000, 2001; Rourke et al., 1999), but we introduced five additional codes. In the cognitive process we introduced two indicators: rote factual response and analysis. These two additional indicators emerged inductively from early coding efforts. Their utility emerged after we began to think of the cognitive process indicators in a hierarchy, as suggested by Aviv (2000). In the discourse process we added three indicators: Identifying the speaker of an utterance (student or teacher), instances of informal speech, and passive voice. Appendix 1 presents the specific codes in the third level and gives definitions and examples of each.

Methods

We observed 120 seniors in a large university in the northeast United States during two case study discussions. They were enrolled in two sections of the capstone course for the B.S. in Information Management degree program. The sections contained 53 and 67 students. Since the capstone course is case-based, these discussions were normal course activities. Each student participated in two discussions. One discussion took the form of a traditional, FTF, in-class discussion. The other was conducted asynchronously, using the bulletin board feature of WebCT, a web-based instructional tool utilized in the course.

For these discussions, each section was randomly divided into two equal subgroups. Section A was divided into A1 (27 students) and A2 (26 students.) Section B was divided into B1 (33 students) and B2 (34 students.) This division into four smaller subgroups allowed us to observe eight individual discussions: four in FTF mode, and four in ALN mode. It also permitted us to control for systematic order effects, group composition effects, and effects due to differences between the two cases used as discussion stimuli. The observation period lasted two weeks, allowing one week for each discussion. In week 1 all four groups discussed Case 1 (2 FTF mode, and 2 ALN mode.) In week 2, all four groups discussed Case 2 (each group changing discussion mode.)

The first author was the class instructor, and the discussion facilitator in both mediums. The instructional goals for each discussion mode were identical, and each discussion was structured by the facilitator into three sequential segments, with identical starting and transitioning questions used in each mode. The facilitator made an effort to cover the issues in both mediums in as similar a fashion as possible. To control for the differences that might arise from the facilitator's interactions with the different groups and over the different mediums, we constructed the following facilitator guidelines.

• Adopt the same strategies in calling on individuals in both classes
• Cover the three broad sections in similar ways:
  • Use the same starting and transition questions
  • Allocated time for each of the three sections should be of approximately equal proportions in FTF and ALN discussions, and of approximately equal time in both FTF sessions.
• Do not direct student answers to go in the same way as in the other medium or other section
• Do not bring up points initiated by the students of another section or medium

The students were informed of the guidelines and details of the study in an earlier class period. They were told that discussions would be graded for participation in the normal fashion by the course graduate assistant. They were also informed that both discussions would be transcribed, that a doctoral student would analyze the texts, that individual student identities would not be used in the analysis, that the results of the analysis available to the professor would contain no student identification, and that the analysis would have no impact on their grades.

In-class discussions were recorded and transcribed. The second author also took detailed notes of FTF discussions, and the notes were used to verify the quality of the recording and transcription. A test recording of the FTF discussions was made in classes prior to the dates of the live observations in order to minimize problems with student reactions to being recorded, and to improve the quality of recording. Complete texts of ALN discussions were extracted from WebCT logs. The transcripts were analyzed by the second author using the coding scheme described above. A second coder was trained to use the coding scheme, and recoded randomly selected portions of the transcripts. The inter-rater agreement was 86% of all coding decisions.

Results

Discourse Process

Table 3 presents a comparison of the discourse processes in FTF and ALN case study discussions, and provides a first indication of how these two modes differ.

The first and most obvious observation is the sheer difference in the number of individual utterances. In the average FTF discussion, there were 287 individual utterances, compared to 74 in the average ALN discussion. The FTF discussion was much more "back and forth" in nature, with the teacher asking questions and students responding, as shown by the relatively equal number of utterances by teacher (141) and students (146).

The presence of the teacher was much more pervasive in the FTF discussions, averaging 141 utterances compared to an average of 11 utterances in each ALN discussion. In the ALN discussions, students carried a much greater share of the discourse. The ratio of student/teacher utterances was 5:1 in ALN, compared to 1:1 in the traditional classroom. In addition, student utterances were longer in ALN (100 words versus 30 words), while teacher utterances were shorter (50 words versus 80 words.)

FTF discussions employed much more informal language and active voice construction, while ALN discussions were more formal and employed much more passive voice. There was relatively little difference between FTF and ALN in the proportions of non-responsive utterances, responses to teacher, and responses to student.

Social Process

Table 4 presents a comparison of the social processes that occurred in these FTF and ALN case study discussions. While there were a greater number of social processes observed in the average FTF discussion (154 versus 124), a larger percentage of ALN utterances contained social indicators. As indicated in Table 5, approximately half of all FTF utterances contain social indicators, while on average, every ALN utterance includes 1.6 instances of social process.

Proportions of the three major categories of social process (affective response, cohesive response, interactive response) were very similar in each mode. The nature of the interactive responses in each mode, however, illustrates a fundamental difference between them. The FTF discussions were much more question-driven (with virtually all questions coming from the teacher), while there was a much greater incidence of continuing a thread in the ALN discussions.

Another interesting difference in the social process between ALN and FTF discussions is that students played a bigger role in creating a social environment in the ALN discussions than they did in the FTF discussions. As indicated in the Table 6 below, there were 104 incidents of social processes in ALN versus 10 in FTF by students. Students referred to each other by name, and replied directly to each other's thoughts, expressed agreement, etc. This social expression by students appeared to be limited by the teacher's presence in the FTF discussions. The teacher, however, had far fewer incidents of social process indicators in ALN than FTF. The teacher presented 20 social process indicators in ALN versus 134 in FTF.

Teaching Process

It is clear that there were many more examples of traditional "teaching" in the FTF discussions. Table 7 shows that, on average, there were 125 instances of direct instruction in FTF, while there were only 18 in ALN. While the majority of these instances were to confirm understanding, the average FTF discussion contained 15 instances of presenting content, while the average ALN discussion contained only two. There were also examples of focusing the discussion in the FTF discussion, which did not occur in the online mode. These focusing actions can be distinguished from larger, preplanned transitions in the discussions, which were coded as discussion strategy, and which occurred in both modes.

There were also more instances of facilitating discourse in the average FTF discussion. Most of these instances took the form of drawing in participants, which typically took the form of calling on specific students, often as "cold calls," a phenomenon that did not occur at all in the online discussions.

Finally, there appeared to be increased occurrence of identifying agreement and disagreement in the online discussions. Closer inspection revealed that, in the online discussions, virtually all of these instances were performed by students themselves, and not by the teacher. In fact, 14 of the 26 instances of Teaching Process (54%) in the average ALN discussion were performed by students. In the average FTF discussion, however, only 8 of 148 instances of Teaching Process (5%) were performed by students.

Cognitive Process

In the average FTF discussion we observed nearly twice as many instances of cognitive process as in the average ALN discussion (139 versus 71). However, Table 8 indicates that the distribution of the instances presents an interesting comparison.

In FTF discussions, the instances of cognitive process were predominantly in the lower-order exploration category. They consisted mainly of rote factual response and information exchange, almost entirely in direct response to questions from the teacher about the "facts of the case." In FTF discussions, exploration instances accounted for 70% of all cognitive instances, compared to 17% in ALN discussions. This suggests that a more leisurely process of information exchange, potentially rich in detail, occurred in the FTF discussions.

In contrast, the ALN discussions contained more high-level Cognitive Process instances, both in absolute and relative terms. The most striking difference was in the Analysis category, with nearly twice as many instances occurring in the ALN discussions. Interestingly, the number of instances of the highest level Cognitive Process, integration, was identical in both modes, suggesting that students were able to synthesize the facts of the case and come to judgment and resolution equally well in both modes.

In addition to differences in frequency of occurrence of different cognitive indicators, there are also interesting differences in the distribution of cognitive indicators over time. To illustrate these differences, we partitioned each discussion into four equal quarters, with each quarter containing 25% of the total discussion utterances. Figure 2 shows that exploration, analysis, and integration indicators occur in different patterns over time in FTF and ALN discussions.

In FTF discussions, exploration of the concrete details of the case is high early in the discussion, while occurrences of analysis and integration remain low. As the discussion progresses, exploration steadily decreases while analysis and integration increase.

In ALN discussions, exploration, analysis, and integration start at approximately equal levels, and analysis and integration continue at steady and relatively high levels relative to exploration throughout. Thus in the ALN discussions, students apparently resisted the instructor's attempts to defer analysis and integration until after the concrete details of the case had been thoroughly explored.

Cognitive activities in peer-to-peer interactions

The social interdependence theory of collaborative learning suggests that responses from students to students may be different than responses from students to instructors. Some research suggests that students-to-student responses may be more likely to be at a higher cognitive level (Aviv, 2000). In this study, student-to-student messages include a higher percentage of analysis and integration indicators than student-to-teacher messages, as indicated in Table 9.

Discussion

The findings show quite clearly that even though the same students discussed the same cases, in discussions led by the same instructor, following identical discussion plans, there were substantial differences between the ALN and FTF discussions. Table 10 summarizes the major findings.

We will first discuss the implications of these differences by exploring three themes: (1) the sequence of dialog, (2) the relative contribution and role of teacher and student, and (3) the nature of cognitive processes in each mode. We will then turn to an assessment of the quality of content analytic studies such as this. We do this in order to suggest future direction for research that includes analysis of computer-mediated transcripts.

Sequence of dialog in FTF and ALN case study discussions

Even though teacher and students were discussing the same cases in each mode, the patterns of dialog in time were very different. The most obvious difference, of course, is that the FTF discussions occurred over 90 minutes, while the ALN discussions occurred over seven days. The FTF discussions were of a steady, linear, turn-taking character (see Figure 3). The teacher [T] asked a question (sometimes preceded by a comment), and a student [S] responded. The alternation of teacher and student was consistent. Each utterance was a direct response, tightly coupled to the immediately previous utterance. Each speaker was often talking directly to someone—students almost always responding to the teacher, teacher often directing a cold-call question, follow-up question, or feedback to a specific student. The time gap between utterances was regular and very short.

The asynchronous dialogs lacked the linear, turn-taking character of the FTF discussion, as shown in Figure 4.

After the teacher initiated the discussions, a number of students responded, sometimes simultaneously, and often with no reference to other student responses. Once a discussion was going, many students might respond to a provocative comment by another student. The teacher occasionally responded to a few student comments, but mainly summarized the discussion and led transitions into new discussion areas. A number of student comments and several teacher comments generated no explicit response. It was possible to have several parallel discussion threads going simultaneously. Several students noted in a follow-up discussion that they "did not have time" to read other student comments before posting, and others complained of duplicated postings. Thus it was not clear that students had received previous utterances in the dialog. The gaps between utterances were irregular.

Role of teacher and student in FTF and ALN case study discussions

These differences in the pattern and sequence of dialog have a profound effect on the conduct of asynchronous case study discussions. Much of the pedagogical literature on the case method instructs the teacher to "choreograph" discussions in order to allow students to make an increasingly complex set of discoveries and syntheses (e.g., Applegate, 1988). Such choreography has traditionally been achieved through an incremental build-up of facts and inferences, and the linear nature of FTF dialog makes this incremental build-up relatively easy to control. The process is typically Socratic in nature, with instructors heavily using the functions of questioning and feedback.

In the asynchronous dialogs observed here, students tended to ignore the lower-level, triggering, fact-based questions initially posed by the instructor, and tended to immediately post problem analysis and solution responses. Because each response in the ALN discussion is larger and more complex, and because many responses occur at once, often near the end of the discussion period, there is difficulty structuring the feedback process. It appears that the traditional Socratic questioning and feedback functions are to some extent dependent on the linear nature of the FTF dialog.

This suggests that instructors must choreograph ALN case study discussions differently from those conducted in FTF mode. Certain structural devices such as requiring students to post daily give the instructor the ability to more closely simulate the linear nature of the FTF discussion. But if asynchronous learning networks are to be truly "any time, any where," such constraints may be artificial and limit some students.

The nature of cognitive process in FTF and ALN case study discussions

There was a higher absolute and proportional incidence of abstract analytical processes in the ALN mode. The FTF discussions, on the other hand, contained a greater incidence of lower-level cognitive processes in the exploration category. This difference was probably a function of the "choreographic" issue noted above. In the linear, FTF discussions, the instructor was able to insure that the facts of the case were initially explored in a leisurely, detailed fashion, but was unable to accomplish this in the ALN mode.

Does this difference in the pattern of cognitive process matter? Should we conclude that the higher incidence of analytical expression in ALN discussions is an indicator of a more successful learning experience? Or does the absence of a richly detailed exploration of the facts make it less successful? The design of this study does not provide additional comparisons of the quality of learning in each mode. In order to keep the process as natural as possible in the context of the course, we did not ask students to prepare additional, identical assignments that could be graded by independent assessors. Nevertheless, the findings suggest two somewhat different speculations about the possible cognitive strengths and weaknesses of each mode.

1. One tenet of the case method is that it is experiential, or situational (Benbunan-Fich & Hiltz, 1999). It is intended to simulate a realistic situation so that the student can get at least some of the benefits of experiencing that situation. If the complexity and ambiguity of realistic situations are to be simulated, a sufficient level of concrete detail must be explored. Thus, we speculate that ALN discussions may better achieve the experiential goal of the case method if they are able to find ways to incorporate more rather than fewer of the concrete exploration processes.
   
2. These findings are consistent with previous work that characterizes written ALN transcripts as demonstrating higher levels of abstract cognitive process. Because they were required to write rather than speak their responses, students appeared to be more careful, more formal, and far more reflective about their answers, as previous research suggests. These benefits of writing may be more difficult to obtain in the traditional case study discussion, but can be incorporated into the FTF case study process, by having students prepare either individual or group written assignments.
   

Quality in ALN Content Analytic Research

In the past decade, a number of studies have used the transcripts from asynchronous technology-mediated bulletin boards to provide data on learning and collaborative environments (e.g., Anderson et al., 2001; Garrison et al., 2000; Hass, 1996; Henri, 1992; Hiltz & Turoff, 1993; Mason, 1992; Newman et al., 1996; Olson, 1994; Rourke et al., 1999; White, 1993). As does the present study, they have employed a content analytic approach to interpreting the meaning or meanings that underlie these transcripts. The transcripts have been viewed as relatively accurate, direct, or manifest indicators of discourse behaviors. On another level, they have been analyzed in order to reveal deeper, or latent levels of meaning that concern cognitive, social, and teaching phenomena.

As data, these transcripts have important advantages. Because they are captured digitally, completely, and in real time, they present an opportunity to study a phenomenon in its purely natural setting. If one wishes to observe what people actually do when collaborating through this computer-mediated technology, the record is preserved indefinitely for all to scrutinize. This availability of naturalistic data holds the promise of several important educational advances. The most significant of these is the ability to more accurately assess educational outcomes. If valid interpretive schema can be developed, these transcripts can reveal much about individual learning outcomes, group learning outcomes, and teacher performance. All three of these assessments have proved problematic in educational practice.

In order to obtain the benefits that are promised for analysis of computer-mediated transcripts, however, it is necessary to assess the quality of our interpretations of them. The following discussion is an attempt to address this issue. It presents a systematic exploration of dimensions that contribute to the reliability, validity, and the utility of the content analytic approach to evaluating learning in asynchronous learning networks. We will address these issues as they pertain to the present study, but it is also our intention to begin consideration of how we should evaluate this stream of inquiry in general. By doing so, it is our objective to contribute to future improvements in this line of research.

Miles and Huberman (1994) identify five standards for assessing the quality of conclusions in a qualitative research project.

1. Objectivity/Confirmability
2. Reliability/Dependability/Auditability
3. Internal Validity/Credibility/Authenticity
4. Utility/Application/Action Orientation
5. External Validity/Transferability/Fittingness

Objectivity/Confirmability

This standard deals with the issue of neutrality and freedom from bias, and the procedures followed by the researcher. To assess this standard, we must ask questions such as: Are the studies, methods, and procedures described explicitly and in detail? Has the researcher been explicit and self aware about personal assumptions, values, and biases (Miles & Huberman, 1994)?

In the present study we have attempted to describe the procedures followed explicitly and in detail. We note several limitations that are inherent in the design we used. First, we emphasize that observations based on these eight discussions do not necessarily reveal invariable attributes of either ALN or FTF learning modes. These discussions were based on a specific form of stimulus, the case study, which may have attributes that are different from other learning activities. In addition, these particular students were predominantly resident undergraduates enrolled in a traditional on-campus program. They were unlikely to have the same experience or motivation for ALN discussions as older, nontraditional, non-resident, and ALN-experienced students. Finally, some outcomes in this study may have been due to the idiosyncrasies of this particular instructor, who is also the first author. Other limitations will be discussed below. But it is important to note that design attributes that lead to limitations may also make positive contributions to the quality of a study. For example, the observation of a single instructor raises concerns about generalizability, but may also provide control against some forms of unwanted variation. The important point is that procedures be described in sufficient detail so that readers can make informed judgments.

Reliability/Dependability/Auditability

This standard addresses the question of whether the assessments used in the study's process are consistent and stable over time across researchers and methods. To assess reliability, qualitative researchers typically ask: Is the researcher's role and status within the research site explicitly described? Were code and checks made, and did they show adequate agreement? Do multiple observers agree in their accounts and interpretations (Miles & Huberman, 1994; Rourke et al., 2001)?

In the present study, an acceptable level of intercoder reliability was attained, and the studies from which the coding scheme was adapted (Anderson et al., 2001; Aviv, 2000; Garrison et al., 2000; Rourke et al., 1999) showed similarly acceptable levels of coder agreement. Thus it is clear that coders can be trained to interpret this scheme with a reasonable level of reliability. However, there is not yet any evidence that the scheme can be used reliably across different studies and researchers. In order to create a cumulative body of knowledge based on this type of content analytic research, efforts should be made to standardize and possibly simplify coding schemes, so that collaborative work spanning multiple research teams can begin the creation of a cumulative tradition.

Internal Validity/Credibility/Authenticity

Internal validity raises issues of the truth value of the conclusions of the study. Do the findings make sense? Are they confirmable? Do they conform to our understanding of the world arrived at from other sources? Researchers can assess internal validity by asking questions such as: Did triangulation among complimentary methods produce generally converging conclusions? Are the findings internally coherent (Miles & Huberman, 1994)?

In content analytic studies, researchers consider both latent and manifest content. Manifest content is directly observable in the transcripts and thus considered relatively "objective." Latent content on the other hand is not directly observable, is open to interpretation, and thus is considered relatively "subjective" (Rourke et al., 2001). Some scholars, such as Mason (1992), used inductive methods to identify a typology of manifest indicators, later linking them to latent variables. Other scholars developed manifest indicators believed to reflect predefined latent constructs (Anderson et al., 2001; Garrison et al., 2000; Rourke et al., 1999). Regardless of whether inductive or deductive approaches are used, the linkage between manifest content and latent content is the first concern of internal validity. For example, we might ask whether manifest content such as conspicuous capitalization or repetitive punctuation are valid indicators of a latent construct called "emotional expression." In other words, we must satisfy ourselves that our observations provide valid indicators of the four latent structuring processes—discourse, teaching, social, and cognitive. Because the theoretical grounding of the present coding scheme has been described in detail by the researchers who developed it (Anderson et al., 2001; Aviv, 2000; Garrison et al., 2000; Rourke et al., 1999) we have some confidence that the scheme is coherent.

However, a question that is larger and of more direct interest remains. Are the latent constructs captured by the coding scheme valid and relevant indicators of the learning process? Different scholars, with some variance, have defined learning as a process of refining mental models through processing and testing information, through individual activities, or through interactions with others (e.g., Alavi, 1994; Benbunan-Fich & Hiltz, 1999). In other words, learning is a process that results in change in behavior and/or knowledge. Linking the four structuring processes to learning is a difficult question, because empirical support for the assertion can only be the result of many studies that employ similar constructs in a variety of settings. In addition, our change-based definition of learning suggests that such studies should be longitudinal. They should provide measures of specific knowledge or skills both before and after the structuring processes occur. The addition of strong, independent, and longitudinal measures of learning will allow us to compare patterns in the structuring processes to independently measured outcomes of learning. Such research will be costly and difficult to execute, but necessary if we are to be confident of the internal validity of this framework.

Utility/Application/Action Orientation

In this standard, Miles and Huberman suggest that over and above the other standards for reliability and internal validity, researchers must ask questions of "pragmatic validity" (Kvale, 1989). Researchers can assess this standard by asking questions such as: Do the findings provide guidance for future action? Are the findings accessible to potential users? Can the findings actually be used in solving problems (Miles & Huberman, 1994)?

The application of this content analysis framework is labor intensive, time consuming, and expensive. It is very unlikely that this method, in its manual form, can be usefully applied to the practical task of assessing educational outcomes. Thus, once we become more confident of the validity and reliability of our frameworks, it is not unreasonable that we turn our attention to finding an automated process to conduct the analysis of transcripts. Natural language processing (NLP) technologies are developing at a rapid pace and are being applied to an ever-widening array of problems. NLP has been used to extract military intelligence from published sources, to interpret and respond to customer service email, and to identify trends in consumer focus group transcripts. These problems present levels of complexity similar to the ALN context. Thus, while we are forced to conclude that this content analytic approach has little practical validity today, we can be hopeful that the research stream is laying the foundation for future advances.

External Validity/Transferability/Fittingness

This standard addresses the question of whether the findings of the study have any applicability in contexts beyond the immediate locus of the study itself. To assess this standard, we can ask questions such as: Are the characteristics of the people's settings and processes sufficiently described to provide comparison with other settings? Does the researcher attempt to define the scope and the boundaries of reasonable generalization from the study? Does the researcher suggest ways and other settings in which the findings can be further tested (Miles & Huberman, 1994)?

This study was confined to a narrow group of students (undergraduate seniors in one residential degree program). It used a single form of discussion stimulus—the management case study—and discussions were led by a single instructor. Thus, we are reluctant to make strong assertions of generalizability to other settings. Nevertheless, there are sufficient similarities between case study discussions and other collaborative learning activities that we can at least suggest a few other ways and settings in which these findings might be further tested. Case study discussions have some similarities to other forms of discussion that proceed from a narrative stimulus. In addition, these FTF and ALN discussions employed entirely conventional environments and technologies. Thus, it should be reasonable to hypothesize similar findings in future observations of discussions planned around other narrative forms.

At a somewhat further remove, we might consider applying aspects of this coding scheme to other settings where technology-mediated collaborative learning is important. One such approach would be to adapt the coding scheme here to technology-mediated student task groups. Collaborative group tasks are assigned in many educational settings. It may be possible to evaluate transcripts of task-oriented student work using adapted forms of the four structuring processes employed here.

And while ALN content analytic studies have been largely limited to groups in the educational setting, their application might also be extended to other groups outside of the educational realm. Decision-making groups in organizations could be studied using such frameworks to better understand the learning processes of these groups. While many studies have been performed to study computer-mediated research and work groups, these studies tend to focus on communication patterns and decision-making processes. An important contribution of this framework could be in its application to the study of learning processes in work groups, using social interdependence, critical thinking, and constructivist learning theories. Organizations have long emphasized learning—the ability to detect errors and change—as a key competency for survival and competitive advantage. Understanding the process of learning among the members of geographically dispersed workgroups could guide the development of technologies and procedures to maximize learning in such groups.

Conclusion

Despite the limitations of this study, the findings provide useful empirical data for those attempting to maximize the learning potential of case study discussions in both FTF and ALN modes. For example, these findings suggest that a combination of FTF and ALN methods might prove beneficial for on-campus students. And they indicate a need to find ways in ALN discussions to provide individual feedback and better access to the rich, concrete details of the case. They also provide additional evidence that ALNs generate high levels of cognitive activity, at least equal to, and in some cases superior to, the cognitive processes in the FTF classroom. This evidence confirming previous findings (Alavi, 1994; Benbunan-Fich & Hiltz, 1999) is important because it comes from a new source of data -a rigorous and detailed comparative content analysis of FTF and ALN case study discussions.

Future research should turn to questions of reliability and validity of the methods used in this and previous content analytic studies. For example, while we have demonstrated that this framework can be used reliably by multiple coders, it remains to be demonstrated that it can be used reliably across multiple studies. Work is also needed to demonstrate that the cognitive processes identified here are true indicators of learning. Until they are triangulated with other, independent measures of learning, we will be unable to make this assertion. Thus, this study and its predecessors provide the foundation for much future research.

Acknowledgments

An earlier version of this paper was presented at the 36th Hawaii International Conference on System Sciences, January 2003.

References

Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, 18 (2), 159-174.

Alavi, M., Yoo, Y., & Vogel, D. R. (1997). Using information technology to add value to management education. Academy of Management Journal, 40 (6), 1310-1333.

Anderson, T., Rourke, L., Archer, W., & Garrison, R. (2001). Assessing teaching presence in computer conferencing transcripts. Journal of the Asynchronous Learning Network, 5 (2). Retrieved December 27, 2004 from http://www.aln.org/alnweb/journal/jaln-vol5issue2v2.htm

Applegate, L. M. (1988). Case Teaching at Harvard Business School: Some Advice for New Faculty. Boston, MA: Harvard Business School Publishing Division.

Aviv, R. (2000). Educational performance of ALN via content analysis. The Journal of Asynchronous Learning Networks, 4 (2). Retrieved December 27, 2004 from http://www.aln.org/alnweb/journal/Vol4_issue2/le/reuven/LE-reuven.htm

Benbunan-Fich, R., & Hiltz, R. (1999). Impacts of asynchronous learning networks on individual and group problem solving: A field experiment. Group Decision and Negotiation, 8 (5), 409-426. Retrieved December 27, 2004 from http://www.alnresearch.org/Data_Files/articles/full_text/benbunan.htm

Bonoma, T. V. (1989). Learning with cases. Note 9-589-080. Boston, MA: Harvard Business School Publishing Division.

Curtis, D. D., & Lawson, M. J. (2001). Exploring collaborative online learning. Journal of Asynchronous Learning Networks, 5 (1), 21-34.

Garrison, R., Anderson, T., & Archer, W. (2000). Critical thinking in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2 (2-3), 87-105.

Garrison, R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence and computer conferencing in distance education. American Journal of Distance Education, 15 (1), 7-23.

Harasim, L. (Ed.) (1990). Online Education: Perspectives on a New Medium. New York: Prager/Greenwood.

Hass, C. (1996). Writing Technology: Studies on the Materiality of Literacy. Manwah, N.J.: Erlbaum.

Heckman, R., & Annabi, H. (2004). How the teacher's role changes in on-line case study discussions. Working paper; available from authors.

Heckman, R., Maswick, D., Rodgers, J., Ruthen, K., & Wee, G. (2000). The impact of information technology on roles and role processes in small groups. In L. A. Petrides (Ed.), Case Studies on Information Technology in Higher Education: Implications for Policy and Practice (pp. 157-167). Hershey, PA: Idea Group Publishing.

Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative Learning through Computer Conferencing: The Najaden Papers (pp. 115-136). New York: Springer.

Hiltz, S. R., & Turoff, M. (1993). The Network Nation: Human Communication via Computer. Cambridge: The MIT Press.

Hiltz, S. R. (1994). The Virtual Classroom: Learning Without Limits via Computer Networks. Norwood, NJ: Ablex Publishing Corporation.

Johnson, D. W., & Johnson, R. (1989). Cooperation and Competition: Theory and Research. Edina, MN: Interaction Book Company.

Kvale, S. (Ed.) (1989). Issues of Validity in Qualitative Research. Lund, Sweden: Studentlitteratur.

Mason, R. (1992). Methodologies for evaluating applications of computer conferencing. In A. R. Kaye (Ed.), Collaborative Learning Through Computer Conferencing: The Najaden Papers (pp. 105-115). New York: Springer.

Miles, M. B., & Huberman, A. (1994) Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks: CA: Sage.

Newman, D. R., Johnson, C., Cochrane, C., & Webb, B. (1996). An experiment in group learning technology: Evaluating critical thinking in face-to-face and computer-supported seminars. Interpersonal Computing and Technology: An Electronic Journal for 21st Century, 4 (1), 57-74. Retrieved December 27, 2004 from http://www.helsinki.fi/science/optek/1996/n1/newman.txt

Olson, D. R. (1994). The World on Paper: The Conceptual and Cognitive Implications of Reading and Writing. Cambridge and New York: Cambridge University Press.

Ragan, V. K. (1996). Choreographing a case class. Note 9-595-074. Boston, MA: Harvard Business School Publishing Division.

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous, text-based computer conferencing. Journal of Distance Education, 14 (3), 51-70. Retrieved December 27, 2004 from http://cade.athabascau.ca/vol14.2/rourke_et_al.html

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12 (1), 8-22.

White, E. M. (1993). Assessing higher-order thinking and communication skills in college graduates through writing. The Journal of General Education, 42 (2), 105-122.

Appendix 1: Coding Scheme

Based on Model of Community of Inquiry (Garrison, Anderson, & Archer, 2000) and ALN Process Model (Aviv, 2000)

About the Authors

Robert Heckman is Associate Professor and Director of the Graduate Program in Information Management at the School of Information Studies, Syracuse University. Before joining the faculty at Syracuse University, Dr. Heckman worked for over 20 years in the financial and information industries, serving as Vice President and Division head for the Mellon Bank Datacenter Group. Dr. Heckman's current research is focused on information management issues, including effective use of asynchronous learning networks, and discretionary technology-mediated collaboration.
Address: School of Information Studies, Syracuse University, 233 Hinds Hall, Syracuse, NY 13244-4100 USA

Hala Annabi is an Assistant Professor in the Information School at the University of Washington. Her academic background is in business administration and management information systems. Her main research interests are in distributed work, asynchronous learning networks, and organizational learning. She is also interested in the scholarship of teaching and implements experiential pedagogy in her teaching.
Address: The Information School, University of Washington, Box 352840, Seattle, WA 98195-2840 USA