Student Satisfaction and Learning Outcomes in E-Learning: An Introduction to Empirical Research

From its humble origins approximately 30 years ago (Hiltz & Turoff, 1978), it is possible that we are now entering what may be a golden age of e-learning. In the United States, 4.6 million students took at least one online course during Fall 2008, a seventeen percent increase from the previous year.  U.S. schools offering these courses have seen increases in demand for e-learning options, with 66 percent and 73 percent of responding schools reporting increased demand for new and existing online course offerings respectively (Allen & Seaman, 2010). Similar reactions to e-learning are occurring across the globe. The European Union’s Lifelong Learning Programme will be investing much of its 7 Billion Euro budget between 2007 and 2013 in the development of and enhancement of e-learning tools and open collaboration initiatives (European Commission, 2010). Institutions such as Ramkamhaeng University in Thailand, the Indira Ghandi National Open University in India and the Open University of Malaysia are adopting e-learning to help manage enrollments approaching 2 million students (Bonk, 2009). With e-learning beginning to expand beyond its historic roots in higher education to K-12 educational settings and populous nations such China, India, and Indonesia only beginning to embrace e-learning, it appears that the demand for online learning will only increase in the future, and likely increase dramatically.

But in spite of such potential promise for e-learning, support for delivering education through this medium is far from unanimous.  Empirical studies suggest that online education is not a universal innovation applicable to all types of instructional situations. Online education can be a superior mode of instruction if it is targeted to learners with specific learning styles (visual and read/write learning styles) (Eom, Ashill, & Wen, 2006), students personality characteristics (Schniederjans & Kim, 2005), and with timely, helpful instructor feedback of various types. Although cognitive and diagnostics feedbacks are all important factors that improve perceived learning outcomes, metacognitive feedback can induce students to become self-regulated learners. Recent meta-analytic studies (Means, Toyama, Murphy, Bakia, & Jones, 2009; Sitzmann, Kraiger, Stewart, & Wisher, 2006) also suggest that learning outcomes now equal, and in some cases, surpass those provided in classroom-based settings. However, concerns regarding this delivery medium’s effectiveness continue to persist (Morgan & Adams, 2009; Sarker & Nicholson, 2005). Some question its appropriateness for the delivery of technically-oriented or skills-based content (Hayes, 2007; Kellogg & Smith, 2009; Marriott, Marriott, & Selwyn, 2004). Others bemoan a lack of direct contact between students and instructors (Haytko, 2001; Tanner, Noser, & Totaro, 2009; Wilkes, Simon, & Brooks, 2006). Still others associate the medium with for-profit universities, and therefore lump its use in with the practices of low standards and high-pressure marketing associated with some of those types of institutions (Bonk, 2009; Stahl, 2004). Still others believe that although the technology itself may be neither good nor bad, the bad or even non-existent training that many of those employed to teach using the medium likely guarantees a poor educational experience for learners and changes the learner-instructor relationship in ways that are not always positive (Alexander, Perrault, Zhao, & Waldman, 2009; Kellogg & Smith, 2009; Liu, Magjuka, Bonk, & Lee, 2007).

The Objective of this Book

One way to address such concerns is through researching the phenomenon to determine whether and under what conditions the use of the medium is most effective.  However, concerns regarding the quality of research of e-learning have long persisted. From concerns such as over-reliance upon single-course studies (Phipps & Merisotis, 1999), to lack of randomized experimental designs (Bernard, Abrami, Lou, & Borokhovski, 2004), to incomplete and/or imprecise measures of student learning (Sitzmann, Ely, Brown, & Bauer, 2010) to more general concerns over methodological quality (Bernard et al., 2009), concerns regarding the rigor of research on e-learning are not new. These concerns regarding research quality are augmented by the trend that although the number of e-learning instructors continues to increase, the number of scholars providing a sustained history of research contributions on the topic has been comparatively few. For example, a recent review of the literature on e-learning in business education reported that fewer than 20 scholars were intensively contributing to this literature (three or more articles), and this number was enhanced in part because these scholars were collaborating with each other (Arbaugh et al., 2009). If such a distorted picture of dedicated e-learning researchers relative to e-learning educational practitioners exists in other disciplines, it is evident that we would greatly benefit from substantially increasing the number of researchers dedicated to examining this increasingly pervasive phenomenon.

The Audience of This Book

This book is for practitioners, managers, researchers, and graduate students in virtually every field of study. Application areas of e-learning are not limited to a specific academic area. E-learning is a worldwide perpetual trend that is being applied to educate employees of non-academic organizations such as governments, profit or non-profit organizations. Needless to say, libraries in university, profit and non-profit organizations around the world will be potential customers. Therefore, we have produced a book that will help introduce these instructors, researchers, practicing managers, and graduate students in the e-learning community to research on satisfaction and learning outcomes in e-learning.  Besides providing insights from previous research on effective instructional practices for new instructors (who, in turn, could be new researchers) that are entering the e-learning realm, why not a book that might help them examine and conduct their work more thoroughly? It is our hope that new (and not so new) instructors, researchers, practicing managers, and graduate students will use the materials in this book to enter the increasingly fascinating field of research in online teaching and learning.

The Contributors of This Book

In compiling this book’s contents, we are particularly pleased that we have both a multi-national and a multi-disciplinary composition of contributors of the book’s chapters. We have authors from institutions in Canada, Mexico, Romania, South Africa, Spain, Turkey, the United Arab Emirates, and the United States. These scholars represent fields such as adult education, computer science, distance education, economics, educational leadership, Information Systems, instructional technology, international management, marketing, and strategy. Considering the diverse backgrounds from which theoretical and methodological perspectives used to develop e-learning research, we feel that incorporating the works of scholars from varied backgrounds not only informs the reader of the breadth of research conducted in this emerging field, but also affords the chapter authors the opportunity to bring the perspectives of this collection of works back to inform scholars in their respective disciplines.

The Structure of This Book

When one seeks to enter a new research field, familiarizing oneself with some of that field’s influential articles is a necessary starting point. However, being able to see those articles in the broader context of the field’s predominant theoretical and methodological influences and potential future directions can help scholars to determine where their expertise and skills can make the most appropriate contribution. Therefore, we organized the book’s chapters to familiarize prospective researchers with processes and topics for conducting research in e-learning.  The book is divided into 4 sections: Theoretical Frameworks, Empirical Research Methods and Tutorial, Factors Influencing Student Satisfaction and Learning Outcomes, and Other Applications of Theory and Method.

The first section, Theoretical Frameworks, introduces readers to emerging methodological and theoretical perspectives for effective empirical e-learning research. The two chapters in the book’s first section present a case for increased use of multi-course, multi-disciplinary studies and provide an overview and application of an increasingly influential model of e-learning effectiveness, the Community of Inquiry framework (Garrison, Anderson, & Archer, 2000). In chapter 1, Arbaugh argues that research in online teaching and learning in higher education should take a multi-disciplinary orientation, especially in settings whose curricula are drawn from several disciplinary perspectives such as business schools.  The benefits of a multi-disciplinary approach include curriculum integration and enhanced communication and collective methodological advancement among online teaching and learning scholars from the disciplines that comprise the integrated curricula. After reviewing multi-disciplinary studies in business education published to date, the chapter concludes with recommendations for advancing research in this emerging stream.  Some of the primary recommendations include the use of academic discipline as a moderating variable, and more studies incorporate samples comprised of faculty and/or undergraduate students, and the development of more comprehensive measures of student learning.

In chapter 2, Akyol and Garrison explain the capability of the Community of Inquiry (CoI) framework as a research model to study student learning and satisfaction. The framework identifies three elements (social, cognitive, and teaching presence) that contribute directly to the success of an e-learning experience through the development of an effective CoI. It is argued that a CoI leads to higher learning and increased satisfaction. The chapter presents findings from two online courses designed using the CoI approach. Overall, the students in these courses had high levels of perceived learning and satisfaction as well as actual learning outcomes.

The second section of the book is titled Empirical Research Methods and Tutorial.  Because empirical research in e-learning is our topic of interest, it seems particularly appropriate that research methods are the focus of the book’s second section. After a review of research methods employed to date in a relatively active discipline, the book’s second section chronicles and provides examples of several of the structural equation modeling techniques whose increased use was called for in the review chapter. This section also include chapter that deals with higher order multivariate techniques, experimental designs, and data mining.

The first chapter in this section, chapter 3, reviews the online teaching and learning literature in business education and it found growing sophistication in analytical approaches over the last 10 years.  We believe researchers are uncovering important findings from the large number of predictors, control variables, and criterion variables examined. Scholars are employing appropriate and increasingly sophisticated techniques such as structural equation models in recent studies within a field setting.  To increase methodological rigor, researchers need to consciously incorporate control variables that are known to influence criterion variables of interest so as to clearly partial out the influence of their predictor variables of interest. This will help address shortcomings arising from the inability to convince sample respondents such as instructors, institutional administrators, and graduate business students on the benefits versus the cost of a fully randomized design approach.

Chapter 4 is an introduction to path analysis modeling using LISREL.   Over the past decades, we have seen a wide range of empirical research in the e-learning literature. The use of multivariate statistical tools has been a staple of the research stream throughout the decade.  Path analysis modeling is part of four related multivariate statistical models including regression, path analysis, confirmatory factor analysis, and structural equation models. This chapter focuses on path analysis modeling for beginners using LISREL 8.70.  Several topics covered in this chapter include foundational concepts, assumptions, and steps of path analysis modeling. The major steps in path analysis modeling explained in this chapter consist of specification, identification, estimation, testing, and modification of models.

Chapter 5, “Testing the DeLone-McLean model of information system success in an e-learning context,” has two important objectives: (a) introduction of structural equation modeling for a beginner; and (b) empirical testing of the validity of the information system (IS) success model of DeLone and McLean (the DM model) in an e-learning environment using LISREL.   The next section presents the research model tested and discussion of the survey instrument. The structural equation modeling process is fully discussed including specification, identification, estimation, testing, and modification of the model. The final section summarizes the test results. To build e-learning theories, those untested conceptual frameworks must be tested and refined. Nevertheless, there has been very little testing of these frameworks. This chapter is concerned with the testing of one such framework. There is abundant prior research that examines the relationships among information quality, system quality, system use, user satisfaction, and system outcomes. This is the first study that focuses on the testing of the DM model in an e-learning context.   

Chapter 6 is an introduction to structural equation modeling (SEM) and the partial least squares (PLS) methodology. Over the past 15 years, the use of Partial Least Squares (PLS) in academic research has enjoyed increasing popularity in many social sciences including information systems, marketing, and organizational behavior.  PLS can be considered an alternative to covariance-based SEM and has greater flexibility in handling various modeling problems in situations where it is difficult to meet the hard assumptions of more traditional multivariate statistics.  This chapter focuses on PLS for beginners.  Several topics are covered and include foundational concepts in SEM, the statistical assumptions of PLS, a LISREL-PLS comparison, and reflective and formative measurement.

Chapter 7, “Using Experimental Research to Investigate Students’ Satisfaction with Online Learning,” discusses the most common experimental designs and threats to internal validity of experimental procedures that must be controlled to ensure that the interventions or programs under investigation are responsible for changes in the dependent variables of interest. A study by Bangert (2008) is used to illustrate procedures for conducting experimental research, controlling potential threats to internal validity, and reporting results that communicate both practical and statistical significance. The use of experimental research in higher education settings for investigating the effectiveness of technology-supported instructional innovations in K-12 and higher education settings is fairly limited. The implementation of the No Child Left Behind Act (NCLB) of 2001, has renewed the emphasis on the use of experimental research for establishing evidence to support the effectiveness of instructional interventions and other school-based programs in K-12 and higher education contexts.  

Chapter 8 introduces a new approach of data mining as an empirical analysis tool for analyzing student performance in e-learning environments. The aim of this chapter is to explore the application of data mining for analyzing performance and satisfaction of the students enrolled in an online two-year Master degree programme in project management. This programme is delivered by the Academy of Economic Studies, the biggest Romanian university in economics and business administration in parallel, as an online programme and as a traditional one. The main data sources for the mining process are the survey made for gathering students’ opinions, the operational database with the students’ records and data regarding students activities recorded by the e-learning platform are. More than 180 students have responded and more than 150 distinct characteristics/ variable per student were identified. Due the large number of variables data mining is a recommended approach to analysis this data. Clustering, classification, and association rules were employed in order to identify the factor explaining students’ performance and satisfaction, and the relationship between them. The results are very encouraging and suggest several future developments.

Chapter 9 is the last chapter of the second section of the book. The purposes of this chapter are three-fold: (1) to present our findings in investigating the success factors for designing, developing and delivering e-learning initiatives, (2) to examine the applicability of information systems theories to study e-learning success, and (3) to demonstrate the usefulness of action research in furthering our understanding of e-learning success.  Inspired by issues and challenges experienced in developing an online course, a process approach for measuring and assessing e-learning success is advanced. This approach adopts an Information Systems perspective on e-learning success to address the question of how to guide the design, development, and delivery of successful e-learning initiatives. The validity and applicability of our process approach to measuring and assessing e-learning success is demonstrated in empirical studies involving cycles of action research.  Merits of our approach are discussed and contributions in paving the way for further research opportunities are presented.

The third section of the book is titled Factors Influencing Student Satisfaction and Learning Outcomes. After presenting prevailing theoretical and methodological perspectives, the book’s third section examines particular influences of e-learning course outcomes in a variety of settings. These chapters examine factors such as learner dispositional and behavioral characteristics, quality assurance frameworks for e-learning effectiveness, course content design and development, and their roles in shaping effective e-learning environments.  To date, much of e-learning research has focused on asynchronous learning environments (exemplified by the Journal of Asynchronous Learning Networks) based in higher education settings. However, these are not the only contexts in which e-learning occurs.  Therefore, we also address the alternative and potentially increasingly important settings of synchronous course delivery and corporate learning environments in this section.

Chapter 10 is concerned with quality assurance in e-learning.  Quality is a subjective concept, and as such, there are many criteria for assuring quality, including assessment practices based on industry standards and accreditation requirements. Most assessments, including quality assurance in e-learning, frequently occur at three levels: individual course assessments, department or program assessments, and institutional assessments; frequently these levels cannot be distinctly delineated. While student evaluations are usually included within these frameworks, student views are but one variable in the quality assessment equation. To offer some plausible perspectives of how students view quality, this chapter will provide an overview of quality assurance for online learning from the course, program, and institutional viewpoints as well as review some of the key research related to students’ assessment of what constitutes quality in online courses.

Chapter 11 presents the synchronous virtual classroom (SVC) success model. The SVC model will help instructors design online courses that incorporate the factors that students need to be successful. This model will also help virtual classroom instructors and managers develop a systematic way of identifying and addressing the external and internal factors that might impact the success of their instruction. The strategies for empirically researching the SVC, which range from qualitative inquiry to experimental design, are discussed along with practical examples. This information will benefit instructors, researchers, non-profit and profit organizations, and academia.

Chapter 12, “Factors Influencing User Satisfaction with Internet-Based E-learning in Corporate South Africa,” examines the factors that influence user satisfaction with Internet based learning in the South African corporate environment. An electronic survey was administered and one hundred and twenty responses from corporations across South Africa were received. Only five of the thirteen factors were found to exert a statistically significant influence on learner satisfaction: instructor response towards the learners, instructor attitude toward Internet based learning, the flexibility of the course, perceived usefulness, perceived ease of use, and the social interaction experienced by the learner in assessments. Interestingly, four of those five were also identified as significant in a similar Taiwanese study, which provides an interesting cross-cultural validation for the findings, even though our sample was different and smaller. Perhaps surprisingly, none of 6 demographic variables exerted significant influence. Hopefully organisations and educational institutions can note and make use of the important factors in conceptualizing and designing their e-learning courses.

Chapter 13 examines the relationships between student personality and e-learning outcomes. Among students enrolled in Web-based courses, some students learn a lot while others do not.  There are many possible reasons for the differences in learning outcomes (e.g., student’s learning style, satisfaction, motivation, etc,).  In the last few decades, student’s personality has emerged as an important factor influencing the learning outcomes in a traditional classroom environment.  Among different personality models, the Big-Five model of personality has been successfully applied to help understand the relationship between personality and learning outcomes.  Because Web-based courses are becoming popular, the Big-Five model is applied to find out if students’ personality traits play an important role in a Web-based course learning outcomes.   

Chapter 14, “A method for adapting learning objects to students preferences,” analyzes the different learning theories and styles, as well as the main standards for creating contents with the goal of developing a proposal for structuring courses and organizing material which best fits students’ needs, in order to increase motivation and improve the learning process. The objective of this chapter has been to analyze different factors that influence student learning. To identify different factors that influence student learning, it was necessary to review different learning theories and different learning styles. After that, the authors analyzed the role of teachers and their main responsibilities, and students’ learning process in order to propose a pedagogical structure for an e-learning course. The relevant roles that both teaching contents and e-learning play were also discussed. An active teacher who participles and creates high quality contents is necessary to prevent the sense of isolation, discouragement, and lack of motivation. Considering all these factors and the special features of each student as regards the way he learns, this chapter has proposed a new method that facilitates teaching and adapts knowledge to special preferences of each student.

The fourth section of the book includes three chapters that deal with other applications of e-learning theory and method. The book’s final section extends the approach of alternative e-learning theory and environments through applying the Unified Theory of Acceptance and Use of Technology (UTAUT) of Venkatesh et al (2003), and blended learning.

The objective of “Chapter 15, Understanding Graduate Students’ Intended Use of Distance Education Platforms,” is to apply the Unified Theory of the Acceptance and Use of Technology to better understand graduate students’ intended use of distance education platforms, using as a case a distance education platform of a Mexican University, the SERUDLAP system. Four constructs are hypothesized to play a significant role: performance expectancy, effort expectancy, social influence and attitude toward using technology; the moderating factors were gender and voluntariness of use. Data for the study was gathered through an on-line survey with a response rate of about 41%. Results suggested that the performance expectancy and attitude towards technology are factors that help us understand graduate students’ intended use of a distance education platform. Future research must consider the impact of some factors, such as previous experiences, age, and facilitating conditions in order to better understand the students’ behavior.

Chapter 16 is concerned with the investigation of critical issues, dynamics and challenges related to project-based learning (PBL) from 49 student perspectives in an online course. The effect of PBL was examined qualitatively with open-ended questionnaire, observations and students’ submissions who were taking an online certificate course. According to the findings, students thought that an online PBL course supports their professional development with provision of practical knowledge, enhanced project development skill, self confidence, and research capability. This support is further augmented with the facilities of the online learning environment. Students mainly preferred team-work more than individual work. Although students were mostly satisfied with the course, they still had some suggestions for prospective students and instructors. The findings are particularly important for those people who are planning to organize course or activities which involve online PBL and who are about take an online or face-to-face PBL course.

Chapter 17 is the last chapter of the book. This chapter is a case study that examines students’ perceptions, interaction and satisfaction in the interactive blended courses. Blended courses that offer several possibilities to students and teachers such as becoming more interactive and more active have become increasingly widespread for both K12 and higher education settings. With the rising of the cutting edge technologies, institutions and instructors have embarked on creating new learning environments with a variety of new delivery methods. At the same time, designing visually impressive and attractive blended settings for students have been easier with extensive learning and content management systems (LMS, CMS, LCMS) such as Blackboard, WebCT, Moodle, and virtual classroom environments (VLE) such as Adobe Connect, Dimdim, and WiZiQ. In this study, we aimed to investigate students’ perspectives and satisfactions towards designed interactive blended learning settings and to find out the students’ views on both synchronous and asynchronous interactive blended learning environment (IBLE).

The Future of e-Learning Research

Although this book examines numerous topics for which research has been conducted, there are several areas in which e-learning research is still in its infancy. To help steer prospective scholars in directions where they might provide immediate impact, we conclude this section with a brief discussion of some of these topics:

Impacts of E-Learning by and on Instructors

There have been numerous studies of student reactions to e-learning and potential predictors of effective learner-related outcomes. However, studies of the other primary participants in e-learning environments, the instructors, have been slow in coming. Fortunately, we are beginning to see studies that focus on instructor motivations and reactions to e-learning (i.e. (Connolly, Jones, & Jones, 2007; Coppola, Hiltz, & Rotter, 2002; Shea, 2007). As instructors continue to increase their knowledge and experience with e-teaching, there likely will be research opportunities for comparing attitudes, motivations, and behaviors of novice versus experienced online instructors. Other instructor-related research opportunities may include consideration of changes in workplace environments, interactions with students outside of class, working relationships with colleagues, and relationships to their host institutions.

Potential Influences of Academic Disciplines on E-Learning Effectiveness

Considering that many of the theoretical foundations of e-learning research have come from the communities of instructional technology, Information Systems, and educational theory, it is not surprising that the potential influence of epistemological, sociological, and behavioral characteristics of academic disciplines may play in shaping effective e-learning environments has received limited research attention to date.  However, recent studies of disciplinary effects in e-learning suggest that they may have distinct influences on both course design(Smith, Heindel, & Torres-Ayala, 2008) and student retention, attitudes, and performance (Arbaugh & Rau, 2007; Hornik, Sanders, Li, Moskal, & Dziuban, 2008). Such initial findings suggest that potential disciplinary effects in e-learning should be a focus of prospective researchers.

Increased Global Coverage and Cross-Cultural Studies

Although our book has a multi-national pool of contributors, regional attitudes toward e-learning are, thus far universally positive.  For example, portions of the Middle East view e-learning and distance education with disdain (Rovai, 2009), as is indicated in part by studies from the region that resort to using prison inmates as research samples (Al Saif, 2007). As universities from other parts of the world collaborate to create branch campuses or joint ventures with institutions in the Middle East, assessing influences of present attitudes toward and potential changes in attitudes toward the conduct of e-learning yields a productive stream of research. Also, although our book does not examine Asian e-learning settings, a review of contributors from prominent e-learning journals such as Computers & Education suggests that Asian scholars and institutions will become increasingly influential in shaping e-learning research agendas. We certainly would welcome collaborations between scholars from these emerging regions and those where e-learning has become comparatively well-established.

Issues in e-Learning Empirical Research

As we are now entering what may be a golden age of e-learning, we have witnessed increasing proportion of e-learning empirical research using highly sophisticated research tools such as structural equation modeling.

A review of the major works of Kuhn (1970a), Kaplan (1964), (Dubin 1969), and Cushing (1990) describes the process by which an academic discipline becomes an establishment in terms of four steps:

1. Consensus building among a group of scientists about the existence of a body of phenomena that is worthy of scientific study (Kuhn 1970a);
2. Empirical study of the phenomena to establish a particular fact or a generalization (Kaplan 1964);
3. Articulation of theories to provide a unified explanation of established empirical facts and generalizations (Kuhn 1970a); and
4. Paradigm building to reach a consensus on the set of elements possessed in common by practitioners of a discipline such as shared commitments, shared values, and shared examples (exemplars) (Kuhn 1970a).  

More than 30 years ago, Keen demanded three prerequisites to be fulfilled for the management Information Systems (MIS) area to be a coherent and substantive field. They are defining the dependent variables, clarifying the reference disciplines, building a cumulative research tradition.  An important objective of this book is an attempt to clearly define the dependent variables in e-learning empirical research. The review of Arbaugh et al. (2009) suggests that the e-learning systems area is building a cumulative research tradition through empirical and non-empirical research during the past decade. In our judgment, we are heading toward the stage of articulating e-learning theories to provide a unified explanation of established empirical facts and generalizations. To articulate e-learning theories, we need consensus building as to what dependent and independent variables are worthy of investigation. During the past decade, a large number of e-learning empirical studies were conducted to investigate the impacts of too numerous factors. To provide a useful lesson to the e-learning community, let us use an example from decision support systems (DSS) and group support systems (GSS) empirical research.  Eom summarized  the state of DSS/GSS empirical over the past decades this way (Eom, 2007, p.436):

A previous study (Farhoomand 1987) shows an increasing proportion of empirically based DSS research. Nevertheless, this accelerating rate of DSS research publication and the steady transition from non-empirical to empirical studies have not resulted in DSS theory building. Some researchers abandoned their efforts to develop context-free DSS theories and suggested that future DSS research should focus on modeling the specific "real world" target environment. This environment   is characterized in terms of organizational contexts, group characteristics, tasks, and EMS environments (Dennis et al. 1990-1991). Other empirical researchers continue their efforts to integrate the seemingly conflicting results of empirical experiments (Benbasat and Lim 1993). However, the considerable amount of empirical research in GDSS, user interface/individual differences, and implementation has produced conflicting, inconsistent results due to methodological problems, the lack of a commonly accepted causal model, different measures of dependent variables, hardware and software designed under different philosophies, etc. (Benbasat et al. 1993; Dennis et al. 1988; Jarvenpaa et al. 1985; Pinsonneault and Kraemer 1989; Zigurs 1993)

This problem of empirical research in the DSS/GSS areas possibly could reoccur in e-learning empirical research. The four causes that produced inconsistent/conflicting empirical results in the DSS are with e-learning empirical research. Some of evident problems in e-learning empirical research stem from the comparison of apples and oranges. Occasionally, some studies compare the results of studies based on samples of different subjects in terms of age and generations, gender, disciplines, course levels (undergraduate vs. graduate), demographics, socio-economic status, et cetera. Perceived e-learning outcomes and level of satisfactions are the results of interplay of many psychological, socio-economic, cultural, and other variables. In future e-learning empirical research, it may be a prudent direction to focus on the specific “real world” target environment, seeking to develop context-specific mid-range theory, rather than context free e-learning theory building. The models need to be parsimonious. At the same time, it must be complex enough to capture the reality of this world. Measurement issues are another area of concerns. These issues include the design of questionnaires, software issues, and as we see more empirical research, there is a need for informing the readers the details of survey instruments and list of questions used. For example, there was no information on survey instruments in some studies (LaPointe & Gunawardena, 2004). Further, there is a need for developing standardized indicator variables. Certainly, the same group of students’ responses may differ significantly when they respond to the two different questions to measure learning outcomes of online education.

• I have learned a lot in this course (Peltier, Drago, & Schibrowsky, 2003).
• I feel that I learned more in online courses than in face-to-face courses (Eom, Ashill, & Wen, 2006).

There are also software issues. For example, the two common approaches for SEM are the covariance-based approach used in LISREL, AMOS, and EQS and the variance-based approach used in PLS-PC, PLS-Graph, Smart-PLS and XLSTAT-PLS.  The fundamental differences between LISREL and PLS are reflected in the PLS and LISREL algorithms and their optimum properties. The same data set may produce different sets of results (unacceptable vs. acceptable) with the covariance-based and variance based approach, due to the differences of the two approaches.


References


Al Saif, A. A. (2007). Prisoners’ attitudes toward using distance education whilst in prisons in Saudi Arabia. Issues in Informing Science and Information Technology, 4, 125-131.

Alavi, M., & Leidner, D. E. (2001). Research commentary: Technology-mediated learning – a call for greater depth and breadth of research. Information Systems Research, 12(1), 1-10.

Alexander, M. W., Perrault, H., Zhao, J. J., & Waldman, L. (2009). Comparing Aacsb faculty and student online learning experiences: Changes between 2000 and 2006. Journal of Educators Online, 6.

Allen, I. E., & Seaman, J. (2010). Learning on demand: Online education in the United States 2009. Wellesley, MA: Babson Survey Research Group.

Arbaugh, J. B. (2005). How much does “subject matter” matter? A study of disciplinary effects in online MBA courses. Academy of Management Learning & Education, 4(1), 57-73.

Arbaugh, J. B., Godfrey, M. R., Johnson, M., Pollack, B. L., Niendorf, B., & Wresch, W. (2009). Research in online and blended learning in the business disciplines: Key findings and possible future directions. Internet and Higher Education, 12, 71-87.

Arbaugh, J. B., & Rau, B. L. (2007). A study of disciplinary, structural, and behavioral effects on course outcomes in online MBA courses. Decision Sciences Journal of Innovative Education, 5(1), 65-94.

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, E., Tamim, R. M., Surkes, M. A., et al. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79, 1243-1289.

Bernard, R. M., Abrami, P. C., Lou, Y., & Borokhovski, E. (2004). A methodological morass? How can we improve quantitative research in distance education. Distance Education, 25(2), 175-198.

Bonk, C. J. (2009). The world is open: How Web technology is revolutionizing education. San Francisco, CA: Jossey-Bass.

Connolly, M., Jones, C., & Jones, N. (2007). New approaches, new vision: Capturing teacher experiences in a brave new online world. Open Learning, 22(1), 43-56.

Coppola, N. W., Hiltz, S. R., & Rotter, N. G. (2002). Becoming a virtual professor: Pedagogical roles and asynchronous learning networks. Journal of Management Information Systems 18, 169-189.

Eom, S. B. (2007). The development of decision support systems research: A bibliometrical approach. Lewiston, NY: The Edwin Mellen Press.

Eom, S. B., Ashill, N., & Wen, H. J. (2006). The determinants of students' perceived learning outcome and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education, 4(2), 215-236.

European Commission. (2010). ICT results, educating Europe: Exploiting the benefits of ICT.   Retrieved August 31, 2010, from http://cordis.europa.eu/ictresults/pdf/policyreport/INF%207%200100%20IST-R%20policy%20report-education_final.pdf

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2, 87-105.

Hayes, S. K. (2007). Principles of finance: Design and implementation of an online course. MERLOT Journal of Online Teaching and Learning, 3, 460-465.

Haytko, D. L. (2001). Traditional versus hybrid course delivery systems: A case study of undergraduate marketing planning courses. Marketing Education Review, 11(3), 27-39.

Hiltz, S. R., & Turoff, M. (1978). The network nation: Human communication via computer. Reading, MA: Addison-Wesley.

Hornik, S., Sanders, C. S., Li, Y., Moskal, P. D., & Dziuban, C. D. (2008). The impact of paradigm development and course level on performance in technology-mediated learning environments. Informing Science, 11, 35-57.

Kellogg, D. L., & Smith, M. A. (2009). Student-to-student interaction revisited: A case study of working adult business students in online course. Decision Sciences Journal of Innovative Education, 7(2), 433-454.

LaPointe, D. K., & Gunawardena, C. N. (2004). Developing, testing and refining of a model to understand the relationship between peer interaction and learning outcomes in computer-mediated conferencing. Distance Education, 25(1), 83-106.

Liu, X., Magjuka, R. J., Bonk, C. J., & Lee, S. (2007). Does sense of community matter? An examination of participants’ perceptions of building learning communities in online course. Quarterly Review of Distance Education, 8(1), 9-24.

Marriott, N., Marriott, P., & Selwyn, N. (2004). Accounting undergraduates’ changing use of ICT and their views on using the internet in higher education – a research note. Accounting Education, 13(S1), 117-130.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: U.S. Department of Education.

Morgan, G., & Adams, J. (2009). Pedagogy first: Making Web technologies work for soft skills development in leadership and management education. Journal of Interactive Learning Research, 20, 129-155.

Peltier, J. W., Drago, W., & Schibrowsky, J. A. (2003). Virtual communities and the assessment of online marketing education. Journal of Marketing Education, 25(3), 260-276.
Phipps, R., & Merisotis, J. (1999). What’s the difference: A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC. Retrieved from www.ihep.com/PUB.htm

Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: A research framework and a preliminary assessment of effectiveness in basic IT skills training. MIS Quarterly, 25(4), 401-426.

Rovai, A. P. (2009). The Internet and higher education: Achieving global reach. Cambridge, UK: Chandos Publishing.

Schniederjans, M. J., & Kim, E. B. (2005). Relationship of student undergraduate achievement and personality characteristics in a total Web-based environment: An empirical study. Decision Sciences: Journal of Innovative Education, 3(2), 205-221.
Shea, P. (2007). Bridges and barriers to teaching online college courses: A study of experienced online faculty in thirty-six colleges. Journal of Asynchronous Learning Networks, 11(2), 73-128.

Simmering, M. J., Posey, C., & Piccoli, G. (2009). Computer self-efficacy and motivation to learn in a self-directed online course. Decision Sciences Journal of Innovative Education, 7(1), 99-121.

Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning & Education, 9, 169-191.

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of Web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, 623-664.

Smith, G. G., Heindel, A. J., & Torres-Ayala, A. T. (2008). E-learning commodity or community: Disciplinary differences between online courses The Internet and Higher Education 11, 52-159.

Stahl, B. C. (2004). E-teaching – the economic threat to the ethical legitimacy of education? Journal of Information Systems Education, 15, 155-162.

Tanner, J. R., Noser, T. C., & Totaro, M. W. (2009). Business faculty and undergraduate students’ perceptions of online learning: A comparative study. Journal of Information Systems Education, 20(1), 29-40.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of Information Technology: Toward a unified view. MIS Quarterly, 27(3), 425-478.

Wan, Z., Wang, Y., & Haggerty, N. (2008). Why people benefit from e-learning differently: The effects of psychological processes on e-learning outcomes. Information & Management, 45, 513-521.

Wilkes, R. B., Simon, J. C., & Brooks, L. D. (2006). A comparison of faculty and undergraduate students’ perceptions of online courses and degree programs. Journal of Information Systems Education, 17, 131-140.