Research methodology Research methodology and issues in e-learningand issues in e-learning

Chin-Chung Tsai

National Taiwan University of Science and Technology, Taipei, Taiwan

OutlineOutline

1.1. Introduction Introduction

2.2. Research purposesResearch purposes

3.3. MethodsMethods

4.4. Results and DiscussionResults and Discussion

5.5. Implication and ConclusionsImplication and Conclusions

6.6. Limitation of the ResearchLimitation of the Research

IntroductionIntroduction Content analysis and citation analysisContent analysis and citation analysis

of published articles in academic journals of published articles in academic journals has been conducted in a variety of has been conducted in a variety of professional fields. We felt that it was professional fields. We felt that it was necessary to conduct a content analysis necessary to conduct a content analysis for the reference of future studies which for the reference of future studies which are interested in related issues. are interested in related issues.

Articles with more citation frequencies are Articles with more citation frequencies are usually those that are better recognized by usually those that are better recognized by others in related fields. They probably others in related fields. They probably present more fundamental ideas about the present more fundamental ideas about the issues for future research. issues for future research.

IntroductionIntroduction

An analysis of technology-enhanced or e-An analysis of technology-enhanced or e-learning studies from a cognitive learning studies from a cognitive perspective is quite essential to acquire a perspective is quite essential to acquire a better understanding of how advanced better understanding of how advanced technology may help learning processes. technology may help learning processes.

Wentling et al. (2000) suggest that “e-Wentling et al. (2000) suggest that “e-learning is the acquisition and use of learning is the acquisition and use of knowledge distributed and facilitated knowledge distributed and facilitated primarily by electronic means” .primarily by electronic means” .

Research purposesResearch purposes

The purpose of the paper was to The purpose of the paper was to investigate the research trends in the investigate the research trends in the field of cognition in e-learning. field of cognition in e-learning.

The research questions addressed by this The research questions addressed by this study were :study were :

1.1. What percentage of the articles What percentage of the articles published in these journals were published in these journals were related to cognition in e-learning?related to cognition in e-learning?

Research purposesResearch purposes2.2. What research topics related to What research topics related to

cognition in e-learning were published cognition in e-learning were published in the journals in these five years?in the journals in these five years?

3.3. What were the relatively popular What were the relatively popular research topics related to cognition in research topics related to cognition in e-learning published in the journals?e-learning published in the journals?

4.4. What were the highly cited research What were the highly cited research articles in the SSCI related to cognition articles in the SSCI related to cognition in e-learning in these journals?in e-learning in these journals?

5.5. What was the trend of the published What was the trend of the published cognition in e-learning articles in these cognition in e-learning articles in these five journals in recent years? five journals in recent years?

MethodsMethods

This study analyzed papers published by This study analyzed papers published by five major educational journalsfive major educational journals that that are categorized as being educational-are categorized as being educational-technology-related in the SSCI from 2001-technology-related in the SSCI from 2001-2005:2005:

1.1. British Journal of Educational Technology (BJET)British Journal of Educational Technology (BJET)2.2. Innovations in Education and Teaching Innovations in Education and Teaching

International (IETI)International (IETI)3.3. Computers and Education (CE)Computers and Education (CE)4.4. Educational Technology Research & Educational Technology Research &

Development (ETR&D)Development (ETR&D)5.5. Journal of Computer Assisted Learning (JCAL). Journal of Computer Assisted Learning (JCAL).

MethodsMethods This study only analyzed papers that were This study only analyzed papers that were

identified as “articles” in the SSCI. Other types identified as “articles” in the SSCI. Other types of papers such as “book reviews”, “reviews”, of papers such as “book reviews”, “reviews”, and “editorial materials” were all excluded and “editorial materials” were all excluded from this study. from this study.

Excluding these other types of papers, there Excluding these other types of papers, there were a were a total of 1,027 articlestotal of 1,027 articles published by published by these five journals from 2001 to 2005. these five journals from 2001 to 2005.

Based on these articles, Based on these articles, a content analysisa content analysis was carried out, using article abstracts and was carried out, using article abstracts and publication information indexed in the SSCI. publication information indexed in the SSCI.

This procedure This procedure identified 444 articlesidentified 444 articles that that were judged to be relevant to the topic of were judged to be relevant to the topic of cognition in e-learning.cognition in e-learning.

MethodsMethods

Based on the methodology of content Based on the methodology of content analysis, the research topics were analysis, the research topics were categorized related to cognition in e-categorized related to cognition in e-learning into seven tentative categories: learning into seven tentative categories:

A. MotivationA. MotivationB. Information ProcessingB. Information ProcessingC. Instructional approachesC. Instructional approachesD. Learning environmentD. Learning environmentE. Prior KnowledgeE. Prior KnowledgeF. MetacognitionF. MetacognitionG. Cognitive psychology characteristics G. Cognitive psychology characteristics

For each category, some sub-categories For each category, some sub-categories were classified.were classified.

MethodsMethods In order to identify articles of higher quality In order to identify articles of higher quality

or impact, the 444 related articles were or impact, the 444 related articles were further analyzed by their citation counts in further analyzed by their citation counts in the SSCI the SSCI (as at October, 17, 2006).(as at October, 17, 2006).

Three articles with the highest citation Three articles with the highest citation counts in each research topic category counts in each research topic category were selected for this analysis. Because were selected for this analysis. Because several of the articles discussed topics from several of the articles discussed topics from several categories, several categories, a total of 16 articles a total of 16 articles were finally chosen.were finally chosen.

Their participants, research settings, Their participants, research settings, research design type, and research research design type, and research methods (data collection and data analysis) methods (data collection and data analysis) were further analyzed in detail.were further analyzed in detail.

OutlineOutline

1.1. IntroductionIntroduction2.2. Research purposeResearch purpose3.3. MethodsMethods4.4. Results and DiscussionResults and Discussion

1)1) Percentage of articles related to cognition in Percentage of articles related to cognition in e-learninge-learning

2)2) Research topic analysisResearch topic analysis3)3) An analysis of highly cited articlesAn analysis of highly cited articles

5.5. Implication and ConclusionImplication and Conclusion6.6. Limitation of the ResearchLimitation of the Research

Percentage of articles related to Percentage of articles related to cognition in e-learningcognition in e-learning （（ 2001-2001-

20052005 ）） Almost half (43.2%, 444 of 1,027 Almost half (43.2%, 444 of 1,027

articles) of the articles were related. articles) of the articles were related. IETI had 58.4%, BJET had 46.8%, JCAL IETI had 58.4%, BJET had 46.8%, JCAL

had 42.5%, CE had 37.1%, and ETR&D had 42.5%, CE had 37.1%, and ETR&D had 28% had 28%

2003 had the highest percentage 2003 had the highest percentage (49.8%), while 2002 had the least (49.8%), while 2002 had the least (37.4%). (37.4%).

Percentage of articles related to Percentage of articles related to cognition in e-learningcognition in e-learning （（ 2001-2001-

20052005 ））

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

2001 2002 2003 2004 2005

BJET

IETI

CE

ETR&D

JCAL

Percentage of articles related to Percentage of articles related to cognition in e-learningcognition in e-learning （（ 2001-2001-

20052005 ））

444/1027444/1027

(43.2%)(43.2%)90/22590/225

(40%)(40%)87/19587/195

(44.6%)(44.6%)

102/205102/205

(49.8%)(49.8%)79/21179/211

(37.4%)(37.4%)86/19186/191

(45%)(45%)Total Total

91/21491/214

(42.5%)(42.5%)14/3814/38

(36.8%)(36.8%)14/4114/41

(34.2%)(34.2%)24/4724/47

(51.1%)(51.1%)16/4516/45

(35.6%)(35.6%)23/4323/43

(53.5%)(53.5%)JCAL JCAL

35/12535/125

(28%)(28%)7/267/26

(26.9%)(26.9%)7/217/21

(33.3%)(33.3%)8/268/26

(30.8%)(30.8%)6/306/30

(20.0%)(20.0%)7/227/22

(31.8%)(31.8%)ETR&D ETR&D

86/232 86/232

(37.1%)(37.1%)15/4715/47

(31.9%)(31.9%)16/4716/47

(34.0%)(34.0%)20/4620/46

(43.5%)(43.5%)19/5019/50

(38.0%)(38.0%)16/4216/42

(38.1%)(38.1%)CE CE

94/161 94/161

(58.4%)(58.4%)13/2813/28

(46.4%)(46.4%)20/3020/30

(66.7%)(66.7%)25/3425/34

(73.5%)(73.5%)20/3220/32

(62.5%)(62.5%)16/3716/37

(43.2%)(43.2%)IETI IETI

138/295 138/295

(46.8%)(46.8%)41/8641/86

(47.7%)(47.7%)30/5630/56

(53.6%)(53.6%)25/5225/52

(48.1%)(48.1%)18/5418/54

(33.3%)(33.3%)24/4724/47

(51.1%) (51.1%) BJET BJET

Total Total 2005 2005 2004 2004 2003 2003 2002 2002 2001 2001

Research Topic analysis (2001-Research Topic analysis (2001-2005)2005)

The three most published research topics are:The three most published research topics are:• learning environment-interactive learning learning environment-interactive learning

environment (110)environment (110)• instructional approaches-collaborative learning instructional approaches-collaborative learning

(92)(92)• metacognition-perception and awareness (75).metacognition-perception and awareness (75).

As the concepts of active learning and lifelong As the concepts of active learning and lifelong learning have become more and more popular learning have become more and more popular and important nowadays, the research topic and important nowadays, the research topic of metacognition requires more attention. of metacognition requires more attention.

Frequencies of cognition Frequencies of cognition category and sub-categorycategory and sub-category3636Problem-Problem-

based learningbased learning

2323 Cognitive Cognitive style style 1616Situated Situated

learninglearning

1414Cognitive Cognitive loadload2626Contextual Contextual

learninglearning

77Mental ModelMental Model9292Collaborative Collaborative learninglearning

1818Concept mapConcept map2121Cooperative Cooperative learninglearning

C. Instructional C. Instructional ApproachesApproaches

1010 SchemataSchemata

G. Cognitive G. Cognitive PsychologyPsychologyCharacteristicsCharacteristics

1111Problem Problem solving solving

2626Self-Self-evaluationevaluation44Decision Decision

makingmaking

7575Perception & Perception & awarenessawareness1919Critical Critical

thinkingthinking

1313VisualizationVisualization2020Information Information management management

1414 PlanningPlanning

F. Meta-F. Meta- cognitioncognition

1919Information Information seekingseeking

3232 Technology Technology experienceexperience2525Individual Individual

differencedifference

B. Information B. Information ProcessingProcessing

1515Technology Technology knowledgeknowledgeE. Prior E. Prior

KnowledgeKnowledge

2828Behavioral Behavioral changechange

4242Learning Learning community community 6565Attitudes Attitudes

110110InteractiveInteractiveLearning Learning

EnvironmentEnvironment D. Learning D. Learning EnvironmentEnvironment

3131BeliefsBeliefs

A. Motivation A. Motivation

FrequencyFrequencyCountCount

Sub-Sub-categorycategoryCategoryCategoryFrequencyFrequency

CountCount Sub-categorySub-categoryCategoryCategory

Research Topic analysis (2001-2005)Research Topic analysis (2001-2005)

7.94%7.94%8.88%8.88%3.27%3.27%13.0813.08%%14.9514.95%%12.1512.15%%11.6811.68%%JCALJCAL

8.00%8.00%8.8%8.8%2.40%2.40%9.69.6%%10.4010.40%%7.27.2%%8.008.00%% ETR&DETR&D

3.88%3.88%9.91%9.91%3.44%3.44%14.2214.22%%16.8116.81%%12.0712.07%%10.7810.78%% CECE

4.97%4.97%18.63%18.63%4.97%4.97%19.8819.88%%26.0926.09%%4.974.97%%8.7 8.7 %%IETIIETI

8.81%8.81%13.22%13.22%4.35%4.35%13.2213.22%%11.5311.53%%5.425.42%%8.478.47%% BJETBJET

GGFFEEDDCCBBAA CategoryCategory

JournalJournal

A: Motivation B: Information Processing C: Instructional Approaches

D: Learning Environment E: Prior Knowledge F: Metacognition

G: Cognitive Psychology Characteristics

OutlineOutline 1.1. IntroductionIntroduction2.2. Research purposesResearch purposes3.3. MethodsMethods4.4. Results and DiscussionResults and Discussion

1)1) Percentage of articles related to cognition in Percentage of articles related to cognition in e-learninge-learning

2)2) Research topic analysisResearch topic analysis3)3) An analysis of highly cited articlesAn analysis of highly cited articles

5.5. Implication and ConclusionsImplication and Conclusions6.6. Limitation of the ResearchLimitation of the Research

Topic Study Participants and Research Setting

Research Design/Type

DataCollection

DataAnalysis

Citation

Motivation Tsai, Lin, & Tsai, (2001). CE, 37, 41-51.

753 high school students at eight schools from northern, central, and southern Taiwan

Descriptive Questionnaire Descriptive, statistical analysis(factor analysis, ANOVA)

17

Motivation/ Metacognition

Mumtaz (2001). CE, 36, 347-362.

360 year 3 and year 5 pupils selected from schools in the UK

Descriptive Questionnaire Descriptive (SD) statistical analysis(ANOVA )

11

Motivation/ Metacognition

Metzger, Flanagin & Zwarun (2003). CE, 41, 271-290.

Study 1: 356 undergraduate students Study 2: 436 undergraduate students

Descriptive Questionnaire Descriptive (mean and SD) statistical analysis (ANOVA)

11

Instructional Approaches

Chang (2001). JCAL, 17, 263-274.

137 tenth gradestudents in Taiwan

Experimental Testing, Questionnaire

Descriptive statistical analysis(MANCOVA, ANCOVA)

10

An analysis of high-citation papers

Topic Study Participants and Research Setting

Research Design/Type

DataCollection

DataAnalysis

Citation

Information Processing / Instructional Approaches

Oliver & Omari (2001). JCAL, 17, 34-47.

240 college students

Descriptive Interviews, online question-naires

Descriptive(frequencies)

18

Information Processing/Cognitive Psychology Characteristics

Ford & Chen (2001). BJET, 32, 5-22.

73 post-graduate students

Experimental Personal survey

Statistical analysis (ANOVA and t tests)

18

Information Processing / Prior Experience /Learning Environment

Papanikolaou, Grigoriadou, Magoulas & Kornilakis (2002). CE, 39, 333-360.

10 Undergraduate students

Developmental (with pilot study)

Participants’ answers to questions embedded in the scenario and log files of interactions

Descriptive(frequencies)

9

Instructional Approaches

Zurita & Nussbaum (2004). CE, 42, 289-314.

48 students ranging 6-7 years in one low-income school.

Descriptive Classroom observations (videotapes), field notes, interviews

Descriptive statistical (mean, SD) analysis (t tests)

11

An analysis of high-citation papers(continued)

Topic Study Participants and Research Setting

Research Design/Type

DataCollection

DataAnalysis

Citation

Instructional Approaches

Ligorio (2001). CE, 37, 103-125.

40 primary or secondary students, 18 teachers, and 6 researchers

Descriptive Chat logs, text-based messages, photos

Discourse analysis

7

Learning Environment

Rovai (2001). ETR & D, 49(4), 33-48.

20 adult learners in an online course

Descriptive Questionnaire, online messages, course statistical data

Descriptive(mean and SD)

9

Learning Environment

Pena-Shaff & Nicholls (2004). CE, 42, 243-265.

22 under-graduates, 8 graduates, and 5 university employees

Descriptive Outline or index of BBS postings, transcripts of electronic discussions

Descriptive 8

Learning Environment/Meta-cognition

Brusilovsky (2003). BJET, 34, 487-497.

64 Under-graduate students in a total of two studies

Descriptive, Developmental

Questionnaire Descriptive 8

An analysis of high-citation papers(continued)

Topic Study Participants and Research Setting

Research Design/Type

DataCollection

DataAnalysis

Citation

Prior Experience

Lazonder, Wilhelm & Ootes (2003). CE, 41, 291-308.

Study 1:36 secondary studentsStudy 2:43 secondary students

Developmental

Questionnaire, chat logs,

Discourse analysis, Descriptive

5

Prior Experience

Steel & Hudson (2001). IETI, 38, 103-111.

11 members of the full-time teaching staff at SHU

Descriptive Interviews Interpretative 5

Cognitive Psychology Characteristics

Chang, Sung & Chen (2001). JCAL, 17, 21-33.

48 seventh-grade students selected from three classes in Taipei

Experimental Testing, questionnaire

Descriptive statistical analysis (, ANCOVA, chi-square and post hoc tests )

10

Cognitive Psychology Characteristics

Lin, Liu, & Yuan (2001). JCAL, 17, 420-432.

58 computer science majors in Taiwan

Experimental Testing, questionnaire

Descriptive statistical analysis (ANOVA)

9

An analysis of high-citation papers(continued)

An analysis of highly cited articlesAn analysis of highly cited articles ：：TopicsTopics

The most highly cited articles fall into the The most highly cited articles fall into the research categories of:research categories of:1.1. ““Information processing/instructional Information processing/instructional

approaches” (Oliver & Omari, 2001; approaches” (Oliver & Omari, 2001; citation counts = 18)citation counts = 18)

2.2. ““Information processing/cognitive Information processing/cognitive psychology characteristics” (Ford & Chen, psychology characteristics” (Ford & Chen, 2001; citation counts = 18) 2001; citation counts = 18)

3.3. ““Motivation” (Tsai, Lin & Tsai, 2001; Motivation” (Tsai, Lin & Tsai, 2001; citation counts = 17) citation counts = 17)

““Instructional Approaches”, “Information Instructional Approaches”, “Information Processing” and “Motivation” may be Processing” and “Motivation” may be perceived as the research foundation for perceived as the research foundation for e-learning studies. e-learning studies.

An analysis of highly cited articlesAn analysis of highly cited articles ：： Participants and Research SettingParticipants and Research Setting

College students (38%) and students in College students (38%) and students in elementary and secondary schools (38%) elementary and secondary schools (38%) were the most popular participants among were the most popular participants among the 16 articles.the 16 articles.

It is reasonable to suggest that future It is reasonable to suggest that future studies should pay more attention to adult studies should pay more attention to adult learners outside of formal school learners outside of formal school environments. environments.

An analysis of highly cited articlesAn analysis of highly cited articles ：： Research DesignResearch Design

These papers were classified into These papers were classified into three major types: three major types:

1.1. Experimental research Experimental research

2.2. Descriptive researchDescriptive research (10)(10)

3.3. Developmental researchDevelopmental research Descriptive research can better describe Descriptive research can better describe

the complicated relationships between the complicated relationships between the different variables involved in a the different variables involved in a certain situation. certain situation.

An analysis of highly cited articlesAn analysis of highly cited articles ：： Data CollectionData Collection

More than half of the studies (ten articles) More than half of the studies (ten articles) used the used the questionnairequestionnaire as their primary as their primary data collection technique or as one of their data collection technique or as one of their data collection methods. data collection methods.

However, there is also a clear trend that However, there is also a clear trend that many studies in this field used many studies in this field used learners’ learners’ log files or online messageslog files or online messages as the as the data sources for analysis.data sources for analysis.

An analysis of highly cited articlesAn analysis of highly cited articles ：： Data AnalysisData Analysis

Among the 16 studies:Among the 16 studies:• Seven used both descriptive and Seven used both descriptive and

statistical analysis to analyze datastatistical analysis to analyze data

• Five utilized descriptive analysisFive utilized descriptive analysis Online technology or databases used in Online technology or databases used in

e-learning can keep records about the e-learning can keep records about the learners’ learning processes (e.g. learners’ learning processes (e.g. discourse, online interactions).discourse, online interactions).

An analysis of highly cited articlesAn analysis of highly cited articles ：： Data AnalysisData Analysis

Therefore, it is more convenient for Therefore, it is more convenient for researchers to carry out researchers to carry out discourse discourse or interpretative researchor interpretative research in in online environments.online environments.

OutlineOutline

1.1. IntroductionIntroduction

2.2. Research purposesResearch purposes

3.3. MethodsMethods

4.4. Results and DiscussionResults and Discussion

5.5. Implication and ConclusionImplication and Conclusion

6.6. Limitation of the ResearchLimitation of the Research

Implication and ConclusionsImplication and Conclusions

It is suggested that future studies It is suggested that future studies focus more on adult learners outside focus more on adult learners outside of formal school settings. of formal school settings.

Some of the highly-cited articles Some of the highly-cited articles reviewed in this paper have used reviewed in this paper have used mixed method research (e.g. mixed method research (e.g. Lazonder, Wilhelm & Ootes, 2003; Lazonder, Wilhelm & Ootes, 2003; Zurita & Nussbaum, 2004). It may Zurita & Nussbaum, 2004). It may become a major trend in this become a major trend in this research field. research field.

Implication and ConclusionsImplication and Conclusions Because online technology or databases in e-Because online technology or databases in e-

learning keep records of learners’ learning learning keep records of learners’ learning processes, it allows researchers to make processes, it allows researchers to make interpretations based on log files, discourse, interpretations based on log files, discourse, or interaction records. It seems that the new or interaction records. It seems that the new study field of e-learning has brought study field of e-learning has brought researchers researchers new methods of data new methods of data collection.collection.

How to maintain and enhance students’ How to maintain and enhance students’ learning motivationlearning motivation and teachers’ and teachers’ teaching motivationteaching motivation in this constantly in this constantly changing educational environment will be an changing educational environment will be an essential issue for future studies.essential issue for future studies.

Limitation of the ResearchLimitation of the Research

It is suggested that a combination It is suggested that a combination of citation analysis and other of citation analysis and other analysis techniques can be applied analysis techniques can be applied in future studies in this domain. in future studies in this domain.

This study focused primarily on This study focused primarily on English-based SSCI journals and English-based SSCI journals and articles.articles.