microsoft word - literature review...

Running head: The Efficacy of Technology-Enhanced Instructional Strategies 1

The Efficacy of Technology-Enhanced Instructional Strategies to

Promote Critical Thinking in Middle School Mathematics

Catherine P. Henderson

Kennesaw State University

2The Efficacy of Technology-Enhanced Instructional Strategies

Results from National Assessment of Educational Progress (NAEP), Program for

International Student Assessment (PISA) and Trends in International Mathematics and Science

Study (TIMSS) provide insight into student achievement in mathematics in the United States as

well as individual states. These findings indicate that math education in the United States is not

as effective as math education in other countries. For example, according to the 2013 NAEP,

only 35 percent of 8th graders in the United States were proficient in mathematics (National

Center for Educational Statistics, 2014). Fifteen-year-old students in the United States

performed at the 27th rank, below average in math among nations in the 34 member nations

belonging to the Organisation for Economic Cooperation and Development (OECD)

(Organisation for Economic Cooperation and Development, 2012). According to the Trends in

International Mathematics and Science Study (TIMSS, 2011), the average mathematics score for

8th-grade students in the United States has not increased significantly since 2007.

Researchers associate the low performance of U.S. students with the low expectations set

by states under the 2002 federal law, No Child Left Behind (NCLB) (Bandeira de Mello, 2011;

Peterson & Kaplan, 2013). Based on a comparison to the 2011 NAEP standards and state testing

standards in reading and math, Georgia was ranked as the state with the lowest expectations

(Peterson & Kaplan, 2013). In 2010, the Georgia State Board of Education adopted new

mathematics content standards, Common Core Georgia Performance Standards (CCGPS), in

order to improve student achievement in a manner that prepares students for college and careers.

Georgia Milestones, a new state assessment system was administered for the first time in spring

2014 to measure the effectiveness of CCGPS (Beaudette, 2014).

The Georgia Milestones assessment will measure the success of instructional strategies

that represent a shift in the way that students learn mathematics. The Common Core State


Standards Initiative (2014) describes one of the main changes in mathematics education as the

need to increase rigor that focuses equally on conceptual understanding, procedural skills and

fluency, as well as application. Educators will need to adapt research-based strategies that

support “the critical-thinking, problem-solving, and analytical skills students will need to be

successful” (Common Core State Standards Initiative, 2014). The delivery and intent of

instruction will need to adapt to new standards that require more than rote memorization of static

processes with shallow levels of content knowledge. The Common Core standards “focus on

skills in critical thinking and problem-solving that will require non-traditional technology-based

strategies” (Roblyer & Doering, 2013, p. 17). Researchers assert that Web 2.0 tools such as

blogs enhance critical thinking skills in the classroom (Anderson P. , 2007; Karsak, Fer, &

Orhan, 2014; Zhang, 2013).

Purpose Statement

The purpose of this study was to determine the efficacy of technology-enhanced instructional

strategies (weblogs or blogs) in increasing critical thinking skills of my math students. The

Common Core State Standards not only require disciplinary knowledge, but also set expectations

for students as critical thinkers who analyze information in the completion of daily activities

(National Governors Association Center for Best Practices & Council of Chief State School

Officers, 2010). Ennis (1987) refers to critical thinking as a reasoning process where reflection

determines actions or beliefs.

The effectiveness of weblogs to facilitate reflection has also been examined. (Anderson,

Rohrer, Taylor, & Trimarco, 2006; Li, Bado, Smith, & Moore, 2013; Muncy, 2014). Weblogs

are communication tools with the potential to bring the student together to collaborate, share

knowledge, reflect and explain (Dos & Demir, 2013). Sevelj (2006) says that, modern

technology such as weblogs provides opportunities to improve how students think. Weblogs


facilitate strategies such as journal writing and peer feedback which promote deep thinking and

learning. (Xie, Ke, & Sharma, 2008).

Research Questions

This study was designed to examine the following research questions:

What is the effect of using a rubric with a critical thinking criterion on student's approach

to learning?

Is there a significant difference in the indicators of cognitive presence when critical

thinking is a criterion for assessment, the indicators of cognitive presence increase?

Is there a significant difference in student performance when critical thinking is a

criterion for constructed response questions?

Importance of the Study

The goal of this research is to conduct a mixed-methods study to investigate and explain

the use of a rubric with weblogs to increase critical thinking in mathematics. This study is

important because it will inform how I support instructional strategies that promote deeper

conceptual understanding in middle school mathematics, the foundation for algebra and

advanced high school mathematics. “Completing advanced math courses in high school has a

greater influence on whether students will graduate from college than any other factor, including

family background; students who take math beyond Algebra II double their chances of earning a

bachelor’s degree” (Adelman, 2006). In fact, future earnings, regardless of college degrees,

grades, family background or any other factors are higher for students that have completed

advanced high school mathematics (Rose & Betts, 2004). Developing critical thinking skills in

middle school mathematics is essential to developing learners who recognize patterns, see

relationships, form generalizations, and master Algebra.


This study investigates the relationship between weblogs that are assessed for evidence of

critical thinking and the level of critical thinking that results. Given the online implementation

of high-stakes, constructed-response testing based on the Common Core standards-based

mathematics curriculum, providing strategies that are support critical thinking with weblogs is

timely and practical. Schools such as mine are faced with unique challenges because we have

received a focus school designation. Focus schools have an achievement gap between the

highest-achieving subgroups and the lowest-achieving subgroups within the school (The State of

Georgia, 2014).

Based on current research, weblogs are not always effective in increasing critical thinking

(Li, Bado, Smith, & Moore, 2013; Richardson W. , 2010). Richardson (2010) states that

students engage in blogging, but critical thinking, analytical writing and reflection are not

evident. Although clear communication of objectives is a significant factor in the success of

weblogs, critical thinking is not an important factor in scoring them (Hsaio, Chen, & Hu, 2013;

Zhang, 2013).

Definition of Terms

Critical thinking is knowledge generation using analytic and quantitative reasoning,

problem-solving and writing (Benjamin et al., 2013). Critical thinking skills are desirable

outcomes of education (Dewey, 1910). They are required for analyzing the information that is

necessary to understand the complex challenges of the 21st century workplace (The New

Commission on the Skills of the American Workforce, 2006; The Secretary’s Commission On

Achieving Necessary Skills, 1991). Reflection is a cycle of inquiry for the purpose of making

meaning or finding solutions for a troubling situation or question (Xie, Ke, & Sharma, 2008).

“Weblogs, or ‘blogs’, as they are called, can best be defined as Web sites that are easily created

and updated by those with even a minimum of technology know-how” (Bookhart, 2010).


Richardson (2004) describes the use of blogs as a new way of writing in learning, which is a

valuable tool for developing writing skills, information literacy, and critical thinking skills.

Literature Review

Search terms

The search terms that were used access articles for this literature include: weblogs, blogs,

asynchronous communication, academic discourse, journaling, constructed response, critical

thinking, cognitive processes, learning theories, and secondary education.

Background

A significant theme of this research review was student learning and specifically the

ways that a student approaches learning. As a result, I identified a theory that focuses on the

relationship between the situation and a person. Biggs (1993a) suggests that student learning

occurs within a model of teaching and learning measured by process inventories within an open

system.

Two instruments have been designed to measure how students approach learning tasks

that promote critical thinking. The Revised Learning Process Questionnaire for secondary

grades has been simplified to examine two of three approaches used prior to the revision. The

table in Appendix A lists the subcomponents of the two approaches, the Surface Approach, and

the Deep Approach. The Surface Approach and the Deep Approach defined in the context of the

Presage-Process-Product (3P) model of student learning (Biggs, 1987a, 1993a, 1993b). The 3P

model refers to the relationship between prior knowledge (Presage factor), the student approach

to the task and level of engagement (Process factor). The last factor in the 3P model is a product

or learning outcome. This framework attributes the learning outcome to both the teacher and

student.

http://eric.ed.gov/?ti=Cognitive+Processes




Based on the literature, two themes that emerged that demonstrate the efficacy of

technology-based instructional strategies (Anderson, Rohrer, Taylor, & Trimarco, 2006; Dos &

Demir, 2013; Karsak, Fer, & Orhan, 2014; Li, Bado, Smith, & Moore, 2013; Muncy, 2014;

Richardson & Ice, 2010; Zhang, 2013). They are critical thinking skills and teaching with blogs.

Critical Thinking

In a recent study, Kun promoted blogs as a critical thinking tool. Focus group and survey

were the instruments used to determine the affect of peer commenting on critical thinking. The

study lasted for eight weeks and had a minimal number of participants. Even so, it yielded a

more effective framework for implementing blogs. (Kun, Niamboue, Smith, & Moore, 2013)

Current Research on Critical Thinking Facilitated by Online Discussion

Cognitive density and higher order thinking skills relationships were investigated,

comparing two populations, in a quantitative analysis. (Lee, 2014). The results were there is no

correlation between cognitive density and higher order thinking skills (Lee, 2014). In a mixed

methods study that utilized online discussion, and compared the effect of instructional techniques

on critical thinking and critical thinking disposition, a similar conclusion was reached.

Instructional techniques Role Playing, Socratic Seminar, Six Thinking Hats, Brainstorming, and

Anyone Here an Expert, were utilized. The results concluded the scores between pre-tests and

post-tests of critical thinking dispositions did not differ between the two groups, in the

quantitative study.

Current Research on the Assessment Methods That Identify Critical Thinking

Bookhart (2010) detailed the background information necessary to assess methods that

identify critical thinking skills. He also investigated the effect of these skills on the growth and

development of academics. Since he required students to engage in quality discourse that


produced exceptional performance, Bookhart provided details on how to intentionally plan for

students to do so (Bookhart, 2010).

System utility, training, and support are harbingers of a favorable perception of classroom

blogs. Software platform and expectations are also considerations that the author mentions.

Zhang suggests that blog objectives are clearly defined, for optimal success. He further

elaborates that the clearly defined objectives will optimize the blog as a feedback tool, and

enhances student success and satisfaction (Zhang, 2013).

A debate, case-based discussion, and an open-ended (or topical) discussion form the three

instructional strategies of this online discussions study. Mixed method, the study details

“… Critical thinking levels as defined in the Community of Inquiry (CoI) framework and

analyzed in the Practical Inquiry Model (PIM). The study evaluated (1) the contribution

of demographic criteria and student background to students' preferred instructional

strategy, (2) participants' preferred instructional strategy and rationales for the selection,

(3) based on the Practical Inquiry Model's (PIM) indicators, the contribution of students'

strategy preferences in determining level of critical thinking, , and (4) comparisons of

participants' critical thinking levels across instructional strategies” (Richardson & Ice,

2010, p. 150).

Teaching with Blogs

Using online blogs as a teaching tool was examined in this mixed method case study. The

group behavior of a group of cross-cultural students was its focus. The study measured

participation rates and content. The novelty of the blogging experience created value for the

students, according to the qualitative data from the study (Meinecke, Smith, & Lehmann-

Wellenbrock, 2013). In the section of his book that focuses on education theory, Sturm concurs


that using blogs in the classroom creates an authentic learning experience for students (Sturm,

Kennell, & McBride, 2009).

Asynchronous and synchronous tools differed in purpose and types of learning supported,

purports Hrastinski. Blogs are asynchronous; states Hrastinski Asynchronous discussion tools

facilitated learning over extended periods of time. When issues require time for reflection or are

complex, asynchronous e-learning and media is preferable to synchronous (Hrastinski, 2008).

Synchronous discussion tools facilitate collaborative learning and allow students to

become acquainted socially. These include messaging and chat. Subrahmanyam’s (2009)

research confirmed this social aspect of blogging. His researchers used coding to analyze the

format, content, tone and style of blog posts. They concluded that adolescents who blog, use the

tool for off-line life themes such as identity, romance, and the future, as well as on-line themes.

(Subrahmanyam, Garcia, Harsono, Li, & Lipana, 2009). Alvermann, (2008) agrees.

His research focuses on analyzing online literacies of adolescents for research and

classroom purposes. Alvermann theories state that adolescents in the United States use of social

media for blogging, sharing photographs, artwork, videos, or stories, and working on web pages.

He states that 93% of U. S. adolescents between 12 and 17 used the Internet for social reasons.

Methodology Design

I completed this study using action research with triangulation of mixed methods. The

research was completed at Mansion Middle School, a public middle school in the Clayton

County Public School district. This is the second cycle of action research that focuses on

students in my Accelerated Math classes. The cyclic model for action research is described by

Creswell as an iterative process that involves gathering data, analyzing the data,

communicating outcomes and working towards a solution.

As a result of the first cycle of action research, I identified the need to evaluate the use


of blogs before and after a critical thinking rubric is implemented. This study will use a mixed

methods design that includes 1) tests before and after treatment, 2) a quantitative content

analysis of weblogs, 3) a questionnaire to assess student approach to learning, and 4) a rubric

for student self-assessment. The Revised Learning Process Questionnaire (Kember, Biggs, &

Leung, 2004), an analytic rubric and weblog analysis will inform the data analysis.

Participants

Students from my accelerated math classes (79 total) participated in the study. The

cultural composition of the classes is 64% Asian, 32% Black (non-Hispanic), 2% Hispanic and

2% Caucasian. This is in contrast with a school population that consists of 61% African-

American, 18% Asian, 15% Hispanic, 3% Caucasian, and 3% Multi-racial students. Students in

the school population are more culturally and economically diverse; about 12% are students with

disabilities and 7% have limited English proficiency. Of the students that participated in the

study, one of the students is a special education student. Seventy-six percent the students in the

study were classified as gifted students. All of the students in the school are eligible for free or

reduced price lunch.

I selected the participants in the study using a convenience sampling method, based on

the composition of the classes that I teach and the participants that submitted completed assent

and consent forms. All student participants gave assent, and all of the parents of the students

gave consent prior to participation in any aspect of the study. Therefore, I will not assert that

the students in the study are a representation of the school’s population (Creswell, 2012). The

goal was to insure the students in the student represented the population of the courses that I

teach.

Instruments

Multiple qualitative and quantitative instruments were used to facilitate the triangulation


of data. According to Mills (2014), using qualitative and quantitative methods affords a fuller,

more synergistic understanding of research results. The triangulation matrix showing the

relationship between data sources, instruments and research questions is shown in Figure 1.

Research Questions Data Source

1 2 3Indicators of Cognitive Presence? Student

CTARWeblog Analysis using PIM (Before Intervention)

Weblog Analysis using PIM(After Intervention)

Student Approach to Learning? R-LPQ-2F(Before Intervention)

R-LPQ-2F(After Intervention)

Math Performance? AIMSWeb®Pre-test

AIMSWeb®Posttest

Figure 1. A triangulation matrix of data sources. This figure illustrates the relationships between data sources, instruments and research questions.

Revised Learning Process Questionnaire (R-LPQ-2F).

The Revised Learning Process Questionnaire used two scales to measure the student

approach to learning (SAL). The first scale is the deep approach, and the second scale is the

surface approach. According to Biggs (1993b), the goal of teaching is to transition students from

a surface approach to processing a task to a deeper more contextual approach to learning.

Appendix B shows The Revised Learning Process Questionnaire (R-LPQ-2F).

Appendix C features the components of the scales, the historical sets of subscales as well

as the methods for calculating scale scores. The approach scale was compared to the results of

the AIMSWeb® probes to measure the correlation between the approach to learning and student

performance. The approach scale was also used to identify any relationship between a student’s

approach to learning and the level of critical thinking in constructed response questions.

Process Inquiry Model.


The weblogs was analyzed to identify the critical thinking indicators. The Process

Inquiry Model (PIM) was one of the methods used to triangulate the results. The Process Inquiry

Model is part of the Community of Inquiry (CoI) framework. The CoI framework uses

categories to describe different levels of critical thinking. According to Garrison, Anderson, and

Archer (2001), the analysis of higher-order thinking processes is possible using specific

categories, indicators, and social-cognitive processes. Reference the list of categories and

associated indicators and social-cognitive processes in Appendix D.

Critical Thinking Analytic Rubric.

The Critical Thinking Analytical Rubric (CTAR) to will be used to assess the weblogs

based on a critical thinking metric. Students submitted an evaluation of each of their weblog

entries using the rubric. Refer to Appendix I for a sample of the CTAR. Each weblog entry was

evaluated by two scorers using the CTAR.

A score was computed based on six factors: interpretation, analysis, evaluation,

inference, explanation and disposition. A comparison of the score on the critical thinking rubric

(CTAR) to the indicators of critical thinking (PIM) was used to determine whether there was a

correlation between a student’s assessment of thinking skills and a higher level of critical

thinking when solving math problems.

AIMSweb® probes.

The pretest and posttest assessments are compatible with the Common Core Standards for

eighth-grade math. Reference the tests and instructions in Appendices F-G. Pearson (2010)

describes the AIMSweb® probes as Curriculum-Based Measures (CBM) of math performance

and notes that results of the test can predict the outcome on state tests.

Procedures


Because the students were under the age of 18, all of the participants signed an assent

form, and all parents signed a consent form before students become participants. The students

and parents were informed that participation in the study was voluntary and the student could

quit before the end of the study. They were also informed that participation or lack of

participation would not factor into their grades in my classes.

I distributed the parent consent forms to all students during homeroom. Teachers were

instructed to read the script that has been provided to ensure that there was not a sense of

obligation. Students returned the consent form regardless of whether parents granted

permission. The homeroom with the highest number of forms returned by the due date (with or

without consent) will received a free dress down day. Students submitted their forms

anonymously in sealed envelopes, so I maintained the confidentiality of who actually granted

consent and assent. I assigned a participant id to each student that submitted the assent and

consent forms. This participant id was used throughout the study.

Participants signed into Edmodo using an account which was defined by participant id,

and maintained anonymity. All blog messages were viewable by other participants based on

participant id only. The association between participants, assessments and questionnaires

remained confidential. This information was not shared with anyone else.

Participation in the study requires approximately 20-30 minutes per week outside of

school hours. At least one session will be completed on-site during the school week, before or

after school in the computer lab. The second session will be completed using a device that

accesses Edmodo at any time outside of normal school hours.

Initially, students completed a pretest and a questionnaire. Then, each week, students

created two separate asynchronous weblog entries in response to two mathematics problems that


required a constructed response. For the second weblog, students were instructed to evaluate the

level of critical thinking present in their response to the mathematics problem using the rubric.

At the end of the evaluation period, students were given the questionnaire again as well

as the posttest. Initially, the same set of weblogs were analyzed and coded by the teacher-

researcher and a second coder. After the first ten weblogs, the coding was compared and

adjusted to align based on a consensus on how the codes should be interpreted. Appendix D

shows the descriptors used to analyze the weblog posts.

To examine the relationship between the use of critical thinking and achievement in

math, the frequency and percentage of critical thinking indicators (per message) were compared

with the percentage of math questions that were answered correctly on the AIMSweb® probes,

before and after the intervention. A paired sample t-test was used to determine the strength of

the relationship between two variables for one sample.

To examine the effect of using a rubric with a critical thinking criterion on student's

approach to learning, an average of the scores using the Critical Thinking Analytic Rubric was

calculated. This average score was compared to the scale scores from the Revised Learning

Process Questionnaire. A paired sample t-test was used to determine the strength of the

relationship between critical thinking and intrinsic motivation.

To determine whether there was a significant difference in the indicators of cognitive

presence when critical thinking was a criterion for assessment, the frequency and percentage of

critical thinking indicators (per message) based on the Process Inquiry Model was compared to

average of the scores calculated using the Critical Thinking Analytic Rubric. To determine

whether there was a significant difference in student performance when critical thinking has been

used as a criterion for constructed response questions, the percentage of math questions that were


answered correctly on the AIMSweb® probes was compared with the scores calculated using the

Critical Thinking Analytic Rubric using a paired sample t-test.

Results

The Impact of the Critical Thinking Intervention based on Math Performance Scores

A paired samples t-test was used to examine the impact of the critical thinking

intervention on math performance. The difference between math performance before the critical

thinking intervention and math performance after the critical thinking intervention was

measured. A paired sample test was used because the data was collected from the the same

group fo students each time which represents repeated measures of the math scores. The results,

t(78) = 1.66, p = .00 (one-tailed), indicated that there was a significant difference, with the math

performance scores after the critical thinking intervention higher than the math performance

scores before the critical thinking intervention (After Intervention mean = 87.47; Before

Intervention mean = 80.96). The output data is shown in Table 1 of Appendix L.

The Impact of the Critical Thinking Intervention based on Critical Thinking Indicators

A paired sample t-test was used to examine the impact of the critical thinking

intervention on the average critical thinking scores assigned to online constructed responses to

math problems. The difference between the critical thinking score before the use of the rubric

and the critical thinking score after the critical thinking interventions was measured. A paired

sample test was used because the data was collected from the the same group of students in

repeated measures. The results, t(78) = 1.66, p = .00 (one-tailed), indicated that there is a

significant difference, with a higher percentage of critical thinking indicators after the critical

thinking intervention (After Intervention mean = 90.38; Before Intervention mean = 84.57). The

output data is shown in Table 1 in Appendix M.


The Impact of the Critical Thinking Intervention on a Deep Approach to Learning

A paired sample t-test was used to examine the impact of the critical thinking intervention on

the scaled score for a deep approach to learning. The difference between the scaled score for a

deep approach to learning before the use of the rubric and the scaled score after the critical

thinking intervention was implemented was measured. A paired sample test was used because

the data was collected from the the same group of students in repeated measures. The results,

t(78) = 1.66, p = .00 (one-tailed), indicated that there is a significant difference, with a higher

scaled score for a deep approach to learning after the critical thinking intervention (After

Intervention mean = 41.49; Before Intervention mean = 35.56). The output data is shown in

Table 1 in Appendix N.

The Frequency of Higher Order Thinking Categories After the Intervention

Histograms were used to analyze the frequency of higher order thinking in online

constructed responses after coding the messages using the Cognitive Presence coding scheme.

Refer to Appendix D for the coding scheme. A sample of a transcript of an online constructed

response and the coding is display in Apprendix O. The frequency of messages in each category

of the Cognitive Presence coding scheme before and after the critical thinking intervention was

compared. Initially, 114 messages were coded to indicate the triggering phase, 364 messages

were coded to indicate the exploration phase, 137 messages were coded to indicate the

integration phase and 17 messages were coded to indicate the application phase. After the

critical thinking rubric was used by the students, 131 messages were coded in the triggering

phase, 328 messages were coded in the exploration phase, 153 messages were coded in the

integration phase and 20 messages were coded in the application phase. Accordingly, the

proportion of messages in the integration and application phases increased from 24.37% to


27.37%. This represents a 10% increase in the proportion of messages that indicate higher-order

thinking. The frequency tables and histograms used in the data analysis are shown in Appendix

P.

Conclusion

This study was conducted to determine the efficacy of technology-enhanced instructional

strategies (weblogs or blogs) in increasing critical thinking skills of my math students. Several

research questions were evaluated to reach a conclusion regarding the efficacy.

Conclusion 1

To examine the effect of using a rubric with a critical thinking criterion on a student's

approach to learning, an average of the scores using the Critical Thinking Analytic Rubric was

compared to the scale scores from the Revised Learning Process Questionnaire. The results of a

paired sample t-test indicated that using a rubric with a critical thinking criterion had a

significant impact on a student’s approach to learning.

Conclusion 2


presence when critical thinking was a criterion for assessment, the difference between the critical

thinking score before the use of the rubric and the critical thinking score after the critical

thinking interventions was measured. The results of a paired sample t-test indicated that using a

rubric with a critical thinking criterion had a significant impact on a indicators of cognitive

presence.

Conclusion 3

To determine whether there was a significant difference in student performance when

critical thinking has been used as a criterion for constructed response questions, the percentage


of math questions that were answered correctly on the AIMSweb® probes was compared with

the scores calculated using the Critical Thinking Analytic Rubric. The results of a paired sample

t-test indicated that using a rubric with a critical thinking criterion had a significant impact on a

indicators of cognitive presence.

Other Considerations

To support the fact that the conclusions indicated by the results are attributed to the use of

the critical thinking criterion in the constructed response rubric, I reviewed multiple factors such

as math performance, cognitive presence indicators and the student approach to learning. Data

was collected before and after several trials as well as between trials. Sources were triangulated

to further investigate and identify the relevant factors causing positive results.

The position of the researcher as the math teacher for the students who participated is a

factor that would have an impact on the motivation of the participations to be more productive or

cooperative. The participants would have also been influenced by the repeated measure during

the research process. Because a disproportionate number of the students may have been

classified as gifted, theis may have impacted the high levels of cognitive presence indicators and

the rate of increase in performance.

Summary

After analyzing the results based on the research questions and considering factors that

may have influenced the data and results, the efficacy of using critical thinking as a criterion for

weblogs is supported by this research study. My classes performed better and created

constructed responses that demonstrated more critical thought as a result of implementing a

rubric with a critical thinking criterion. While previous research recommended the use of rubrics,

and critical thinking was considered important, insufficient research cited the impact of using


critical thinking as an expectation when assessing constructed responses in math. Further

research with more representative samples of the school’s population should be part of a follow

up to this study.


References

Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through

college. Washington, D.C.: U.S. Department of Education.

Anderson, K., Rohrer, M., Taylor, T., & Trimarco, R. (2006). Content analysis final report:

Evidence of learning in blogs. Retrieved from

http://tmiketaylor.files.wordpress.com/2008/02/690-learningblogs.pdf

Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. JISC

Technology and Standard Watch.

Bandeira de Mello, V. (2011). Mapping state proficiency standards onto NAEP scales: Variation

and change in state standards for reading and mathematics 2005-2009. Washington, DC:

National Center for Education Statistics, Institute of Education Sciences, U.S.

Department of Education.

Beaudette, P. (2014). Georgia Milestones: Georgia's new standardized test. Retrieved from

Georgia's Office of Student Achievement News and Research:

https://gosa.georgia.gov/georgia-milestones-georgia%E2%80%99s-new-standardized-test

Biggs, J. (1993a). What do inventories of students' learning processes really measure? A

theoretical review and clarification. British Journal of Educational Psychology, 63, 1-17.

Biggs, J. (1993b). From theory to practice: A cognitive systems approach. Higher Education

Research and Development, 12, 73-86.

Bookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom.

Alexandria, VA: ASCD.

Bookhart, S. M. (2010). How To Assess Higher-order Thinking Skills in Your Classroom.

Alexandria, Va: ASCD.

Common Core State Standards Initiative. (2014). Key Shifts in Mathematics. Retrieved from


Common Core State Standards Initiative:

http://www.corestandards.org/other-resources/key-shifts-in-mathematics/

Common Core State Standards Initiative. (2014). What Parents Should Know. Retrieved from

Common Core State Standards Initiative: http://www.corestandards.org/what-parents-

should-know/

Creswell, J. W. (2012). : Planning, Conducting, and Evaluating Quantitative and Qualitative

Research. Boston: Pearson.

Dewey, J. (1910). How we think. Boston, MA: D. C. Heath.

Dos, B., & Demir, S. (2013). The analysis of the blogs created in a blended course through the

reflective thinking perspective. Educational Sciences: Theory & Practice, 13(2), 1335-

1344.

Ennis, R. (1987). A taxonomy of critical thinking dispositions and abilities. In J. B. Baron, & R.

J. Sternberg, Teaching Critical Thinking Skills: Theory and Practice (pp. 9-26). New

York: Freeman.

Hrastinski, S. (2008). Asynchronous & synchronous e-learning. Educause Quarterly, 4, pp. 51-

55. Retrieved from https://net.educause.edu/ir/library/pdf/EQM0848.pdf

Karsak, H. O., Fer, S., & Orhan, F. (2014). The effect of using cooperative and individual

weblog to enhance writing. Educational Technology & Society, 17(4), 229-241.

Kember, D., Biggs, J., & Leung, D. Y. (2004). Examining the multidimensionality of approaches

to learning through the development of a revised version of the learning process

questionnaire. British Journal of Educational Psychology, 74, 261-280.

Li, K. (2010). Integrating weblogs in a pedagogy model for enhancing students' critical thinking

skills. Research and Practice in Technology Enhanced Learning, 35-49.


doi:10.1142/S1793206810000803

Li, K., Bado, N., Smith, J., & Moore, D. (2013). Blogging for teaching and learning: An

examination of experience, attitudes, and levels of thinking. Contemporary Educational

Technology, 4(3), 172-186.

Meinecke, A. L., Smith, K. K., & Lehmann-Wellenbrock, N. (2013, 05 05). Sage Publications.

doi:10.1177/1046496413487020

Mills, G. E. (2014). A Guide for the Teacher Researcher, Fifth Edition. Upper Saddle River,

New Jersey: Pearson Education, Inc.

Muncy, J. A. (2014). Blogging for reflection: The use of online journals to engage students in

reflective learning. Marketing Education Review, 24(2), 101-113.

National Center for Educational Statistics. (2014, May). The Condition of Education. Retrieved

from National Center for Educational Statistics:

http://nces.ed.gov/programs/coe/indicator_cnc.asp

National Governors Association Center for Best Practices & Council of Chief State School

Officers. (2010). Common Core State Standards. Washington D.C: National Governors

Association Center for Best Practices & Council of Chief State School Officers.

NCS Pearson, Inc. (2010). Mathematics computation: Administration and technical manual.

Bloomington, MN: NCS Pearson, Inc. Retrieved from www.aimsweb.com

Organisation for Economic Cooperation and Development. (2012). PISA 2012 Results. Retrieved

from PISA: http://www.oecd.org/pisa/keyfindings/PISA-2012-results-US.pdf

Peterson, P. E., & Kaplan, P. (2013). Despite Common Core, states still lack common standards:

students proficient on state test but not national. Education Next, 44, 44.

Richardson, J. C., & Ice, P. (2010). Investigating students' level of critical thinking across


instructional strategies in online discussions. Internet and Higher Education, 13(1-2), 52-

59.

Richardson, W. (2010). Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms.

Thousand Oaks, CA: Corwin Press.

Roblyer, M. D., & Doering, A. H. (2013). Integrating Educational Technology into Teaching

(6th ed.). Pearson Education, Inc.

Rose, H., & Betts, J. R. (2004). The effect of high school courses on earnings. The Review of

Economics and Statistics, 86(2), 497-513.

Saxton, E., Belanger, S., & Becker, W. (2012). The Critical Thinking Analytic Rubric (CTAR):

Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical

thinking performance assessments. Assessing Writing, 17, 251-270.

Shea, P., Hayes, S., Gozza-Cohen, M., Uzuner, S., Mehta, R., Valchova, A., & Rangan, P.

(2010). A re-examination of the community of inquiry framework: Social network and

content analysis. Internet and Higher Education, 13, 10-21.

Shea, P., Hayes, S., Gozza-Cohen, M., Uzuner, S., Mehta, R., Valchova, A., & Rangan, P.

(2010). A re-examination of the community of inquiry framework: Social network and

content analysis. Inernet and Higher Education, 13, 10-21.

The State of Georgia. (2014). Priority, Focus, and Alert Schools. Retrieved from Georgia Office

of Student Achievement: http://gosa.georgia.gov/

Xie, Y., Ke, F., & Sharma, P. (2008). The effect of peer feedback for blogging on college

students' reflective learning processes. Internet and Higher Education, 18-25.

Zhang, S. (2013). An empirical study of the factors affecting weblog success in higher education.

Journal of Information Systems Education, 24(4), 267-279.


Appendix A

Table A1

The Surface and Deep Approaches of The Revised Learning Process Questionnaire

Note: Adapted from “Examining the multidimensionality of approaches to learning through the development of a revised version of the learning process questionnaire” by D. Kember, J. Biggs, and D. Leung, 2004, British Journal of Educational Psychology, 74, p. 262.


Appendix B

Figure 1. Revised Learning Process Questionnaire (R-LPQ-2F). From “Examining the multidimensionality of approaches to learning through the development of a revised version of the learning process questionnaire” by D. Kember, J. Biggs, and D. Leung, 2004, British Journal of Educational Psychology, 74, p. 277-278. Copyright 2001 by John Biggs and David Kember.


Appendix C

Figure 2. Scales in the Revised Learning Process Questionnaire (R-LPQ-2F). From “Examining the multidimensionality of approaches to learning through the development of a revised version of the learning process questionnaire” by D. Kember, J. Biggs, and D. Leung, 2004, British Journal of Educational Psychology, 74, p. 278-279. Copyright 2001 by John Biggs and David Kember.


Appendix C (continued)

Figure 2. Scales in the Revised Learning Process Questionnaire (R-LPQ-2F). From “Examining the multidimensionality of approaches to learning through the development of a revised version of the learning process questionnaire” by D. Kember, J. Biggs, and D. Leung, 2004, British Journal of Educational Psychology, 74, p. 278-279. Copyright 2001 by John Biggs and David Kember.


Appendix D

Table D1

Coding Scheme for Cognitive Presence

Note: From “A re-examination of the community of inquiry framework: Social network and content analysis“ by P. Shea, S. Hayes, M. Gozza-Cohen, S. Uzuner, R. Mehta, S. Valchova, & P. Rangan, 2010, Internet and Higher Education, 13, p. 20.


Appendix E

Table E2

Descriptive and Reliability Statistics for AIMSweb® Probes

Note: From “Mathematics Computation: Administration and Technical Manual “ by NCS Pearson, Inc., (2010). Bloomington, MN.


Appendix F

8th Grade Math Pretest

From “AIMSweb® Math Concepts and Applications: Progress Monitor Grade 8, Probe 4” by NCS Pearson, Inc., 2010. Bloomington, MN.


Appendix F

8th Grade Math Pretest (continued)



Appendix F




Appendix F


From “AIMSweb® Math Concepts and Applications: Progress Monitor Answer Key, Grade 8, Probe 4” by NCS Pearson, Inc., 2010. Bloomington, MN.


Appendix G

8th Grade Math Posttest



Appendix G

8th Grade Math Posttest (continued)



Appendix G




Appendix G


From “AIMSweb® Math Concepts and Applications: Progress Monitor Answer Key, Grade 8, Probe 33” by NCS Pearson, Inc., 2010. Bloomington, MN.


Appendix H

8th Grade Math Administration Directions

From “AIMSweb® Math Concepts and Applications: Math Administration Directions” by NCS Pearson, Inc., 2010. Bloomington, MN., p 11.


Appendix I

Critical Thinking Analytical Rubric (CTAR)

Adapted from “The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments” by E. Saxton, S. Belanger, and W. Becker, 2012, Assessing Writing, 17, p. 267-269.


Appendix J

From “The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments” by E. Saxton, S. Belanger, and W. Becker, 2012, Assessing Writing, 17, p. 261.


Appendix K

Table K1

Power Analysis Table:

Approximate Sample Size Per Group at Alpha=.05

Power Criterion

Effect Size .80 .90 .95

.20 395 525 650

.30 175 235 290

.40 100 130 165

.50 65 85 105

.60 45 60 75

.70 35 45 55

.80 25 35 45

Note: Adapted from “Design Sensitivity: Statistical Power for Experimental Research” by M. W. Lipsey, 1990, Sage Publications, Inc., p. 137.

Appendix LTable 1 The Impact of the Critical Thinking Intervention based on Math Performance Scores

T-Test: Paired Two Sample for Means

AIMSWEB® Pretest Score

AIMSWEB® Posttest Score

Mean 80.96 87.47Variance 165.50 122.51Observations 79 79Pearson Correlation 0.90Hypothesized Mean Difference 0df 78t Stat -10.33P(T<=t) one-tail 0.00t Critical one-tail 1.66P(T<=t) two-tail 0.00t Critical two-tail 1.99


Note: This table displays the t value and indicates the significance of the difference between the score that students received based on a math performance test before and after a critical thinking intervention.


Appendix M

Table 1 The Impact of the Critical Thinking Intervention based on Critical Thinking Indicators

T-Test: Paired Two Sample for Means

Before Rubric After Rubric

Mean 84.57 90.38

Variance 52.08 31.03

Observations 79 79Pearson Correlation 0.47Hypothesized Mean Difference 0df 78t Stat -7.70P(T<=t) one-tail 0.00

t Critical one-tail 1.66P(T<=t) two-tail 0.00t Critical two-tail 1.99

Note: This table displays the t value and indicates the significance of the difference between the average percentage of critial thinking indicators before and after students referenced a rubric for online constructed responses to math problems.


Appendix N

Table 1. The Impact of the Critical Thinking Intervention on a Deep Approach to Learning

t-Test: Paired Two Sample for MeansBefore Intervention After Intervention

Mean 35.56 41.49

Variance 77.76 75.74

Observations 79 79

Pearson Correlation 0.98

Hypothesized Mean Difference 0

df 78

t Stat -27.92

P(T<=t) one-tail 0.00

t Critical one-tail 1.66

P(T<=t) two-tail 0.00

t Critical two-tail 1.99Note: This table displays the t value and indicates the significance of the difference between the scaled score for a deep approach to learning before and after students used a rubric for online constructed responses to math problems.


Appendix O

Document Description: Transcript of an Online Constructed Response to a Math Problem from Class Blog Website

Msg ID Transcript Coding

030615289171 1) In the Journey to the Bus Stop, Tom walked along a straight road

with a distance of 160 miles from his home. In the start of his journey,

his pace would gradually increase by a rate of 20 meters for every 10

seconds that he walked. Suddenly he felt a wave of exhaustion come

over him, and slowed his pace to a rate of 30 meters for every 10

seconds. By midway of his journey, he suddenly noticed that he was

running late, leading him to walk at a faster pace of 40 meters for every

10 seconds he walked. As he saw the bus stop coming in sight, he

walked the constant pace and decided to walk at a moderate pace.

2) All the sections of the graph are not realistic because it is not

possible for Tom to go backwards in pace and forwards in time during

his journey. If this were to be realistic, Tom would keep a steady pace

throughout the whole journey and not decrease dramatically midway.

Instead, the story goes that Tom had increased his pace, decreased his

pace, and then picked up at a regular pace once again, which is

inaccurate.

CP-IN-4


Appendix P

Coding Category Frequency Cumulative %

Triggering 114 18.04%Exploration 364 75.63%Integration 137 97.31%Application 17 100.00%

Triggering Exploration Integration Application0

50

100

150

200

250

300

350

400

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Histogram

Coding Category

Freq

uenc

y

Figure R1. Histogram of the Frequency of Coding Categories Before the Intervention. This figure shows the histogram based on the cognitive presence coding categories for online constructed response to math problems.


Appendix P (continued)

Coding Category Frequency Cumulative %

Triggering 131 20.73%

Exploration 328 72.63%

Integration 153 96.84%

Application 20 100.00%

Triggering Exploration Integration Application0

50

100

150

200

250

300

350

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Histogram

Coding Category

Freq

uenc

y

Figure R2. Histogram of the Frequency of Coding Categories After the Intervention. This figure shows the histogram based on the cognitive presence coding categories for online constructed response to math problems after a critical thinking rubric was used to score responses.


To address the research questions, the data points and units of measurements are used as

follows:

the relationship between the use of critical thinking and achievement in math.

To examine the relationship between the use of critical thinking and achievement in

math, the frequency and percentage of critical thinking indicators (per message) were compared

with the percentage of math questions that were answered correctly on the AIMSweb® probes,

before and after the intervention. A regression analysis was used to determine the strength of the

relationship between two variables for one sample. The scatter plot and trend line for the

regression analysis are in Appendix L. The rate of change value for the trend line indicated that

the AIMSWEB® Math Pretest score increased 1.3 points for each percentage increase in critical

thinking indicators per message. Based on the R2 value for the line, 85% of the variation in the

math pretest scores can be attributed to critical thinking.

To examine the effect of using a rubric with a critical thinking criterion on student's approach

to learning, an average of the scores using the Critical Thinking Analytic Rubric was

calculated. This average score was compared to the scale scores from the Revised Learning

Process Questionnaire. A regression analysis was used to determine the strength of the

relationship between critical thinking and intrinsic motivation. The scatter plot and trend line

for the regression analysis are in Appendix M. The rate of change of the trend line indicated

that the level of intrinsic motivation increased .4 units for each percentage increase in critical

thinking indicators per message. 18% of the variation in the level of intrinsic inBased on the

R2 value for the line, 85% of the variation in the math pretest scores can be attributed to

critical thinking.



presence when critical thinking was a criterion for assessment, the frequency and percentage

of critical thinking indicators (per message) based on the Process Inquiry Model was

compared to average of the scores calculated using the Critical Thinking Analytic Rubric.

To determine whether there was a significant difference in student performance when critical

thinking has been used as a criterion for constructed response questions, the percentage of

math questions that were answered correctly on the AIMSweb® probes was compared with

the scores calculated using the Critical Thinking Analytic Rubric.

Reliability and Validity

AIMSweb® Probe

The source of the pretests and posttests was the AIMSweb® assessment system. Pearson

states that numerous criteria were used to choose the most psychometrically-sound

equivalent probes for the system. Pearson’s evaluation of the internal consistency of the

probes, Cronbach’s alpha, and split-half reliability produced the results listed in Appendix D.

Cronbach's alpha for the 6th, 7th and 8th grade probes is .89, .91 and .90 respectively.

Revised Learning Process Questionnaire(R-LPQ-2F)

The two scales, the deep approach, and the surface approach achieve Cronbach alpha values

of 0.82 and 0.71. Because the values are above the accepted cut-off values, the two scales

have to be accepted as reliable. According to Kimber et al. (2004), the R-LPQ-2F, is

recognized as an instrument with acceptable psychometric properties, suitable for classroom

evaluation

Practical Inquiry Model

Inter-rate reliability was reviewed based on coding at the message level. A previous

quantitative content analysis using the coding scheme for Cognitive Presence, featured in


Appendix D, had adequate inter-rater reliability on 944 posts (Shea, et al., 2010). In a

separate study (Garrison, Cleveland-Innes, Koole, & Kappelman, 2006), coders were able to

achieve agreement on approximately 80% of all messages.

Critical Thinking Analytical Rubric

The CTAR has six rubric categories, including interpretation, analysis, evaluation, inference,

explanation, and disposition. The basis of the inter-rater reliability is the scoring of four sets

of performance-based student work samples by two trained raters. The work samples were

from two separate studies. Research results show that the CTAR is a reliable instrument for

scoring student work consistently (Saxton, Belanger, & Becker, 2012).

Appendix J shows the results for inter-rater reliability. The results for the inter-rater

reliability demonstrate levels of consistency of at least 0.70 in all rubric categories (Saxton,

Belanger, & Becker, 2012). For intra-rater reliability, the analysis category had different

levels of consistency. One rater showed a consistency of .91, which is excellent, but the

other rater showed a poor consistency of .56 in the analysis category. In the remaining five

rubric categories, the intra-rater reliability was at least 0.71 (Saxton, Belanger, & Becker,

2012). Table J2 of Appendix J shows the results of the intra-rater reliability of each

category.

microsoft word - literature review...

Documents