evaluation of multimedia software and a workbook designed to improve 3-d spatial skills of...
TRANSCRIPT
Evaluation of Multimedia Evaluation of Multimedia Software and a Workbook Software and a Workbook
Designed to Improve 3-D Spatial Designed to Improve 3-D Spatial Skills of Engineering StudentsSkills of Engineering Students
Sheryl A. Sorby & Thomas DrummerSheryl A. Sorby & Thomas Drummer
Michigan Technological UniversityMichigan Technological University
BackgroundBackground
Spatial skills are important for success in Spatial skills are important for success in technical fieldstechnical fields Especially important for success in engineering Especially important for success in engineering
graphicsgraphics
Significant gender differences in 3-D spatial Significant gender differences in 3-D spatial skills, favoring malesskills, favoring males Gender differences cross international boundariesGender differences cross international boundaries
Poorly developed spatial skills may lead women Poorly developed spatial skills may lead women to drop out of engineering at a disproportionate to drop out of engineering at a disproportionate raterate
Assessing Spatial Skills-Purdue Spatial Assessing Spatial Skills-Purdue Spatial Visualization Test: Rotations (PSVT:R)Visualization Test: Rotations (PSVT:R)
Assessing Spatial Skills-Mental Assessing Spatial Skills-Mental Cutting Test (MCT)Cutting Test (MCT)
Assessing Spatial Skills-Differential Aptitude Assessing Spatial Skills-Differential Aptitude Test: Space Relations (DAT:SR)Test: Space Relations (DAT:SR)
Spatial Skills Class at Spatial Skills Class at Michigan TechMichigan Tech
Initially developed in 1993 with funding Initially developed in 1993 with funding from the National Science Foundationfrom the National Science Foundation
Additional funding in 1998 led to Additional funding in 1998 led to development of multimedia software and development of multimedia software and workbookworkbook
Class modified significantly in 2000 to Class modified significantly in 2000 to include software and workbookinclude software and workbook
Multimedia Software and Multimedia Software and WorkbookWorkbook
Developed in 1998-99 with funding from Developed in 1998-99 with funding from the National Science Foundationthe National Science Foundation
Consist of 9 separate modules designed to Consist of 9 separate modules designed to enhance 3-D spatial skillsenhance 3-D spatial skills Each module consists of Background and Each module consists of Background and
Exercises sectionsExercises sections Software is stand-alone and works on Software is stand-alone and works on
either a Mac or PCeither a Mac or PC Published by Delmar Learning in 2002Published by Delmar Learning in 2002
Assessment of Multimedia Assessment of Multimedia SoftwareSoftware
Students given attitudinal surveys rating Students given attitudinal surveys rating various aspects of the softwarevarious aspects of the software Ratings given by module and overallRatings given by module and overall
Ratings were on a 1-4 scaleRatings were on a 1-4 scale Students asked to rate such things as Students asked to rate such things as
ease of understanding, ease of use, etc.ease of understanding, ease of use, etc.
Results from Attitudinal Surveys of Results from Attitudinal Surveys of Individual ModulesIndividual Modules
ModuleModule BackgroundBackground ExercisesExercisesIsometric SketchesIsometric Sketches 3.523.52 3.353.35
Orthographic ProjectionOrthographic Projection 3.583.58 3.443.44
Flat Pattern FoldingFlat Pattern Folding 3.523.52 3.663.66
Rotation about One AxisRotation about One Axis 3.473.47 3.493.49
Rotation about Two AxesRotation about Two Axes 3.623.62 3.793.79
Reflections and SymmetryReflections and Symmetry 3.523.52 3.363.36
Cross Sections of SolidsCross Sections of Solids 3.633.63 3.583.58
Surfaces & Solids of RevolutionSurfaces & Solids of Revolution 3.653.65 3.543.54
Combining SolidsCombining Solids 3.503.50 3.533.53
Student Overall RatingsStudent Overall Ratings
StatementStatement AgreeAgree DisagreeDisagree UncertainUncertain
The modules helped me learn the The modules helped me learn the course materialcourse material
3232 11 22
The multimedia modules were fun to The multimedia modules were fun to useuse
3131 22 22
The topics covered in the modules were The topics covered in the modules were appropriate to the courseappropriate to the course
3535 00 00
The multimedia modules were easy to The multimedia modules were easy to useuse
2828 44 33
I would rather learn using the I would rather learn using the multimedia materials than using written multimedia materials than using written materialsmaterials
2121 55 99
Student Overall Ratings (cont.)Student Overall Ratings (cont.)
StatementStatement AgreeAgree DisagreeDisagree UncertainUncertain
The exercises in the multimedia modules The exercises in the multimedia modules were at the right difficulty level for mewere at the right difficulty level for me
1818 1717 00
There were enough exercises at the end There were enough exercises at the end of each module to find out whether or not I of each module to find out whether or not I understood the materialunderstood the material
2222 88 55
It took me the entire hour of lab time to It took me the entire hour of lab time to complete each modulecomplete each module
11 3131 33
This is my first time using multimedia This is my first time using multimedia educational softwareeducational software
2424 1010 11
Multimedia software is an effective tool for Multimedia software is an effective tool for helping students develop their 3-D spatial helping students develop their 3-D spatial abilityability
3232 11 22
Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-PSVT:RSkills Test-PSVT:Rnn Pre-Pre-
TestTestPost-Post-TestTest
GainGain Significance Significance of Gainof Gain
Original Original Course Course (93-99)(93-99)
186186 50.550.5 76.976.9 26.426.4 p<0.0001p<0.0001
Modified Modified Course Course (00-02)(00-02)
157157 48.348.3 73.773.7 25.425.4 p<0.0001p<0.0001
Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-MCTSkills Test-MCT
nn Pre-Pre-TestTest
Post-Post-TestTest
GainGain Significance Significance of Gainof Gain
Original Original Course Course (93-99)(93-99)
9999 37.937.9 51.451.4 13.513.5 p<0.0001p<0.0001
Modified Modified Course Course (00-02)(00-02)
109109 34.834.8 52.652.6 17.817.8 p<0.0001p<0.0001
Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-DAT:SRSkills Test-DAT:SRnn Pre-Pre-
TestTestPost-Post-TestTest
GainGain Significance Significance of Gainof Gain
Original Original Course Course (93-99)(93-99)
9999 62.362.3 78.378.3 16.016.0 p<0.0001p<0.0001
Modified Modified Course Course (00-02)(00-02)
110110 62.162.1 77.977.9 15.815.8 p<0.0001p<0.0001
Improvement in Retention RatesImprovement in Retention Rates
WomenWomen MenMen
EGEG CGCG EGEG CGCG
Original Original CourseCourse
88.9%88.9% 68.3%68.3% 75.3%75.3% 69.0%69.0%
Modified Modified CourseCourse
87.4%87.4% 71.7%71.7% 76.8%76.8% 70.0%70.0%
Differences in Retention Rates for Women wereStatistically Significant but those for Men were not
Current ProjectCurrent Project
New course using multimedia software and New course using multimedia software and workbook seems to be effective workbook seems to be effective Similar or better gains on spatial skills testsSimilar or better gains on spatial skills tests Similar retention rates for men and women in Similar retention rates for men and women in
engineeringengineering
Research Questions: Is the software alone as Research Questions: Is the software alone as effective as when it is used in conjunction with effective as when it is used in conjunction with the workbook in developing spatial skills? and the workbook in developing spatial skills? and Can the materials be used effectively with non-Can the materials be used effectively with non-engineering students?engineering students?
Study GroupsStudy Groups
170 non-engineering students participated in the study170 non-engineering students participated in the study Most from either Biology or Computer ScienceMost from either Biology or Computer Science
Divided into four groupsDivided into four groups Software alone, Workbook alone, Software and Workbook, and Software alone, Workbook alone, Software and Workbook, and
ComparisonComparison Treatment groups participated in ten weeks of training in Treatment groups participated in ten weeks of training in
2-hour sessions each week2-hour sessions each week Tried to balance groups for gender, major, and PSVT:R Tried to balance groups for gender, major, and PSVT:R
pre-test scorepre-test score Comparison group invited back for post-test at end of 10-Comparison group invited back for post-test at end of 10-
week periodweek period
Gains on Spatial Skills Tests-Gains on Spatial Skills Tests-PSVT:RPSVT:R
Pre-testPre-test Post-testPost-test GainGain SignificanceSignificance
EG-SoftwareEG-Software 69.569.5 79.479.4 7.57.5 p<0.005p<0.005
EG-WorkbookEG-Workbook 75.975.9 86.686.6 11.611.6 p<0.0005p<0.0005
EG-Software & EG-Software & WorkbookWorkbook
74.374.3 85.685.6 11.211.2 p<0.0005p<0.0005
Comparison Comparison GroupGroup
72.672.6 77.977.9 5.35.3 p<0.03p<0.03
Gains on Spatial Skills Tests-MCTGains on Spatial Skills Tests-MCT
Pre-testPre-test Post-testPost-test GainGain SignificanceSignificance
EG-SoftwareEG-Software 44.544.5 51.051.0 6.56.5 p<0.01p<0.01
EG-WorkbookEG-Workbook 53.953.9 62.362.3 8.48.4 p<0.001p<0.001
EG-Software & EG-Software & WorkbookWorkbook
46.446.4 60.260.2 13.813.8 p<0.0005p<0.0005
Testing DataTesting Data
Each group had one or two data points that Each group had one or two data points that appeared to be outliersappeared to be outliers
For example in SW group, one foreign-born For example in SW group, one foreign-born student had pre-test of 17% and post-test of student had pre-test of 17% and post-test of 92%. 92%. Her English skills improved dramatically over 10-week Her English skills improved dramatically over 10-week
periodperiod Large gain could be due to improvements in language Large gain could be due to improvements in language
skills more than to improvements in spatial skillsskills more than to improvements in spatial skills Data analyzed using a nonparametric logistic Data analyzed using a nonparametric logistic
regression modelregression model
Logistic Regression ResultsLogistic Regression Results
Both groups that used the workbook were Both groups that used the workbook were significantly better than the CGsignificantly better than the CG
Software only group was not better than CGSoftware only group was not better than CG
Attitudinal Survey and ResultsAttitudinal Survey and Results
Students in EG were administered an Students in EG were administered an attitudinal survey at end of training periodattitudinal survey at end of training period
62% would have preferred using the 62% would have preferred using the software alone for training even though it software alone for training even though it was least effectivewas least effective Less time involvedLess time involved
Female students spent more time Female students spent more time completing the training materials for all completing the training materials for all groupsgroups
Attitudinal Survey and ResultsAttitudinal Survey and Results
Male students expressed greater confidence in Male students expressed greater confidence in their spatial skills at beginning of studytheir spatial skills at beginning of study Question asked retrospectivelyQuestion asked retrospectively
Level of ConfidenceLevel of Confidence Males (n=66)Males (n=66) Females (n=29)Females (n=29)
Not confidentNot confident 1.5%1.5% 27.6%27.6%
Somewhat ConfidentSomewhat Confident 28.8%28.8% 51.7%51.7%
ConfidentConfident 37.9%37.9% 20.7%20.7%
Most ConfidentMost Confident 31.8%31.8% 0%0%
Attitudinal Survey and ResultsAttitudinal Survey and Results Women preferred the use of manipulatives and working Women preferred the use of manipulatives and working
in groupsin groups Female students expressed higher level of learningFemale students expressed higher level of learning
Level of LearningLevel of Learning Males (n=66)Males (n=66) Females (n=29)Females (n=29)
Did not learn anything newDid not learn anything new 6%6% 3%3%
Learning limited to a few Learning limited to a few aspectsaspects
47%47% 17%17%
Learned something on most Learned something on most daysdays
26%26% 45%45%
Learned something every Learned something every sessionsession
21%21% 35%35%
ConclusionsConclusions
Multimedia software and workbook developed Multimedia software and workbook developed for use with engineering students is suitable for for use with engineering students is suitable for use with non-engineersuse with non-engineers
Software alone does not seem to be as effective Software alone does not seem to be as effective as when used in conjunction with the workbookas when used in conjunction with the workbook
There were gender differences in the type of There were gender differences in the type of training preferred and in the perceived level of training preferred and in the perceived level of learning learning
AcknowledgementAcknowledgement
The authors gratefully acknowledge the The authors gratefully acknowledge the support of this project from the National support of this project from the National Science Foundation through grant number Science Foundation through grant number HRD-0429020HRD-0429020