work in progress: suitability of spatial visualization
TRANSCRIPT
Paper ID #33761
Work in Progress: Suitability of Spatial Visualization Training forRemote Learning
Dr. Lelli Van Den Einde, University of California, San Diego
Van Den Einde is a Teaching Professor in Structural Engineering at UC San Diego and the President ofeGrove Education, Inc. She incorporates education innovations into courses (Peer Instruction, Project-based learning), prepares next generation faculty, advises student organizations, hears cases of academicmisconduct, is responsible for ABET, and is committed to fostering a supportive environment for di-verse students. Her research focuses on engagement strategies for large classrooms and developing K-16curriculum in earthquake engineering and spatial visualization.
Prof. Nathan Delson, University of California at San Diego
Nathan Delson is a Teaching Professor at the University of California at San Diego. His research inter-ests include robotics, biomedical devices, and engineering education. He teaches introductory design,mechanics, mechatronics, capstone design, medical devices, and product design & entrepreneurship. Hisinterests in design education include increasing student motivation, teamwork, hands-on projects, andintegration of theory into design projects. In 1999 he co-founded Coactive Drive Corporation (currentlyGeneral Vibration), a company that provides force feedback solutions. In 2016 Nate co-founded eGroveEducation, Inc. an educational software company focused on teaching sketching and spatial visualizationskills.
Elizabeth Rose Cowan, eGrove Education
Elizabeth Cowan is a User Experience Researcher and Designer for eGrove Education, Inc., which de-velops Spatial Visualization Training software. Her degree is in Cognitive Science and Human-ComputerInteraction from the University of California San Diego.
c©American Society for Engineering Education, 2021
WIP: Suitability of Spatial Visualization Training for Remote
Learning
This work in progress (WIP) describes the impact that a spatial visualization training app had on
learning outcomes and engagement during the COVID-19 pandemic. When the pandemic hit,
classes with computer aided design (CAD) instruction had two challenges to overcome as people
transitioned to remote or online learning. One challenge was that the remote nature of the
instruction made it difficult for teachers to provide feedback to students as quickly as they did in
a classroom setting thereby maintaining student engagement. The second challenge was the lack
of student access to computers or technology necessary to run the CAD packages. To address
these challenges, a number of middle schools, high schools and post-secondary schools adopted
for the first time the use of Spatial Vis, a freehand sketching and spatial visualization app that ran
on smartphones, tablets, and Chromebooks. Since Spatial Vis automatically grades students’
sketches and provides immediate feedback, it has the potential to increase student engagement
during remote instruction and prepare them for CAD instruction. For many of these newly
adopted courses, as well as for courses that had used the app prior to the pandemic, the use of the
app resulted in an increased amount of spatial visualization instruction to compensate for the
technical constraints and reduced capabilities to teach CAD. Due to the rush to switch to remote
instruction, the courses that adopted the Spatial Vis app were not part of a controlled study.
However, the digital nature of the app allows for the quantification of data such as how many
assignments the students completed in each module and quantification of student persistence in
the app. Furthermore, instructor feedback provides qualitative information regarding learning
outcomes. In this study, student performance was evaluated in two introductory CAD courses at
San Diego State University and University of California, San Diego, both of which used the app
prior to COVID-19 and during the pandemic. In both classes, pre and post testing was done with
a standardized test to measure spatial visualization ability. Comparisons were made of app
performance and test performance before COVID-19 and during COVID-19. Conclusions were
drawn regarding the suitability of the Spatial Vis app for remote learning.
1. Introduction
The COVID-19 pandemic resulted in educational institutions across the world pivoting quickly
to online learning. Initially, teachers had to adapt to emergency remote teaching and were
overwhelmed with the additional work required to develop their remote classes [1]. One survey
conducted to evaluate the successes and challenges teachers faced as they moved to virtual
instruction in spring 2020 showed that although teachers transitioned immediately to virtual
instruction, they were not properly trained or given enough time to effectively redesign their
pedagogy. Teachers were inundated with information and resources to help with virtual
instruction and overwhelmed by how quickly they had to transition [2]. Some courses, especially
in the area of career and technical education (CTE) or other science, technology, engineering,
and mathematics courses such as Computer Aided Design (CAD) that normally have hands-on or
laboratory learning were not simple to convert to a virtual environment [3]. During the initial
move to remote instruction, these courses were challenged with providing access to high-end
equipment or computing resources requiring high-speed internet, and they no longer could
benefit from in-person, close-up demonstrations of proper technique or the opportunity for
teachers to ascertain students’ proficiency of the material in real time [4]. There were concerns
that some students might not be receiving quality instruction in these courses, which would
negatively impact their college and career readiness upon graduation [5] or their fundamental
CAD skills in support of higher education engineering degrees.
To address these challenges, eGrove Education, Inc. [6] offered their Spatial Vis app to teachers
for free when the pandemic hit. The authors of this paper are affiliated with eGrove Education,
Inc. and have developed the Spatial Vis app. Middle Schools, high schools and higher education
institutions jumped at the opportunity to use the Spatial Vis app for the first time. Spatial Vis
teaches freehand sketching on smartphones, tablets, and Chromebooks to improve spatial skills.
Studies have shown that spatial visualization training can increase grades in introductory science,
technology, engineering, and math (STEM) courses [7], and can improve computer aided design
(CAD) and design skills. The app had the potential to increase student engagement during
remote instruction since it automatically grades students’ sketches and provides immediate
feedback. Many classes had to compensate for the technical constraints and reduced capabilities
to teach their traditional hands-on curriculum. For example, some teachers that normally teach
hand sketching had to require students to scan or take photographs of their sketches and submit
them to the teacher for grading, which would take several days to provide feedback to the
students. Therefore, the use of the Spatial Vis app in many of these newly adopted courses in
spring and fall 2020 resulted in an increased amount of spatial visualization instruction for the
students.
Because the move to remote instruction occurred quickly, the courses that adopted the Spatial
Vis app during the pandemic were not part of a controlled study. However, through instructor
and student feedback, qualitative information regarding learning outcomes was established and
the impact that the Spatial Vis app had on learning outcomes and engagement during the
COVID-19 pandemic was ascertained. These results are described in [8] which concluded that
despite rapid adoption in these middle school, high school and higher education institutions, the
Spatial Vis app provided a resource that was not only effective in the classroom but worked
seamlessly outside of the classroom and provided flexibility to teachers.
In order to provide quantitative measurement of the impact that the pandemic had on spatial
visualization training, two higher education courses that had used the Spatial Vis app prior to the
pandemic and also during the pandemic were analyzed. This paper particularly focuses on two
freshman engineering CAD courses taught at two different institutions, San Diego State
University and the University of California, San Diego. A primary objective for both of these
courses is to teach students spatial visualization and CAD skills. One major difference between
these two courses is that when instruction was in-person, the UC San Diego course taught CAD
through physical design-build projects, while the other focused on purely computer-based design
assignments. In both classes the Spatial Vis app was used as supplemental material to other
course content.
This paper describes the quantitative analysis conducted on these two CAD courses for the year
prior to the pandemic and during the pandemic to evaluate the Spatial Vis app’s suitability not
only for in the classroom but also for remote instruction. Additionally, feedback from teacher
interviews is included to provide instructor perspectives.
2. Spatial Vis Sketching App
Spatial visualization has been shown to be a fundamental skill for student success. A spatial
visualization app was developed by eGrove Education, Inc. [9] to make use of the prevalence of
touchscreen devices in education and to make spatial visualization training more engaging and
easier to teach. The app teaches freehand sketching to improve spatial skills for students in pre-
engineering, manufacturing, and CAD courses. The app was motivated by Sorby’s finding that
the “importance of sketching in developing 3-D spatial skills cannot be understated” [7].
Sketching provides an added benefit beyond spatial visualization, and has been correlated to
communication, teamwork, and creativity [10]. During the study, the app ran on iOS and
Android phones and tablets, and Chromebooks (as of this publication the app can also run from
any web connected computer). Students sketch isometric and orthographic assignments, and the
app provides automatic grading and personalized feedback on student work making it especially
well-suited for remote learning as well as in-class instruction. Studies with the Spatial Vis app
have demonstrated its efficacy when used in class or as homework ([11] through [16]).
Figure 1. Spatial Vis™ App
2.1. Features Suitable for Remote Instruction
2.1.1. Automatic Grading Algorithm with Hint and Peek Capabilities
The Spatial Vis app provides a sketching window and an assignment window, each with a
reference dot so students know the position in the sketching window they need to draw their
solution for the grading algorithm to correctly score their submission (Fig. 2a). The grading
algorithm is initiated when the user selects the submit button at the bottom of the screen. If the
solution is correct, the student can move on to the next assignment. If incorrect, a pop-up
window is produced by the grading algorithm that gives the student immediate feedback by
providing them with a circular progress indicator showing them how much of their sketch is
correct and how much is incorrect (Fig. 2b). The student has the option to retry the assignment as
many times as they would like without losing any credit. If the student would like more help,
they can select the Hint button or take a Peek at the solution [8].
The Hint tells the user which parts of their drawing are correct by highlighting them in green
(Fig. 2c). If the Hint shows that most of their submission is highlighted in green, then the student
is close to the solution. However, if the Hint shows that most of their submission is not green,
then they know they need to rethink the problem, ask for help, or possibly use a Peek.
The Peek highlights the correct parts of the drawing in green, incorrect parts in red, and missing
lines in blue (Fig. 2d). Additionally, gamification was added to the app to encourage persistence.
Students receive three stars for each assignment they solve correctly without any assistance
regardless of how many times they attempt the sketch. If a Hint is used, then they are awarded
two stars for the problem and if a Peek is used then students earn only one star when they
successfully complete the sketch (Fig. 3) [8]. The motivation for gamification is based on a study
that showed that students who tried to solve a problem on their own before asking for a Peek at
the solution had increased gains in spatial visualization ability. The average number of stars per
assignment that a student receives is one indication of their persistence in working to solve a
problem on their own.
Figure 2. Orthographic Sketch Assignment with Different Feedback Options
Figure 3. Star Reward System
2.1.2. Personalized Feedback (Mini Hints)
During in-person instruction, a student struggling on an assignment will often receive a small
hint from an expert teacher looking over their shoulder. There is value for creating hints that
provide only a small amount of help. This small hint will keep the student productively working
on the assignment while reducing frustration, but still elicits effective struggle to solve a difficult
assignment. To try to replicate a teacher looking over the students’ shoulder, the Spatial Vis app
provides Mini Hints to students as “free hints” without the loss of stars. The grading algorithm
identifies the appropriate hint based on specific known errors of incorrect sketches. If these free
mini hints are not sufficient, a student will still have the option of looking at the original Hint
and Peek options that were shown in Fig. 2 but will lose stars. An ideal mini hint will move a
student forward in the solution, but still require them to actively engage and struggle to solve an
assignment [17]. For remote instruction, mini hints could potentially take upon increased
pedagogical importance.
The most common mini hint is a progress indicator indicating the percentage of the lines in the
student’s sketch that are correct, incorrect, and/or missing. This was previously shown in Fig. 2b.
The progress indicator helps quantify to the student how much of the various categories of lines
are present giving the student a sense of how close they are to finishing the sketch. However, it
does not provide specific graphical information as to what part of the sketch is correct or what
the actual solution is. If a student sees that their initial lines are correct, then they will know that
they are on the right track and feel encouraged to continue. Alternatively, a student may see early
on that much of their approach is incorrect which may encourage them to start over.
Another type of mini-hint provided by the grading algorithm is specific textual messages such as
“Hidden Line Incorrect”, “You may be missing a line”, “You may have an extra line”, or “Close!
Draw more carefully” (Fig. 4). Other hints that are anticipated in the future include “Shape is
drawn correctly but in the wrong location”, “The object is rotated incorrectly”, “Error in right
side (or top, front, etc.) view”, etc. The purpose of a mini hint is to provide some guidance to the
student without showing them the solution.
Figure 4. Textual Mini Hints Provided to Students Based on Errors in Submitted Sketch
2.1.3. 2.1.3 Other Features Within the App
To facilitate instruction, eGrove Education also has created Lecture Slides and Videos that
introduce the basic concepts necessary to solve the assignments in each lesson. These slides and
videos are provided on the eGrove Education website [9] or can be directly accessed by the
students within the app on the first page of each lesson.
Additionally, test questions are provided as the last three questions of each lesson. These
assessments were chosen to be of moderate difficulty, and the Hint and Peek features are
disabled. These test questions try to incentivize students to learn the material in each lesson
rather than relying on the Hint and Peek features because students are aware that they will have
to eventually complete three assignments without help features before they can move on to the
next chapter.
Furthermore, a challenge with remote learning is ensuring all students have equitable access to
the Internet. The Spatial Vis app can work in offline mode if access to the Internet is a problem.
Essentially, grading of assignments can occur locally and progress in the app is saved until the
app can connect with the internet and sync student progress with eGrove Education’s servers.
3. Evaluation Methods
This evaluation takes place in two college engineering CAD courses in San Diego, CA. The first
course denoted as Class A (CIVE 121 Computer Graphics for the Built Environment) was taught
at San Diego State University. It was on a semester system and was approximately 5-6 weeks
into the semester when the pandemic hit. The other course denoted as Class B (SE 3 Graphical
Communication for Engineering Design) was taught at University of California, San Diego. It
started the quarter remotely. Students were given a standardized spatial skills test, the Purdue
Spatial Visualization Test: Rotations (PSVT: R) [18], before doing lessons in the app and
participants were assigned all nine lessons as part of their course. The lessons, total number of
assignments, and number of required assignments in each lesson, as well as the aggregate
average time for required assignments are summarized in Table 1. The assignments in each
lesson increased in difficulty. Both courses required the same Spatial Vis assignments: assigning
all of the odd-numbered problems in the app and the three test questions in which the assignment
reverts back to medium level difficulty and the hint and peek features are disabled. However, to
encourage persistence, students were tasked to complete these required problems without losing
any stars. If they needed to take a hint or peek and subsequently lose stars, they were able to
make up the lost stars by doing extra even-numbered problems. After completion, a post-test,
PSVT:R, was administered. Class A assigned one app module per week while Class B assigned
three modules per week and completed the app in the first three weeks of the quarter.
Table 1. Spatial Vis Lessons and Assignments
Lesson Total Number of
Assignments in
Lesson
Number of
Required
Assignments in
Lesson
Average Time for
Required
Assignments
Lesson 0: Introduction 6 6 3m 48s
Lesson 1: 2D Rotations 30 16 13m 48s
Lesson 2: Iso Cubes 35 19 57m 12s
Lesson 3: Ortho Cubes 33 18 53m 24s
Lesson 4: 2D to 3D 29 16 53m 42s
Lesson 5: Slopes and Curves 31 17 47m 48s
Lesson 6: Flat Patterns 31 17 38m 0s
Lesson 7: Rotations about 1 Axis 30 16 37m 30s
Lesson 8: Rotations about 2 Axes 30 16 44m 12s
Lesson 9: Assembly 20 11 21m 48s
Because the move to online was immediate, a controlled study regarding the impact that Spatial
Vis had on these institutions and students was not possible. However, qualitative information
regarding learning outcomes was established through instructor feedback. Both courses had
similar learning outcomes such as understanding engineering design methodology and
demonstrating the role of graphics in the engineering design process, organizing and delivering
effective verbal, written and graphical communication, and applying relevant sketching, 2D and
3D techniques and skills using modern engineering tools. Developing spatial visualization and
reasoning skills, related directly to the Spatial Vis app, was also one of the primary learning
outcomes for both courses.
Additionally, the digital nature of the app allows for the quantification of data such as how many
assignments the students completed in each module as well their performance on all questions.
These data points were used to establish the impact that the Spatial Vis app had on learning
outcomes and engagement during the COVID-19 pandemic to demonstrate its suitability not only
in the classroom but also for remote instruction.
3.1. Participants
San Diego State University is a large public institution with more than 30,000 undergraduate
students that has a very high racial diversity. 66 percent of students are minorities or people of
color (BIPOC). While diversity information for Class A was not readily available it was assumed
consistent with the Civil Engineering graduating class of 2018-2019, which consisted of 28.5%
White, 26.3% Hispanic or Latino, 22.6% Non-resident, 13.1% Asian, 8% Other races, and 1.5%
Black or African American [19]. During the Spring 2019 semester, there were 255 enrolled in
Class A of which 208 were participants in this analysis because they completed the pre and post
PSVT:R test. In Spring 2020 when COVID-19 hit, 195 of the 247 enrolled students were
included as part of the analysis. There were no hands-on building projects in this course.
Class B is a Structural Engineering Graphics course at the University of California, San Diego,
which is also a large public institution with more than 30,000 undergraduate students that has
very good racial diversity. 59% of students are minorities or people of color. Again, the diversity
statistics specifically for Class B were unavailable but was fairly consistent with the university
breakdown consisting of 33.8% Asian, 20.1% Hispanic or Latino, 19.4% White, 18.4% Non-
resident, 1.6% Black or African American, 0.4% Other and 6.2% ethnicity unknown [20].
During the Spring 2019 quarter, 129 students were enrolled in the course with 89 being included
in this analysis because they completed both the pre- and post- PSVT:R test, while in Spring
2020, 92 out of 105 enrolled students were included in the analysis. While the learning
objectives for Class B were similar, in the 2019 offering, it also included a team-based hands-on
project where students designed in CAD and did rapid prototyping using 3D printers and 2D
laser cutters to build a Rube Goldberg machine. In 2020, because the course was remote, the
team project was software based rather than the prior hands-on rapid prototyping project.
3.2. Instruments
Students’ performance was assessed using the Purdue Spatial Visualization Test: Rotations
(PSVT:R) [18] to measure student proficiency on spatial visualization skills. The PSVT:R is a
timed 30 multiple choice question assessment where students identify the correct object
orientation based on a particular rotation. Participants had 20 minutes to complete the PSVT:R
using an online learning management system (Canvas for both courses). The same online test
was used both pre-COVID-19 and during the pandemic. In both cases Spatial Vis lessons were
then assigned as part of the students’ regular schoolwork. After completion of the Spatial Vis
app, the PSVT:R was administered again as a post-test. For Class A the pre- and post-tests were
administered during Week 2 and Week 15 of the semester while Class B administered the pre-
and post-tests during Week 1 and Week 10 of the quarter. Both instructors were interviewed to
provide their personal opinion on the development of their student’s spatial visualization skills
and the impact that the app had during the pandemic.
3.3. Analysis
The average scores on the pre- and post- PSVT:R test were analyzed for Class A and Class B.
Students who scored low on their pretest were of specific interest. In this analysis the students
whose pre-test was < 70% are categorized as "At-Risk" for low graduation rates in future STEM
undergraduate programs due to low Spatial Visualization skills [7].
The average gains in PSVT:R scores do not tell the complete story, especially within the “At-
Risk” group. It has been shown in previous studies that learning gains between the pre- and post-
tests are correlated to students’ persistence [12], which is measured by how often a student
completes an assignment without using a hint or peek. Therefore, to assess persistence, the
average stars that students earned on assignments within the app were also evaluated. Persistent
students who try an assignment as many times as needed to get to the solution without taking
hints or peeks will have average stars close to 3.
Another metric to assess the effectiveness of the spatial visualization training is the percentage of
students who scored less than 70% on the PSVT:R but then raised their PSVT:R to above this
threshold. This metric captures gains in spatial visualization skills among students that need it
the most, since the 70% threshold corresponds to the skill level that is correlated to higher
graduation rates in STEM. This metric can be seen as an indicator of the percentage of students
who moved out of the "At-Risk" category for low graduation rates due to low spatial
visualization skills, and these large gains in PSVT:R scores can be used to characterize
effectiveness of spatial visualization training [12].
For both classes, the analysis separated the “At-Risk” students and evaluated pre- and post-
PSVT:R average scores, percent gains on the PSVT:R, average number of assignments
completed in the App, average number of test assignments (test assignments have Hint and Peek
disabled) completed in the app, average stars earned on assignments, and percent of students who
rose out of the “At-Risk” category. These results are summarized in the next section.
3.4. Results
The results of the analyses are provided in Tables 2 through 5. Table 2 shows the performance of
all of the students in Class A, while Table 3 summarizes the performance of the “At-Risk”
students in Class A. Similarly, Table 4 shows the performance of all of the students in Class B
while Table 5 shows the results for only the “At-Risk” students. For Class A, in 2019 there were
123 out of 208 students in the “At-Risk” category representing 59% of the students having low
spatial visualization skills, and 112 out of 195 students in 2020 representing 57%. For Class B,
37 out of 89 students in 2019 (42%) and 39 out of 92 students in 2020 (42%) were categorized in
the low spatial skills category.
Table 2. Overall Performance Results for All Students in Class A
Class A: Complete Class Spring 2019 (Before
COVID, n=208)
Spring 2020 (During
COVID, n=195)
Average Pre-PSVT:R 62.2% 62.2%
Average Post-PSVT:R 74.3% 68.6%
Average PSVT:R Gain 12.1% 6.4%
Average No. of Assignments Completed* 203.5 232.6
Average No. of Test Assignments Completed* 18.5 23.7
Average Stars 2.80 2.77
Correlation Between No. of Assignments Completed and PSVT:R gain r=-0.055, p=0.434 r=-0.129, p=0.072
Correlation Between Test Assignments Completed and PSVT:R gain r=-0.079, p=0.259 r=-0.037, p=0.604
Correlation Between Persistence Indicator (Ave. Stars) and PSVT:R gain r=-0.074, p=0.291 r=-0.053, p=0.466
Table 3. Overall Performance Results for “At-Risk” Students in Class A
Class A: At-Risk Students (Pre-PSVT:R < 70%) Spring 2019 (Before
COVID, n=123)
Spring 2020 (During
COVID, n=112)
Average Pre-PSVT:R 48.2% 47.3%
Average Post-PSVT:R 67.1% 58.6%
Average PSVT:R Gain 18.9% 11.3%
Average No. of Assignments Completed* 196.6 229.2
Average No. of Test Assignments Completed* 17.1 23.0
Average Stars 2.76 2.71
Percent of Student Who Rose to Be Out of Risk 48.8% 35.7%
Correlation Between No. of Assignments Completed and PSVT:R gain r=0.129, p=0.157 r=0.200, p=0.034*
Correlation Between Test Assignments Completed and PSVT:R gain r=0.192, p=0.033* r=0.117, p=0.221
Correlation Between Persistence Indicator (Ave. Stars) and PSVT:R gain r=0.165, p=0.069 r=0.137, p=0.149
* Statistically Significant when p < 0.05
Table 4. Overall Performance Results for All Students in Class B
Class B: Complete Class Spring 2019 (Before
COVID, n=89)
Spring 2020 (During
COVID, n=92)
Average Pre-PSVT:R 73.6% 71.8%
Average Post-PSVT:R 87.0% 84.7%
Average PSVT:R Gain 13.4% 12.9%
Average No. of Assignments Completed* 240.8 227.1
Average No. of Test Assignments Completed* 25.9 25.8
Average Stars 2.90 2.93
Correlation Between No. of Assignments Completed and PSVT:R gain r=-0.047, p=0.665 r=0.152, p=0.148
Correlation Between Test Assignments Completed and PSVT:R gain r=-0.062, p=0.566 r=0.128, p=0.224
Correlation Between Persistence Indicator (Ave. Stars) and PSVT:R gain r=-0.227, p=0.032* r=0.162,p= 0.122
* Statistically Significant when p < 0.05
Table 5: Overall Performance Results for “At-Risk” Students in Class B
Class A: At-Risk Students (Pre-PSVT:R < 70%) Spring 2019 (Before
COVID, n=37)
Spring 2020 (During
COVID, n=39)
Average Pre-PSVT:R 57.3% 56.1%
Average Post-PSVT:R 79.6% 78.3%
Average PSVT:R Gain 22.3% 22.2%
Average No. of Assignments Completed* 238.6 228.0
Average No. of Test Assignments Completed* 26.0 26.3
Average Stars 2.83 2.91
Percent of Student Who Rose to Be Out of Risk 78.4% 69.2%
Correlation Between No. of Assignments Completed and PSVT:R gain r=-0.205, p=0.224 r=0.385, p=0.015*
Correlation Between Test Assignments Completed and PSVT:R gain r=0.114, p=0.502 r=0.133, p=0.418
Correlation Between Persistence Indicator (Ave. Stars) and PSVT:R gain r=-0.084, p=0.621 r=0.504, p=0.001*
* Statistically Significant when p < 0.05
A key observation is that both before COVID-19 and during COVID-19 students increased their
spatial visualization ability after using the app. However, under the remote learning environment
there were some reductions in gains. In Class A, across all students who participated in the study,
the PSVT:R gains dropped from 12.1% in 2019 to 6.4% in 2020. For the At-Risk students the
gains dropped from 18.9% in 2019 to 11.3% in 2020. While these drops are not desirable, it was
still a positive indication that under the COVID-19 environments students still increased their
spatial visualization scores. In the At-Risk group the gains were still at 2/3 of the pre COVID-19
class.
For Class B, COVID-19 had only a very slight negative effect. Over the whole class, the
PSVT:R score gains dropped from 13.4% to 12.9%. For the At-Risk students the gains dropped
from 22.3% to 22.2%. The reason for why Class B performed higher than Class A is unknown,
but it is interesting to note that the persistence indicator of average stars was higher in Class B
than Class A. For example, the average stars for the At-Risk students during COVID-19 for
Class B was 2.91 while for Class A it was 2.71. This result should not detract that in both
classes, students had significant gains in spatial visualization ability.
Correlations were performed between gains in the PSVT:R scores and the average stars, number
of assignments completed, and number of test assignments completed. Those which were
statistically significant p<0.05 are indicated with an asterix in Tables 2 through 5. Plots in
Figures 5 and 6 show the correlations.
a) Class A b) Class B
Figure 5. Change in PSVT:R Score vs. Persistence Indicator for “At-Risk” Students
a) Class A b) Class B
Figure 6. Change in PSVT:R Score vs. Number of Assignments “At-Risk” Students
3.5. Instructor Feedback
The instructors (Class B instructor is an author on this paper) had already used the Spatial Vis
App prior to the pandemic and already knew that the app provided the ability for students to
draw by hand without being in the classroom, but nevertheless had very positive feedback about
the app’s usefulness for remote learning. Instructor for Class A said, “When we went online,
students couldn’t access the engineering graphics software that we used for the course without
access to our computer labs. But Spatial Vis worked as well online as in person. I didn’t have to
change anything and was able to seamlessly transition with Spatial Vis.”
The two instructors indicated that their implementation was not that different when their course
transitioned to online. Instructor for Class A said, “It was nice to have something that students
already had access to remotely. The only thing I did was make Spatial Vis a larger amount of
their grade since I was unsure about how to do online testing of CAD software”.
4. Conclusions
Spatial visualization training and specifically freehand sketching isometric and orthographic
shapes has been shown to increase graduation rates in engineering for students with low spatial
visualization skills. However, the ability to teach hand sketching during the pandemic and
remote instruction can be difficult. Accordingly, the potential for spatial visualization training in
which students can work independently is attractive. The Spatial Vis app was designed to meet
this need as it allows for freehand sketches to be graded automatically and for hints to be
provided when needed. The Spatial Vis app provides a resource that is not only effective in the
classroom but works seamlessly outside of the classroom providing flexibility to teachers.
The digital nature of the app allows for the quantification of data such as how many assignments
the students completed in each module as well their performance on test questions. This
quantifiable data was used to establish the impact that the Spatial Vis app had on learning
outcomes and engagement during the COVID-19 pandemic.
It is understood that the act of learning and practicing 3D CAD skills alone can improve spatial
visualization skills even without specific sketching training using the Spatial Vis app. However,
a Fall 2017 trial was conducted at the University of San Diego in which the Spatial Vis app was
assigned as homework for an experimental section (n=23), while a second section (n=22) was
used as a control. Among the low-performing students, those who used the app showed
remarkable progress. 8 of 13 (62%) raised their test scores above 70% compared to just 2 of 14
(14%) in the control group. While this study needs to be duplicated with a larger number of
students, this represents a 341% increase in the number of students who moved out of an “At-
Risk” category due to low spatial visualization skills [11] after using the Spatial Vis app and
demonstrates the efficacy of the app in increasing PSVT:R scores.
Also, it is not entirely clear why the COVID-19 classes (Spring 2020) resulted in slightly lower
scores than those of the pre-COVID-19 classes (Spring 2019). It could be attributed to low
engagement and challenges with struggling students when courses went to emergency remote
instruction. But the results pre-COVID-19 and during COVID-19 which essentially represent the
difference between face-to-face and remote learning were not significantly different.
Since the pandemic hit, eGrove Education has made significant improvements to the Spatial Vis
app and their teacher resources to support remote and in-person instruction. The teacher interface
was improved and now allows instructors to track student persistence (whether they are hinting
and peeking too often) and intervene as necessary.
Additionally, the experience of moving to remote instruction during the pandemic resulted in an
increased interest by middle schools and a much wider range of classes that use the Spatial Vis
app. Rather than developing a separate curriculum for the younger ages or for different types of
courses, teachers are now provided with the capability to customize the assignments their
students do within the app and take advantage of the scaffolding built into the system to assign
problems of relevant difficulty and applicability to their courses. Students can identify within the
app which assignments are required by their teachers and which are optional, and grade reports
for the teachers correlate directly to the required assignments.
5. Acknowledgements & Disclosure
Nathan Delson and Lelli Van Den Einde have equity interest in eGrove Education, Inc., a
company that may potentially benefit from the research results. The terms of this arrangement
have been reviewed and approved by the University of California, San Diego in accordance with
its conflict of interest policies. In addition, a Small Business Innovation Research (SBIR) grant
was awarded to eGrove Education, Inc., by the NSF (Award # 1648534), that also supported the
research effort of this publication.
References
[1] E. Nordmann, C. Horlin, J. Hutchison, J.A. Murray, L. Robson, M.K. Seery, J. MacKay,
"Ten simple rules for supporting a temporary online pivot in higher education”, PLOS
Computational Biology, October 1, 2020, Retrieved from
https://doi.org/10.1371/journal.pcbi.1008242.
[2] D. Schaffhauser, “Educators Feeling Stressed, Anxious, Overwhelmed and Capable”, The
Journal: Transforming Education, June 6, 2020, Retrieved from
https://thejournal.com/articles/2020/06/02/survey-teachers-feeling-stressed-anxious-
overwhelmed-and-capable.aspx.
[3] C. Cahill, S. Jackson, N. Summerall, K. Harruna, “Helping Career and Technical Education
Programs Meet this Moment”, JFF, September 1, 2020, Retrieved from:
https://www.jff.org/what-we-do/impact-stories/pathways-to-prosperity-network/helping-
cte-programs-meet-moment/
[4] E. Richmond, “Shop Class, Over Zoom: How one career and technical high school is going
remote”, The Atlantic, Retrieved from:
https://www.theatlantic.com/education/archive/2020/05/remote-career-technical-school-
COVID-19-pandemic/611047/
[5] S. Blackburn, “How schools can improve CTE during COVID-19 Online learning could
have increased the equity gap for students enrolled in career technical education courses”,
District Administration, Retrieved from: https://districtadministration.com/how-schools-
navigate-cte-online-cte-access-during-COVID-19/
[6] Spatial Vis App, www.egrove.education
[7] S. Sorby, “Educational research in developing 3‐D spatial skills for engineering students”.
International Journal of Science Education. Vol 31(3), pp 459-80, Feb. 2009.
[8] L. Van Den Einde, N. Delson, L. Cowan, “Sketching App to Teach Spatial Visualization
Skills Suitable for Remote and In-Person Instruction”, Proceedings of INTED 2021, virtual
conference, March 8-9, 2021.
[9] N. Delson, L. Van Den Einde, E. Cowan, J. Tara “eGrove Education.” [Online] Available
www.egrove.education.
[10] J. Power, J. Buckley, and N. Seery. “Visualizing Success: Investigating the Relationship
between Ability and Self-Efficacy in the Domain of Visual Processing”. 70th ASEE
Engineering Design Graphics Division Midyear Conference, Embry-Riddle Aeronautical
University, FL, January, 2016.
[11] G. Hoople, E. Cowan, L. Van Den Einde, J. Tara, and N. Delson. “Teaching Spatial
Visualization: A Controlled Trial of a Touchscreen App Implemented as Homework”,
Conference proceedings for Frontiers in Education: Fostering Education through Diversity,
October 3-6. San Jose, CA. 2018.
[12] N. Delson and L. Van Den Einde, “Tracking student engagement with a touchscreen app
for Spatial Visualization Training and freehand sketching,” 122nd ASEE Annual
Conference & Exposition proceedings, Seattle, WA, 2015.
[13] E. Cowan, N. Delson, R. Mihelich, and L. Van Den Einde, “Improvement in Freehand
Sketching Application for Spatial Visualization Training,” in Conference on Pen and
Touch Technology in Education, Illinois, 2017.
[14] L. Van Den Einde, N. Delson, E. Cowan, and D. Yang, “Increasing Student Persistence In
A Sketching App For Spatial Visualization Training,” ICERI2017 Proceedings, pp. 5373–
5381, October 2017.
[15] N. Delson, and L. Van Den Einde, “Sketching, Assessment, and Persistence in Spatial
Visualization Training on a Touchscreen”, in 124th ASEE Annual Conference &
Exposition proceedings, Salt Lake City, UT, 2018. 10.18260/1-2--30968.
[16] L. Van Den Einde, N. Delson, E. Cowan, “Freehand Sketching on Smartphones for
Teaching Spatial Visualization”, in 126th ASEE Annual Conference & Exposition
proceedings, Tampa, FL, 2019. 10.18260/1-2--32859
[17] N. Delson, L. Van Den Einde, E. Cowan, B. Mihelich, “Mini-Hints for Improved Spatial
Visualization Training”, in 126th ASEE Annual Conference & Exposition proceedings,
Tampa, FL, 2019, 10.18260/1-2--33112.
[18] R. Guay, (1977). “Purdue Spatial Visualization Test - Visualization of Rotations”. W.
Lafayette, IN. Purdue Research Foundation.
[19] Civil Engineering at the San Diego State University, College Factual, Accessed April 29,
2021, https://www.collegefactual.com/colleges/san-diego-state-university/academic-
life/academic-majors/engineering/civil-engineering/index.html
[20] University of California, San Diego Undergraduate Ethnic Diversity Breakdown, College
Factual, Accessed April 29, 2021, https://www.collegefactual.com/colleges/university-of-
california-san-diego/student-life/diversity/chart-ethnic-diversity.html
[21] J. Ottway, “Integrating CAD and CAM for Design-build Projects”, in 124th ASEE Annual
Conference & Exposition proceedings, Salt Lake City, UT, 2018. Retrieved from:
https://www.asee.org/public/conferences/106/papers/22132/view