interactive stem learning with zspace
DESCRIPTION
Joseph, Will and Sarah will present a discussion on developing a software solution for the zSpace device that combines the 3D head tracked stereo display with the ability to write on the surface of the device with the stylus. Their focus is in STEM education where users can write down mathematical expressions and have them recognized by the system. These recognized expressions can then be used to help drive 3D simulations. They will present how the software was developed, lessons learned, and provide examples dealing with the graphing of 3D functions, torque, projectile motion, and kinetic energy.TRANSCRIPT
Interactive STEM Learning with zSpace
Will Holderness, Sarah Buchanan, Dr. Joseph J. LaViola Jr.
Overview• Motivation
• Related Work
• System Overview
• Challenges
Motivation• Incorporate handwriting input on
zSpace Display
• Educational application – Math and Physics
• zSpace Display incorporates 3D visualization & interaction
Related Work• Physics Book: a sketch-based
interface for animating physics diagrams by Salman Cheema– Allows users to sketch diagrams and
equations and then animate the results using a 2D physics engine
• Math Pad: another sketch based math tutor by Dr. Joseph J. LaViola Jr.
Math PadProjectile Motion Example
Physics Book Examples
3D Physics Book• Our goal is to make a 3D version of
Physics Book using the zSpace display
• The zSpace stylus will work as pen and gestural input
• Unity 3D to create the environment and leverage existing 3D physics engine
System Components• Handwriting Recognition– StarPad handwriting recognition
• Runs as server in separate process• Recognizes mathematical equations, returns
results• Same engine as Physics Book
• Stylus stroke collection– Stylus position is tracked using IR
cameras
– Challenge: No touch input on the screen• Need to detect touch input by stylus up &
down
System Components• Mouse Emulation by Stylus
– Implemented by Jerry Tu and his team at zSpace
– Track Stylus optically and with onboard inertial sensors
– Taps can be detected as rapid changes in acceleration
– Combination of position, orientation, and tap-detection allows us to determine when stylus starts writing on screen and where it goes.
– Inject these measurements into events on the Windows mouse event queue • mouse down, mouse up, double-click, and position
System Components• 3D Stereo & Head Tracking– zSpace SDK provides Unity Prefabs for
3D Stereo & Stylus setup.
– Objects in scene must use realistic sizes for 3D to work properly. • For the object to appear 4inches from the
screen, it must be 4 inches from the virtual camera in Unity.
Math and Physics Lessons• Projectile Motion
• Kinetic vs. Potential Energy
• Torque
• 3D Graphing
Projectile Motion
Kinetic Energy
Torque
Graphing
Challenges
1. Taps and lift are difficult to detect
2. Not all .NET libraries compatible with Unity
3. Changing object size and location affected the stereo & head tracking experience
4. The stylus position is relative to the main camera’s position
Challenge – .NET Libraries in Unity• StarPad is based on F# – Not compatible
• Windows built in Ink & Recognition libs– Not compatible
• Sockets that allow us to connect to external servers– Easy and Fast!
– Doesn’t cause lag in Unity
Challenge – Stereo & Head Tracking• Motion Parallax incorrect
• Can cause the stylus pointer not to follow the stylus exactly
• The zSpace Unity SDK includes a number of variables such as World Scale– Still difficult to find exactly right values
• zSpace Diagnostic Window has sliders for each variable
• Plan scenes in advance
Challenge – It’s All Relative.• We wanted to be able to “pick up”
the camera with the stylus and move the viewpoint around
• The stylus position is relative to the camera position– This prevented us from simply moving
the camera the same delta distance as the stylus
Summary• Adding the zSpace package to Unity
was trivial– zSpace elements to a project is drag and
drop
• Handwriting is still a challenge
• Graphing turned out to be the most interesting/engaging lesson – promoted most exploration from users
Questions?