Augmented Reality Equation Plotter Final Project EE 368 Spring 2013 Salman Naqvi Uzair Sikora

salman.naqvi@stanford.edu usikora@stanford.edu

Motivation:

A student in a math, science, or engineering class will undoubtedly come across an equation in the

course of his or her studies that the student will want to quickly visualize. Plotting the equation allows the

student to better understand some property of the real variables the equation relates. However, this may still

be difficult for younger students that are working on building the intuition to be able to visualize simple

equations quickly in their thoughts. In addition, some three dimensional expressions are difficult to picture

in one’s thoughts even with the intuition. While there are several tools available to the student to accomplish

this plotting, all involve leaving the context the equation originally appears in; i.e. the notebook. The goal

of this app is to provide the capability for the student to visualize an equation in his or her notes using the

Android platform.

Methods:

We will implement this using a Motorola DROID smartphone. The general approach we will take

is to first detect and identify the equation to be plotted in the image from the camera. Then detect and

identify the axis the user has drawn and any features the user has specified. For example, the presence of

any labeled tic marks and their spacing on the axis, axis labels, or the number of axis drawn. Finally, we

will then update the screen with the plot in the context of the axis defined by the user. This can be broken

down into a number of steps:

1. Equation Detection

a. Binarize the image using locally adaptive thresholding.

b. Segment the individual characters.

c. Use Optical Character Recognition to classify the characters. (Implemented by the

Tesseract OCR engine.) [1]

d. Analyze equation. (Check for consistency, number of dimensions, etc.)

2. Axis Detection

a. Binarize as before.

b. Segment out the axis regions.

c. Determine if there are any other markings around like tic marks by using a

morphological detector.

d. Track the axis location using feature tracking implemented in OpenCV. [2]

3. Plotting

a. Update the user’s screen with the plot in the correct region with rotation and resizing

applied as necessary.

As a start to the proposed project we will focus on detecting and plotting 2D functions that are

printed on paper as opposed to hand written. As we polish our approach, we will add 3D function

capabilities and hand-written equation detection as time permits.

References

[1] R. Smith, "An Overview of the Tesseract OCR Engine," in Proc. Ninth Int. Conference on Document Analysis

and Recognition (ICDAR), 2007.

[2] Itseez, "Android | OpenCV," 2013. [Online]. Available: http://opencv.org/platforms/android.html.

[3] R. Zanibbi and D. Blostein, "Recognition and retrieval of mathematical expressions," International Journal on

Document Analysis and Recognition (IJDAR), vol. 15, pp. 331-357, 2012.

[4] S. S. Tsai, H. Chen, D. Chen, R. Vedantham, R. Grzeszczuk and B. Girod, "Mobile visual search using image

and text features," in Signals, Systems and Computers (ASILOMAR), 2011 Conference Record of the Forty

Fifth Asilomar Conference on, 2011.