In-Place Vision-Based Skewed 1D Barcode Scanning &

Effective Use of Nutrition Labels on Smartphones

Vladimir KulyukinDepartment of Computer Science

Utah State University

vkedco.blogspot.com CS Graduate Research Seminar, USU, 10/30/14

Outline

● Introduction● R&D Background● An Algorithm for In-Place 1D Skewed Barcode Scanning with Relaxed

Pitch, Roll, and Yaw Constraints● 1D Skewed Barcode Scanning Experiments & Results● Effective Use of Nutrition Labels on Smartphones● Current Challenges

CS Graduate Research Seminar, USU, 10/30/14

Introduction: Camera Alignment

● A common requirement for many mobile barcode scanners, both commercial and open source, is camera alignment: the camera must be aligned with a target barcode to obtain at least one complete scanline

● This requirement presents difficulties for visually impaired smartphone users or for users who do not have adequate control of their hands

● Another challenge is lack of coupling of barcode scanning to comprehensive nutrition label databases

CS Graduate Research Seminar, USU, 10/30/14

Introduction: Nutrition Labels

● Many products sold worldwide have nutrition labels (NLs) ● In the U.S., the display of nutrition information is mandated by the

Nutrition Education and Labeling Act (NLEA) of 1990● EU FLABEL is a similar legislative act in EU● Even highly motivated consumers find it difficult to locate and

comprehend nutrition information on many products● Due to their ever increasing presence, smartphones are an

excellent platform for proactive nutrition management

CS Graduate Research Seminar, USU, 10/30/14

Two Great Books

CS Graduate Research Seminar, USU, 10/30/14

R& D Background

CS Graduate Research Seminar, USU, 10/30/14

Dominant Orientation of Gradients● In our previous work (Kulyukin et al., IPCV 2013), we

developed a vision-based algorithm for localizing skewed barcodes on smartphones (pdf)

● The algorithm localizes skewed barcodes in captured frames by computing dominant orientation of gradients (DOGs) of image segments and collecting smaller segments with similar DOGs into larger connected components (DOGs eat barcodes :-))

● The algorithm was successfully evaluated on 1,066 images

CS Graduate Research Seminar, USU, 10/30/14

Dominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of GradientsDominant Orientation of Gradients

Regions with similar DOGs are combined into clusters on the basis of their geometric proximity

CS Graduate Research Seminar, USU, 10/30/14

An Algorithm for In-Place 1D Barcode Scanning

with Relaxed Pitch, Roll, and Yaw Constraints

CS Graduate Research Seminar, USU, 10/30/14

Pitch, Roll, & Yaw

CS Graduate Research Seminar, USU, 10/30/14

Algorithm in Action

Online demo videos:- lab video 1- lab video 2- supermarket video 1- supermarket video 2

CS Graduate Research Seminar, USU, 10/30/14

Algorithm's Flowchart

CS Graduate Research Seminar, USU, 10/30/14

Growth of Scanlines

Scanlines are grown orthogonally to the skew line detected by the the DOG algorithm

CS Graduate Research Seminar, USU, 10/30/14

Growth of Scanlines

The number of grown scanlines is a parameter currently set to 10.

CS Graduate Research Seminar, USU, 10/30/14

Linux Cluster for Image Processing &

Data Storage

CS Graduate Research Seminar, USU, 10/30/14

Hardware

● A Linux cluster was originally built out of four Dell computers; two nodes were added later

● Each node computer has an Intel Core i5-6503.2 GHz dual-core processor that supports 64-bit computing

● The node's processor has 3MB of cache memory● Each node has 6GB DDR3 SDRAM and 320GB of hard

disk space

CS Graduate Research Seminar, USU, 10/30/14

Software

● Ubuntu 12.04 LTS was installed on each machine● JBoss was used to build and configure the cluster● JBoss AS 7.1.1 was installed on every node● Load balancing is done with the Apache mod_cluster

module● The cluster has one master and five slaves● All machines are part of a local area network

CS Graduate Research Seminar, USU, 10/30/14

Software ● The system's front end was implemented as a smartphone

application that runs on Android 4.3 or higher● The system's back end was deployed on the Linux cluster● A Java servlet for image recognition was deployed on the

master node● The servlet receives images from smartphones through

HTTP POST requests and sends nutrition labels back to the smartphones

CS Graduate Research Seminar, USU, 10/30/14

1D Skewed Barcode Scanning Experiments &

Results

CS Graduate Research Seminar, USU, 10/30/14

Experiment Design ● The system was evaluated on a set of 506 video recordings

of common grocery products● The videos have a 1280x720 resolution, an average

duration of 15 seconds and were recorded on an Android 4.2.2 Galaxy Nexus smartphone in a supermarket

● The videos covered four product categories: bags, boxes, bottles, and cans

● The videos are publicly available here

CS Graduate Research Seminar, USU, 10/30/14

Experiment Design

● There are 130 box videos, 127 bag videos, 125 bottle videos, & 124 can videos

● Frames were extracted at a rate of 1 frames per second, which resulted in 7,545 frames

● Frames were manually classified into 1950 box frames, 1905 bag frames, 1875 bottle frames, and 1860 can frames

● Manually classified frames were automatically classified as blurred or sharp with an algorithm based on 2D Haar filter (Python source is at https://github.com/VKEDCO/PYPL/blob/master/haar_blur)

CS Graduate Research Seminar, USU, 10/30/14

Average Request-Response Times (ms)

16 sessions of 100 frames per session; tests were done over a Wi-Fi network with a download speed of 72.31 Mpbs and an upload speed of 24.64 Mbs

CS Graduate Research Seminar, USU, 10/30/14

Impact of Blurriness

CS Graduate Research Seminar, USU, 10/30/14

Average Scan Times in Product Categories

CS Graduate Research Seminar, USU, 10/30/14

Results● In all four categories of products, the true negative and false positive

percentages are 0● The false negative percentages are relatively high: the algorithm

rejects frames on the slightest chance that it does not contain a barcode

● The algorithm scans barcodes of any orientation in the yaw plane, of up to 73 degrees in the pitch plane, and of up to 55 degrees in the roll plane

CS Graduate Research Seminar, USU, 10/30/14

NutriGlass 1.0 on Google Play

The free front end application is available on Google Play; search “NutriGlass on Google Play” to download; make sure that OpenCV libraries are available on your Android platform; online demo is here.

CS Graduate Research Seminar, USU, 10/30/14

Effective Use of Nutrition Labels on

Smartphones

CS Graduate Research Seminar, USU, 10/30/14

Barriers to NL Comprehension

● Recent investigations of NL use by consumers have used digital cameras to track consumers' eye movements to understand how consumers locate and understand NLs

● Four key barriers have been identified in the literature: 1) label's location on the package; 2) presentation of information within the label; 3) label's surface size; 4) surrounding visual clutter

CS Graduate Research Seminar, USU, 10/30/14

Confusing NL Examples

CS Graduate Research Seminar, USU, 10/30/14

Confusing NL Examples

CS Graduate Research Seminar, USU, 10/30/14

Recommendations from NL Use Literature

1) Central positions of NLs2) Nutrients sorted by health relevance3) Explanation of nutrients4) Reduced visual clutter around NLs 5) Increased visual salience through contrast and orientation6) Increased surface size of NLs

CS Graduate Research Seminar, USU, 10/30/14

Implementation Details

● The front end application sends captured frames to the back end wirelessly

● The back end cluster does image processing (1D skewed UPC and EAN barcode scanning) and, if the scanning is successful, the scanned barcode is used to retrieve a corresponding NL from a database of 230,000 NLs

● The retrieved NL is displayed on the smartphone

Sample Screenshots

CS Graduate Research Seminar, USU, 10/30/14

NutriMenuNXT

NutriMenuNXT is an Android application that allows the user to get nutritional information on menu items from 296 restaurant chains; search “NutriMenuNXT on Google Play” to download; online demo is here

CS Graduate Research Seminar, USU, 10/30/14

Back to Four Targeted Recommendations ● NutriGlass & NutriMenu satisfy four out of six recommendations

found in the literature on effective use of nutrition labels:

1) Central positions of NLs: NLs are displayed centrally on the smarpthone's screen3) Explanation of nutrients: Wiki links4) Reduced visual clutter around NLs: no visual clutter around the label 6) Increased surface size of NLs: NLs are displayed on the entire touchscreen

CS Graduate Research Seminar, USU, 10/30/14

Current Challenges

CS Graduate Research Seminar, USU, 10/30/14

Image Blur Detection● Blurred images, as experiments show, have a

negative impact on skewed barcode detection● A current challenge is to detect and filter blurred

images before sending them to the cloud● If image blur can be detected on the smartphone,

data throughput and recognition accuracy will likely improve

CS Graduate Research Seminar, USU, 10/30/14

Blurred vs. Sharp Images

CS Graduate Research Seminar, USU, 10/30/14

This image should be thrown away This image should be sent to the cloud for processing

Image Blur Detection● We can estimate a degree of change in various

regions of the image (this can be done with wavelets)● Hypothesis: sharp images will have clusters of

regions with pronounced changes; blurred images will either have no such regions or small clusters

● Initial experiments indicate that the hypothesis may be validated

CS Graduate Research Seminar, USU, 10/30/14

High-Change Clusters in Blurred & Sharp Images

CS Graduate Research Seminar, USU, 10/30/14

Blurred Image: no high-change clusters Sharp Image: pronounced high-change cluster in the middle

Nutrition Information Extraction

Line Segmentor

Table Segmentor

TTS Engine

Image

Text

Table Lines

Open CVOCR

Engine

CS Graduate Research Seminar, USU, 10/30/14

Skewed Nutrition Labels & Texts● We have solved the problem of localizing aligned and slightly skewed

nutrition labels:– Kulyukin, V. and Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed

Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.

– Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.

● Our current challenge is to detect highly skewed nutrition labels and to estimate the amount (in degrees) of text skew: this will allow us to detect rows and use OCR to extract information

CS Graduate Research Seminar, USU, 10/30/14

References● V. Kulyukin & T. Zaman. Vision-Based Localization and Scanning of 1D UPC and EAN Barcodes with

Relaxed Pitch, Roll, and Yaw Camera A lignment Constraints. International Journal of Image Processing (IJIP), V olume (8) : Issue (5) : 2014, pp. 355-383.

● Kulyukin, V. & Blay, C. An Algoritm for Mobile Vision-Based Localization of Skewed Nutrition Labels that Maximizes Specificity. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 3-9, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.

● Kulyukin, V. & Zaman, T. An Algorithm for In-Place Vision-Based Skewed 1D Barcode Scanning in the Cloud. In Proceedings of the 18th International Conference on Image Processing and Pattern Recognition (IPCV 2014), pp. 36-42, July 21-24, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-280-1.

● Kulyukin, V., , Kutiyanawala, A., Zaman, T, & Clyde, S. (2013). Vision-Based Localization & Text Chunking of Nutrition Fact Tables on Android Smartphones. In Proceedings of the International Conference on Image Processing, Computer Vision, & Pattern Recognition (IPCV 2013), pp. 314-320, ISBN 1-60132-252-6, CSREA Press.

● Kulyukin, V., Zaman, T., and Andhavarapu, S. Effective Use of Nutrition Labels on Smarphones. In Proceedings of the 15th International Conference on Internet Computing and Big Data (ICOMP 2014), pp. 93 - 99, July 21-24, 2014, Las Vegas, NV, USA, CSREA Press, ISBN: 1-60132-227-1.

CS Graduate Research Seminar, USU, 10/30/14

Thank you!

CS Graduate Research Seminar, USU, 10/30/14