autolabel: matching physical sites with web sites for semantic localization rufeng meng, sheng shen,...
TRANSCRIPT
![Page 1: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/1.jpg)
AutoLabel:Matching Physical Sites with Web Sitesfor Semantic Localization
Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury
![Page 2: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/2.jpg)
Semantic Localization
Physical Location40.137675, -88.241373
?Semantic LocationBest Buy
?
![Page 3: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/3.jpg)
Wi-Fi Based Semantic Localization
![Page 4: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/4.jpg)
Wi-Fi Based Semantic Localization
![Page 5: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/5.jpg)
Core Problem
?
![Page 6: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/6.jpg)
Intuition: Manual Labeling
AP List Store NameMacy’sAT&T
Starbucks… …
Manual Labeling
![Page 7: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/7.jpg)
Intuition: AP Name Mining
AP List Store NameMacy’sAT&T
Starbucks… …
AP NameMining
Best Buyon Street
Macy’sin Mall
![Page 8: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/8.jpg)
Opportunity
![Page 9: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/9.jpg)
Intuition: Store Logo Searching
AP List Store NameMacy’sAT&T
Starbucks… …
Store Logo Searching
![Page 10: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/10.jpg)
Intuition: Matching In-store Text with Website Text
AP List Store NameMacy’sAT&T
Starbucks… …
Website Text Matching
![Page 11: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/11.jpg)
Intuition: Matching In-store Text with Website Text
Fusion Cafe Whole Foods Panera Starbucks
AP List Store Name
Starbucks
Starbucks
Starbucks
Starbucks
… …
Matching In-store Textwith Website Text
OCR
Text From Webpages
AutoLabelMatching In-store Text
with Website Text
![Page 12: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/12.jpg)
A Bigger Picture
AP List Store NameStarbucksStarbucksStarbucksStarbucks
… …StaplesStaplesStaples
… …Dollar TreeDollar TreeDollar Tree
… …AutoLabel
Matching In-store Text with Website Text
Best Buy StaplesDollar Tree Starbucks
?
![Page 13: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/13.jpg)
In-store Text Extraction
Blurred Image Filter
OCR
![Page 14: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/14.jpg)
Website Text Extraction
Google Map Search
Google Web Search
Fusion Cafe Whole Foods
Panera Starbucks
![Page 15: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/15.jpg)
Bag-of-Words Text Matching
Fusion Cafe Whole Foods Panera Starbucks
Extract Nouns and Proper Names
Text From Webpages
Assign TF-IDF Weight
Calculate Cosine Similarity
Text From In-store Photos
![Page 16: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/16.jpg)
Preliminary Evaluation
![Page 17: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/17.jpg)
Data Collection• 18 stores in a shopping mall
- 2 nutrition stores - 3 sportswear stores - 3 electronics stores - 4 appeal stores - 6 other businesses
• Google Glass takes panoramic videosSmartphone records Wi-Fi AP data
• 16 – 72 photos with text / store
![Page 18: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/18.jpg)
Evaluation• Different stores have dissimilar text.
MC Sports Store Finish Line Store
![Page 19: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/19.jpg)
Evaluation• Different stores have dissimilar text.• Different stores’ websites have dissimilar text.
MC Sports Website Finish Line Website
![Page 20: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/20.jpg)
Evaluation• Different stores have dissimilar text.• Different stores’ websites have dissimilar text.• Each store matches best with its own website
MC Sports StoreMC Sports Store
MC Sports Website
![Page 21: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/21.jpg)
How Similar Are Text In Different Stores?
![Page 22: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/22.jpg)
How Similar Are Text In Different Websites?
![Page 23: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/23.jpg)
Does Each Store Matches Best with Its Own Website?
Match one physical store (GNC) with 18 websites
![Page 24: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/24.jpg)
Improvement: Using Store Names
![Page 25: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/25.jpg)
Matching with Partial Data• Matching with limited number of photos
- Varying # of photos per store: 5, 10, …, all• Matching with specific portion of text
- In-store: above/at eye level
Above eye level
At eye level
Below eye level
![Page 26: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/26.jpg)
Matching with Partial Data• Matching with limited number of photos
- Varying # of photos per store: 5, 10, …, all• Matching with specific portion of text
- In-store: above/at eye level- Website: menus
![Page 27: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/27.jpg)
Matching with Partial Data: Evaluation
Partial Text
Full Text
10 photos with text per store are good enough!
![Page 28: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/28.jpg)
Limitations and On-going Work
• Need to evaluate with more stores, in various areas• Need to understand website and physical store structure
well for better text matching• Need to have a good clustering algorithm
Clustering
Matching
![Page 29: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/29.jpg)
Conclusion• AutoLabel is a solution for semantic localization.
?
![Page 30: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/30.jpg)
Conclusion• AutoLabel is a solution for semantic localization.• AutoLabel matches physical sites with web sites through
text matching, and gives a mapping from Wi-Fi access points to semantic store names.
![Page 31: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/31.jpg)
Conclusion• AutoLabel is a solution for semantic localization.• AutoLabel matches physical sites with web sites through
text matching, and gives a mapping from Wi-Fi access points to semantic store names.• Feasibility is verified through preliminary evaluation.
![Page 32: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/32.jpg)
Conclusion• AutoLabel is a solution for semantic localization.• AutoLabel matches physical sites with web sites through
text matching, and gives a mapping from Wi-Fi access points to semantic store names.• Feasibility is verified through preliminary evaluation.
Phys
ical S
tore
s Websites
TEXT
Physical Locations Semantic Locations
![Page 33: AutoLabel: Matching Physical Sites with Web Sites for Semantic Localization Rufeng Meng, Sheng Shen, Srihari Nelakuditi, Romit Roy Chouhury](https://reader038.vdocuments.mx/reader038/viewer/2022110323/56649d745503460f94a54029/html5/thumbnails/33.jpg)
Thanks for your attention!
Questions?
Paper available at http://synrg.csl.illinois.edu/