Download - 20090914 Petamedia Irp5
![Page 1: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/1.jpg)
IRP5: Social Media Data Acquisition
Presented by: Arjen P. de Vries
![Page 2: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/2.jpg)
20/10/2008 2
PROBLEM
• Each of us can think of many research questions related to the Petamedia objective of integrating the data network, the user network and the physical network...
![Page 3: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/3.jpg)
20/10/2008 3
![Page 4: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/4.jpg)
20/10/2008 4
PROBLEM
... but, today, these three networks are mostly disparate – they do not overlap!
So, how to evaluate the effectiveness of our ideas?!
![Page 5: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/5.jpg)
20/10/2008 5
IRP5 OBJECTIVES
• Develop a common data-set that is large enough to allow meaningful research, and contains video content as well as explicit information generated by a social network of sufficient density
• Organize a PetaMedia ‘data-set yellow pages’irp5petamedia.pbworks.com
Functions also as crawling code repository!
![Page 6: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/6.jpg)
20/10/2008 6
![Page 7: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/7.jpg)
20/10/2008 7
DATASET REQUIREMENTS
• Availability of video data• Creative Commons (CC) licenced data only• Data sources with developer-friendly APIs
only
• Availability of social data• Users creating the data should be organized
in a social network, and provide feedback about their preferences in relation to the data (comments, ratings, ...)
![Page 8: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/8.jpg)
20/10/2008 8
CANDIDATE DATA SOURCES
• Blip.tv• high quality, 25-60% CC, poor social data
• Revver.com• Medium quality, 100% CC, poor social data
• Flickr.com• 200K CC,• No API for access to video content
![Page 9: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/9.jpg)
20/10/2008 9
DECISIONS
1) Join Blip.tv and revver.com with Del.icio.us, digg and Twitter to get richer social data
2) Crawl links to videos, as well as the social networks of users creating those links, up to 4 levels of social network but only 2 levels of metadata (bookmarks/posts/profiles)
3) Crawl Flickr irrespective of missing API
Comments,User info,Friend info
Video datablip.tv
revver.com
Social ‘mention’ in•Digg•Del.icio.us•Twitter
![Page 10: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/10.jpg)
20/10/2008 10
FLICKR
• Outline:• Video data acquired through scraping the mobile flickr
site (m.flickr.com)• At most 90 seconds each• Only mp4 (no flv)
• Metadata acquired through API• Typical download rate: 1 video with metadata per
minute• Approach:
• Query for travel-related tags; 10 videos per tag• Leads to 211 videos from 162 uploaders;• Leads to 4143 videos in total from these uploaders• Leads to 17598 videos from their 32K contacts
![Page 11: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/11.jpg)
20/10/2008 11
BLIP+REVVER
• Most popular videos:• Blip-10,000 and Revver-10,000 data sets
• Social mentions of additional videos:• 175 blip.tv and 45 revver.com downloadable
videos mentioned at Del.icio.us (out of 5GB of social data, reached by starting with ‘joshua’ – its founder – and recursively following network fan links)
• 1250 blip.tv and 9198 revver.com video clips digg-ed (by 3602 unique users)
• ~850 blip.tv links posted by Twitter users per week
![Page 12: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/12.jpg)
20/10/2008 12
PRELIMINARY RESULTS
• Del.icio.us (now delicious.com) is better queried over the TU Berlin DAI-Labor lab collection (bookmarks from 2003-2007)• 14K distinct links to blip.tv (from 22K
bookmarks)• 4K distinct links to revver (from 10K
bookmarks)
• Reason: API truncates results to only 100 per item
![Page 13: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/13.jpg)
20/10/2008 13
PRELIMINARY RESULTS
• Twitter is better ‘crawled’ through Topsy, a search engine over Tweets• 27K links to blip (from 42K indexed by Topsy)• 300 links to revver (from ~1200 indexed by
Topsy)
• Reason: API usage limited to #queries per IP address, but, more importantly, API access only to msgs at most 7 days old• BTW: Topsy has been queried using ~2100
‘popular’ `travel-related’ tags, to circumvent 500 results per query limitation
![Page 14: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/14.jpg)
20/10/2008 14
NEXT
• Crawl actual social network data and tweets corresponding to the Twitter V2 and Del.icio.us V2 links, by re-using QMUL and EPFL code
![Page 15: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/15.jpg)
20/10/2008 15
TO BE DONE
• Analyse data and methods• Useful?• Complete?
• Create data repository• Legal check for public sharing• Twitter data• Flickr data html-scraped
• Advertise data sets (SIGIR-Forum?)
![Page 16: 20090914 Petamedia Irp5](https://reader033.vdocuments.mx/reader033/viewer/2022061217/54b41af84a7959620e8b4588/html5/thumbnails/16.jpg)
20/10/2008 16
TEAM
• TUD:• Pavel Serdyukov (coordination, Del.icio.us V2, Twitter V2)• Stevan Rudinac (blip.tv, revver.com)• Ronald Poppe (blip.tv, revver.com)• Maarten Clements (blip.tv, revver.com)• Arjen P. de Vries (coordination)
• TUB:• Sebastian Schmiedeke (flickr.com)
• EPFL:• Ivan Ivanov (Twitter V1)
• UEP:• David Chudan (Digg)• Tomas Kliegr (Digg)
• QMUL:• Naeem Ramzan (Del.icio.us V1)• Muhammad Akram (Del.icio.us V1)