pig at linkedin
DESCRIPTION
Pig at LinkedIn by Chris Riccomini from LinkedInPig is an integral part of data analytics at LinkedIn. Learn about LinkedIn’s analytic stack, and see how Pig is used to design, develop, and deliver data products at LinkedIn. We’ll explore a successful example of Pig deployment at LinkedIn, pain points, and integration with Azkaban, Voldemort, Hadoop, and the rest of LinkedIn’s ecosystem.TRANSCRIPT
![Page 1: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/1.jpg)
Pig at LinkedinChris Riccomini
9/29/10
![Page 2: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/2.jpg)
Who?
![Page 3: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/3.jpg)
What?
![Page 4: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/4.jpg)
LinkedIn Analytics
![Page 5: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/5.jpg)
![Page 6: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/6.jpg)
Pig at LinkedIn
![Page 7: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/7.jpg)
Why?
![Page 8: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/8.jpg)
Production Quality
![Page 9: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/9.jpg)
Streaming
![Page 10: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/10.jpg)
Serialization
![Page 11: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/11.jpg)
VoldemortStorage ~ Avro
![Page 12: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/12.jpg)
views = LOAD '/data/awesome' USING VoldemortStorage();
![Page 13: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/13.jpg)
Voldemort ♥ Pig
![Page 14: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/14.jpg)
Partitioning
![Page 15: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/15.jpg)
YYYY/MM/DD
![Page 16: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/16.jpg)
Last N days?
![Page 17: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/17.jpg)
views = LOAD '/data/etl/tracking/extracted/profile-view' USING VoldemortStorage('date.range', 'num.days=90;days.ago=1’)
![Page 18: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/18.jpg)
Some-file-YYYY-MM-DD
![Page 19: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/19.jpg)
member_position = LOAD '/data/etl/replicated/member/member_position/#LATEST' USING VoldemortStorage()
![Page 20: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/20.jpg)
Scheduling
![Page 21: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/21.jpg)
Azkaban
![Page 22: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/22.jpg)
type=pig
pig.script=myscript.pig
![Page 23: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/23.jpg)
![Page 24: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/24.jpg)
![Page 25: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/25.jpg)
Ad hoc?
![Page 26: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/26.jpg)
Future at LinkedIn
![Page 27: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/27.jpg)
Wishes
![Page 28: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/28.jpg)
Dates
![Page 29: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/29.jpg)
Fix Data Types
![Page 30: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/30.jpg)
JSON
![Page 31: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/31.jpg)
Cross Platform
![Page 32: Pig at Linkedin](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54be2e754a795949228b458c/html5/thumbnails/32.jpg)
Questions?
• [email protected]• http://www.riccomini.name• http://www.sna-projects.com• http://www.project-voldemort.com• @criccomini• LinkedIn is Hiring! Email me!