Download - Open Government Data and MongoDB
![Page 2: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/2.jpg)
![Page 3: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/3.jpg)
Question? @LuigiMontanez
![Page 4: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/4.jpg)
Question? @LuigiMontanez
Open Data + Open Source = Open Government
![Page 5: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/5.jpg)
Question? @LuigiMontanez
MongoDB enablesopen data
![Page 6: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/6.jpg)
Question? @LuigiMontanez
Opening Up Data
✴ Gather data from disparate sources✴ Data dumps (SQL, Fixed-width columns)✴ Web scraping✴ Text/PDF parsing
✴ Serving RESTful JSON APIs
![Page 7: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/7.jpg)
Question? @LuigiMontanez
JSON
✴ Tree structure, not tabular✴ Still relational✴ JSON for data, XML for documents✴ Closely resembles native data structures✴ No manual parsing needed
![Page 8: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/8.jpg)
Question? @LuigiMontanez
Three Projects
✴ Poligraft✴ Real Time Congress API✴ Open State Project
![Page 9: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/9.jpg)
Question? @LuigiMontanez
Three Projects
✴ Poligraft✴ Real Time Congress API✴ Open State Project
![Page 10: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/10.jpg)
Question? @LuigiMontanez
App designdrives
schema design
![Page 11: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/11.jpg)
![Page 12: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/12.jpg)
![Page 13: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/13.jpg)
![Page 14: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/14.jpg)
![Page 15: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/15.jpg)
Text
{ "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com"}
![Page 16: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/16.jpg)
![Page 17: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/17.jpg)
Text
{ "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com",
"slug": "EOsc","source_url": "http://www.politico.com/news/stories/0810/40534.html","content": ".................",
}
![Page 18: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/18.jpg)
![Page 19: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/19.jpg)
![Page 20: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/20.jpg)
Text
{ "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com",
"slug": "EOsc","source_url": "http://www.politico.com/news/stories/0810/40534.html","content": ".................","entities": [...]
}
![Page 21: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/21.jpg)
Text
{ "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com",
"slug": "EOsc","source_url": "http://www.politico.com/news/stories/0810/40534.html","content": ".................","entities": [
{"name": "Barack Obama","type": "politician",},...
]}
![Page 22: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/22.jpg)
![Page 23: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/23.jpg)
Text
{ "title": "President Obama's climate 'Plan B' in hot water - Darren Samuelsohn - POLITICO.com",
"slug": "EOsc","source_url": "http://www.politico.com/news/stories/0810/40534.html","content": ".................","entities": [
{"name": "Barack Obama","type": "politician","breakdown": {"indiv": "33", "pac": "67"}"top_industries": ["Lawyers/Lobbyists","Finance/Insurance/Real Estate","Misc. Business"]},...
]}
![Page 24: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/24.jpg)
![Page 25: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/25.jpg)
Question? @LuigiMontanez
Natural Schemas
![Page 26: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/26.jpg)
Question? @LuigiMontanez
Three Projects
✴ Poligraft✴ Real Time Congress API✴ Open State Project
![Page 27: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/27.jpg)
Real-Time Congress API
Credit: vgm8383 on Flickr
![Page 28: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/28.jpg)
Android App: “Congress”
![Page 29: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/29.jpg)
Politiwidgets
![Page 30: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/30.jpg)
Question? @LuigiMontanez
Requirements✴ Aggregate lots of data
Biographical, Bills, Votes, Earmarks, Video Clips, Floor Updates, Legislative Documents, Committee Schedules, Contributions, Interest Group Ratings
✴ Lightweight responses
![Page 31: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/31.jpg)
{legislator: { in_office: true, title: "Rep", nickname: "", district: "9", bioguide_id: "L000551", govtrack_id: "400237", phone: "202-225-2661", website: "http://lee.house.gov/index.html", twitter_id: "", last_name: "Lee", name_suffix: "", last_updated: "2010/04/13 00:00:14 +0000", party: "D", chamber: "house", state: "CA", youtube_url: "http://www.youtube.com/RepLee", first_name: "Barbara", gender: "F", congress_office: "2444 Rayburn House Office Building", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } ...}
![Page 32: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/32.jpg)
// limit selection to a subset of fieldsdb.people.find( { 'first_name' : 'john' }, { 'last_name' : 1, 'address' : 1 } );
// use dot-notation to dig into an objectdb.people.find( { 'state': 'CA' }, { 'address.zip_code': 1 } );
![Page 33: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/33.jpg)
{legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, }}
?sections=last_name,first_name,state,earmarks
![Page 34: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/34.jpg)
{legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { total_amount: 10000000, total_number: 28 }}
?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number
![Page 35: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/35.jpg)
Question? @LuigiMontanez
Partial responses make payloads
smaller
![Page 36: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/36.jpg)
Question? @LuigiMontanez
Three Projects
✴ Poligraft✴ Real Time Congress API✴ Open State Project
![Page 37: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/37.jpg)
![Page 38: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/38.jpg)
Question? @LuigiMontanez
50 States =50 Formats
![Page 39: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/39.jpg)
Question? @LuigiMontanez
Schemalessness allows for granular
control
![Page 40: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/40.jpg)
Question? @LuigiMontanez
Custom Fields✴ Traditional RDBMS
✴ Update the schema for new fields, run a migration, feel icky
✴ Create a custom_fields table✴ MongoDB
✴ Just store it
![Page 41: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/41.jpg)
Question? @LuigiMontanez
Speaking JSONnatively
![Page 42: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/42.jpg)
Source Scraped JSON PythonTransform PostgreSQL
![Page 43: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/43.jpg)
Source Scraped JSON MongoDB
![Page 44: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/44.jpg)
Question? @LuigiMontanez
Three Projects
✴ Poligraft✴ Real Time Congress API✴ Open State Project
![Page 45: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/45.jpg)
Developer Happiness
![Page 46: Open Government Data and MongoDB](https://reader033.vdocuments.mx/reader033/viewer/2022061221/54bf76aa4a795940398b456d/html5/thumbnails/46.jpg)
Question? @LuigiMontanez
Thanks!sunlightlabs.com@LuigiMontanez