building data-centric businesses
TRANSCRIPT
![Page 1: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/1.jpg)
Daniel Aragao & Simon Hope
![Page 2: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/2.jpg)
Daniel Aragao Simon Hope@dear_dr_dan @mapbutcher
![Page 3: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/3.jpg)
REALESTATE.COM.AU
6BMarket Cap
11MAustralian Properties
55MVisits in September
4.7MApp Downloads …and counting
![Page 4: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/4.jpg)
3,500PEOPLE
13COUNTRIES
34OFFICES
TECHNOLOGY &
SOCIAL JUSTICE
![Page 5: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/5.jpg)
• In the beginning…
• Organising our Data
• Implementation approaches
• Hipster Batches
• Reactify
• Bring Your Own Data
• Finding the Data
• What we have learned so far
THIS IS WHAT THE STORY IS ABOUT
![Page 6: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/6.jpg)
SORRY… IT’S OK TO LEAVE NOW
• Nope, we didn’t create a new Hadoop
• No hardcore Data Science
• There are some implementation details
• REA embraced the Cloud. AWS everywhere
• Under construction
![Page 7: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/7.jpg)
IN THE BEGINNING…
![Page 8: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/8.jpg)
![Page 9: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/9.jpg)
![Page 10: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/10.jpg)
![Page 11: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/11.jpg)
![Page 12: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/12.jpg)
![Page 13: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/13.jpg)
![Page 14: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/14.jpg)
![Page 15: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/15.jpg)
![Page 16: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/16.jpg)
![Page 17: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/17.jpg)
![Page 18: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/18.jpg)
![Page 19: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/19.jpg)
![Page 20: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/20.jpg)
![Page 21: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/21.jpg)
![Page 22: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/22.jpg)
![Page 23: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/23.jpg)
![Page 24: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/24.jpg)
![Page 25: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/25.jpg)
![Page 26: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/26.jpg)
![Page 27: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/27.jpg)
![Page 28: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/28.jpg)
![Page 29: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/29.jpg)
![Page 30: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/30.jpg)
ORGANISING OUR DATA
Increasingly, content is being distributed through searchand social platforms... There’s less visiting of publishers as destinations.
Jeff Weiner, CEO, Linkedin
![Page 31: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/31.jpg)
Data sources
Data warehouse
PROBLEM…
![Page 32: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/32.jpg)
STRATEGY…
![Page 33: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/33.jpg)
STRATEGY…
![Page 34: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/34.jpg)
STRATEGY…
![Page 35: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/35.jpg)
Data Warehouse
StagingSSIS Dim Fact
PROBLEM…
![Page 36: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/36.jpg)
Data Warehouse
StagingSSIS Dim Fact
PROBLEM…
Star schema leaky details
![Page 37: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/37.jpg)
No Data Warehouse
StagingSSIS Dim Fact
STRATEGY…
![Page 38: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/38.jpg)
STRATEGY…
Data Warehouse Facade
StagingSSIS Dim Fact
![Page 39: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/39.jpg)
???
WHAT’S IN THE BOX?
![Page 40: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/40.jpg)
Good things come in small packages services
THE HIPSTER BATCH
???
Hipster Batch
![Page 41: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/41.jpg)
Hipster Batch
THE HIPSTER BATCH
• Small and short lived
• Decoupled via flat files via S3
• Single purpose
• Idempotent
• Polyglot
• Minimal runtime dependencies
• Discoverable
![Page 42: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/42.jpg)
SNS, SQS
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 43: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/43.jpg)
SNS, SQS
ASG, ECS, Lambda
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 44: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/44.jpg)
SNS, SQS
ASG, ECS, Lambda
KMS
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 45: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/45.jpg)
Logs
SNS, SQS
ASG, ECS, Lambda
KMS
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 46: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/46.jpg)
Logs
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 47: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/47.jpg)
Logs
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
S3 buckets
Data
A ‘TYPICAL’ IMPLEMENTATIONHipster Batch
![Page 48: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/48.jpg)
Hipster Batch
HIPSTER BATCH DOES SCIENCE
• Behavioural models for targeted marketing
• Recommendation engine
• External channels
![Page 49: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/49.jpg)
Hipster BatchSCIENCE!
![Page 50: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/50.jpg)
x 20
Hipster Batch
Stats models
SCIENCE!
![Page 51: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/51.jpg)
x 20
API
Hipster Batch
Stats models
SCIENCE!
![Page 52: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/52.jpg)
API
x 20
API
Hipster Batch
Stats models
SCIENCE!
![Page 53: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/53.jpg)
API
x 20
API
Hipster Batch
Stats models
SCIENCE!
![Page 54: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/54.jpg)
API
x 20
API
Hipster Batch
Stats models
GoogleNowAPI
SCIENCE!
![Page 55: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/55.jpg)
From legacy to reactive
REACTIFY
Reactify
???
![Page 56: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/56.jpg)
Reactify
http://www.reactivemanifesto.org
REACTIFY
• Manage Data flow with messages
• Protect consumers and care about isolation
• Resilience is important and Data replication is just fine
• Demand is elastic - and your components should be too
![Page 57: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/57.jpg)
Reactify
Listings
Data coupling
No resilience or elasticity
Coupling
PROBLEM…
![Page 58: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/58.jpg)
Reactify
Listings
SOLUTION…
![Page 59: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/59.jpg)
Reactify
Listings Reactify
SOLUTION…
![Page 60: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/60.jpg)
Reactify
Listings Reactify
SOLUTION…
![Page 61: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/61.jpg)
Reactify
Listings ReactifyHipster Batch
SOLUTION…
![Page 62: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/62.jpg)
Reactify
Listings ReactifyHipster Batch
Shielded consumers
IsolationDecoupled
SOLUTION…
![Page 63: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/63.jpg)
Reactify
Listings
IMPLEMENTATION…
![Page 64: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/64.jpg)
Reactify
ListingsRESTAPI
IMPLEMENTATION…
![Page 65: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/65.jpg)
Reactify
ListingsRESTAPI
IMPLEMENTATION…
![Page 66: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/66.jpg)
Reactify
ListingsRESTAPI Dynamo
Event Maker
Event Differ
IMPLEMENTATION…
![Page 67: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/67.jpg)
Reactify
ListingsRESTAPI Dynamo
Event Maker
Event Differ
Kinesis
2
IMPLEMENTATION…
2
![Page 68: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/68.jpg)
• Exposes current state only
• Stream of change notifications
• Hypertext Application Language - HAL
• Clear entity types
• Linking over embedding
• Cacheable and discoverable
REST API
REACTIFY REST API
![Page 69: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/69.jpg)
REST API
https://feeds.listings.realestate.com.au/combined-listings/120449689
![Page 70: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/70.jpg)
REST API
https://feeds.listings.realestate.com.au/combined-listings/120449689
![Page 71: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/71.jpg)
REST API
https://feeds.listings.realestate.com.au/combined-listings/120449689
![Page 72: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/72.jpg)
REST API
https://feeds.listings.realestate.com.au/combined-listings/120449689
![Page 73: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/73.jpg)
REST API
Event Maker
https://feeds.listings.realestate.com.au/combined-listings/-/changes
![Page 74: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/74.jpg)
REST API
Event Maker
https://feeds.listings.realestate.com.au/combined-listings/-/changes
![Page 75: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/75.jpg)
REST API
Event Maker
https://feeds.listings.realestate.com.au/combined-listings/-/changes
![Page 76: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/76.jpg)
REST API
Event Maker
https://feeds.listings.realestate.com.au/combined-listings/-/changes
![Page 77: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/77.jpg)
Reactify
Event Differ
![Page 78: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/78.jpg)
Reactify
Event Differ
![Page 79: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/79.jpg)
Reactify
Event Differ
![Page 80: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/80.jpg)
Reactify
Event Differ
![Page 81: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/81.jpg)
The octopus in the box
— Did you use that data set? — Errr… No, we have another one
BRING YOUR OWN DATA
![Page 82: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/82.jpg)
BRING YOUR OWN DATA - BYOD
• Allow data to flow freely
• Help the business to get what they need when they need it
• Self-service
![Page 83: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/83.jpg)
BYOD
![Page 84: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/84.jpg)
BYOD
CSV
![Page 85: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/85.jpg)
BYOD
CSV
x 5
![Page 86: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/86.jpg)
BYOD
CSV
x 5
Smarts on datatypes
![Page 87: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/87.jpg)
BYOD
CSV
x 5
TableauServer
Smarts on datatypes
![Page 88: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/88.jpg)
BYOD
CSV
x 5
TableauServer
Smarts on datatypes
![Page 89: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/89.jpg)
BYOD
CSV
x 5
TableauServer
Audit, auth, share…
Smarts on datatypes
![Page 90: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/90.jpg)
These were the implementation approaches, now to…
FIND THE DATA
Meaningful, automated, and easy-to-search metadata
![Page 91: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/91.jpg)
WE TRIED
![Page 92: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/92.jpg)
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
Logs
MORE THAN DATAHipster Batch
![Page 93: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/93.jpg)
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
Logs
MORE THAN DATAHipster Batch
![Page 94: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/94.jpg)
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
Logs
Dataz
Ancestry
MORE THAN DATAHipster Batch
![Page 95: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/95.jpg)
SNS, SQS
ASG, ECS, Lambda
KMS
Cloudwatch
Logs
Dataz
Ancestry
Metadata
MORE THAN DATAHipster Batch
![Page 96: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/96.jpg)
Ancestry
![Page 97: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/97.jpg)
Ancestry
![Page 98: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/98.jpg)
Ancestry
![Page 99: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/99.jpg)
Ancestry
![Page 100: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/100.jpg)
Ancestry
![Page 101: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/101.jpg)
RESTAPI
METADATA PIPELINE
Producers
![Page 102: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/102.jpg)
RESTAPI
Ancestry
Ancestry
Ancestry
METADATA PIPELINE
Producers
![Page 103: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/103.jpg)
RESTAPI
Ancestry
Ancestry
Ancestry
METADATA PIPELINE
Producers
![Page 104: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/104.jpg)
RESTAPI
Ancestry
Ancestry
Ancestry
METADATA PIPELINE
Producers
Scrapy
![Page 105: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/105.jpg)
RESTAPI
Ancestry
Ancestry
Ancestry
METADATA PIPELINE
Producers
Scrapy
![Page 106: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/106.jpg)
RESTAPI
Ancestry
Ancestry
Ancestry
METADATA PIPELINE
Producers
Scrapy
![Page 107: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/107.jpg)
WHAT WE HAVE LEARNED SO FAR
• Consumers create the last-mile data as needed
• We must work with external, independent delivery channels
• Push quality back to source/producer systems
• Data belongs to the entire organisation, not to a single team
![Page 108: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/108.jpg)
I’ll give you my Data Warehouse when you can pry it from my cold dead hands.
![Page 109: Building Data-Centric Businesses](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587414111a28abcb5b8b4eaf/html5/thumbnails/109.jpg)
THANK YOU
Daniel Aragao Simon Hope@dear_dr_dan @mapbutcher
REALESTATE.COM.AU