cloud scale analytics with azure · • power bi meets azure sql data warehouse: amplifying your...
TRANSCRIPT
Darwin Schweitzer
Cloud Scale Analytics with Azure
Frank J Sprague
Massachusetts Bay Transportation Authority
Boston Subway 1897
NY Subway 1904
Artificial Intelligence:
Changing the Game for Everyone
Metro
http://mapa-metro.com/en/Taiwan/Taipei/Taipei-MRT-map.htm
Start or continue your Data & AI Journey
Agenda
• Define Cloud Scale Analytics (CSA)
• Common architectural patterns
• Roles involved in building solutions
• Demo of Azure services used to build CSA
• Customer example of CSA
• Resources
© Microsoft Corporation
Today, 80% of
organizations adopt cloud-first
strategies
AI investment
increased by
300% in 2017
Data will grow to
44 ZB in 2020(1 ZB = 1 trillion GB)
Today, 80% of
organizations adopt cloud-first
strategies
AI investment
increased by
300% in 2017
Data will grow to
44 ZB in 2020(1 ZB = 1 trillion GB)
Cloud AIData
© Microsoft Corporation
Cloud
Data AI
Organizations that harness data, cloud, and AI outperform
AIIoT
ABCs of Cloud Scale Analytics
Infrastructure
Data
Apps
ABCs of Cloud Scale Analytics
© Microsoft Corporation
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
Cloud
Ingest Store Prep & train Model & serve
Data orchestration and monitoring
Big Data store
Analytics engines
Data warehouse
BI + Reporting
Real Time Analyticsin Custom Apps
Big Data and data warehouse
Azure Data Factory Azure Blob storage
Azure Data Lake Store
Azure Databricks
Azure HDInsight
Azure Data Lake Analytics
Azure SQL Data Warehouse
Azure Analysis Services
© Microsoft Corporation
Implementing common architectural patterns
© Microsoft Corporation
Advanced analytics on Big Data
Ingest Store Prep & train Model & serve
Cosmos
DB
Business/custom
apps
(structured)
Files
(unstructured)
Media
(unstructured)
Logs
(unstructured)
Azure Blob StorageAzure Data Factory Power BI
SparkR
Azure Databricks
Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure Machine Learning, and
Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.
Apps
Code-free data integration at scale for Azure SQL Data Warehouse with Azure Data Factory
© Microsoft Corporation
Ingest Store Prep & train Model & serve
Modern data warehouse
Azure Blob Storage
Logs
(unstructured)
Azure Data Factory
Azure Databricks
Microsoft Azure also supports other Big Data services like Azure HDInsight and Azure Data Lake
to allow customers to tailor the above architecture to meet their unique needs.
Media
(unstructured)
Files
(unstructured)
PolyBase
Business/custom
apps
(structured)
Azure SQL Data
Warehouse
Azure
Analysis
Services
Power BI
Code-free data integration at scale for Azure SQL Data Warehouse with Azure Data Factory
© Microsoft Corporation
Ingest Store Prep & train Model & serve
Real time analytics
Sensors and
IoT
(unstructured)
Apache Kafka for
HDInsight
Cosmos
DB
Files
(unstructured)
Media
(unstructured)
Logs
(unstructured)
Azure Blob StorageAzure Data Factory
Azure Databricks
Real-time
apps
Business/custom
apps
(structured)Microsoft Azure also supports other Big Data services like Azure IoT Hub, Azure Event Hubs,
Azure Machine Learning, and Azure Data Lake to allow customers to tailor the above
architecture to meet their unique needs.
HSBC PayMe
Build Cloud native apps using MySQL that scales up to 500M transactions a day
Azure MySQL
Cosmos DB
Click
Stream
APP DBs
APP DBs
AKSAzure APIM
Social
timeline
Azure
SQL
Analytical platform
Events
EventHub
EventHub
Azure
Storage
Azure Databricks
Model
development
& testing
Data engineers & scientist
(Python, R, Scala …)
Application platform
Tableau
Insights
generation
Azure MySQL AKS
> To APP
Grap
h
data
Technical Roles
• Developer
• Data Engineer
• Data Scientist
• Data Analyst
© Microsoft Corporation
Demo of Cloud Scale Analytics
Azure Databricks
Azure Data Factory
Azure Machine Learning service
Azure Machine Learning service
Build and Train Models
Automated ML
Validate
Containerize and test on Azure
Container Instance
Train Models
Azure ML Compute
Monitor
Monitor models with Azure ML
Deploy Models
Scale with Azure Kubernetes Service
CI/CD and model retraining
Azure DevOps integration
Azure SQL
Power BI
Seamless integration, everywhere
Business analysts
No code, low code
IT professionals, Data scientists
Low to high code
CDM folders
Azure Data Lake Storage
Visualize and report
Power BI
Power BI
Dataflows
Ingest
Orchestrate & move
Azure Data
Factory
Ingest
Model & serve
Azure SQL Data
Warehouse
Train & predict
Azure Machine
Learning
Advance data prep
Azure
Databricks
Azure MySQL
Cosmos DB
Click
Stream
APP DBs
APP DBs
AKSAzure APIM
Social
timeline
• Current platform supports all risk, regulatory and self-service reporting on all PayMe data.
• We also have a data science platform which does adhoc insights generation in batch mode (on near-realtime data).
• Graph-database modelled to store customer-merchant-transaction relationships catering to the social-timeline users.
Azure
SQL
Analytical platform
Events
EventHub
EventHub
Azure
Storage
Azure Databricks
Model
development
& testing
Data engineers & scientist
(Python, R, Scala …)
Application platform
Tableau
Insights
generation
Azure MySQL AKS
> To APP
Grap
h
data
© Microsoft Corporation
Retail &
consumer goods
Discrete
manufacturing
Government &
education
Professional
services
Banking &
financial services
Healthcare
Big Data + DW solutions in action
Workshop
Cloud Scale Analytics with Azure Workshop
Time : 2019-05-22 15:30 - 16:30
Workshop Azure Service
SQL Server 2019 Big Data Clusters
• Read and write directly to
HDFS using SQL or Spark
• Elastically scale compute
and storage with K8s
• Combine and cache data
from relational and non-
relational data sources
https://www.microsoft.com/en-us/sql-server/sql-server-2019
ResourcesBuild Sessions• Code-free data integration at scale for Azure SQL Data Warehouse with
Azure Data Factory
• Managing your ML lifecycle with Azure Databricks and Azure ML
• Azure AI: Making AI real for every developer and every organization
• Power BI Meets Azure SQL Data Warehouse: Amplifying your data stack with petabyte-scale analytics
• Deploying apps for on-premises and hybrid ML+AI on SQL Server 2019 big data clusters
• Build cloud native apps using MySQL that scale to 500 million transactions a day on Azure
GitHub Repos
• Cloud Scale Analytics Workshop
https://aka.ms/cloud.scale.analytics
• Azure Machine Learning Notebooks
https://aka.ms/aml.notebooks
• SQL 2019 Big Data Clusters
https://aka.ms/sqlworkshops
Not Matter what your Role
• Business Owner
• Architect
• Implementor
• End-user
You must Start. Your must Continue
THANK YOU!
Darwin Schweitzer | WW INTELLIGENT CLOUD – Big Data/AI Advanced Workload LeadWorldwide Commercial Business (WCB) – Intelligent [email protected] | Twitter @DataSnowman | GitHub DataSnowmanPlease check out Data and AI and IoT resources at https://github.com/Azure/data-ai-iot