real-time data analysis with azure stream analytics
TRANSCRIPT
![Page 1: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/1.jpg)
Real-time data analysis with Azure Stream Analytics
Michael Johnson
![Page 2: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/2.jpg)
What is streaming data?
Batch data Streaming data
![Page 3: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/3.jpg)
Where do we find streaming data
![Page 4: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/4.jpg)
What is Azure Stream Analytics and why would you want to use it
• What• Real-time event processing engine that helps to unlock deep insights from
data.
• Why• Easy to use
• Scalable
• Reliable, Repeatable and quick to recover
• Low cost
![Page 5: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/5.jpg)
Azure stream analytics architectureAzure stream analytics
Inputs Outputs
![Page 6: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/6.jpg)
Input source
• Streaming Data• Event hubs
• Blob storage
• IOT Events hubs
• Reference data• Blob storage
![Page 7: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/7.jpg)
Output sinks
Azure data lake
Power BI
Azure SQL database Azure document DB Azure Blob storage
Azure Service bus
Azure Event hubAzure table storage
![Page 8: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/8.jpg)
Query language overview
• DML Statements• SELECT• FROM• WHERE• GROUP BY• HAVING• CASE• JOIN• UNION
• Date and time• DATEPART
• DAY
• MONTH
• YEAR
• DATEDIFF
• DATEADD
• Aggregation functions• SUM
• COUNT
• AVG
• MIN
• MAX
• Scaling functions• WITH
• PARTITION BY
• Windowing functions• WITH
• PARTITION BY
• String functions• LEN
• CONCAT
• CHARINDEX
![Page 9: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/9.jpg)
What does a query look like?
![Page 10: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/10.jpg)
Tumbling windowsTUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] )
![Page 11: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/11.jpg)
Hopping windowHOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] )
![Page 12: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/12.jpg)
Sliding windowSLIDINGWINDOW ( timeunit , windowsize )
![Page 13: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/13.jpg)
Javascript UDF intergration
• Supports stateless, compute only functions
• No external connectivity
• Why• REGEX
• Math libraries
![Page 14: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/14.jpg)
Machine learning integration
![Page 15: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/15.jpg)
Geospatial Queries (In public preview)• Geospatial Functions
• types• CreatPoint
• CreatePolygon
• CreateLineString
• Functions• ST_Distance
• ST_WITHIN
• ST_OVERLAP
• ST_INTERSECTS
![Page 16: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/16.jpg)
Deployment options
Azure portal .Net SDK PowerShell
![Page 17: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/17.jpg)
Pricing
• Streaming unit• 1 streaming unit = 1MB/Second
• 1 x 60 x 60 x 24 = 84.375 GB/day
• Data volume + streaming unit cost• Data volume = $ 0.001 or R 0.014 per GB
• Streaming unit = $ 0.12 or R1.656 per hour
• If run at max through put for a month• Total data ingested = 2.47 TB
• Data = R 35.44
• Streaming Unit = R1 192.32
• Total cost = R 1 227.76
![Page 18: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/18.jpg)
DEMO
![Page 19: Real-time data analysis with Azure Stream Analytics](https://reader031.vdocuments.mx/reader031/viewer/2022022809/621b4ce0dfaed139fe4216ba/html5/thumbnails/19.jpg)
Questions