iot and microservice

Download IoT and Microservice

If you can't read please download the document

Post on 13-Apr-2017

754 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

Slide 1

Cloud Native Application, Microservices and Realtime Streaming DemoPivotal Cloud FoundryPivotal Big Data SuiteKapil Shukla

#Pivotal ConfidentialInternal Use Only

#Pivotal ConfidentialInternal Use Only

#TITLEMonth Year

AgendaProblem StatementArchitectureProblem SolutionDemo

#Pivotal ConfidentialInternal Use Only

#TITLEMonth Year

Problem Statement

Data ModelTaxi data streamed for New York regionData contains details like taxi number, pickup time, dropoff time, pickup and dropoff lat/long, fare, taxesArea to be divided as squares. Each square is 1kmx1km

Find out EVERY 10 SECONDSInconsistent dataTop 10 areas where taxies are plying the most (Report starting and ending area and number of taxies that traveled in these areas)Total data processed Free taxies available in different areas (only 50 taxies)

Analytical QueriesWhich taxi driver is not reporting data correctlyTop 10 taxi driver earning the most

#Pivotal ConfidentialInternal Use Only

#Pivotal ConfidentialInternal Use OnlySpringXD(Fast Ingestion)Real Time Analytics Demo - FlowSpark Streaming(In Memory analytics)Gemfire(In memory data store)FilterBusiness logic(filter, transformation)10s moving windowMonolithic AppData is streamed to network port where springXD is listeningData stream is forwarded to SPARK streaming which collects data for every 10seconds and process the dataAfter processing, various aggregated metrics are uploaded to GemfireA php webapp then shows the data and refreshes the data every 10 seconds.Pivotal HD (long term storage)Terminal OutputNPPCF

#Pivotal ConfidentialInternal Use OnlyReal Time Analytics Demo Tools used

SpringXD Data IngestionSpark Streaming In-memory stream computingGemfire In- memory data storeHAWQ Analytic SQL queries on HadoopSpring Boot MicroservicesGcharts PHP based web applicationPivotal Cloud Foundry Application Runtime Platform

#Pivotal ConfidentialInternal Use OnlyGithub code could be accessed here

https://github.com/kgshukla/Realtime-Streaming-PCF-Microservices-BDS/blob/master/problem.adoc

#Pivotal ConfidentialInternal Use OnlyChallenges with Monolithic App?1. INEFFICIENT SCALING2. ERROR HANDLING4. CONFIGURATION MANAGEMENT5. APPLICATION UPGRADE6. AGGREGATION3. TIGHTLY COUPLED

#Pivotal ConfidentialInternal Use OnlyThere could be many other challenges that could be shown here. Change accordingly#TITLEMonth Year

Monolithic AppSpringXD(Fast Ingestion)Real Time Analytics Demo - FlowSpark Streaming(In Memory analytics)Gemfire(In memory data store)FilterBusiness logic(filter, transformation)10s moving windowMicroservices Oriented WebappWe will change this monolithic application to microservices.Pivotal HD (long term storage)Terminal OutputNPPCF

#Pivotal ConfidentialInternal Use OnlySERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKERLets build the architectureARCHITECTUREWe need these components for microservices management, Circuit breaker, Service Registry and Configuration server

#Pivotal ConfidentialInternal Use OnlySpring Cloud Services SuiteSpring Cloud Services

Config Server

Service Registry

Circuit Breaker Dashboard

#Pivotal ConfidentialInternal Use OnlyThis is stuff we have tiles forRibbon and Hystrix are really client libraries hence no tile.

This is whats in the box for 1.0 Service Registry: Eureka (not any others, need something that just works for devsConfig Server can use SVN, GitCircuit Breaker Hystrix Dashboard + turbine. #TITLEMonth Year

Spring Cloud Config Server

Config ServerSpring Cloud Config ServerService Binding via Spring Cloud ConnectorGit URL for Config Repo provided via Service Dashboard (post-provisioning)Use Case: Move app from one environment to other without change

#Pivotal ConfidentialInternal Use OnlyASK#TITLEMonth Year

Spring Cloud Service Registry

Service RegistryService Registration and Discovery via Netflix OSS EurekaService Binding via Spring Cloud ConnectorSingle-tenant, scoped to CF spaceRegistration via CF RouteUse Case: Register/Discover a microservice to/at a central place

#Pivotal ConfidentialInternal Use OnlyAsk

Consul, Zoookeeper

#TITLEMonth Year

Spring Cloud Circuit Breaker

Circuit BreakerDashboardNetflix OSS Turbine + Hystrix DashboardHandle expected failures gracefullyAggregation via AMQP (RabbitMQ)Binding via Spring Cloud ConnectorSingle-tenant, scoped to CF spaceUse Case: Handle Microservice failure gracefully

#Pivotal ConfidentialInternal Use Only

#TITLEMonth Year

TaxiService (SB)Events Service (SB)Analytics Service (SB)Aggregator Service (SB)/routes/top10routes/freetaxies/list/events/total/events/missed/events/quality/top10routes/freetaxies/totalevents/missedevents/quality/top10incorrectdrivers/top10earningdriversWeb application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKER/analytics/top10incorrectdrivers/analytics/top10earningdriversARCHITECTURETAXISERVICE http://ts.abc.comEVENTSERVICE https://es.abc.comANALYTICSERVICE https://as.abc.com

#Pivotal ConfidentialInternal Use OnlyTaxiService (SB)Events Service (SB)Analytics Service (SB)Aggregator Service (SB)/routes/top10routes/freetaxies/list/events/total/events/missed/events/quality/top10routes/freetaxies/totalevents/missedevents/quality/top10incorrectdrivers/top10earningdriversWeb application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKER/analytics/top10incorrectdrivers/analytics/top10earningdriversARCHITECTURETAXISERVICE http://ts.abc.comEVENTSERVICE https://es.abc.comANALYTICSERVICE https://as.abc.com 2. http://TAXISERVICE/route/top10routes3. http://ts.abc.com/route/top10routes1. http://aggservice.abc.com/top10routesHow a request to a route works out

#Pivotal ConfidentialInternal Use OnlyGEMFIREPHD/HAWQWeb application (PHP)M O N O L I T H I C

M I C R O S E R V I C E S

#Pivotal ConfidentialInternal Use OnlyEvents ServiceAnalytics ServiceAggregator ServiceWeb application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKER1. SCALINGAnalytics ServiceAnalytics ServiceTaxi ServiceTaxi ServiceTaxi ServiceScale microservices independently based on load/usage

#Pivotal ConfidentialInternal Use OnlyYou could now scale each microservice independently.

In the demo, scale taxi and analytic service. Show them how they get registered in service registry upon startup. Show in logs, request coming to different app instances.#TITLEMonth Year

Taxi ServiceEvents ServiceAnalytics ServiceAggregator Service/top10routes/freetaxies/totalevents/missedevents/quality/top10incorrectdrivers/top10earningdriversWeb application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKER2. GRACEFUL ERROR HANDLING

If any microservice fails, then circuit breaker kicks in and runs your default code on failure. Graceful error handling

#Pivotal ConfidentialInternal Use OnlyIf any microservice goes down, circuit breaker takes over and showcase default behavior that you have implemented for microservice failure.

Bring down eventsservice, and show them the default behavior#TITLEMonth Year

3. LOOSELY COUPLEDTaxi Service (SB)Events Service (SB)Analytics Service (SB)Aggregator Service (SB)Web application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKERTAXISERVICE http://ts.abc.comEVENTSERVICE https://es.abc.com2. http://TAXISERVICE/route/top10routes3. http://ts.abc.com/route/top10routes1. http://aggservice.abc.com/top10routesAgg Service does not have to know where taxi service is running. It asks service registry for microservice named TAXISERVICE and service registry returns the URL

#Pivotal ConfidentialInternal Use OnlyShow in code how Aggregator service is referring to taxi or event service. #TITLEMonth Year

Taxi ServiceEvents ServiceAnalytics ServiceAggregator ServiceWeb application (PHP)SERVICE REGISTRYPHD/HAWQCONFIG SERVERCIRCUIT BREAKER4. CONFIGURATION MANAGEMENTGEMFIRETESTPROFILE = TESTExternalize configuration parameters, put them together under different profiles. For example here, profile test lets all microservices to connect to a test gemfire server

#Pivotal ConfidentialInternal Use OnlyShowcase how without changing the application code, you could change the behavior of application via configuration server. In this example we are showcasing that using profiles, we could connect to different data sources.#TITLEMonth Year

Taxi ServiceEvents ServiceAnalytics ServiceAggregator ServiceWeb application (PHP)SERVICE REGISTRYPHD/HAWQCONFIG SERVERCIRCUIT BREAKER4. CONFIGURATION MANAGEMENTGEMFIREPRODPROFILE=PRODWhen profile is PROD, the same application gets pushed into prod environment with PROD profile.

#Pivotal ConfidentialInternal Use OnlyShowcase how without changing the application code, you could change the behavior of application via configuration server. In this example we are showcasing that using profiles, we could connect to different data sources.#TITLEMonth Year

Taxi ServiceEvents ServiceAnalytics ServiceAggregator ServiceWeb application (PHP)SERVICE REGISTRYGEMFIREPHD/HAWQCONFIG SERVERCIRCUIT BREAKER5. VERSION UPGRADE

#Pivotal ConfidentialInternal Use OnlyLets just focus on one microservice upgrade. Im just picking up one way of doing upgrades (may not be right), change accordingly.#TITLEMonth Year

Aggregator ServiceTaxi Service (v1)Web application (PHP)SERVICE REGISTRYGEMFIRECONFIG SERVERCIRCUIT BREAKER5. VERSION UPGRADETAXISERVICE-v1 http://ts.abc.comTaxi Service (v2)TAXISERVICE-v1 http://t

Recommended

View more >