iot and microservice

1Pivotal Confidential–Internal Use Only 1Pivotal Confidential–Internal Use Only

Cloud Native Application, Microservices and Realtime Streaming Demo

Pivotal Cloud FoundryPivotal Big Data Suite

Kapil Shukla

2Pivotal Confidential–Internal Use Only

Agenda Problem Statement Architecture Problem Solution Demo


Problem StatementData Model1. Taxi data streamed for New York region2. Data contains details like taxi number,

pickup time, dropoff time, pickup and dropoff lat/long, fare, taxes

1. Area to be divided as squares. Each square is 1kmx1km

Find out EVERY 10 SECONDSa. Inconsistent datab. Top 10 areas where taxies are plying the

most (Report starting and ending area and number of taxies that traveled in these areas)

c. Total data processed d. Free taxies available in different areas (only

50 taxies)

Analytical Queriese. Which taxi driver is not reporting data

correctlyf. Top 10 taxi driver earning the most


SpringXD(Fast Ingestion)

Real Time Analytics Demo - Flow

Spark Streaming(In Memory analytics)

Gemfire(In memory data store)

Filter

Business logic(filter, transformation)

10s moving window

Monolithic App

Data is streamed to network port where springXD is listening

Data stream is forwarded to SPARK streaming which collects data for every 10seconds and process the data

After processing, various aggregated metrics are uploaded to Gemfire

A php webapp then shows the data and refreshes the data every 10 seconds.

Pivotal HD (long term storage)

Terminal Output

NP

PCF


Real Time Analytics Demo – Tools used

SpringXD Data Ingestion

Spark Streaming In-memory stream computing

Gemfire In- memory data store

HAWQ Analytic SQL queries on Hadoop

Spring Boot Microservices

Gcharts PHP based web application

Pivotal Cloud Foundry Application Runtime Platform


Github code could be accessed here

https://github.com/kgshukla/Realtime-Streaming-PCF-Microservices-BDS/blob/master/

problem.adoc

https://github.com/kgshukla/Realtime-Streaming-PCF-Microservices-BDS/blob/master/problem.adoc




Challenges with Monolithic App?

1. INEFFICIENT SCALING

2. ERROR HANDLING

4. CONFIGURATION MANAGEMENT

5. APPLICATION UPGRADE

6. AGGREGATION

3. TIGHTLY COUPLED


Monolithic App

SpringXD(Fast Ingestion)

Real Time Analytics Demo - Flow

Spark Streaming(In Memory analytics)

Gemfire(In memory data store)

Filter

Business logic(filter, transformation)

10s moving window

Microservices Oriented Webapp

We will change this monolithic application to microservices.

Pivotal HD (long term storage)

Terminal Output

NP

PCF


SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

Let’s build the architectureARCHITECTURE

We need these components for microservices management,

Circuit breaker, Service Registry and Configuration server


Spring Cloud Services Suite

Spring Cloud Services

Config Server Service Registry Circuit Breaker Dashboard


Spring Cloud Config Server

Config Server

• Spring Cloud Config Server• Service Binding via Spring Cloud

Connector• Git URL for Config Repo provided via

Service Dashboard (post-provisioning)

Use Case: Move app from one environment to other without change


Spring Cloud Service Registry

Service Registry

• Service Registration and Discovery via Netflix OSS Eureka

• Service Binding via Spring Cloud Connector

• Single-tenant, scoped to CF space

• Registration via CF Route

Use Case: Register/Discover a microservice to/at a central place


Spring Cloud Circuit Breaker

Circuit BreakerDashboard

• Netflix OSS Turbine + Hystrix Dashboard

• Handle expected failures gracefully• Aggregation via AMQP (RabbitMQ)• Binding via Spring Cloud Connector• Single-tenant, scoped to CF space

Use Case: Handle Microservice failure gracefully


TaxiService (SB)

Events Service (SB)

Analytics Service (SB)

Aggregator Service (SB)

/routes/top10routes/freetaxies/list

/events/total/events/missed/events/quality

/top10routes/freetaxies/totalevents/missedevents/quality/top10incorrectdrivers/top10earningdrivers

Web application (PHP)

SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

/analytics/top10incorrectdrivers/analytics/top10earningdrivers

ARCHITECTURE

TAXISERVICE – http://ts.abc.comEVENTSERVICE – https://es.abc.comANALYTICSERVICE – https://as.abc.com

http://rs.abc.com/

https://ft.abc.com/

https://as.abc.com/


TaxiService (SB)

Events Service (SB)



/routes/top10routes/freetaxies/list

/events/total/events/missed/events/quality



SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

/analytics/top10incorrectdrivers/analytics/top10earningdrivers

ARCHITECTURE

TAXISERVICE – http://ts.abc.comEVENTSERVICE – https://es.abc.comANALYTICSERVICE – https://as.abc.com

2. http://TAXISERVICE/route/top10routes

3. http://ts.abc.com/route/top10routes

1. http://aggservice.abc.com/top10routes

How a request to a route works out

http://rs.abc.com/

https://ft.abc.com/

https://as.abc.com/


GEMFIRE PHD/HAWQ

Web application (PHP) M O N O L I T H I C

M I C R O S E R V I C E S


Events Service

Analytics Service

Aggregator Service


SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

1. SCALING

Analytics ServiceAnalytics Service

Taxi Service

Taxi Service

Taxi Service

Scale microservices independently based on load/usage


Taxi Service Events Service

Analytics Service

Aggregator Service



SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

2. GRACEFUL ERROR HANDLINGIf any microservice fails, then

circuit breaker kicks in and runs your default code on failure.

Graceful error handling


3. LOOSELY COUPLED

Taxi Service (SB)

Events Service (SB)




SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

TAXISERVICE – http://ts.abc.comEVENTSERVICE – https://es.abc.com…

2. http://TAXISERVICE/route/top10routes

3. http://ts.abc.com/route/top10routes

1. http://aggservice.abc.com/top10routes

Agg Service does not have to know where taxi service is

running. It asks service registry for microservice named

“TAXISERVICE” and service registry returns the URL

http://rs.abc.com/

https://ft.abc.com/


Taxi Service

Events Service

Analytics Service

Aggregator Service


SERVICE REGISTRY

PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER


GEMFIRE TESTPROFILE = TEST

Externalize configuration parameters, put them together

under different profiles. For example here, profile “test” let’s all microservices to connect to a

test gemfire server


Taxi Service

Events Service

Analytics Service

Aggregator Service


SERVICE REGISTRY

PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER


GEMFIRE PRODPROFILE=PROD

When profile is PROD, the same application gets

pushed into prod environment with PROD

profile.


Taxi Service

Events Service

Analytics Service

Aggregator Service


SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

5. VERSION UPGRADE


Aggregator Service

Taxi Service (v1)


SERVICE REGISTRY

GEMFIRE

CONFIG SERVER

CIRCUIT BREAKER

5. VERSION UPGRADE

TAXISERVICE-v1 – http://ts.abc.com

Taxi Service (v2)

TAXISERVICE-v1 – http://ts.abc.comTAXISERVICE-v2 - http://tsv2.abc.com

http://TAXISERVICE-v1/route/top10routes

TaxiService version 2 could be deployed in prod. Both v1 and v2 could coexist. It is up to aggregator service which

one it uses. Right now aggregator service refers to

v1

http://rs.abc.com/

http://rs.abc.com/

http://rsv2.abc.com/


Aggregator Service (v2)

Taxi Service (v1)


SERVICE REGISTRY

GEMFIRE

CONFIG SERVER

CIRCUIT BREAKER

5. VERSION UPGRADE

TAXISERVICE-v1 – http://ts.abc.com

Taxi Service (v2)

TAXISERVICE-v1 – http://ts.abc.comTAXISERVICE-v2 - http://tsv2.abc.com

http://TAXISERVICE-v2/route/top10routes

When aggregator service is upgraded, v2 version starts referring to taxi service v2. You could now delete taxi

service v1.

http://rs.abc.com/

http://rs.abc.com/

http://rsv2.abc.com/


Taxi Service

Events Service

Analytics Service


SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

6. AGGREGATION

If you don’t have agg service, your web app needs to make three calls to three diff service. Causes battery

drain on mobiles/ipads


Taxi Service

Events Service

Analytics Service

Aggregator Service


SERVICE REGISTRY

GEMFIRE PHD/HAWQ

CONFIG SERVER

CIRCUIT BREAKER

6. AGGREGATIONOne call to agg service which calls all other microservices,

prepares relevant data and sends back to webapp


Boon for DevelopersDeploying an applica-tion


Just push your apps


Saviour for OperatorsScaling an application


PCF alleviates many operational concernsIn-built functionalities

In-built app scaling

In-built self healing

Logging aggregation & audit trail

Strong IT and user governance

App lifecycle mgmt (perf, upgrade)


Container Scheduling

Application Framework

ServicesPlatform Runtime

Routing DatabaseConfigurationService Discovery

Big Data

Object Storage

Mobile

Build CI

User ProvidedLogging & Metrics Messaging

Circuit Breakers

12 Factor Apps

RESTful Services

Microservices

.NETSpring Boot Node.jsRuby on Rails

Operations

Zero Downtime

Deployments

Failover & Recovery Scaling Security

PatchingPlatformUpgrades

Infrastructure

OpenStack AmazonVMware


Pivotal Big Data Suite Is a Complete Tool Set for Data-Driven Enterprises

A NEW PLATFORM FOR A NEW ERA

iot and microservice

Technology