Cloud Mashups

Michael Heydt – Senior Technology Manager, SCS ATGMichael.heydt@sungard.com, mike@heydt.org42spikes.com, @mikeheydt

Building cloud-cloud and cloud-premises mashups

Defining a new application paradigm

Via lessons learned with cloud systems

AgendaHistory

Future

Idea

Demo

How-To

Closing

HISTORY

Mashups up to now

Legacy Mashups

ConsumerPlatformEnterprise

Consumer Mashups

Mashup software is public via a web site,All services are “external”, Likely not to use custom services,And consumed through service portal or Iframe / portlets

Yahoo Pipes

Platform Mashups

Provide a single provider face to multiple cloud services

Facebook + EC2 storage / Salesforce

Enterprise Mashups

Mashing of services within an enterprise,Likely with some web data and custom services,A single identity system,Exposure of data through corporate portals

JackBe Presto

Guess What?

These are so 2009They are not “cloud mashups”, and…

I’m not talking about any of these!

FUTURE

C2C and C2P Mashups,A Brave New World

New Architectures

Cloud-2-CloudCloud-2-Premises

Mash across cloud providers and your private systemsEnabled with new technologies

A New Paradigm

Quickly building applications byIntegrating services from multiple cloudsAnd data from on-premises systemsInto a new serviceHosted in the cloud and Used in your applicationAnd scalable beyond anything before

Requires new skills and technologiesThis is what I’ll be talking about!

ETROCLOUD-2-CLOUD MASHUPS ARE SERVICES AND DATA ORCHESTRATED ACROSS MULTIPLE CLOUDSServices and data orchestrated between multiple cloudsOrchestration services in the cloudMultiple identitiesData and transforms in the cloud

The reality: There will be multiple cloud platforms

ETROCLOUD-2-PREMISES MASHUPS EXTEND CLOUD MASHUPS INTO YOUR PRIVATE SERVICES AND DATA

Communications from the cloud to on-premises systemsBe they private cloud systems or non-cloud systems

Your stuff is safe at home but still can be used by the cloud

ETROTOGETHERTHEY REPRESENT THE NEXT EVOLUTION OF SOA, PLATFORM AND APPLICATIONS

Business and consumer collaboration exemplified

IDEA

What can I do with this?

CONSUMER / PROVIDER ENERGY COLLABORATION SYSTEMSCollect price data from grid operators,Expose uniform data feed via cloud standards (oData, …),Providers visualize prices with graphs and charts in real-time,Collect meter data from smart grids and dynamically analyze,Process contracts from on-premises systems,Consumers can view usage, pricing and billing info, and Homes integrate with the smart grid

An ecosystem mashed from multiple subsystems in different clouds and private systems

Concepts

Small independent services wired together on a busMultiple cloud providersConsume public data into cloud dataScheduled execution of tasks in the cloudOn-premises data access from the cloudFederated access controlLarge scale event processingScalable cloud servicesIntegration of cloud and on-premises dataData as a Service

This is the stuff you have to know

Subsystems

Price collectionPrice visualizationMeter event processingConsumer PortalHome and smart grid integrationContracts processing

These are the primary parts of the system

Price Collection

Scrape data from energy regulatorsStore in cloud storageProvide access via oData, REST, SOAP

Get the data from elsewhere and make it DaaS

Price Visualization

Access price data from rich clients of various formsStandard data formats allow any clients to consumeStandard “mashlets” allow the same view across platforms

Consume DaaS with mashups and RIA components

Meter Data Event Processing

Thousands if not millions of meters sending data,Every 15 minutes,Across the smart grid,Massive flows of real-time events,How to know what’s going on?How to handle the volume?How to store it?

Flex the ISB and CEP

Consumer Portal

Scalable web application in the cloud,Supporting an ever growing user base,Mashing on-premises customer data and cloud pricing dataAccess control from on-premises custom STS

Scalable cloud web app with data tunnels and federated ID

Home / Smart Grid CollaborationConsumers use portal to establish rules for their homeHome collaborates with the grid via cloud agentsGrid delivers commands based upon events

Having the grid charge your car when prices are lowestControl your home remotely

Contracts Processing

Daily and monthly analytics for tradingUsually requiring big iron on siteMany hours or days to completeData needs to stay out of the cloud

Save money on the big iron and feel safe about the data

DEMO

Show some stuff already will you!

Architecture

HOW-TO

The Building Blocks

Toolbox

AppFabric Service BusMicrosoft AzureAmazon EC2Access Control ServicePort Bridge / On-premises AFSB services

Microsoft Azure

.NET Platform as a Service (PaaS)Scalable Web and Worker RolesRuns .NET applications in the cloudBut also provides capabilities for PHP, Java, and more

Provides Internet Service Bus and Access Control ServiceThese are unique to Azure and Critical for C2C/P

Amazon EC2

Infrastructure as a Service (IaaS)Essentially virtual machine hosting in the cloudMany pre-built virtual machines to pick from (AMIs), orRoll your ownScalable, Durable, Totally Customizable

You are likely using it but don’t even know it

AppFabric Service Bus

Internet scale messaging busEnables inter/intra cloud, cloud-premisesNamed endpoints for services to rendezvousNegotiates end-to-end network traversalsOptimizes network traversalTightly integrated with the Access Control ServiceSynchronous, fire and forget, and pub/sub multicastAccess via WCF or REST

This is the backbone of C2C/P systems

AFSB: NetEventRelay

AFSB: NetTcpRelay

Access Control Service

Abstracts orchestration of auth[*]Claims based modelControls access to the ISBPerforms claims mapping between providersYour apps and services can integrateCan integrate with your auth servicesYou can integrated custom auth services via WIF

Ensures all services on the ISB are only used by who you allow

Access Control Service

Premises #1 AFSB Service

A WCF service that connects to the AFSBAPI exposed to cloud applicationsRequests securely routed to on-premises serviceService can interoperate with on-premises system directly

Makes premises systems accessible to your cloud app

Premises #2 Port BridgeTunneling of level 3 traffic To/from cloud to non-cloud / non-service systemsPort Agent in cloud, Port Bridge on premisesAgent and Bridge use AFSB for connectivity and tunnelingBecause it uses ISB, it also uses ACSCloud enables apps that don’t talk cloud

Makes premises systems accessible to your cloud app

Deep-dive

If we have time

Closing

Wrap up and where is this going?

Some Comments

Plumbing is in place for c2c/p mashupsBuild small servicesGlue them together with AFSBLet ACS handle the securityFocus on dynamic code allocationExpose data through oData services

Better tools are still needed

Future Tools

Declarative and graphical designers for applications Dynamic code storage and allocation to cloud servicesAutomatic deployment of services and wiringEvolutionary upgrade of servicesSemantics data repositoriesAdaptive systems that auto-wire based on usage patternsIntentional systems that auto build based upon needs

This is going to be great!

REFERENCES

WWW.MICROSOFT.COM/WINDOWSAZUREAWS.AMAZON.COMODATA.ORG42SPIKES.COM (my blog)

PRINCIPLES

Architecture for C2C/P mashups

The Principles

ReuseFunctional ServicesData As A ServiceIdentitySecurityInterconnection

DiscoverabilityWiringMashupMashletTransformationSemantics

Reuse

Fundamentally all mashups are reused Either in user interfaces,Other services, orIn other mashups

Nothing lives alone

Functional Services

Provide an operation,An interaction with another application,A retrieval of data from another system, or A change of state

SOA Classic

Data as a Service (DaaS)

Mashups need dataUI’s need dataServices can expose data

Data itself is a service

No need to code!

Identity

Multiple entities each requiring different credentialsNeed Identify FederationProviding Identity Management and mappingUsed to implement security

You or your system will have many

Security

Use of IdentityTo authenticate a user or serviceAgainst another serviceTo control access to services, andWhat can be done with the service

Information wants to be free, but you may not want it to

Interconnection

Services need to talk to each otherProtocols need to be bridgedTunnels need to be createdBuilding the nervous system of the mashup

This is the Internet Service Bus

Wiring

The definition of the flow of the data through the mashupFrom one service to anotherUtilizing an interconnection

These are the rules of how the data flows, not how it is moved

Transformation

Services often disagree on the structure of dataThis is a service that changes the format of dataUsing specific rules

The physical transformation of data according to semantics

Semantics

A description of the meaning of dataNot the content of dataOften used to understand data in a Mashlet, orChange data for another service

The rules of the vocabularies of data between the services

PATTERNS

Things you will do all the time

The Patterns

Classic MashupInternet Service BusDynamic ServiceDeploy ServiceIdentity FederationPremises Tunnel

WiringMashupMashletData as a ServiceHubPlug Into Hub

Classic Mashup

Ingest data from multiple feedsMerge dataTransform dataFilter dataExpose results

Assumes all web services

Internet Service Bus

Backbone for cloud-cloud and cloud-premises composition,Providing discovery through standard naming techniques, andInteroperation through dynamic routing / binding of services,Provides protocol bridging across heterogeneous clouds, andAbility for two services to attach without knowing their actual locations or protocols

The nervous system

Dynamic Service

A service that is provisionedOn a schedule, orOn demand, andIs discoverable and connected automatically to mashups,To provide scalability and availability, whichMay be autonomous, orAccessed through it’s own web service, orVia an endpoint on the ISB

Your mercenaries

Deploy Service

Dynamic spin up of a service in a cloud provider,Connecting the service to and endpoint on the ISB, and Configuration of the ACS and IF

Run it only when you need it

Identity Federation

A service that provides unified mapping of credentials from multiple services and clouds,To provide seamless access controls across all services

Presenting yourself in the light of another

Premises Tunnel

Establishes protocol level tunnel for native protocols,From a cloud service, To an on-premises service,Via the Internet Service Bus,Forming a secure means on integration cloud and premisesCommonly using the port bridge

Safely expose private data, systems and services to the cloud

Mashup

An ordered sequence of wiresThat pass data into servicesUsing the interconnectsRetrieving results from the servicesTransforms data according to semanticsFlows the results into other servicesAnd exposing the result as a new service

Secretly (or no so) a business process

Mashlet

Reads the results of a mashup via its addressApplying a visualizationMay be exposed itself as a web construct, orA platform specific user controlAnd knows how to display one or more types of data

Somewhere, sometimes, you need to see the results

Data As A Service

Data being directly exposed as a serviceWithout the ceremony (code) of normal services

Information wants to be free (within your permission)

Hub

Service integration through Internet Service Bus, withAutomatic data transformation, Automatic wiring and interconnection,Dynamic deployment of services,Abstraction of cloud and premises systems,Automated orchestration of services,Identify federation, andAccess control

No one or thing lives alone anymore

Plug Into Hub

Configuration of a deployed service, To communicated with a Hub,Attaching the service to the hub’s orchestration, Providing data transforms and Access control and identify mapping to the hub

I want to play with others like me

Transform!

Change data from one form to another Using a specific transform, orSemantic information

The Optimus Prime