data modeling in the api economy

Data Modeling in theAPI EconomyPRESENTATION TO DATA MODELING ZONE, OCTOBER 2014

TED EPSTEIN, FOUNDER AND CEO

MODELSOLV, INC.

All Eyes on APIsParadigm shift with integration architecture, happening on two fronts:◦ REST as a recommended architectural style for internal services

◦ Emergence of the API Economy

Internal RESTWeb-Style REST APIs are replacing SOAP for internal services.

REST brings some useful design constructs for data-oriented APIs.

REST is another wave of change in integration architecture◦ New trends bring new projects, modernization programs

◦ A chance to improve on whatever we did before.

◦ In this case, a chance to improve on SOA

We need you!◦ Large scale integration needs data modelers!

◦ We need to make the case for that, and get our IT organizations on a better path.

◦ This talk will help get you started.

Emergence of the API EconomyAPIs have entered the mainstream of business and marketing awareness.

Business can profit from APIs◦ As products and services.

◦ As sales channels.

◦ As points of customer engagement.

Massive Integration on Web and MobileEverything integrating with everything else◦ Mashups

◦ Common Services◦ Identity (Google, Facebook, Twitter, OpenID)

◦ Transactions (PayPal, Google Wallet, Amazon Payments)

◦ Location (Google Maps)

◦ Document Sharing (Dropbox, Google Drive, SkyDrive)

◦ Social (LinkedIn, Facebook, Twitter)

Integration Trends in Business and GovernmentCloud/SaaS

Big Data

Mobility

Internet of Things

Open Government

Mapping the Service Landscape

Internal and External ServicesInternal: We're somewhere on the REST adoption curve.◦ REST is the new SOA

◦ No, it's an evolution of SOA◦ No, it's different, for this reason or that reason◦ Just don't call it SOA, because we don't like SOA◦ Fine, but it's still the new SOA◦ ... and that's OK. Really.

External: We're jumping on the API bandwagon.◦ May be headed by marketing◦ May be a separate technology team

Meeting in the Middle

source: http://www.soatothecloud.com/

There is no hard line:◦ We know that some

private APIs will become public.

◦ We don’t know which ones, or when.

◦ We should be converging towards a common architecture for internal & external services.

What's gov got to do with it?Why do service APIs need “governance”?

APIs are data conduits: ◦ Services account for a significant portion of data flow

Major technology initiatives are integration-intensive:◦ Mobile, Big Data, Cloud Computing /SaaS

◦ Regulatory compliance

Integration is the #2 challenge in Mobile and SaaS:◦ #1 is data security, #2 is integration

sources: MuleSoft and SnapLogic surveys

Why is integration such a headache?Our systems are not speaking the same language.

Every time a new team builds a new API, the IT landscape gets more complex.

Billions spent on remedial solutions to aggregate disparate data, but we have a leaky roof.◦ Increasing complexity◦ Poor data quality◦ High integration cost◦ Slow time to market

What are we doing to stop these leaks?This is why data modelers need to get involved.

We need unified canonical data models in our APIs.

We need data quality constraints.

Governance and Collaboration: two complementary paths to convergence.

REST Standards -Exploring the Stack

A Simple Data Modelstructure TaxFiling

id : stringyear : gYearnetTax : decimalcurrency : stringtaxpayer : reference to Person

structure Person taxpayerID : stringlastName : stringfirstName : stringaddresses : reference to Address*

structure Address street1 : stringstreet2 : stringcity : stringstateOrProvince : stringpostalCode : string

REST: An Architectural StyleSome Essential Modeling Constructs:

◦ Resources

◦ Identifiers

◦ Representations

◦ Media Types

◦ Methods

◦ Hyperlinks

Fielding DisclaimerStrict REST vs. REST in Practice◦ URIs: opaque vs. transparent

◦ Out-of-Band Information: media types, link relations vs. documentation

◦ Processing Model: hypertext-driven vs. client-driven

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

Data Model as Resource Model

CollectionResource

Data Type

ObjectResource

HTTPSome Essential HTTP Modeling Elements:◦ URIs

◦ Methods

◦ Headers

◦ Status Codes

REST/HTTP Resource Model

URI Template

Method

Request

Response

Status Code

Media TypesAbout Media Types◦ MediaType(Resource) --> Representation

◦ Appears in the content-type HTTP header

◦ Used for content negotiation

◦ Standard media types registered at:

http://www.iana.org/assignments/media-types/media-types.xhtml

Low-Level Media TypesEssentially Wire Formats◦ Technology-specific

◦ Domain-neutral

Examples:◦ application/xml

◦ application/json

many others◦ text/html

◦ image/jpeg

◦ application/protobuf

◦ …

Generic Structured Media TypesWhat are they?◦ Data-oriented media types a level above basic wire formats

◦ Many of these basic formats are not hypermedia-aware

◦ Intended to express generic structured data, but with some additional abstractions:Hyperlinks, embedded representations, collections, items…

◦ Different media types will emphasize different things

Sidebar: Hyperlinks◦ Hyperlinks make REST genuinely different.

◦ Hyperlinks are your friend.◦ A better way to model relationships◦ Tends to de-emphasize big, deeply nested messages◦ Helps neutralize some of the perspective mismatch problem◦ Hides identifiers in hyperlinks

◦ Isn't that expensive?◦ Cache can go a long way

◦ When to embed vs. hyperlink:◦ For network efficiency (even with caching)◦ For client developer convenience, if the embedded data will most often be required◦ For encapsulation (Think DDD Composites)

Hyperlinked Resource Model

Hyperlink

Domain-Specific MetadataOptions◦ Domain-Specific Media Types (Sun Cloud, Atom)

◦ Schemas (XSD, JSON, etc.)

◦ Profiles

◦ Microformats

◦ Documentation

Diagram with Domain-Specific Data

Sidebar: Schemas and RESTA cultural issue: uncomfortable relationship between REST and formal domain-specific data contracts.

XML Schema was a sore spot for SOA.

Domain-specific data contracts present a dilemma for strict REST:

◦ Out-of-band information is supposed to be concentrated in media types and link relations,

◦ ... but we don't want an explosion of media types.

REST is silent on the question of what to do with domain-specific data.

REST is silent on whether your payload can or should include a schema.

An opinionated perspective:

◦ There's always a schema, whether you document it or not.

◦ There are new options (JSON Schema, technology-independent schemas).

◦ Defining a data contract formally just makes sense, much of the time.

API Description LanguagesOptions◦ Swagger

◦ API Blueprint

◦ RAML

What they have in common◦ Allow modeling of resources, data, methods, links, etc.

◦ Public domain specifications, open-source web-based tooling.

◦ Generate interactive documentation / testing sandbox.

◦ Integration libraries and code generators.

◦ Highly recommended as learning tools, to help you explore how REST APIs are put together.

Beyond REST as a Wrapper

REST as a Silver BulletThe fantasy:◦ Exactly one resource for everything

◦ The golden source is the only source

◦ Everything perfectly hyperlinked

The Reality◦ One size fits none

◦ Need to balance centralization and agility

◦ Different APIs for different functions

API as a Domain Model

API as a workflow modelAffordances:◦ Action-oriented hyperlinks

◦ Transaction-oriented resources

◦ Navigational links to related resources

Workflow and affordances vary by service context.

Variations often require different APIs for different clients.

API as a Client ViewIt’s not necessarily just a domain model dump.

It’s the domain as the client sees it: Respect the Client’s vocabulary

Points of Divergence:◦ Property subsets◦ Flattening◦ Renaming◦ Change of perspective.◦ Reification: example, refund

Canonical Models and VariationsThere’s a common theme (canon)underneath these variations.

See how far you can go with canonical models and realizations:◦ Can we describe the theme and

variations separately?

◦ Can we model the variations as adaptations, augmentations of the theme?

◦ If so, maybe we don’t needAPI-specific data types

Realization: Property Subset

API requests or responses may only need a subset of properties defined in the canonical model.

Realization model may specify a list of included properties.

Realization: Perspective

Message and resource structures project different views from the same logical data model

Canonical model should support bi-directional references.

Realization model should allow embedded or linked representations.

Realization: Metadata

Business Information Model

Account ID

Balance

Margin

Status

Account

Party ID

Party1

Data Aspects

· Deltas

· Data Source

· Data Security

· Explicit Null Values

Message Structure

</account>

</accounts>

</party>

APIs may need to augment essential data with descriptive metadata.

Data aspects are cross-cutting concerns that may be woven together with canonical data as part of the interface realization.

Realization: Contextual Constraints

<=$10MMAsset

Class = Bond

TradeServices may have specific constraints that are not intrinsic to the data definitions.

Realization model may specify constraints on requests or responses. Constraints may take different forms: range, subtype, logical expression, etc.

Conclusion: Get involved!We need your modeling skills.

Done right, new API initiatives can have huge positive impact.

Done poorly, they can actually make things worse.◦ Net Negative Contributing API (NNCAPI)

◦ … or Convergence-Resistant API ;-)

ResourcesREST:◦ Basic REST in action:

http://www.youtube.com/watch?v=7YcW25PHnAA

◦ HTTP and REST:http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340

◦ REST API Design Principles:http://www.infoq.com/articles/rest-introduction

◦ REST Idioms and Patterns:http://www.narwhl.com/

API Blogs◦ http://apievangelist.com

◦ http://programmableweb.com

Resources (continued)Canonical Models:◦ http://www.slideshare.net/TedEpstein/canonical-modeling-for-api-interop

◦ http://www.digitalml.com/resources/Service_Design_Forum

API Description Languages◦ http://swagger.io/

◦ http://apiblueprint.org/

◦ http://raml.org/

◦ http://modelsolv.com/reprezen

QuestionsTHANK YOU!

ted.Epstein@modelsolv.com

data modeling in the api economy

data modelers

data conduits

data security

disparate data

dataoriented apis

data modeling zone

data quality constraints

rest standards

Technology

a short journey to the api economy

tracking the growth of the api economy

scale the api economy with the open api definition leap of...

embracing the api economy

บทความ "api economy...

emerging trends in the api economy and api management

ibm redguide api economy

winning in the api economy

open api economy in financial services · open api economy...

bold predictions for the 2016 api economy

a topic modeling approach to categorizing api customer value...

modeling the economy

economy and network_effect_(modeling _and_analysis)

the app economy - api first?

the api economy

welcome to the api economy: developing your api strategy

winning in the api economy api world 2015

benvenuti nella “api economy”

climate-energy-economy modeling: scenarios for designing a...

the api economy and how to optimize your organization's...