Naviga&ng  SAP’s  Integra&on  Op&ons  Lessons  Learnt  Integra&ng  SAP  into  a  Complex  Landscape

Sascha  Wenninger@sufw    

About Me

Technical Architect

Focus: SAP Integration

Opinionated

Co-founder of Blue T

SAP Mentor

Wannabe Performance Engineer

Enterprise IT !== Boring

3

4https://secure.flickr.com/photos/good_day/159927879/

There were files.

5

Then came RFC

6

7

So What Do You Choose?

8

https://secure.flickr.com/photos/manannan_alias_fanch/315630347/ 9

10h1p://www.flickr.com/photos/freddiebrown/6296214984/

11h1ps://secure.flickr.com/photos/92583675@N00/2322009245/

12h1ps://secure.flickr.com/photos/pascalbovet/4375204691/

?

TINLOTR

One Integration Approach to Rule them All

One Integration Approach to Rule them All

One ____________ to Rule them All

One ____________ to Rule them All

Use the Right Tool for the Job!“Right  Tool  for  the  Job?”,  by  Bruce  Murray

21h1p://www.flickr.com/photos/rideyourbike/2275993007/

22

Back to First Principles

Push or Pull?That is the Question.

23

24

Data  Flow

A BSource DesVnaVon

25

A B

Actual  IntegraVon?

A  sends  to  B

A BB  fetches  from  A

or

26

A knows B

Low latency easy

A is active party

A has control

A may not know B

Low latency possible

B is active party

B has control

A B A B

27

or

h1p://www.flickr.com/photos/gcwest/2686928531/

Synchronous:

Source is responsible to make itself understood

28h1p://www.flickr.com/photos/48975388@N07/7163918352/

Asynchronous:

Target is responsible to ensure it understands

29h1p://www.dumpaday.com/wp-­‐content/uploads/2013/02/funny-­‐pictures-­‐dumpaday-­‐28.jpg

They’re the only binary choices you have to make!

30

<sidebar>

31

Synchronous == Best Effort, “Kiddie Stuff”

Asynchronous == Reliable, Guaranteed Delivery, “Enterprisey”

32

Synchronous == Best Effort, “Kiddie Stuff”

Asynchronous == Reliable, Guaranteed Delivery, “Enterprisey”

33

Remember the OSI Model?

34

7 Application

6 Presentation

5 Session

4 Transport

3 Network

2 Data Link

1 Physical

More or less leaky abstractions

Inherently Synchronous

Everything in Computing is Ultimately Synchronous!

35“Turtle  Tower”,  by  Andreas  Al

36

(It’s all about error handling!)

</sidebar>

37

For Example:

Getting data into an Enterprise Data Warehouse

(using SAP BW as an example)

38

SAP  BW

39

Inbound Data Layer

Transformation Layer

Reporting Data Layer

Reporting Tool Visualisation Tool

Source System, e.g. SAP ERP

Extractor

Process Chain

Process Chain

Queries

Browser access

SAP  BW

40

Source System A

Transformation Layer

Reporting Data Layer

Reporting Tool Visualisation Tool

Source System B

Source System C

Source System D

Source System E

Inbound Data Layer

41

A

B

CD

BWE

“Blue  Marble  Next  GeneraVon,  Raw  Bathymetry”,  by  NASA  Visible  Earth

42

43

A

B

CD

BW

“Blue  Marble  Next  GeneraVon,  Raw  Bathymetry”,  by  NASA  Visible  Earth

E

44

Best Bets for Push vs. Pull Decisions

n Sources, 1 Target

1 Source, n Targets

“dumb” Source

No/few intermediaries

Many intermediaries

Low latency needed

✔

✔

✔

✔

✔

✔

45

Shared Database

ETL (Extract, Transform, Load)

File Transfer

Message-Based

SOA (Service Oriented Architecture)

REST

h1p://www.flickr.com/photos/marktee/7545627352

46

Shared Database

47

Shared Database

ETL: Extract, Transform, Load

48

File Transfer

49

Message-Based

50

SOA: Service Oriented Architecture

51

REST: Representational State Transfer, aka “Web APIs”

52

When to Use Which Approach?

53Image  credit:  Jenny  Shirey

54

Criteria: Data Volume

1 kB 10 kB 100 kB 1 MB 10 MB 100 MB 1 GB

ETL

File Transfer

Message-Based

SOA

REST

55

Criteria: Frequency

1/week 1/day 1/hour 1/min 1/sec 10/sec

ETL

File Transfer

Message-Based

SOA

REST

56

Criteria: Application Capabilities

None Rudimentary Intermediate Sophisticated

ETL

File Transfer

Message-Based

SOA

REST

Criteria: Synchronicity

57

Fire  &  Forget! Asynchronous@ Synchronous$

ETL ✔ ✔

File Transfer ✔

Message-Based ✔

SOA ✔ ✔

REST ✔ ✔

!      No  technical  acknowledgment  received  by  Sender@  Technical  (delivery)  acknowledgment  received  by  Sender$    Request/Response  communicaVon

Criteria: Data or Functionality?

58

Data-­‐Centric Func&onality-­‐Centric

ETL ✔

File Transfer ✔

Message-Based ✔

SOA ✔

REST ✔

59

Our Tools

h1p://www.flickr.com/photos/tunnelarmr/2435107204/

60

61

Best for Caution with

• SAP’s Middleware product. –Origins in EAI, Message-based integration–XML-centric, many protocol adapters–Design-time SOA features, file transfer capabilities

62

PI: Process Integration

•Push-oriented integration•XML data formats•Stateless processing

•e.g. Message Router, Channel or Translator patterns

•High-volume synchronous scenarios•Large (~200MB+) messages•Scenarios requiring keeping state•e.g. collect, distributed transactions, etc.

•B2B integration

Best for Caution with

• SAP’s “middleware stack”: PI 7.3 + BPM + BRM – Java-only installation with much improved (10x!) runtime performance–Next-gen NetWeaver BPM runtime for stateful processing

63

PO: Process Orchestration

•Push-oriented integration•XML data formats•Stateless processing

•e.g. Message Router, Channel or Translator patterns

•High-volume synchronous scenarios•Large (~200MB+) messages•Short timeframe projects (skills availability)•B2B integration (although investment is increasing)

Best for Caution with

• SAP’s ETL Tooling: Extract > Transform > Cleanse > Load–Bulk data transfers at the database level–Useful for replicating content of data warehouses

64

BODS: BusinessObjects Data Services

•Data-centric integration•Pull-oriented integration•Large volume of data•Low frequency, high latency•Data quality enforcement•Bulk loads into HANA

•Application-to-Application integration•Lower latency requirements•Infrequently-changing data•Granular information

ALE - specifically IDocs

65

• SAP-proprietary message-based integration• Well integrated into SAP applications, some heavily rely on it. –Slowly being supplanted by web services, but not yet.

• Mature and feature-rich

Best for Caution with•Integrating standard functionality of different SAP applications with each other

•Forward Error Handling•Processing messages in bulk, or in sequence

•Integrating applications not built by SAP.•Can enhance “Fortress SAP” perceptions•Enhancing/extending standard IDocs

66

ABAP Batch Jobs

• "The 80's called, they want their integration back"• Nevertheless still useful in some cases–But use XML. Tab-delimited files should have gone extinct in the 80s.

67

Best for Caution with•Fire & Forget asynchronous transfer•Long-running "message" creation•Often lowest-common denominator•Outbound from SAP ;-)

•SAP on the inbound side; error handling is generally bespoke.

•High-volume or high-frequency interactions

Best for Caution with

• SAP’s Web Service Layer– translates XML to ABAP, and back

68

ABAP Proxies

•Proper outside-in web service design•Logging•Idempotency•Forward Error Handling•WS-* support

•Relying only on ESR modeling •Some industry-standard XML Schemas

NW ABAP (e.g. ECC, CRM)

BAPI

ABAP Classes

ABAP Proxy Layer

SOAP Client

Best for Caution with

• SAP’s REST-inspired OData API Layer for:–Business Suite–HANA–NW BPM, Business Workflow–etc.

69

NetWeaver Gateway

•Functionality-centric integration•Pull-oriented integration•Client/server architectures•Multi-request interactions

•Exposing public APIs directly.•Formats other than OData or JSON

NW ABAP (e.g. ECC, CRM)

BAPI

ABAP Classes

Workflow

ECC/CRM etc.

NW BPM

NW Gateway

JSON/OData Client

Best for Caution with

• SAP’s generic HTTP Server Layer–direct access to HTTP requests–hand-craft responses to include any content.

70

ABAP HTTP Handlers

•Complex, functionally-rich REST APIs•Intimate control over content:

•from Plain-text to Binary•Interesting 'hacks'

•Large numbers of 'cookie-cutter' interfaces•requires hand-crafting

•Learning curve

NW ABAP (e.g. ECC, CRM)

BAPI

ABAP Classes

ABAP HTTP

Handler

HTTP Client

71

Assessing Fit

h1p://www.flickr.com/photos/90369723@N00/5580038871/

72

PI PO BO  DS ALE/IDocs

ETL ✔

File Transfer ✔ ✔ ✔

Message-Based ✔ ✔ ✔

SOA ✔ ✔

REST

73

Batch  Jobs ABAP  Proxies Gateway HTTP  Handler

ETL ✔

File Transfer ✔

Message-Based ✔

SOA ✔ ✔ ✔

REST ✔ ✔

74

Use the right tool for the job.

75h1p://cheezburger.com/3535420672

Think XML Schema

76

Modeling XML using ESR Data Type objects is too restrictive:

Time-consuming

Cannot extend elements

No abstract types

No xs:choice, xs:all, xs:any, etc.

Poor support for industry-standard XML SchemasLots of work-arounds...

77

78

0%! 20%! 40%! 60%! 80%! 100%!

ABAP Proxy!

ESR!Supported!

Partially Supported!Not Supported!

♲Plan for reuse at the right level!

79

80

♲

♻✔

♻✔ ♻?

Don’t aim for reuse.

Let it emerge.

81h1p://www.flickr.com/photos/yukonblizzard/2926897104/

Reusableor

82

Future Legacy?Cumbersome?One Size Fits None?Regression Testing Nightmare?

YAGNI+Refactor

83h1p://www.flickr.com/photos/mcgraths/3248483447/

You Ain't Gonna Need It+Refactor

84h1p://www.flickr.com/photos/mcgraths/3248483447/

Don’t try to predict the future. You will be wrong.

If in doubt, leave it out.

“As simple as possible”

Plan to refactor to improve!

85

Build tests.

86h1p://www.flickr.com/photos/kwl/4809326028/

87h1p://agilemanifesto.org

88

89

Business value over technical strategy

Evolutionary refinement over pursuit of initial perfection

Main Message

90http://tomfishburne.com/2012/07/one-size-fits-none.html

Key Points to Take Home

You will need more than one tool.

Get the interaction right.

Aim for simplicity. This helps implementation and support.

91

Sascha Wenninger

@sufwsascha@BlueT.com.au+61 403 933 472

Further Reading

Enterprise Integration Patterns (Hohpe & Woolf). The Book and the Site.

Positioning Process Integration and Data Services, by SAP

The Practical Science of Batch Size, by Don Reinertsen. Video and slides.

Comments on the SOA Manifesto, by co-author Stefan Tilkov

94