networked systems and services - courses · networked systems and services fall 2018 chapter 6:...

Networked Systems and ServicesFall 2018

Chapter 6: Cross-organisationalcollaboration and interoperability

Jussi Kangasharju

Markku Kojo

Lea Kutvonen

9/19/2018 Lea Kutvonen 1

Chapter 6: Cross-organisationalcollaboration and interoperability

• 4moves• Interoperation and collaboration of business services at OSI 7th

• Autonomy of organisations --> plug and "pray" --> required utilities for interoperability

• Reliability concepts widened to business level

• How middleware hides lower levels and supports software engineering• How to build a business service

• different levels of computing and communication middleware to choose from

• Communication channel becomes 1st class citizen, to be managed even at runtime

• Composition of services into collaborations: glued together with business processes

• What kind of contract breaches are to be expected, fixed by whom and how

• Reliability concepts in cross-organisational collaborations

• Business process modeling advise in brief (design task involved)


Outline version

# 1

Some comments on the material• 1st part focuses on current trend vision of interoperability between independent organisations

• Viewpoints cover platform support, software engineering, runtime control , and application domain

• Take home point: everyone should be open for discussing all aspects, make teams with different expert role members

• Overview, no short reading material (multiple active research areas)

• Context in which the design exercise takes place

• 2dn part focuses on the middleware stack as growing concept

• Each layer provides additional concepts and patterns known by experts of that layer

• Key pattern that keeps repeating: open implementation

• 3rd part on reliability requirements design ideas, some shared patterns for alleviating or recovering from risks and failures

• 4th part gives a bit of help for your first business process modeling task

• Article reading assumption: open implementation paper (continues from end-to-end paper)

• Gregor Kiczales, John Lamping, Christina Videira Lopes, Chris Maeda, Anurag Mendhekar, and Gail Murphy. 1997. Open implementation design guidelines. In Proceedings of the 19th international conference on Software engineering (ICSE '97). ACM, New York, NY, USA, 481-490. DOI=http://dx.doi.org/10.1145/253228.253431. Scientific conference paper.

• Design assignment: Business process design in breach recovery situation

• Slides presenting the case and providing some business process modeling hints included in the end of these slides (not presented, questions answered though)


Outline version #

2

http://www.eecg.toronto.edu/~jacobsen/courses/ece1770/reader/p481-kiczales.pdf

Contents1. Introduction

• System view at OSI 7th layer for networked business• Requirements: generic facilities for aligning business and computing in networked business• Goals of OSI 7th layer for this

2. Layering of the OSI level 7 environment with middleware• Middleware layers: overall approach• Facilities for service-oriented computing• Extending the end-to-end argument to open platforms

→ open middleware → open bindings• Change in software engineering• Summarising: Inter-enterprise collaboration management requirements

3. Reliablity in inter-enterprise collaboration cases• Reliability aspects• Reliability support

4. Intro to BPM modeling for the design task


How to build up from sockets and RPC

Extending with new arguments (patterns) to SE and platformsEnvironment

+reliabiilty

Outline version #

3

Role of more detailed elements in the big picture

Q1: How middleware hides socket programming and supports software engineering

• Featuring middleware types• (RPC MW already done)

• MOM MW

• Transactional MW

• Object-oriented MW

• Component-oriented MW

• Service-oriented MW

• Model-driven MW

Q2: Open platform (mw, binding) argument

• Reflective systems • connecting abstract, self-managing

model and real system management

• Ways of guiding such system: parameters incl. models, alternative implementations

• Open implementation patterns


Solving Q1

One agreed learningpoint: transaction mgmt

Business process modeling

• Basic concepts• How to proceed• Relationship to transactions • Opportunities: verification, validation,

generation, execution, …

Extending end-to-end argument

New basicpattern

Current workenvironment

Outline version

# 3

Q3: We have so many levels to choose from, competing solutions on each level. Which one is the right one to use?


Chapter 6.1:Interoperation and collaboration of business

services at OSI 7th

Jussi Kangasharju

Markku Kojo

Lea Kutvonen


1. Introduction • Different environment: large scale networked systems• Endpoints are business services, provided by

autonomous organisations

• Collaboration arranged by interoperabilityinstead of integration

• Applying to various business areas

• Capable of hearing the clients, discussing within the team of business people, software engineers, modeling experts, enterprise architects, global solution architects, competing product engineers, …

• Different meanings for reliability• Building reliable interoperability instead of

assuming it, business level NFP


End-point

Channel

N-to-M Communication

End-point

End-pointEnd-

point

Endpoint Endpoint

Channel

EndpointEndpoint

Consortia, organisation, end-user agent

Business / computational service

Enhanced middleware stack

Ecosystem infrastructure services

Interoperability utilities


Goals of OSI level 7 approachesfor networked business

• Separate business solutions from computing solutions• Current software engineering practices tend to choose

computing and engineering technique for everything(efficiency) (PSM = platform specific model)

• Some decisions cannot be judged by software engineers• business decisions, regulatory rules etc → prepare for

variation and evolution in design• Situational busienss decisions→ runtime decisions

• Use of open solutions (PIM = platform independent m, CIM = computation independent m)

• Who decides in the collaboration?• single resposible partner → single decision-maker for

business aspect• → orchestrated solutions• Peers independent as business partners →

choreographied , federated solutions→ collaboration control and management must takeplace at CIM level

• Aim for platform-agnostic solutions• abstract away from the

technical middlewares (open platform?)

• Minimize the expectations on platform facilities required (avoiddomino effect of changes)

• Who are the end-points in thenetworked system?• Not hosts, but for

example applications running• Individual services → instance per

client or collaboration group


End-point End-point

Chan-nel

End-pointEnd-point

Application layer solutions in use

• For computing the business services …

• … and the channel (here, we call it binding object)• Use a stack of middlewares

• Use them as “open platform” (modifiable behaviour)

• Also middleware remains within OSI layer 7

Physical

Link

Network

Transport

Application

User


Layers of infrastructure / middleware / utilities

How can computing and communicationbe organised effectively?

global transport of messages, media streams, and massive data sets

global network of computing nodes●sources of data●usable as computing resources

● What concepts are supported in programming environments?

● Which system qualities become transparently supported?

● Which properties can be considered configurable?

● How to implement these effectively?

● How are organisational issues, autonomy and dynamics supported by system models and management facilities?

●How are architecture quality properties, such as privacy preservation, supported?

pervasive services at each node: ●programming and management platform (middleware)●OS, network protocol stack

● collaboratively provided ● management for collaborative application systems● reflects relationship of independent system and its environment

Distribution middleware

How are independent, reusable modules or applications composed to form context-sensitive and adaptive services?

Internet and wireless devices

Collaboration coordination infrastructure

Lea Kutvonen

Runtime / Config composition ofbusiness services , Application level(with business processes)

9/19/2018 11

What is different?Solutions built on• low levels (oversimplification):

• programmers decide on technical choises, business methods, information formats

• software engineering team a single decisionmaking unit

• top level (overcomplication):• software engineers work independently• business and information representation

defined by someone else• Business rules and policies defined elsewhere

and modified as the system is used• middleware support dynamic adaptation to

hetegogeneous, dynamic environment

● collaboratively provided ●management for collaborative application systems● reflects relationship of independent system and its environment

Collaboration coordination infrastructure

Runtime / Config composition ofbusiness services, Application level(with business processes)


Networked business needs in society

• Social networks

• Contents (e.g. Flickr)

• Expert nets

• Scientific societies, standardisation

• Multicasting, crowdsourcing

• Entertainment, nets of newsagens

• Business networks

• Supply chains, value nets, ...

• Virtual organisations

• Creation supported by breeding environments

• Business service collaborations

• Roles, gains, responsibilities

• In addition, runtime supported by ecosystem facilities

Example domains

• Chrisis management

• National heath-care

• Big data networks

• Innovative nets

AU

TOM

ATE

D C

ON

TRO

L OF C

OLLA

BO

RATIO

N

Trends in enterprise / business computing

• Networked business pull• Value networks• Business networks• Service science

• Technology push• Web Services• Internet of Things /

Services• BPM• Service-oriented

computing

Business processes

Business applications

Enterprise system

Business processes &

services

Business processes &

services

Ecosysteminfrastructure

Ecosysteminfrastructure

General purpose

platforms

General purpose

platforms

9/19/2018 14Lea Kutvonen

Moving towards inter-enterprise collaborations (compositions of business services)

Examples from networked businesspoint of view

• eCommerce• Supply chain

→

Multiple partners + (in)tangible values →

• Value net• Virtual organization

breeding environment• Collaboration (case)

Key concepts• Business process

• Business service

• Exchanged documents (msgs)

• Vocabulary

→ i.e.x-ecosystem rules→

Individual adaptation• Case management


Examples of ecosystemssingle driver organisation vs democratic peers

Software ecosystem

• software producers make sharedchoise of platform, e.g. Apple, Microsoft, Android

• easy, focus only on development of software-based application services• Shared set of standard protocols in use

for communication transport• Often shared agreement on business

processes, ontologies for informationexchange → additional middleware to support this knowledge

• Middlewares help developers in theirwork

Supporting evolution amongst peers

• More support for evolution of newbusiness ideas and applications• Companies develop new business

ideas and tackle them together

• Free choise for platforms, encapsulation, open middleware

• Enterprise architecture management kind of support• models of business processes

• key terminology of the business field,

• contracting on the new business area


Features supported Concepts supported

2. Layering the OSI level 7 with middleware


RPC mw

MOM mw

Transacti-onal mw

Object-oriented mw

Componenent mw

Service-oriented mw

Dockers etcWith dynamicConnections,Deployments, Bindings

Additional layerswithin application layer

* See arch

Model-driven mw

WhateverReliablity not achieved here,May bite you on the upper layers

Adding“genericbusinesssupport”

Separationof PIMand platform→middleware-agnisticity

Morelater

Service-oriented computing architecturesuggests middleware for business domains in general

S1

S2

S3

S1

S2

S3

Portfolio of services

Service selection and interoperability testing

Process

model 2

ST1

ST2

ST3

ST4

ST5

Processmodel

3

ST4

ST5

Portfolio of business process / network types

Network type selection and interoperability enforcing / testing

IdentityCapabil-

itiesTrust /

reputationPortfolio of partners

Selection of partners

OntologiesProcessmodel

3

ST4

ST5

SOASOCStatefull services

Key concepts:ServiceService ecosystems+ others, likeeContracting,Breach management,Trust, privacy


Service sciences definition of service is more elaborate

◼ eContract management functions

− Service selection

− Negotiation

− Monitoring, breach detection

− Breach management

− Trust management

− Reputation management

− Identification management

− Privacy management

◼ Meta-information repositories

− Service type repository

− Business process model repository

− Service offer repository

– Reputation information flow / repository

Example view onService ecosystem infrastructure

* Platform issues

- ESB, WS*, cloud, agent systems

- Open binding platform

Private support

Private support

Global ecosystem infrastructure


Takes into consideration:who decides?

Explicit binding: selective & adaptable properties for each channel


interface → server interface, client interface, peer interface;

may differ but be mapped

late bindingno addresses available at compilation or deployment time; recall DNS

open bindingcommunication channel between objects is explicit, configurable object

service (object, component)

interface

client(object,agent, ...)

interface

binding managementinterface

Recall


Service engineering on top of the new middlewareDoes it differ from the traditional software engineering practices?

Changes:• Service focus: horisontal interoperability, dynamic• Platform independent operation support

©IJH

Service focus: verticalPortability, reusability

Summarising:

Inter-enterprise collaboration management reqs

• Requires middleware that is • Service-oriented,• Model-driven and• Component-driven

• Requires business domain in which definitions are engineered and made available for standard (best practice) • Business processes• Vocabulary for service types and

collaboration types• Information schema• Contracting processes• Identities for known partners, known

tangibles

• Examples• Swift (banking)• RosettaNet (Nokia, mobiles)• Health-care (Obamacare systems)• Crisis management, international

• Functions• Middleware utilities above• Alternative, selectable business

processes, implementations of different business services, open bindings and binding architecture elements


3.Reliablity in inter-enterprise collaboration cases• Collaboration

• late, dynamic integration of business services through business processes• Reliability requirements include

• Safety in plug and play / pray patterns since failures become manageable• Detecting unexpected situations and considering business level consequences

• Interoperability• Technical interoperability allow messages exchanged reliably

• Transactionality as needed, understanding of reliability of the platform• Bridging between platforms

• Semantic interoperability support document/message and service identification• Accuracy of communication, transformations (translations)• Heterogeneity support + open bindings

• Pragmatic interoperability enforce shared understanding on business processes and protect from unwanted access to service (use, provision)• Trust, privacy• Only valid collaborations within the ecosystem, acceptable for partners


Reliablity in inter-enterprise collaboration cases• Recognition and control of non-functional properties

• Extension of QoS (such as throughput, jitter) to business level concepts (in contracts)

• Examples: • non-repudiation, privacy, timeliness, …

• business transactions (different from database transactions, always dirty)

• Conformance to regulatory systems, …

• Defined metrics for monitoring → additional concepts for reliability

• New set can be defined separately for each business domain

• Support mechanisms• New concepts → ontology support for vocabulary

→ new middleware facility implementations

• Protocols for negotiation, business transaction coordination

• Intercepting operations + Monitoring + policies to monitor

• Hidden binding architecture partners


Middleware supports reliability in multiple ways

• Reliablity between case / collaboration partners’ business services• Trust management seen as part of pragmatic

interoperability• Subjective acceptance criteria by each partner

(organization not forced by others)• Market position bases on feedback: regular breaches

not accepted

• Reliable binging objects• Properties selectable by peers• Properties may include non-repudiation, encryption,

…• Collaborations may be monitored based on the

models available in repositories → breach detection• Failures can be monitored normally → recovery

processes modeled → self-recovery, self-management, … → area of autonomic computing

• Reliability of X-ecosystem as provider of a certain type of business service• Only interface (service type) matters as a contract on

syntax and behaviour tender• Service offers by multiple members

• Partners can be changed (recovery by choosing a new partner with same service type) - note statefulness!

• coopetition• “Automation” of composition

• hiding monolithic vs modular composed services

• Monitoring and breach detection• Collaborations may be monitored based on the models

available in repositories → breach detection• Failures can be monitored normally → recovery

processes modeled → self-recovery, self-management, … → area of autonomic computing

9/19/2018 Lea Kutvonen Open system paradigm applied several times over

26

• Case management approaches• Case (instance) has only part of its business processes known • Decision-points allow insertion of next chunks• Traditional way: business process defined fully, but includes alternative

paths, decision points as variation points


Chapter 6.2: How middleware hides lower levels and supports software engineering

Jussi Kangasharju

Markku Kojo

Lea Kutvonen


Background: Featuring middleware types recollected

• RPC MW • MOM MW

• Transactional MW + transaction management techniques (because above this failures on preceeding layers hurt)


• Component-based MW + Q1 + facilities for enabling Q2


• Model-driven MW + facities needed for using open platform use (Q2)

• Reflective MW + connecting abstract, self-managing model and real system management, guiding open platform use (Q2)


Background


Featuring middleware types• RPC MW

• MOM MW

• Transactional MW


• Component-oriented MW


• Model-driven MW

• Reflective MWRPC mw

MOM mw

Transacti-onal mw

Object-oriented mw

Componenent mw

Service-oriented mw

Dockers etcWith dynamicConnections,Deployments, Bindings

* See arch

Model-driven mw

WhateverReliablity not achieved here,May bite you on the upper layers

Separationof PIMand platform→middleware-agnisticity

Questions to consider• Observation 1:

• MW support for software engineering• component MW adds horizontal view for interoperability and composability,

• developers focus on business logic, componentization (microservices) and late composition

• Q2: Extending end-to-end argument to open implementation (open platform) argument• Component mw, (e.g. dockers) support deployable application units and local mw

support• Q3: Socket programming hidden in open component mw and open binding concepts• Q4: Supporting reliability

• How do I design a reliable business process?• Reliability of collaborations? ecosystem? business processes? business services?


The answers first ….

• … but we need to take a look on the MW stack first to understand what actually is going on

• Course members have very different backgrounds on this area

• We return to these in the end

9/19/2018 Lea Kutvonen ‹#›

Q3: How middleware hides socket programming


Find service with type SBind (C, S1, bindingarch)

Formulatemessage& splitto writesize parts

FindChannelArchitecture&Msgschema Intercept for

monitoring

marshall

ESB

encrypt

T

T

TTT

Intercept for monitoring

marshall

ESB

encrypt

Distributiontransparencies

Peer servicefailures

Click to add text

Having first made the problem more difficult than it was ...

Q2: Use of reflective MW and model-hierarchy facilities for open binding pattern

• Recall open binding

• Runtime check for adaptation coherence based on model-hierarchy

• Binding architecture specifies• Structural rules• Behavior restrictions and

obligations• Policies to be followed by

partners (e.g. using same security protocol)

9/19/2018 Lea Kutvonen / Service ecosystems 32

Choises

• Programmatic adaptation (not so safe, additions through model space for check-up)

• Policy-based adaptation (model names)

• Automatic adaptation (reflective mechanisms)

Q4: Autonomy of partners andrequirements of reliability• In your designs, consider

• Start from the need of service, remember to stick with a clearly cut abstraction level• A shared external business process between partners is a good place to fix first

• Consider the correctness of your business processes as a verifiable protocol

• Your services have state, so you can utilise theories on extended state machines

• What are the interoperability challenges in your case• Why and when partners may trust each others, or each others' services

• Current technology choices may not be final

• How well trusted is your source of vocabulary in messaging?


Autonomy of partners andrequirements of reliability

• For interactions with external services• Plan for check points for policies (rules to restrict or modify behaviours at borders)

• Design recovery processes

• Keep the policies stored, not part of the process definitions, in case of (regular) changes

• Consider replacing access granted / denied being replaced by prohibited/obligated/permitted - and plan reactions in all three cases (negations are not useful, inadequate logics behind)

• On suitable levels, further down the implementation path, remember to check the "call semantics" failures the assumed platform may have – but this is not to be solved on business process model level



• Following regulations (legal aspects, best business practices, cultural habits)• A software agent capable to several services may be wrapped behind a decision-

making wall that under certain conditions refuses the service to be run

• Note deontic logic in policies (rules): permitted, obligated, prohibited

• In business context, additional considerations include any contracted aspects, nonrepudiation ("cannot deny"), "transactionality" (joint understanding of the outcome), timeliness, trust, privacy

• Application domain usually has some specific quality metrics to consider



• Reliability of collaborations? ecosystem? business processes? business services?• Define reliability metrics and concepts in ontologis / models• Use quality ensuring methods: verification and validation, search of bad information flow

patterns, analyse regulatory conformance

• Use runtime monitoring (contracts, reliability metrics, enterprise and ecosystem policies)


RPC middleware summary

9/19/2018 ©Lea Kutvonen / Service Ecosystems 37

• Various versions, recall call semantics

• Essentially, can always cause partial executions

• We cannot be sure whether a full, exactly once semantics has been reached or not

• Need for more exactness → transactions

Transactional middleware

• What is a transaction?

• ACID Properties

• How transactional middleware works?

• Two-Phase Commit

• Two-Phase Locking

1.

What’s a transaction?

• The execution of a program that performs an action by accessing a shared database, usually on behalf of an on-line user.

• For example• Withdraw money from ATM

• Verify credit card sale

• Place bid at an on-line auction

• Submit purchase order

• Further, it is required (as a technical term) refer to ACID properties of the action

ACID Properties

40

• Atomicity – compute fully or not at all

• Consistency - preserve database integrity

• Isolation – execution results are as if computations were run in agreed sequence

• Durability - results keep, failures do not change

Atomicity

41

• No partial results.

• Commit and abort: irrevocable actions

• Abort rolls back operations• Restore data values from before

• For example, temporary private copy that becomes official only at commit, otherwise discarded

• Business services often break atomicity by “dirty actions” that effect the real world or change things outside of the system scope• Money given out from ATM cannot be retrieved

Compensating Transactions

42

• A transaction that reverses the effect of another transaction (that committed) or could not be rolled back properly

• Not always available• Like for real world effects, business transactions

• Still, well-designed applications try to have this

• Also possible to do “forward recovery”• Seek a new acceptable state of affairs

• Flight overbooked → ticket discount, accommodation for delay, etc for volunteers to wait for next flight

Consistency

43

Transactions maintain DB consistency

When each transaction preserves consistency, then the DB stays consistent Referential integrity (no incomplete operations/tables)

Invariants over key data (debits=credits)

Serial execution preserves consistency

Isolation

44

• effect of a set of transactions the same as if they ran independently:• an interleaved execution of transactions is serializable if its effect is equivalent to

a serial one• interleaved computing that just reads or modifies separate datasets can take place with any

interleaved sequence

• Modifications and reads of modified values for further computing should be excluded from interleaved sequences

• Possible methods include pessimistic and optimistic methods• Locking

• Checking sequences for improper orderings and aborting

Durability

45

• Committeed transaction results are permanent

• Methods usually utilise a log• All updates + commits

• When log data physically written on stable storage, results permanent

Two phase locking for avoiding deadlocks

• In transaction systems, resources may become locked as need for writes occur (effects on serialisation)

• Risk of deadlocks arise: processes wait for resources to be released by each other

• Deadlock resolution requires aborting at least one of the transactions

• Deadlocks can be avoided by using 2PL: • Phase 1: collect your resources in given order

• Phase 2: release all resources

• Do not mix phases → no circular waiting situations occur


Two-Phase Commit

47

• Distributed DB system needs control of all DB units succeeding on commits• Pre-commit ensure willingness and stores data• After that, technical failures recoverable

• Phase 1:• Coordinator asks all to “prepare to commit”• Others store data and ack “prepared”

• Phase 2:• When everyone has “prepared”, coordinator tells everyone to do the actual

commit• Any “abort” responses cause global abort

Two phase commit protocol for distributed systems

Application program

Transaction manager

Resource manager (RM)

commit

prepare tocommit response

acknowledgecommitabort


Two phase commit protocol for distributed systems

Application program

Transaction manager


commit

prepare tocommit response

acknowledgecommitabort


Transaction manager

Application program

prepare tocommit etc...


Transaction processing systemTransactional middleware

Front End Program

Request Controller(routes requests and

supervises their execution)

Database System

Client

Back-End

(Server)

End-User

Transaction Server

requests

Transaction monitors

◼ Transaction processing monitor◼ Supports ACID properties

◼ atomicity – performed completely no not at all◼ consistency – no confused data around◼ isolation – intermediate states not visible, locking ◼ durability – effects of operations permanent

◼ can be dealt with◼ recoverable processes◼ 2PC – two phase commit protocol (commit-prepare, commit)◼ 2PL for locking in strict order to avoid deadlocks at resource reservation

◼ Service aspects◼ management interfaces◼ connections to a variety of databases

Traditional mainframe processing

9/19/2018 Lea Kutvonen / Service Ecosystems 52

network

message manager

request controller

application server

database system

BERNSTEIN198x?

collects transaction input

constructs standard msgsends transaction output

starts and commits transaction

determines type of requestroutes request to proper appl.

executes requested application

Manages shared data

Properties

• Reliability - system should rarely fail

• Availability - system must be up almost all the time

• Response time – second(s)

• Throughput - thousands of transactions/second

• Scalability – #clients → internet size, transaction sizes varies a lot (amount of data, messages, code)

• Load – limited number of transaction types / system

• Security – confidentiality

• Configurability

• Distribution - of users and data

Transactional middleware summary


• Transactions provide exactly once semantics when they are “clean”• Protected area of program between begin and commit

• Transaction management relies on two protocols• 2 phase locking for avoiding deadlocks on resources

• 2 phase commitment for surviving distribution and independent decision making by distributed database nodes

• For real use, also need stable snapshots as recovery points in case the system crashes• Snapshot collected from all database nodes between the transactions

• Can be done because of serialisability

• Database protocols assume atomic, failure free messaging → MOM

Message-Oriented Middleware (MOM)

• Communication using messages

• Messages stored in persistent storage based message queues

55

Client App.

local messagequeues

Server App.

local messagequeues

messagequeues

Network Network Network

Message Servers

Properties of MOM

• asyncronous messages• reliable, fault-tolerant• no loss, duplication,

permutation, cluttering

• persistent subscriptions

• models supported• message queue• request-response• multicast• publish-subscribe

◼ violates access transparency

◼ no support for data heterogenity unless in programming language mapping

◼ no support for transactions

◼ persistent message queues support fault tolerance

56

9/19/2018 Lea Kutvonen / Service Ecosystems 57

Properties of MOM

◼ Basic model: client-server, pipe◼ compare with socket level programming

◼ Topics for variation and development◼ persistent/transient msgs◼ FIFO/priority queues◼ translations of msgs◼ abstractions on msg ordering◼ multithreading, automatic load balancing◼ msg routing (source, cost, changes in topology etc) ◼ secure transfer of msgs (at least between msg servers)

Example: IBM MQSeries

Middleware 58

• One-to-one reliable message passing using queues• Persistent and non-persistent messages

• Message priorities, message notification

• Queue Managers• Responsible for queues

• Transfer messages from input to output queues

• Keep routing tables

• Message Channels• Reliable connections between queue managers

• Messaging API: MQopen Open a queue

MQclose Close a queue

MQput Put message into opened queue

MQget Get message from local queue


MOM not very flexible, provides only the low level transport of messages. Does not suit into situations where multiple independent parties around the distributed system wants to listen to particular notifications.

A more flexible model is needed → publish/subscribe model

Publish/Subscribe communication patternEvent-based middleware

• Publishers (advertise and) publish events (messages)

• Subscribers express interest in events with subscriptions

• Event Service notifies interested subscribers of published events

• Events can have arbitrary content (typed) or name/value pairs

• Consumer event filtering, event batching, event priority, event expiration, logging, internationalization, flow control mechanism

Event Service(event-broker

network)

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher

publish

publish

publish

subscribe

subscribe

subscribe

notify

notify

notify

Filter n

Typed events

constraint

Composite patterns

Topic-Based and Content-Based Pub/Sub

• Event Service matches events against subscriptions• Publishers do not know subscribers + vice-versa

• Dynamic subscriptions and withdrawals

Topic-Based Publish/Subscribe

– Publishers publish events belonging to a topic or subject

– Subscribers subscribe to a topicsubscribe(PrintJobFinishedTopic, …)

(Topic and) Content-Based Publish/Subscribe

– Publishers publish events belonging to topics and

– Subscribers provide a filter based on content of eventssubscribe(type=printjobfinished, printer=‘aspen’, …)


Implementation

• Controller with knowledge on • the subscription rules called filters• registered subscribers for each filter

• Interfaces for• subscribing and adding a filter• withdrawing a subscription

• Operation• Message sent to controller• All appropriate filters found (method critical for performance)

(hash table, for example)• Message forwarded to all subscribers

Object-oriented middleware

• local or remote objects, identified by object references• Remote objects located by remote interfaces, represented by

proxy objects and used in RPC way through Remote method invocation

• Designer view dominated by server interface

object A

proxy object B

OOM OOM

skeleton object B

object B

local remote

objectrequestbroker /object

manager

objectrequestbroker/object

manager

Examples:Java RMICORBADCOM.NET

Object-oriented middleware


• Middleware services• Naming (remote/interoperable object references)

• Trading service for locating based on attributes

• Persistency service

• Transactdion service

• Event and notification service

• Pros and cons• Platform and language independence

• Object constellations static

• remote interface specification by server provider ( only 1 view)

Distribution middleware


• Distribution middleware avoids hard-coding client & server application dependencies on object location, language, OS, protocols, & hardware

Interface

Repository

IDL

Compiler

Implementation

Repository

ClientOBJ

REF

Object

(Servant)in args

operation()

out args +

return

DIIIDL

STUBS

ORB

INTERFACE

IDL

SKELDSI

Object Adapter

ORB CORE GIOP/IIOP/ESIOPS

Object request broker core

Interface

repositoryCORBA services

Object

adapter

Server objects

Client

objects

CORBA architecture


•Common middleware services support many recurring distributed system capabilities, e.g.,

•Transactional behavior

•Authentication & authorization,

•Database connection pooling & concurrency control

•Active replication

•Dynamic resource management

•Examples•CORBA Component Model & Object Services, Sun’s J2EE, Microsoft’s .NET

68

Component middleware

•Components encapsulate application “business” logic

•Components interact via ports•Provided interfaces, e.g.,facets•Required connection points, e.g., receptacles

•Event sinks & sources•Attributes

•Containers provide execution environment for components with common operating requirements

•Components/containers can also

•Communicate via a middleware bus and

•Reuse common middleware servicesSecurityReplication NotificationPersistence

SchedulingA/V Streaming Load Balancing

…

Container

… …

Middleware Bus

Container

…

Component middleware from software developer perspective:Docker as example


• Containers and micro-services (technical)

• Platform services• Building and pushing images to image repository

• Pulling images, provisioning and scheduling containers

• Discovering and binding to services running in containers

• Containers discovering and binding to other containers

• Operating and managing services in containers

Integratable to•Amazon Web Services, •Google Cloud Platform,•IBM Bluemix,•Microsoft Azure,•OpenStack,•Cloud Foundry PaaS

Q1: How middleware supports software engineering

9/19/2018 Lea Kutvonen

• Container role• Deployment of applications on to hosts or registries for clusters or clouds• instantiating application code and maintaining private resources, namespace, dependencies,

connections in+out• platform element that hides underlying technology, for software engineers isolates

• Application logic above bindings, service calls and streams and selected middleware facilities (instead of direct use of sockets)

• Middleware functionality on each technology platform separately

• Docker• Application lifecycle management• Packaging, deyployment, portability• Deployment of IaaS, PaaS, SaaS packages

• Sources, for example:• Kai Wähner, Relation of Middleware to Microservices, Docker, and Cloud-Native Architectures.

(Practitioners)

https://dzone.com/articles/relation-of-middleware-to-docker-and-cloud-native

Open implementation and its adaptation strategies


Programmatic adaptation• Insert new program elements• Allows introduction of new functionality• Risk for errors, security gaps

Policy-based adaptation• use parameters to indicate strategic

decisions on which modules to utilise• no user code insertion, easy

Automatic adaptation• The implementation includes strategies for

selecting the right configuration for each situation

Q2: Component MW and dockers support open platform and open binding argument

• Instead of fixed ESB products components and dockers allow local selection of • communication channel type

• local technology choices

• For interoperability and ease of deployment• Encapsulation of private

dependencies and resources

• Ease of introduction of new code, registering it, redeploying it


Service-oriented middleware• Service is composable

• Providers• Produce implementations• Provide service descriptions• Provide runtime environment

• Service description• Advertises capabilities• Interface signature + behaviour• QoS, security, availability, ..

• Technology neutral

• Technology transparency

75

Service directory

Service consumer Service

FindRegister

Invoke


Service-oriented middleware

Within service middlware: Workflow / Business process engines


van der AalstComprehensiveSurvey 2013

ObtainingProcessmodels

Process repository activities


BP enactment in WS stack


WfMC reference model,operations and interfaces


WfMC RM1999

WfMC reference model


WfMC RM1999

WfMC distributed solution


WfMC RM1999



Data flow &Command flow

Explicit binding concepts


Enterprise service bus ESB

• Take the following incredients and mix• MOM mw + RPC feel• Transactional mw• routing, identities, name service• SOA, i.e. service registry, service-type based

routing• Publish/subscribe• Business process engines for orchestration• Semantic interoperability support (incl.

ontology)• Bridges for the essential vendor specific

communication platforms• SLA management (QoS monitoring and

control)• Encryption• Multi-language support


Something missing? New themes?

• Replace by Cloud?• Communication protocols and SOA services missing

• Storage and transaction processing on DB natural?

• Bridge ESBs to I-ESB• Joining ontologies?

• Security solutions? Limits between intra- and inter-enteprise collaborations

• Architecting communication patterns separately and using open binding concept for injecting all that into the ecosystem

• In comparison, IoT protocols more specific to context, still grasping some of the same (old) patterns

Distributed system support mechanisms

Consensus formation• Agreeing on a decision requiring preferrably

all patners (but some can be omitted, due to unreachability or non-cooperative state)

• Collect required Quorum• Quorum = (n/2) +1 • Necessary in case of network / group

partititioning situations → at recovery state, the decision cannot change, all will eventually obey → eventual consistency

• Ensure leader available for reporting decision state• Leader (or contract agent replica) always present• When leader fails, re-election of leader in

identity order


Distributed system support mechanisms

Ontology is• formal, explicit specification of a shared

conceptualization

• semantic structure which encodes the implicit rules constraining the structure of a piece of reality

• Ontology defines• A common vocabulary• The meaning of terms• Interrelation of terms

• extensional class• if and only if it is characterized solely by its

membership

• intensional class• Created by categorization rules

Ontologies are used in computer science

• Expert systems with knowledge databases (facts, axioms) for example for• Diagnostics• Business decisions• Predictions

• Decision-making support based on “big data”• Business intelligence• Innovation support• Social knowledge management


Model-driven middleware key elements

Model Model is about• Structure:

• Data, architecture, configuration

• Behaviour• User interface• Access control• Business process

• On any system aspect


Interpretation:

Mental model

Reality

(observed

or imagined)

Described model

Model specification• Precise instructions for construction, for example,

code generation

Metamodel hierarchy and model hierarchy

Metamodel • Metamodel is a model of a model• Metamodel determines for a category

of models• structure, properties and relationships or• representation

• Metamodel enables• Comparison or categorisation of models• Determination of belonging to such a

category• Refinement of meta-model to

alternative models• Transformation of representational

formats of models of the same category

Model hierarchy (MOF) is an ontology too


Ongologies on entitities and their representation for interoperable comptuing

Model-driven middleware key elements

Model driven architecture

Model driven hierarchy

and engineering• Metamodel enforces

conformance

• Model transformations• Potentially automated• For example, code generation, or

reversing: changes in code reflected back to more abstract model

• Models can be used for monitoring too (runtime use)

• Transformations• require mapping rules on

metalevel• transformation specifications

expressed in specific language


Reflective middleware (reflective system pattern)


Reflective systems include

• Model of their own structure, state, behaviour

• Self control• Deduction of need for structure or

state change• Capability of performing such

change

Source: P Grace, G Blair, Reflective middleware. In the Handbook of Mobile Middleware. CRC Press 2006.


Chapter 6.3: Business process modeling advise in brief for the design task

Jussi Kangasharju

Markku Kojo

Lea Kutvonen


Outline• Why business processes are

needed?

• Business process model vs information flow model

• How a business transaction differs from a transaction (with ACID properties)?

• My own BPM• Basic concepts of BPML• Requirements on autonomy• Reliability definitions:

• by whom & how

Networked business needs in society

• Social networks

• Contents (e.g. Flickr)

• Expert nets

• Scientific societies, standardisation

• Multicasting, crowdsourcing

• Entertainment, nets of newsagens

• Business networks

• Supply chains, value nets, ...

• Virtual organisations

• Creation supported by breeding environments

• Business service collaborations

• Roles, gains, responsibilities

• In addition, runtime supported by ecosystem facilities

Example domains

• Chrisis management

• National heath-care

• Big data networks

• Innovative nets

AU

TOM

ATE

D C

ON

TRO

L OF C

OLLA

BO

RATIO

N

From enterprise centric automation to value nets


Business process properties

• Why business processes are needed?• Societal and business needs• Technical opportunities:

verification, validation, generation, execution, model-driven engineering

• Business process model vs information flow model• Languages for both, focus choice• E.g., cumulating nationwide big data, utilising

adaptive case management

• How a "business transaction" differs from a (database style) transaction (with ACID properties)?• Business processes do have real world

consequences --> no rollback, but compensations

• Sometimes just agreed acceptable future state possible to reach

• e.g. passanger left on an airport mid-trip due to an oversale of tickets

• Resolution may have wider scope

• Business transaction may mean too many things, including running an individual case (instance) of business process (and whatever supporting processes and services it associates with)


Concepts for inter-enterprise collaborations (I.e., concepts for composition of business services)

Examples from networked businesspoint of view

• eCommerce

• Supply chain

• Multiple partners + (in)tangible values

• Value net

• Virtual organization breeding environment

• Collaboration (case)

Key concepts• Business process

• Business service

• Exchanged documents (msgs)

• Vocabulary

i.e.

• x-ecosystem rules

• Individual adaptation

• Case management


My own BPM:

Languages: BPMN as an example


• Version numbers and alternative languages have the same key concepts

• Business Process Modeling Notation

• By BPMNI (... Initiative) → OMG (by org. merger)

• Does:• Supports business process management for technical and business users• Bridge communication gap between business process design and implementation

• Does not:• No state transitions, no functional decomposition, no organisational hierarchies, no

data modeling

• Process flows can be either executable or non-executable

• Finnish public sector agencies must use for their enterprise architecture work and for publishing their business service interface behaviour descriptions

Basic concepts• Process

• Flow of activities, decisions, data and events

• Collaboration• Conversations and interactions (also processes)

• Choreography– Between pools– Participant

interactions using atomic tasks

Basic concepts


• Pools• Independent

organisation

• Black box option

• Lane• Independent actor

(role) within organisation

• Organise and categorise activities

Private and public processes


Collaboration modeling aspects


Pools with a gap in between: pools represent independent "organisations"

Within each pool, orchestrated workflows (single decision-making & responsibiilty domain) running (note begin and end circles, compare with "daemons" in computing system)

Exchange of messages (information or control notifications) across "gap"forms choreography, I.e. cross-organisational discussion

Process: Activity sequence & “token”


• Activity, process, task

• Normal flow

• Uncotnrolled flow

• Conditional flow

• Exception flow

• Default flow

• Compensation flow

• Fork – merge; sequence flow looping

• Multiple instances

• Process break

• Transactions, nested, ...

Activities


• atomic, non-atomic (compound)

• no need to show further detail

• variations: service task; send, receive; User, manual; Business rule; script, ...

• Has self-contained start and end events, sequence flows from parent process must not cross boundary (no overall reusability)

• Completed fully or compensated

Gateways


• Controls sequence, decisions for branching, merging,

• Can be based on events or evaluation of expression

• Exclusive, inclusive, parallel gateways• One or more alternative parallel

executions allowed

Events: start, intermediate, end


• Error: throw error, close all parent activities hierarchically

• Escalation: code to be passed to nearest parent Activity

• Cancel: send cancel to all involved entities in transaction

• Compensation: indicates which task to perform for compensating the target activity (transaction)

Message flows


• Processes are abstract

• Runtime performances are concrete instances• Correlation, correlation keys to keep isolated operations

associated

How to design business processes and independent services?


• Process definition example video (4 mins)• capturing business process model from a use case; not enough

• Between collaborating processes (5 mins)• Want each organisation (or role) to have independent business processes with

start and stop, and choreographies in between of the swimlines

• Seek for a suitable level of abstraction: Abstract models must• Suit the ecosystem environment in which the service must collaborate with

others, be adaptable for multiple partners,• Be adaptable to multiple heterogeneous environments by wrappers,

transformations of messages, intelligent bridges etc• Complete enough for verification ()• Support the engineering methodology (e.g. MDE, model-driven engineering,

• Notation poster v.1

• Notation poster v.2

https://www.youtube.com/watch?v=y1O1K_zrQ5o

https://www.youtube.com/watch?v=_2K-txcDteU

http://bpt.hpi.uni-potsdam.de/foswiki/pub/Public/BPMNCorner/BPMN1_1_Poster_EN.pdf

http://www.bpmb.de/images/BPMN2_0_Poster_EN.pdf

Verification:Finite automata and global state analysishelp seeing whether your process makes sense


a b

start

0?

1!

A B

start0!

1?

aA – bB – bB – bB – bB - ....

D C

start0?

1!

0!

DC - BC – BC – BC –!

D RR (locked, likely to be a design error)

Shared state graph for 1st and 2nd

Shared state graph for 1st and 3rd

Must check all paths:All reached states acceptable and sufficient?

9/19/2018 Service Ecosystems / Lea Kutvonen 112

Service and collaboration model engineeringDoes it differ from the traditional engineering practices?

9/19/2018 Service Ecosystems / Lea Kutvonen 113

Service and collaboration model engineeringDoes it differ from the traditional engineering practices?

Design task 3:Breach recovery in cross-

organizational business process

Lea Kutvonen

Fall 2017

Notes on the design task

• The situation is not realistic, for a few reasons• The limitations bring up a conditions under which collaboration reliability

threats become present

• The limitations keep it small enough to handle

• The limitations make the solutions similar enough to walk through the intended learning processes

• See the grading hints


Outline

• Context: Networked business and open business service ecosystem

• Business scenario: cross-organizational collaboration

• Design task

• Grading hints

• Recall challenge:• Balancing of system reliability and business expectations

Context: Networked business

• Autonomy (independence) of organisations• Platform• Business services provided• Decision on whom to contract collaborations

with• Setting internal (private) enterprise policies

(related to strategies): what business processes to invest in, which orgs to trust/distrust or through which process to decide on trusting, loyalty program member identification and variations in business process for the members etc)

• Local regulatory policy availability

• X-ecosystem support facilities• Service discovery by type• Repository of business process models (and

chunks for them)• Ecosystem policies

• Instead of access-rights, organisationsuse deontic model in contracting• Negotiated contract on the collaboration

states:• Orgs X and Y are obligated to provide

service Type1 to other collaboration members

• Org ZY is permitted to provide service Type1 (identified as “S1-byZY-policiesP1”)

• Org ZN is prohibited from providing service Type1

• Org A is permitted to use service Type1 (but not prohibited or obligated from using it from X)

• Contract shows that things may work, but do not quarantee it in all runtime situations and circumstances → runtime breach detection and recovery needed

Business scenario:cross-organizational collaboration• Group of organizations from different countries sell and transport bulky tangible

items across national boundaries• Business model realistic (existing): international selling and transportation requires knowledge

of international and multiple national regulatory systems and failures to follow those laws have sanctions (loss of merchandize, prison sentence, huge cost)• Regulatory system → directives, legal system, sometimes even business best practices

• Collaboration focuses on knowledge of contracting, regulation and support facilities for import and export• Organizations make contracts themselves, bill clients, split costs and profits• However, they outsource transportation

• Outsourcing: external service provider takes care of that part of operation• Transportation can only be arranged by national carriers by national client

• Collaboration may involve several business processes between required partners• Internal business processes of the partners may be hidden or schetched only in the “need to

know” basis from the other partners point of view

Design task

• Work in steps:1. Design the key business processes between the partners using BPMN

(business process modeling notation) • When identifying the partners, note also the outsourcing needs and regulators (through

stored rules)• In BPMN modeling

• an autonomous organization must have a role (separate set of swimlines) while a unit or computing system may have a swimline associated to that role

• Each role (and swimline, unless you have a justified reason) must have a start node and an end node as the behaviours of separate services of organisations and units are independent (choreographies → starts and stops per role, orchestrations → may have sequences crossing swimlines within a single role)

• Supporting processes can be left out for simplicity and size, also use composed task blocks for parts that do not include interesting elements• exchange of key information, exchange of control signals (start, your turn, stop, did you start

processing my request?, etc as you design)

Design task (cont)

2. Identify potential breaches • Breach = failure to operate according to the contract• Note that the business transactions (requests and activities belonging together between

business partners) are not clean in the database transaction sense• If your transport fails, the bulky goods are in wrong place, and you need activities for moving them

to an acceptable location and the cost and sanctions needs to be determined in a way that depends on the situation, partners, country in which the breach took place, etc.

3. Design a couple of alternative recovery business processes4. Consider where in your system architecture you can place them and why. What

benefits and drawbacks there are for these choises?5. Did you remember to consider who in the collaboration scenario gets to decide

which of your recovery processes is used? 6. Complete your overall design by ensuring all collaboration partners know when (if)

the purchase reached the final location and know whether any breach recovery (with potential sanctions) took place during the collaboration operation.

7. Make sure you discuss the choices you made and assumptions behind them.

Recall challenge + grading hints

• Remember to include into your submission commentary on balancing of system reliability and business expectations in the scenario• What is your definition of reliability here?• What are the facilities for supporting reliability?• What kind of costs is caused? • While the cost seems huge, the same facilites support engineering, correct local operation,

agility on market place, following and adapting to regulatory systems and other well worth goals in addition to the reliability goal.

• There is a lot of space for using your imagination here, no need to check any laws etc for this design. The scenario is not fully realistic.

• Focus on the use of the business scenario page ideas

• From BPMN, very basic use skills suffice, syntax errors do not matter. • May use paper and pen or some BPMN free tools• Some tool examples here

networked systems and services - courses · networked systems and services fall 2018 chapter 6:...

Documents