Download - BAM for Developers
Business Activity Monitoring in Depth for Developers
Published: January 2009
Authors: Jesus Rodriguez and Joe Klug, Tellago, Inc.
Technical Reviewers: Ofer Ashkenazi, Microsoft Marcel Fernee, Microsoft Stephen Kaufman, Microsoft Brian Loesgen, Neudesic LLC Jonathan Moons, Microsoft Allan Naim, Microsoft Paolo Salvatori, Microsoft Andy Shen, Microsoft
Applies to: BizTalk Server 2006 R2 and BizTalk Server 2009
To download a copy of this document and the sample code, go to
http://go.microsoft.com/fwlink/?LinkId=139086.
Summary
Microsoft added Business Activity Monitoring (BAM) to Microsoft® BizTalk® Server as a way for business analysts to track information that is important to them. This paper provides a deep, low-level review of how BAM works and how BAM can be extended. It reviews BAM basics, shows ways in which BAM data can be collected, discusses the BAM infrastructure, demonstrates methods for collecting BAM data, and shows how BAM can be extended to non-Microsoft technologies.
CopyrightThe information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
© 2009 Microsoft Corporation. All rights reserved.
Microsoft, BizTalk, Excel, Internet Explorer, MSDN, Outlook, PerformancePoint, PivotTable, SharePoint, and Windows are trademarks of the Microsoft group of companies.
All other trademarks are property of their respective owners.
Table of Contents
Executive Summary.....................................................................................................................................5
Business Activity Monitoring Overview.......................................................................................................6
BizTalk Server BAM: Exploring the activity model...................................................................................7
Defining a BAM sample activity model..................................................................................................10
How BAM differs from data warehousing.............................................................................................16
Instrumenting Applications by Using BAM................................................................................................18
Using BAM declaratively........................................................................................................................18
Using BAM from BizTalk Server.............................................................................................................19
Using BAM from Windows Communication Foundation and Windows Workflow Foundation.............23
BAM WF interceptor..............................................................................................................................29
Using BAM programmatically................................................................................................................34
BAM event stream API...........................................................................................................................34
Using the BAM interceptor API..............................................................................................................37
Inside the BAM Infrastructure...................................................................................................................40
Storing BAM data: BAMPrimaryImport database..................................................................................40
Archiving BAM Data: BAM Archive database.........................................................................................42
Partitioning BAM data...........................................................................................................................42
Aggregating BAM data...........................................................................................................................45
The BAM Event Bus...............................................................................................................................51
Querying BAM Data...................................................................................................................................53
Overview...............................................................................................................................................53
Using BAM with data access libraries....................................................................................................54
Using BAM with SQL Server Reporting Services.....................................................................................55
Using BAM with Microsoft Office PerformancePoint Server.................................................................57
Using BAM with Microsoft Office SharePoint Server.............................................................................62
Accessing BAM by using Web services..................................................................................................69
Building a BAM RESTful API.......................................................................................................................73
Why do we need a new API?.................................................................................................................73
Embracing the Web programming principles........................................................................................74
Brainstorming a BAM RESTful API..........................................................................................................75
A sample BAM RESTful API....................................................................................................................77
Summary...................................................................................................................................................86
Authors......................................................................................................................................................86
Executive SummaryMicrosoft added Business Activity Monitoring (BAM) to Microsoft® BizTalk® Server as a means for business analysts to track information that is important to them. One common example of this is tracking the dollar amounts of every transaction processed (i.e., total sales from a particular region). Since BAM was first introduced in 2004, there have been a number of white papers, webcasts, and examples about the basics of BAM. Unfortunately, what’s been missing is a deep, low-level review of how BAM works and how BAM can be extended.
In this white paper we will take you beneath the covers of BizTalk Server Business Activity Monitoring. After reviewing the basics, we will show you the various methods by which BAM data can be collected, from the pre-built interceptors to the APIs that can be used in custom solutions. The second part of the paper digs into the BAM infrastructure, explaining the BAM databases and what happens to them when BAM models are deployed, and the role of SQL Server® Integration Services in BAM. In the third part of this paper we will demonstrate a number of methods for querying BAM data, covering the use of data access libraries, SQL Server Reporting Services, Microsoft Office PerformancePoint® Server and Microsoft Office SharePoint® Server, and Web services. Finally, this paper covers how BAM can be extended to non-Microsoft technologies through the use of Web services and programming models such as Representational State Transfer (REST).
The four sections of this paper are relatively independent of each other. You don’t have to read the section on the BAM infrastructure to understand how to extend BAM with REST. Our only suggestion is that you cover the basics at the beginning of this paper. We have also included with the white paper the samples that contain the code that is referenced in this paper. These examples can be compiled and deployed against a basic BizTalk Server installation.
Business Activity Monitoring OverviewBusiness Activity Monitoring (BAM) is one of the rapidly evolving areas of Business Process
Management (BPM). Conceptually, BAM focuses on providing nearly real-time monitoring
and analysis of business process activities. We can think of BAM as a bridge between
integration and business intelligence (BI) techniques. Ultimately, the goal of BAM is to
enable sophisticated BI techniques based on the information collected from business
processes.
In order to achieve this goal, BAM uses BI models based on the information from different
business processes. These models are typically known as event or activity models and are
normally translated into relational and multidimensional databases. BAM technologies
populate those models using events that describe important milestones of a business
process. BAM technologies are responsible for triggering those business events and mapping
them to the activity models. The following figure conceptually illustrates a typical BAM
architecture.
Figure: BAM concepts
As a technology, BAM technology stacks have emerged as a fundamental component of
integration and BPM suites such as Microsoft BizTalk Server, IBM WebSphere Process Server,
Oracle SOA Suite, etc. This is mostly a strategic decision given that many of the business
processes in the industry today are implemented by using those technology suites.
In the case of BizTalk Server, the first version of BAM was introduced with the 2004 release
and followed by subsequent versions in 2006 and 2006/R2.
BizTalk Server BAM: Exploring the activity modelAlthough BAM is released as part of BizTalk Server, it provides a flexible framework that can
be used to instrument any .NET Framework-based application. Additionally, as we will
explore in the last section of this paper, BizTalk Server BAM can be extended to interoperate
with non-.NET technologies such as J2EE or dynamic languages.
BizTalk Server BAM is built around the concept of an activity model. This type of model
defines relevant business information about a business process. The fundamental element of
a BizTalk Server BAM activity model is an activity. Activities represent atomic information
units that need to be monitored. As an analogy in the database world, we can think of
activities as entities of an entity relationship model. In that sense, activities are described by
a collection of fields that represent business data or important business milestones. For
instance, a banking transaction activity can contain a data field that indicates the monetary
amount of the transaction and another milestone field that represents the time in which the
transaction was processed.
Milestones are a key concept of BAM that, as explained in the previous example, capture
important states of a business process. In addition to business milestones (DateTime),
activity fields can be of type Text, Float, and Integer. Activities can have relationships with
other activities. For instance, the transaction activity can have a relationship with the
customer activity.
The data included in a set of activities can serve to provide relevant information to different
domains. For instance, a bank executive might use a BAM activity set to obtain some
financial analysis metrics while an IT manager might be more interested in seeing the
average number of failed and successful transactions. To address these scenarios BizTalk
Server BAM introduces the concept of an activity view, which projects a specific
representation of a set of BAM activities.
One of the main advantages of activity views is that they can combine the information of
various BAM activities, providing different real-time perspectives of a business process.
Additionally, activity views can capture historical metrics incorporating multidimensional
elements such as measures and dimensions. These concepts are the equivalent of the
measures and dimensions included in most OLAP engines.
Measures are numeric aggregations of an activity field; for instance, a measure
AVG(Transaction Amount) will calculate the average of the financial amount of the
transactions represented by the banking transaction activity. Measures are typically
segmented by dimensions, which are information hierarchies that represent areas of interest
in the information; for instance, we can use a Time dimension with three levels (year,
month, and day) to analyze the information provided by the measure AVG(Transaction
Amount) across different time intervals. The current release of BizTalk Server BAM includes
the concept of time, data, numeric range, and progress dimensions.
Progress dimension: Represents the creation of aggregations with respect to the
progress of activities that are still in process. For instance, a progress dimension can
represent the status of a financial transaction with the levels (RECEIVED,
PROCESSING, APPROVED, DENIED).
Data dimension: Used to categorize an aggregation. Data dimensions are based on
the value of string-formatted data items in the BAM activity.
Time dimension: Used to create aggregations across defined time segments. For
instance, a time dimension with levels Year, Month, Day can be used to segment the
measures of the banking transaction activity.
Numeric range dimension: Used to categorize aggregations based on friendly
names of given numeric ranges. For instance, a numeric range dimension can
categorize the amount of a banking transaction into levels Small ($1-$10000),
Medium ($10001-$100000), and Large ($100001-$1000000).
Dimensions and measures can be used to define real-time or scheduled aggregations. A
scheduled aggregation represents a time snapshot of an activity view. In order to execute all
the calculations required for the measures and dimensions, this type of aggregation is
scheduled to run based on specific time intervals. There are some scenarios in which
different parts of an aggregation need to be available nearly in real time. For those
scenarios, BizTalk Server BAM introduces the concept of a real-time aggregation (RTA),
which executes some of the required calculations as the activity data is being populated.
The “Inside the BAM Infrastructure” section drills down into the aggregation model.
BizTalk Server BAM represents the activity model using an XML-based language known as
the BAM definition language. This language is described by the BAM definition schema. The
BAM definition schema defines the constraints of the elements of a BAM activity model
including activities, views, cubes, etc. Developers can create BAM activity models by using
their favorite XML editor or specialized tools like the BAM Add-in for Microsoft Office Excel®
illustrated in the following section.
It is important to notice that the process of defining a BAM activity model is completely
independent of BizTalk Server. Developers or analysts can define activity models to
instrument applications implemented using heterogeneous technologies. Once created, an
activity model is deployed to BizTalk Server so that it can be used at run time. In that sense,
the elements of the BAM activity model are translated into relational artifacts deployed in
different BizTalk Server databases.
Throughout this paper we are going to explore in detail the intricacies of BAM activity
models. Specifically, the “Inside the BAM Infrastructure” section of this paper drills down into
the different components of a BAM activity model and how they are represented in the BAM
infrastructure.
In order to get a better understanding of the process of creating a BAM activity model, the
next section will walk us through the process of creating a sample activity model that we will
use in the remaining sections of this paper.
Defining a BAM sample activity modelA traditional process of some financial institutions like banks is to process credit card
transactions. As part of this process a transaction needs to be approved or denied based on
certain criteria like the debt or credit limits associated with the account. Additionally, if the
characteristic of a specific transaction matches some well-defined fraud pattern, an alert is
triggered in order to take the appropriate actions. In this section, we are going to define an
oversimplified transaction activity model that will serve as the basis for all the examples
included in this paper.
Conceptually, the fundamental elements of our model are the Transaction and Alert
activities detailed in the following tables.
Field name DescriptionAmount Decimal: Amount of the intended transaction
Transaction_Approved Milestone: Time at which the transaction was
approved
Transaction_Created Milestone: Time at which the transaction was
created
Transaction_Failed Milestone: Time at which the transaction
failed
Transaction_Failed_Alert_Triggered Milestone: Time at which the transaction
failed and an alert was fired based on a fraud
criteria
Customer_Age Integer: The age of the customer initiating
the transaction
Customer_Name Text: The name of the customer initiating the
transaction
Figure: Transaction activity
Field name DescriptionCode Text: Code associated with the alert
Description Text: Description of the alert
Alert_Created Milestone: Time at which the alert was
created
Alert_Processed Milestone: Time at which the alert was
processed
Figure: Alert activity
Using the BAM Excel Add-in we can create the activity definition that contains the previous
elements as illustrated in the following figure.
Figure: BAM activity definition
At this point, we can extend the information of the TRANSACTION activity with some
multidimensional elements that facilitate more sophisticated historical analysis over the
activity data. To achieve that, we will create the following dimensions and measures.
Element Type Description
Count_Transaction Measure Number of processed
transactions
Avg_Transaction Measure Transaction amount average
Total_Transaction Measure Total amount of the
processed transactions
Count_Alert Measure Number of triggered alerts
Transaction_Time_Dim Dimension A time dimension to segment
the transaction information
based on Year, Month, Day,
Hour levels
Transaction_Stg_Dim Dimension A process dimension to
segment the transaction
information based on its
different milestones
Amount_Dim Dimension A numeric range dimension
categorizing the amount of a
transaction into Small,
Medium, or Large
Alert_Time_Dim Dimension A time dimension
segmenting the alert time
into Year, Month, Day, Hour
levels
Figure: Transaction dimensions and measures
Using the Business Activity Monitoring View Wizard of the Excel Add-in we can define those
elements as illustrated in the following figure.
Figure: BAM activity view aggregation elements
Each of the multidimensional elements is defined separately.
Figure: Defining measures and dimensions
After defining the multidimensional elements, we can add the final touches to our BAM
activity view by creating specific PivotTable® reportsusing the multidimensional elements of
the activity model. The following figure illustrates this step.
Figure: Customizing a BAM activity view
After exporting the Excel definition to XML we obtain a representation of the TRANSACTION
activity model compatible with the BAM definition schema. Although the BAM Excel Add-in
is, undoubtedly, the preferred tool for creating BAM activity models, developers can build
custom tools that provide a more sophisticated user experience to help domain experts to
create specific activity models. Similarly to the Excel Add-in, these would translate the
graphical representation into the XML that can be processed by BizTalk Server. The following
code illustrates sections of the XML representation of our sample activity model.
<BAMDefinition xmlns="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM"> <Activity Name="TRANSACTION" ID="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC">…Sections have been omitted for brevity.. </Activity>
…Sections have been omitted for brevity..<View Name="TRANSACTION" ID="ID12CBC52A28684F3BAE8F742DCF545A18"> <ActivityView Name="ViewTRANSACTION"
ID="ID94EBCE7C807F48BBA5BA96F3F65B9619" ActivityRef="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC"> <Alias Name="AMMOUNT" ID="ID0399A8CD04CF49E0AF981322275908F9"> <CheckpointRef>ID7691FFDDA6074E2F88597ED5EF3967F5</CheckpointRef> </Alias> <Alias Name="CUSTOMER_AGE" ID="IDD82D0B7C03E14DAAAC7F245195F248EE"> <CheckpointRef>ID44ED38B3A28A4D04A674F9D8CE3238A6</CheckpointRef> </Alias> <Alias Name="CUSTOMER_NAME" ID="IDBF7F4BABE42A4CD0BC1B7F5B5BB941E9"> <CheckpointRef>ID949D996AC41F4E499944F8C388E2C439</CheckpointRef> </Alias> <Alias Name="TRANSACTION_APPROVED" ID="IDB74B11B4122A4C4D8523DFA1BB607D5C"> <CheckpointRef>ID604617A21C82456AB8BE76DBA9B2DCA8</CheckpointRef> </Alias> </ActivityView></View>
…Sections have been omitted for brevity..
<Cube Name="TRANSACTION1" ID="IDB9050B5EB8BB4661ABE1CDA4BA729D55" CreateOlapCube="true" ActivityViewRef="ID94EBCE7C807F48BBA5BA96F3F65B9619"> <Measure Name="AVG_AMMOUNT" ID="ID0A47D6F650FA49AC8FDAC18D7F5E78EE" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Avg"/> <Measure Name="TOTAL" ID="IDD191BAAA7E964CE0B16710AC932C7343" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Sum"/> <Measure Name="COUNT_TRANSACTION" ID="IDB3E446EFDFFC4AC6BD364FBAA34EBA44" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Count"/> <TimeDimension Name="TRANSACTION_TIME_Dim" ID="IDC2CAA2E2B68C4C429CD8DEC19E12745B" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"> <TimeLevel>Year</TimeLevel> <TimeLevel>Week</TimeLevel> <TimeLevel>Day</TimeLevel> <TimeLevel>Hour</TimeLevel> <TimeLevel>Minute</TimeLevel> </TimeDimension> <NumericRangeDimension Name="AMMOUNT_Dim" ID="IDE79FC5B4EBE848BCA78F67FD886E0A5D" NumericAliasRef="ID0399A8CD04CF49E0AF981322275908F9"> <Range Name="SMALL" ID="ID3853D9BC675A436CB2A1C7ACDB4E873D" From="5" To="100"/> <Range Name="MEDIUM" ID="IDB6567ACBFEF14C91BCD2073D569AD85E" From="101" To="1000"/> <Range Name="LARGE" ID="IDBFCC85280801407AB692DD7AFD76A4D2" From="1001" To="10000"/> </NumericRangeDimension> <ProgressDimension Name="TRANSACTION_STG_Dim" ID="ID7BD612325E5C4845B8F716469958AFC4"> <ProgressStage Name="TRANSACTION_CREATED"
ID="IDFC5CBDAE550F4F5EBA6C3BBB6C864E3B" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"> <ProgressStage Name="CREATED" ID="ID1DD5128DC0B14CDFBE049A514666A782" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"/> <ProgressStage Name="FAILED" ID="ID0788A06545854B3298FC410DAE1AF04D" TimeStampAliasRef="IDAC79D047E6134D3E922B23FC650D13D5"/> <ProgressStage Name="FAILED_ALERT" ID="IDDD2D200E51744439A65DA2CB4C2950B4" TimeStampAliasRef="ID18F90EE60D894A739CD13D081AB63387"/> <ProgressStage Name="APPROVED" ID="IDEB527B961B864E7C96E770AEB0D2AD68" TimeStampAliasRef="IDB74B11B4122A4C4D8523DFA1BB607D5C"/> </ProgressStage> </ProgressDimension> </Cube>
…Sections have been omitted for brevity..
</BAMDefinition>
Figure: BAM XML definition
After the activity model is in a form compatible with the BAM definition schema, it can be
deployed to BizTalk Server by using the BAM deployment utility with the following syntax:
>bm.exe –deploy-all DefinitionFile:<BAM activity model file>. Once deployed, the
BAM activity model can be updated and redeployed by using a similar mechanism.
At this point, the BizTalk Server BAM runtime will create the necessary artifacts to enable
the activity model so that it can be used by different applications. Technologies like BizTalk
Server, Windows® Communication Foundation, and Windows Workflow Foundation have
embraced BAM into their programming model, providing tools and technologies that
facilitate the instrumentation of applications using purely declarative mechanisms. More
sophisticated scenarios or applications developed using other frameworks can also populate
the BAM activities by taking advantage of the BAM APIs. The following sections explore the
different alternatives that developers can use to instrument their applications using BAM.
How BAM differs from data warehousingThe use of BAM enables near real-time, business-centric visibility of enterprise processes. In
that sense, BAM extrapolates meaningful business metrics based on the events produced by
business processes. This level of visibility enables executives or other domain experts to
monitor business processes, correlate metrics and analytics down to the actual business
process, and react to and predict business scenarios based on real-time analysis.
It is important to highlight a distinction between BAM and traditional data warehousing
techniques. Although both techniques are enabled by relational and multidimensional
database infrastructures, BAM is centered on capturing, processing, and analyzing real-time
business process events, while data warehousing techniques are traditionally more focused
on enabling historical aggregation and analysis of a variety of data sources. Along those
lines, BAM implementations typically handle a very specific set of information related to a
series of business processes. Data warehouses, on the other hand, are commonly used for
representing aggregated data views containing a broader range of information than what is
normally associated with a business process. Following that thinking, data warehouses can
be seen as a complement to BAM technologies in order to enrich the information included in
the BAM model. For instance, a BAM process might keep track of the business activities
associated with one or more specific accounts. This information could then be
complemented with elements from an ERP data warehouse that details information about
the opportunities associated with those accounts as well as metrics about their historical
behavior.
Instrumenting Applications by Using BAMAs we’ve explored in the previous section, BizTalk Server Business Activity Monitoring (BAM)
provides very flexible mechanisms to express activity models that describe the information
that needs to be monitored. However, the BizTalk Server BAM model does not impose any
constraints on how the monitored information is collected and how the activity model is
populated. This flexibility allows multiple technologies to leverage the benefits of BizTalk
Server BAM in an optimal way. In order to make that integration even more flexible, some of
those technologies have adopted domain-specific declarative mechanisms to indicate what
information needs to be collected and how that information is mapped to a BAM activity
model. BizTalk Server, Windows Communication Foundation (WCF), and Windows Workflow
Foundation (WF) rank among the Microsoft technologies that have adopted declarative
mechanisms to interact with BAM. However, BAM scenarios can be arbitrarily complex and
some of them might require the use of programmatic interfaces to interact with a BAM
activity model. This section drills down into the details of the declarative and programmatic
technologies that developers can leverage to instrument their applications by using BAM.
Using BAM declarativelyInstrumenting an application by using BAM is a combination of two basic principles: knowing
when to capture the application data and knowing how to map that data to BAM activities.
Those two principles are easily applicable when we are talking about data or message-driven
applications such as those implemented with technologies like BizTalk Server, WCF, or WF.
What makes these types of applications special? First, the data processed in those
applications is represented by using a queryable format from which, at any point, the
information can be extracted. Examples of that format are SOAP envelopes, BizTalk
messages, or context properties. Secondly, applications implemented by using these
technologies follow a series of execution points at which the flow can be intercepted and
consequently the information can be extracted. For instance, WCF services flow messages
through a dispatcher pipeline on which they can be intercepted by using components like
channels or message inspectors.
Following these two principles, we can think of developing generic technology components
that can be configured to intercept the application data at different points, query the data
for specific information, and finally map that information to a BAM activity model. Ideally,
the applications will have no dependencies on BAM and the whole process will be driven
declaratively through configuration. Obviously, the complete picture is more complex but it
follows the same basic concepts.
As you might already know, technologies like BizTalk Server, WCF, and WF have taken the
approach explained previously, providing declarative mechanisms that accelerate the
instrumentation of applications by using BAM. The following section takes a deep look at the
internals of those mechanisms and how developers can benefit from using them.
Using BAM from BizTalk Server
As the technology hosting the core components of the BAM infrastructure, BizTalk Server
provides broad support for BAM in terms of tools, APIs, and implementation scenarios. The
BizTalk Server BAM runtime provides the components for populating BAM activity instances
from orchestration persistence points or pipeline components. Essentially, BizTalk Server will
populate a BAM activity model using the data contained in BizTalk messages or context
properties.
Let’s start with the following orchestration that processes a financial transaction. (The entire
source code is included in the samples that accompany this paper.)
Figure: Financial transaction business process
Our sample orchestration receives a message with the transaction information and applies
some simple rules to determine whether the transaction can be approved. Luckily for us, the
financial transaction processes in the real world are more complex.
Having this orchestration, we can start thinking about how to populate our sample BAM
activity model using the data contained in the BizTalk messages. For most of the scenarios,
the easiest way to accomplish that is by using the BizTalk Server Tracking Profile Editor
(TPE). This tool allows developers to create sophisticated mappings between BizTalk
artifacts and a BAM activity model. The following figure illustrates the use of the TPE for
populating our sample BAM activity model. (The entire tracking profile can be found with the
samples that accompany this paper.) You can find more details about TPE in the online
documentation.
Figure: Using TPE with the transaction activity model
The mappings created by using the TPE are serialized into an XML file that can be
interpreted by the BAM components. The following code illustrates a segment of our sample
TPE file.
<TrackingProfile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" VersionGuid="00000000-0000-0000-0000-000000000000" Name="TRANSACTION"> <Dimension Name="ActivityID" DataType="TraceID" /> <Dimension Name="TRANSACTION_CREATED" DataType="DATETIME"> <DataLevel Name="Created" SourceTypeSelected="Orchestration Payload" TargetAssemblyName="BAM-BTS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=79de3c67dc9393f3" OrchestrationReference="BAM_BTS.FinancialTransactionOrchestration" ShapeID="401586f7-f645-4936-a471-64d825edf85d" MessageName="InMsg" MessagePart="part" SchemaName="BAM_BTS.TransactionRequest" MessageDirection="Out" SomXPath="/*[local-name()='<Schema>' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[local-name()='Request' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[local-name()='Created' and namespace-
uri()='']" XPath="/*[local-name()='Request' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[local-name()='Created' and namespace-uri()='']" /> </Dimension> Other elements...
</TrackingProfile>
Looking at the code we can notice that it describes how the TRANSACTION_CREATED activity
field is populated from a BizTalk message payload. The highlighted sections of the tracking
profile XML detail the orchestration shape, message, and XPath sentence that are used to
populate the TRANSATION_CREATED activity.
At this point, the TPE profile can be deployed instructing the BAM runtime where and how to
populate the different elements of the BAM activity model.
Inside the BAM BizTalk interceptorThe architecture of the BizTalk Server BAM runtime is complex enough to require a separate
paper. We will try to cover some of the most important components in this section. After it’s
deployed, a BAM tracking profile is stored in the bam_Metadata_TrackingProfiles table in the
BAMPrimaryImport database, with a reference to it in the bam_TrackingProfiles table of the
BizTalk Management database (BizTalkMgmtDb). By default the BizTalk Server runtime uses
the different interceptors that can be found in the Trackinginterceptor table of the BizTalk
Management database.
Fundamentally, the BizTalk engine uses the Microsoft.XLANGs.Mozart.NativeInterceptor and
Microsoft.XLANGs.Mozart.BIWrapperInterceptor, which inherit from the
Microsoft.XLANGs.RuntimeTypes.BaseInterceptor class. The BaseInterceptor class is the
main interface that the BizTalk Server runtime components use to interact with the BAM
infrastructure.
Figure: BizTalk Server BAM interceptor
The complete architecture of the BizTalk Server BAM runtime is far more complex than what
has been explained in this section, but we intended to illustrate some of its fundamental
components.
Using BAM from Windows Communication Foundation and Windows Workflow FoundationWith the release of BizTalk Server 2006 R2, the use of BAM has been extended to Windows
Communication Foundation (WCF) and Windows Workflow Foundation (WF). Both
technologies introduced a declarative mechanism to express the element that should be
monitored by BAM. This mechanism is based on an interceptor configuration file that
declares how the BAM activities are populated from WCF or WF elements. The interceptor
configuration files for both WCF and WF share the same structure based on the
CommonInterceptorConfiguration.xsd XML schema, which describes generic events that
need to be intercepted and mapped to BAM activities.
The common configuration file provides a schema that can be reused across the WCF and
WF BAM interceptors. However, given the differences between the WCF and WF
programming models, that common schema needs to be extended to incorporate the
specific interception mechanisms for collecting the BAM data. For instance, developers using
the BAM WCF interceptor will, most likely, extract the data from a SOAP or XML message
intercepted at different stages of the client or dispatcher runtimes. On the other hand,
developers using the WF interceptor will extract the activity data from WF-specific
components such as workflow properties or user data types. Essentially, in order to provide
a complete declarative mechanism for populating a BAM activity model, the WCF and WF
interceptors need to extend the common configuration schema with specific elements of
their programming model.
BAM WCF interceptor The BAM WCF interceptor is a flexible technology that allows developers to populate BAM
activity models based on the information exchanged by WCF services and clients. In order to
model the interactions with the BAM model, the BAM WCF interceptor extends the basic
configuration model with WCF-specific elements. Essentially, the interceptor configuration
file models how WCF messages are mapped to BAM activities throughout the different
stages of the client or dispatcher runtimes.
Let’s take the following WCF service that processes a financial transaction.
public class FinancialTransactionService: IFinancialTransactionService { public TransactionResponse ProcessTransaction(TransactionRequest transactioninfo) { //Implementation omitted for brevity.... } }
[ServiceContract] public interface IFinancialTransactionService { [OperationContract] TransactionResponse ProcessTransaction(TransactionRequest transactioninfo); }
[DataContract] public class TransactionRequest { private string transactionId; private double amount; private string customerName; private int customerAge;
[DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; }
}
[DataMember] public double Amount { get { return amount; } set { amount= value; } }
[DataMember] public string CustomerName { get { return customerName; } set { customerName= value; } }
[DataMember] public int CustomerAge { get { return customerAge; } set { customerAge = value; } } }
[DataContract] public class TransactionResponse { private string transactionId; private bool success; private bool alertFired; private string alertDescription;
[DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; } }
[DataMember] public bool Success { get { return success; } set { success= value; } }
[DataMember] public bool AlertFired { get { return alertFired; } set { alertFired= value; } }
[DataMember]
public string AlertDescription { get { return alertDescription; } set { alertDescription= value; } }
}Figure: Sample WCF service (the complete implementation can be found in the
samples included with this paper)
At this point we can create a WCF interceptor configuration file that maps the WCF
messages to the sample activity model created in the previous section. The following figure
highlights sections of the interceptor configuration file that indicate how the
TRANSACTIONID element is populated based on the contents of the upcoming message.
<ic:InterceptorConfiguration xmlns:ic="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorConfiguration" xmlns:wcf="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WcfInterceptorConfiguration">
<ic:EventSource Name="TransactionEventSource" Technology="WCF" Manifest="BAM_WCF.IFinancialTransactionService, BAM-WCF, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"> <wcf:NamespaceMappings> <wcf:Namespace Prefix="soap" Uri="http://schemas.xmlsoap.org/wsdl/soap/" /> <wcf:Namespace Prefix="cs" Uri="http://tempuri.org/"/> </wcf:NamespaceMappings> </ic:EventSource> <ic:BamActivity Name="TRANSACTION"> <ic:OnEvent IsBegin="true" IsEnd="false" Name="TransactionRequest" Source="TransactionEventSource"> <ic:Filter> <ic:Expression> <wcf:Operation Name="GetServiceContractCallPoint" /> <ic:Operation Name="Constant"> <ic:Argument>ServiceRequest</ic:Argument> </ic:Operation> <ic:Operation Name="Equals" /> <wcf:Operation Name="GetOperationName" /> <ic:Operation Name="Constant"> <ic:Argument>ProcessTransaction</ic:Argument> </ic:Operation> <ic:Operation Name="Equals" /> <ic:Operation Name="And" /> </ic:Expression> </ic:Filter>
…Rest of the configuration omitted for brevity…
<ic:Update DataItemName="TRANSACTIONID" Type="NVARCHAR"> <ic:Expression> <wcf:Operation Name="XPath">
<wcf:Argument> //*[local-name()= 'TransactionId']/text() </wcf:Argument> </wcf:Operation> </ic:Expression> </ic:Update> </ic:OnEvent> </ic:BamActivity></ic:InterceptorConfiguration>
Figure: BAM WCF interceptor configuration file (the complete file is included in the samples provided with this paper)
Looking at the preceding interceptor configuration file, we can notice that it is using a
combination of elements from the namespaces
http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorConfiguration (ic:
prefix) and
http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WcfInterceptorConfiguration (wcf:
prefix). The elements prefixed by ic: are part of the common configuration schema, which
includes the instructions that are common for the WCF and WF interceptors like updating an
activity. Additionally, the elements prefixed by wcf: refer to the elements of the WCF BAM
configuration schema, which states WCF-specific instructions like detailing the operation
name that needs to be intercepted. In that sense, the <wcf:Operation> element is the
fundamental extension of the BAM WCF interceptor schema and encloses the necessary
logic to extract the activity data from WCF messages. The interception logic can be applied
at different levels of the client and service message processing pipeline such as
ServiceRequest, ServiceReply, ClientRequest, ClientReply, etc. More details about how to
use the WCF and WF interceptors can be found on MSDN® at (http://go.microsoft.com/fwlink/?
LinkId=134485)
The interceptor configuration file can be deployed by using the BAM deployment utility with
the following arguments:
>bm.exe deploy-interceptor –filename:<Interceptor configuration file…>
The BAM interceptor configuration is added to a WCF service by using an endpoint behavior
that can be configured declaratively as the following:
<configuration> <system.serviceModel> <extensions> <behaviorExtensions> <add name="baminterceptor" type="Microsoft.BizTalk.Bam.Interceptors.Wcf.BamEndpointBehavior,
Microsoft.BizTalk.Bam.Interceptors, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/> </behaviorExtensions> </extensions> <services> <service name="BAM_WCF.FinancialTransactionService" behaviorConfiguration="ExposeContract"> <endpoint behaviorConfiguration="BAMTracking" /> </service> </services> <behaviors> <endpointBehaviors> <behavior name="BAMTracking"> <baminterceptor ConnectionString="Initial Catalog=BAMPrimaryImport;Data Source=.;Integrated Security=SSPI;" /> </behavior> </endpointBehaviors> </behaviors> </system.serviceModel></configuration>
Additionally, the BAM interceptor can be added programmatically.
private static void StartHost() { using (ServiceHost host = new ServiceHost(typeof(FinancialTransactionService))) { WCFBAMProgExtension bamExtension= new WCFBAMProgExtension(); bamExtension.PrimaryImportConnectionString = "Connection String…"; bamExtension.PollingIntervalSec= 1500; foreach(ServiceEndpoint endpoint in host.Description.Endpoints) endpoint.Behaviors.Add(bamExtension.CurrentBAMBehavior); host.Open(); Console.ReadLine(); host.Close(); } }
internal class WCFBAMProgExtension : BamBehaviorExtension { public IEndpointBehavior CurrentBAMBehavior { get { return (IEndpointBehavior)base.CreateBehavior(); } } }
As illustrated in the preceding code, adding the BAM WCF interceptor programmatically is no
different from adding any other WCF endpoint behavior to the host’s endpoint collection.
Given that the actual endpoint behavior is implemented by the BamEndpointBehavior
internal class, we need to access it through the BamBehaviorExtension class. To achieve
that, we need to create a class that inherits from BamBehaviorExtension to access the
protected CreateBehavior method that returns an instance of the BamEndpointBehavior
class. You can find more details about this in the next section.
From a developer perspective, the BAM endpoint behavior is the mechanism used to instruct
the WCF runtime that a particular service will be monitored by using BAM. The specific
instructions about how the service will be monitored are abstracted through the BAM
interceptor configuration file. To get a more detailed understanding of how all these
components fit together we should take a look at the BAM WCF interceptor architecture.
Inside the BAM WCF interceptor architectureAs we explained in the previous section, the BAM WCF interceptor leverages endpoint
behaviors as the extensibility point used to insert the BAM interception mechanisms into the
WCF client and dispatcher runtimes. When the endpoint behavior is loaded by the client or
service host, it plugs in other WCF components that execute the BAM monitoring logic. The
following figure illustrates a high-level view of the BAM WCF interceptor.
Figure: BAM WCF interceptor high-level architecture
Like any other WCF behavior, the BAMEndpointBehavior is configured by a behavior
extension element (BAMBehaviorExtension). Once instantiated, the BAMEndpointBehavior
behavior inserts a WCF message inspector (BAMMessageInspector, which inherits from the
BAMInspector class) into the client or dispatcher runtime. At run time, the message
inspector intercepts a WCF message and creates a BAM interceptor (BAMWcfInterceptor),
which interprets the interception configuration file in order to log the data to the
BAMPrimaryImport database. As you might imagine, there are other components involved in
the BAM WCF interceptor architecture, and it is not our goal to cover them all in detail. From
a developer standpoint, the complexity of this architecture is completely abstracted behind
the BAMBehaviorExtension behavior extension element.
BAM WF interceptorSimilar to the BAM WCF interceptor, the BAM WF interceptor extends the BAM programming
model into WF. Using this interceptor, developers can populate BAM activity models based
on the information processed by WF workflows. The mechanism to declare how a BAM
activity model will be populated is also based on an interceptor configuration file, which
extends the basic interceptor configuration model with specific WF elements.
To better understand how to use the BAM WF interceptor let’s start with the following
workflow that processes a financial transaction.
Figure: Sample transaction workflow
This workflow receives the financial transaction information and executes a very simple logic
to approve or reject the transaction. For this example, the fields that we would like to
monitor are declared as properties of the workflow class as illustrated in the following code.
public sealed partial class FinancialTransactionWorkflow: SequentialWorkflowActivity{ private double transactionID; ...rest of the implementation has been omitted for brevity public double TransactionID { get { return transactionID; } set { transactionID = value; } }
private void TransactionApproved_ExecuteCode(object sender, EventArgs e) { transactionID= sample transaction id….; ...rest of the implementation has been omitted for brevity }}
Figure: Transaction workflow implementation (you can find the complete implementation as part of the samples included with this paper)
In order to populate a BAM activity model based on the state of the WF workflow we need to
create a BAM WF interceptor configuration file like the one in the following figure.
<ic:InterceptorConfiguration xmlns:ic="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorConfiguration" xmlns:wf="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WorkflowInterceptorConfiguration"> <ic:EventSource Name="FinancialTransactionWorkflow" Technology="WF" Manifest="BAM_WF.FinancialTransactionWorkflow, BAM-WF, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/> <ic:BamActivity Name="TRANSACTION"> <ic:OnEvent Name="MyWorkflowEventEnd1" Source="FinancialTransactionWorkflow" IsBegin="false" IsEnd="true"> <ic:Filter> <ic:Expression> <wf:Operation Name="GetActivityName"/> <ic:Operation Name="Constant"> <ic:Argument>TransactionApproved</ic:Argument> </ic:Operation> <ic:Operation Name="Equals"/> <wf:Operation Name="GetActivityEvent"/> <ic:Operation Name="Constant"> <ic:Argument>Closed</ic:Argument> </ic:Operation> <ic:Operation Name="Equals"/> <ic:Operation Name="And"/> </ic:Expression> </ic:Filter>
…Rest of the configuration omitted for brevity… <ic:Update DataItemName="TRANSACTIONID" Type="NVARCHAR"> <ic:Expression> <wf:Operation Name="GetWorkflowProperty"> <wf:Argument>TransactionId</wf:Argument> </wf:Operation> </ic:Expression> </ic:Update>
</ic:OnEvent> </ic:BamActivity></ic:InterceptorConfiguration>
Figure: BAM WF interceptor configuration file (the complete file is included in the samples provided with this paper)
Although it is not an objective of this paper to explain in detail the different sections of the
interceptor configuration file, there are a few aspects that are worth highlighting. The BAM
WF interceptor captures different events at the workflow, activity, and user levels. On each
one of those tracking points, we can extract the data from the workflow by using some of
the custom operations provided by the interceptor. For instance, in this case, we are
interested in the Closed event of the TransactionApproved WF activity (see filter section).
When that event occurs, the interceptor will extract the TransactionID field of the
Transaction activity and map it to the TransactionID activity field. It is important to notice
that there are many other events and functions that can be combined to address diverse
BAM WF monitoring scenarios.
After the interceptor configuration file is created, it can be deployed by using the BAM
deployment utility.
>bm.exe deploy-interceptor –filename:<Interceptor configuration file…>
At this point, we can instruct the WF runtime to use the BAM WF interceptor. The interaction
between the WF runtime and the BAM WF interceptor is abstracted by the
BamTrackingService tracking service, which can be added declaratively as the following:
<WorkflowServiceContainer> <Services> <add type="Microsoft.BizTalk.Bam.Interceptors.Workflow.BamTrackingService, Microsoft.BizTalk.Bam.Interceptors, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" ConnectionString="Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport" PollingIntervalSec="5"/> </Services></WorkflowServiceContainer>
Figure: Adding the BAM tracking service declaratively
Similar to other WF tracking services, the BamTrackingService can be also added by using
code.
string connectionString = "My Connection String….";int pollingInterval = 5;WorkflowRuntime workflowRuntime = new WorkflowRuntime();workflowRuntime.AddService(new BamTrackingService(connectionString, pollingInterval));
Figure: Adding the BAM tracking service imperatively
After this, every time an instance of the WF workflow executes, the BAM WF interceptor will
populate the BAM activity model based on the instructions declared in the interceptor
configuration file. From a developer standpoint, adding the BAM WF interceptor is no
different from adding any other WF tracking service. This service abstracts the complexities
of the BAM WF interceptor architecture that we will explore in the following section.
Inside the BAM WF interceptor architecture
As explained in the previous section, the BAM WF interceptor is based on the WF tracking
services framework. This framework is designed to enable hosts to observe workflow
instances during execution by capturing events that are raised during workflow execution.
The framework is based on a pluggable design pattern that enables hosts to facilitate
complex tracking scenarios by including different tracking services. At a high level, a
tracking service uses two fundamental components: tracking profiles and tracking channels.
Tracking profiles are used by the tracking service as a mechanism to communicate the
events it would like to receive from the runtime. This guarantees that the tracking service
will only be called for specific events instead of the entire event set produced by a workflow
during its execution. The second component used by the tracking service is the tracking
channel. These components are responsible for receiving the tracking events and data about
the execution of a workflow instance.
With this knowledge of a tracking service model it is not difficult to imagine what the high-
level architecture of the BAM WF interceptor looks like.
Figure: High-level architecture of the BAM WF interceptor
In the previous section we explained how applications need to add the BAM tracking service
(BamTrackingService class) in order to indicate to the WF runtime that they want to monitor
certain events by using BAM. At this point, the WF runtime calls the GetProfile operation of
the BamTrackingService, which uses the BamProfileManager class to instantiate a tracking
profile based on the configuration declared in the interceptor configuration file. The WF
runtime also invokes the GetTrackingChannel operation of the BamTrackingService in order
to get the tracking channel that will be used to receive the events. During its construction,
the BamTrackingChannel instantiates a BAM interceptor (WorkflowInterceptor class) object,
which abstracts the interaction with the BAM event streaming APIs. At run time, the data is
sent to the tracking channel, which uses the interceptor to efficiently populate the BAM
activity model.
Using BAM programmatically Undoubtedly, the use of declarative mechanisms like the ones provided by BizTalk Server,
WCF, and WF eliminates a lot of the complexities in the implementation of BAM solutions.
However, one of the tradeoffs with using those mechanisms is that it requires a static
correlation between the application data and the BAM activity model. Often, applications
need to interact with BAM in a more dynamic manner that can’t be easily described in a
declarative way. Those scenarios are a better fit for the use of BAM APIs as mechanisms to
dynamically populate the BAM activity model.
In the current BAM release there are two fundamental APIs that developers can use to
programmatically populate activity models: streaming and interceptor. As you might
imagine, both APIs are the foundation of the runtime mechanisms implemented by BizTalk
Server, WF, and WCF.
BAM event stream API
The event stream API is the fundamental mechanism that developers use to
programmatically interact with a BAM activity model. These APIs provide the interfaces to
execute operations like those that modify the state of a BAM activity model. The following
table summarizes some of the fundamental operations of the BAM EventStream API.
Operation signature(C#) Description
public virtual void BeginActivity (string activityName,string activityInstance
)
Instantiates a new BAM activity with a
specific activity identifier
public virtual void EndActivity (string activityName,string activityInstance
)
Completes a specific instance of a BAM
activity
public virtual void UpdateActivity (string activityName,
Updates fields of a BAM activity
string activityInstance,params Object[] data
)
public virtual void EnableContinuation (string activityName,string activityInstance,string continuationToken
)
Enables continuation for a BAM activity
instance
public virtual void Flush () Flushes the changes to the BAM database
Although the operations listed in the preceding table are not the only ones included in the
BAM event streaming API, they summarize some of the most common functionalities needed
to interact with a BAM activity model. Those interactions can be notably different based on
different factors such as the invoking application (orchestration, pipeline, component, etc.),
synchronization model (synchronous vs. asynchronous), or the information spectrum
(message, context properties, etc). This is why the current release of BizTalk Server 2006 R2
includes the following variations of the event stream API that are optimized for specific
scenarios.
DirectEventStream
From an infrastructure perspective, the DirectEventStream class is the simplest way to
interact with a BAM activity model. This class is used to log BAM activity data synchronously
to the BAMPrimaryImport database. The following code illustrates a simple case of how to
use this API.
string activityID = Guid.NewGuid().ToString();DirectEventStream eventStream = new DirectEventStream("BAM Primary Import DB connection string…", 1);eventStream.BeginActivity("TRANSACTION", activityID);eventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", “123456”, "TRANSACTION_CREATED",DateTime.UtcNow);eventStream.EndActivity("TRANSACTION", activityID);eventStream.Flush();
Figure: Using DirectEventStream
The synchronous nature of this API can have a direct impact on the performance of the
calling application. After updating a BAM activity, the calling application normally needs to
wait until all the database artifacts, such as stored procedures and RTA triggers associated
with that activity, execute. This API is normally recommended for applications that need
near real-time processing and minimum latency in their BAM infrastructure.
BufferedEventStream
The BufferedEventStream class is a great alternative in order to mitigate some of the
performance problems that might occur when using the DirectEventStream class. Instead of
logging the data directly to the BAMPrimaryImport database, the BufferedEventStream
inserts the data asynchronously into the BizTalk MessageBox database. The activity data is
stored in an optimized binary format into specific tables named
Tracking_Data_<number>_<number>. After that, the BAM Event Bus service can extract
the data and insert it into the appropriate tables in the BAMPrimaryImport database. The
event bus keeps transaction consistency throughout the entire process, which guarantees
the integrity of the data. The following code shows a basic example of how to use the
BufferedEventStream class.
string activityID = Guid.NewGuid().ToString();BufferedEventStream eventStream = new BufferedEventStream ("BAM Message Box DB connection string…", 1);eventStream.BeginActivity("TRANSACTION", activityID);eventStream.UpdateActivity("TRANSACTION ", activityID, "TRANSACTIONID", “123456”, "TRANSACTION_CREATED",DateTime.UtcNow);eventStream.EndActivity("TRANSACTION", activityID);
Figure: UsingBufferedEventStream
The BufferedEventStream API can be really advantageous for applications that need to
maximize the performance of the interactions with the BAM API. As a tradeoff, the use of
asynchronous mechanisms introduces an extra level of latency on the processing of the BAM
activity data.
OrchestrationEventStream
The OrchestrationEventStream (OES) is one of the artifacts developers should consider in
order to programmatically populate a BAM activity model from within a BizTalk
orchestration. The OrchestrationEventStream (OES) class provides similar mechanisms to
the ones implemented by the BufferedEventStream (BES) class. In that sense, the OES class
also uses an asynchronous model to first log the data to the BizTalk MessageBox database
so that an external process can move it to the BAMPrimaryImport database.
OrchestrationEventStream.BeginActivity("TRANSACTION", activityID);OrchestrationEventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", “123456”, "TRANSACTION_CREATED",DateTime.UtcNow);OrchestrationEventStream.EndActivity("TRANSACTION", activityID);
Figure: Using OrchestrationEventStream
The fundamental difference between the OES and the BufferedEventStream is that the
former is optimized to work in the context of an orchestration. In that sense, the activity
data created by using the OES class will be persisted when the orchestration reaches a
persistent point. For instance, assume we call the OES class within an Expression shape and
that is followed by a Send shape (persistent point). If an exception occurs before we reach
the Send shape, the BAM data won’t be persisted because the orchestration will be
rehydrated from the previous persistent point. This behavior is different from the BES, which
persists the data as soon as the Flush operation is called.
MessagingEventStream
The MessagingEventStream (MES) API follows a similar approach to the
OrchestrationEventStream (OES) and BufferedEventStream (BES) with the difference that
MES is optimized for processing the BAM activity within a pipeline context. In that sense, the
BAM activity data is persisted within a pipeline transaction. The following code illustrates the
use of MES.
string activityID = Guid.NewGuid().ToString();IPipelineContext context;EventStream eventStream = context.GetEventStream();eventStream.BeginActivity("TRANSACTION", activityID);eventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", “123456”, "TRANSACTION_CREATED",DateTime.UtcNow);eventStream.EndActivity("TRANSACTION", activityID);
Figure: Using MessagingEventStream
Using the BAM interceptor API
As we’ve explored in the previous section, the BAM event stream APIs abstract different
interaction models used by applications to populate BAM activities. When using those APIs,
applications still need to implement the logic that captures the data of interest and map it to
the BAM activity model. BAM streamlines this process by using the BAM interceptor API,
which allows developers to model how an application will be instrumented by using BAM.
The first task in instrumenting an application with a BAM interceptor is to create a BAM
configuration (ActivityInterceptorConfiguration) that declares a set of relevant “steps” that
need to be monitored. Each step also expresses what data needs to be captured and what
element of the BAM activity model should be populated. The following code illustrates a
sample configuration for our activity model.
ActivityInterceptorConfiguration configuration = new ActivityInterceptorConfiguration("TRANSACTION");configuration.RegisterStartNew("Start", "");configuration.RegisterDataExtraction("TRANSACTIONID", "LogInfo", "//*[local-name()= 'TransactionID']");configuration.RegisterDataExtraction("TRANSACTION_CREATED", "LogInfo", "//*[local-name()= 'Transaction_Created']");configuration.RegisterEnd("End");BAMInterceptor interceptor= new BAMInterceptor();configuration.UpdateInterceptor(interceptor);
Figure: Configuring a BAM interceptor
In the preceding code, you can see that the configuration does not express how the data
should be extracted from the application data structures (XML, objects, etc.) in order to
populate the corresponding BAM activities. A BAM interceptor typically delegates this
process to an implementation of the IBAMDataExtractor interface provided by the
application. The following code shows a sample BAM data extractor that uses XPath to
extract the relevant information from an XML document.
public class SampleDataExtractor : IBAMDataExtractor{ public object GetValue(object trackItem, object location, object data) { if (location == "Start") return Guid.NewGuid().ToString(); string xpath = (string)trackItem; if (xpath.Length > 0) { XPathNavigator xn = (XPathNavigator)data; XPathNavigator dataItem = xn.SelectSingleNode(xpath); if (null == dataItem) return null; else return dataItem.InnerXml; } else return DateTime.UtcNow; } }
Figure: Sample IBAMDataExtractor implementation
At run time the application calls the OnStep operation of the BAM interceptor for each
relevant step, passing the BAM data extractor as a parameter. At that point the interceptor
will call the GetValue method of the BAM data extractor and map the results to the specific
BAM activity fields declared in the ActivityInterceptorConfiguration object.
MemoryStream stream = new MemoryStream();
byte[] buffer= Encoding.UTF8.GetBytes(@"<Transaction> <amount>10000</amount> <customer>John Doe 2</customer> <creditcard>8829749823748</creditcard> </Transaction>");
stream.Write(buffer, 0, buffer.Length); stream.Seek(0, SeekOrigin.Begin); XPathDocument msg = new XPathDocument(stream); XPathNavigator navigator= msg.CreateNavigator(); BufferedEventStream eventStream= new BufferedEventStream("MsgBox DB connection string", 1);
ActivityInterceptorConfiguration configuration = new ActivityInterceptorConfiguration("TRANSACTION"); configuration.RegisterStartNew("Start", ""); configuration.RegisterDataExtraction("AMOUNT", "LogInfo", "//*[local-name()= 'amount']"); configuration.RegisterDataExtraction("CUSTOMER_NAME", "LogInfo", "//*[local-name()= 'customer']"); configuration.RegisterDataExtraction("CREDITCARD", "LogInfo", "//*[local-name()= 'creditcard']"); configuration.RegisterEnd("End");
BAMInterceptor interceptor= new BAMInterceptor(); configuration.UpdateInterceptor(interceptor); SampleDataExtractor extractor= new SampleDataExtractor(); interceptor.OnStep(extractor, "Start", navigator, eventStream); interceptor.OnStep(extractor, "LogInfo", navigator, eventStream); interceptor.OnStep(extractor, "End", navigator, eventStream);
Figure: Using the BAM interceptor
The combination of the BAM interceptor, configuration, and data extraction interfaces serves
as the foundation for the declarative models provided by BizTalk Server, WCF, and WF.
Those technologies provide specific implementations of the BAM interceptor API that
facilitate the instrumentation of applications using complete declarative models.
Inside the BAM Infrastructure Monitoring is by nature a heterogeneous process. Different applications deployed on the
same infrastructure can have completely different business semantics, and consequently
the BAM models for those applications will be completely different. To address this
heterogeneity of BAM models, the BAM infrastructure creates a series of model-specific
components that keeps a level of isolation between the data infrastructure of the different
activity models. In that sense, each activity model will interact with specific artifacts such as
tables, views, stored procedures, SSIS packages, dimensions, cubes, etc. that are created
when an activity model is deployed. These artifacts are deployed across different parts of
the BAM infrastructure such as the BAMArchive, BAMPrimaryImport, and BAMStarSchema
relational databases, BAMAnalysis multidimensional database, and SQL Server Integration
Services. In this section, we will take an in depth look at some of those components and the
relationships among them.
Storing BAM data: BAMPrimaryImport databaseThe BAMPrimaryImport database is the main data store that contains the data structures to
store BAM activities. The most important characteristic of this database is that it is
partitioned by a set of tables, views, and stored procedures that are specific to a BAM
activity. If you think in database terms, you can think of the BAMPrimaryImport database as
a first-level multitenant model in which each tenant is represented by a BAM activity. In that
sense, when we deploy an activity model a set of new tables, views, and stored procedures
are created that keep the interaction with different activities in complete isolation.
Activity tablesThis set of tables form the fundamental component of the BAM activity infrastructure
because they store the records related to a BAM activity model. When deploying an activity
model, multiple tables are created to store the instances of the BAM activities and, at the
same time, provide a level of isolation between different activities. Some of the fundamental
tables are summarized in the following list:
bam_<Activity Name>_Active: This table stores the active instances of a specific
BAM activity.
bam_<Activity Name>_Completed: This table stores the completed instances of a
specific BAM activity.
bam_<Activity Name>_ActiveRelationships: This table stores the active
relationship references of a specific BAM activity.
bam_<Activity Name>_CompletedRelationships: This table stores the
completed relationship references of a specific activity instance.
bam_<Activity Name>_Continuations: This table stores the continuations of a
specific activity.
Activity viewsThe BAM activity views are also created when an activity model is deployed as a mechanism
to abstract the interaction with the activity tables. These views enable two key
functionalities in the BAM infrastructure: At the first level, activity views are the fundamental
mechanism to expose BAM data to other applications, including other components of the
BAM infrastructure like the aggregation process. Additionally, these views include some of
the calculations required by elements such as measures that are not directly included in the
activity tables. Some of the most relevant BAM activity views are summarized in the
following list:
bam_<Activity Name> _ActiveInstances: Returns the list of the active activity
instances based on the records stored in the active activity table.
bam_<Activity Name> _CompletedInstances: Returns the list of the completed
activity instances based on the records stored in the completed activity table.
bam_<Activity Name>_AllInstances: Joins the active and complete activity views,
returning a list of all the BAM activity instances.
bam_<Activity Name> _AllRelationships: Joins the active and complete activity
relationship tables, returning a list of all the relationships of the BAM activity
instances.
The views listed above are directly derived from the deployment of a BAM activity.
Additionally, a new set of views are created for each activity view that we include in the BAM
activity model. Some of the most important views are summarized below:
bam_<ActivityName>_<ActivityView Name>_ActiveView: Returns the list of active
activity instances including the calculations of the required dimension levels.
bam_<ActivityName>_<ActivityView Name>_CompletedView: Returns the list of
completed activity instances including the calculations of the required dimension
levels.
bam_<ActivityName>_<ActivityView Name>_View: Joins the two previous views,
returning the list of all activity instances included the calculation of the required
dimension levels.
The BAMPrimaryImport database serves as the first host for the activity instance data. As
those instances complete, the data needs to be archived and aggregated in order to keep-
up the performance of the BAM infrastructure.
Archiving BAM Data: BAM Archive databaseThe BAM Archive database (BAMArchive) stores the historical business data that is recycled
from the BAM Primary Import database. This database is used mostly as a mechanism for
controlling the performance of the applications interacting with the BAM Primary Import
database without losing the historic business data. From the activity schema standpoint, the
table structure of the BAM Archive database is relatively similar to the BAM Primary Import
database. The main differences rely on the fact that the BAM Archive database does not
keep a separate set of tables per state (Active, Completed) because all the activities are
supposed to be completed at this stage. We can find the following tables in the BAM Archive
database; note that each run of the SSIS package generates additional tables by suffixing the base
activity table with the time stamp of the run. These tables contain the data archived from the latest run:
bam_<Activity Name>_Instances: Stores all the recycled activity instances.
bam_<Activity Name>_Relationships: Stores all the recycled relationships.
bam_<Activity Name>_Active_Uncompleted: Stores all the uncompleted activity
instances.
bam_<Activity Name>_ActiveRelationships_Uncompleted: Stores all the
uncompleted activity relationships.
The process of archiving BAM data is executed by the BAM_DM_<Activity Name> SQL Server
Integration Services (SSIS) package that is also generated as part of the activity model
deployment process. We will explore the details in the next section.
Partitioning BAM dataAlthough the recycling of data between the BAM Primary Import and BAM Archive databases
is a key process of the BAM infrastructure, it is not the only mechanism used to partition the
BAM data in order to control performance. Consider a high-throughput scenario in which a
large number of activities are being logged to the BAM system. In that scenario, the number
of completed activities can rapidly increase, affecting the performance of the applications.
To address this, the BAM infrastructure uses an algorithm to partition the completed
activities based on its time stamp. In essence, when that algorithm executes it will swap the
completed instances table with an empty table that has an identical structure. This
mechanism allows controlling the number of records stored in the completed instances
table, which directly impacts the performance of BAM applications. The following diagram
illustrates that concept.
Figure: Conceptual view of the BAM partition process
Given that the BAM infrastructure uses specific tables and stored procedures per activity, it
is obvious that the process of recycling and partitioning that data needs to also be specific
per activity because they depend on activity-specific database artifacts. The time window
that specifies how often the data needs to be partitioned can be specified by using the
bm.exe tool. In the current release of BAM, these processes are controlled by a SQL Server
Integration Services (SSIS) package that is generated when an activity is deployed. This SSIS
package controls both the partition of the activity instances within the BAMPrimaryImport
database and the recycle of the activity instances to the BAMArchive database. The
following figure illustrates the SSIS package generated for our sample BAM activity.
Figure: BAM activity partition and archiving SSIS package
In the previous figure, we’ve highlighted a section of the SSIS package. The first section
executes the bam_Metadata_SpawnPartition stored procedure, which creates a new set of
tables to store the completed activity instances and relationships. To determine the name of
the new tables, this stored procedure appends a GUID to the name of the original tables. For
instance, if the original set of tables are bam_<Activity Name>_Completed and
bam_<Activity Name>_CompletedRelationships, every time we run the BAM_DM_<Activity
Name> SSIS package it will create a new set of tables with the names bam_<Activity
Name>_GUID and bam_<Activity Name>_GUID_Relationships. Additionally, the SSIS
package will modify other artifacts of the BAM databases in order to incorporate these new
tables. For instance, the bam_<Activity Name>_CompletedInstances and bam_<Activity
Name>_CompletedRelationships views need to be reconstructed to reflect the records
stored in the new tables. The following code illustrates the SQL script of the completed
instances view after the SSIS package executes. As you can see, the new SQL statement will
query all the records in both tables, keeping the consistency from the client perspective.
ALTER VIEW [dbo].[bam_<Activity Name>_CompletedInstances] AS
SELECT * FROM dbo.[bam_<Activity Name>_Completed] WITH (NOLOCK)
UNION ALL
SELECT * FROM [dbo].[bam_<Activity Name>_GUID] WITH (NOLOCK)
The second part of the SSIS package (red) is responsible for archiving the records stored in
the BAMPrimaryImport database into the BAMArchive database. This process will select the
records from the bam_<Activity Name>_InstancesForArchive and bam_<Activity
Name>_RelationshipsForArchive views and store them into the bam_<Activity
Name>_Instances and bam_<Activity Name>_Relationships tables in the BAMArchive
database respectively.
The BAM infrastructure uses the mechanisms described in this section to partition and
archive the activity instances in order to control the performance of BAM applications. It is
the responsibility of the BizTalk Server administrator to correctly schedule the execution of
the SSIS packages by implementing the corresponding SQL jobs. Among the direct
consumers of the BAM relational data stored in the BAMPrimaryImport database are
infrastructure mechanisms that aggregate that data into multidimensional formats in order
to provide the foundation for more complex business intelligence processes.
Aggregating BAM data The purpose of BAM data is to monitor real-time information that describes the behavior of
different business processes. Over time, that real-time information can be aggregated to
enable more sophisticated business intelligence mechanisms that analyze the historical
behavior of the business processes. Along those lines, BAM provides the ability of
aggregating activity data in multidimensional formats such as OLAP cubes. There are two
fundamental mechanisms for implementing those aggregations: scheduled and real-time.
Real-time aggregationsReal-time aggregations (RTA) are a simpler type of aggregation that provides relational
views of the multidimensional model. This type of aggregation is typically used on time-
sensitive scenarios in which the information should be aggregated in real time. Obviously,
the fact that the aggregated information is stored in a relational (and not multidimensional)
representation limits the complexity of the business intelligence procedures that can be
applied to the BAM data. When designing a BAM activity model, you can declare a real-time
aggregation based on a BAM view as illustrated in the following figure.
Figure: Creating a real-time-aggregation using the BAM Excel Add-in
When the activity model is deployed, the BAM infrastructure creates a separate table and
view, named bam_<View Name>_<Pivot Table Name>_RTATable and bam_<View
Name>_<Pivot Table Name>_RTAView respectively, in the BAMPrimaryImport database to
store the aggregated information. The purpose of the RTA table is to store the activity
instance data partitioned by the different dimension levels. Specifically, the RTA table will
contain a column for each level of the dimensions included in the BAM view. For instance, if
as a part of a BAM view you are using a time dimension with levels Year, Month, and Week
the RTA table will include the following columns: <Dimension Name>_Year, <Dimension
Name>_Month, and <Dimension Name>_Week. The RTA table does not calculate the
measures needed to aggregate the data; that part is done by the RTA views, which include
the T-SQL statements needed to calculate all the measures of the BAM view.
Additionally, the BAM infrastructure adds SQL Server triggers to the activity Completed and
Active tables in order to aggregate the information into the RTA table. These triggers
execute the T-SQL statements that extract the information from the activity tables, calculate
the different dimension levels, and insert the records in the RTA table. The following figure
illustrates the RTA aggregation process.
Figure: Real-time aggregation process
The aggregated data is stored in the RTA table for a predefined period of time before it is
deleted. This time interval can be configured by modifying the
bam_Metadata_RealTimeAggregations table in the BAMPrimaryImport database.
Although the use of RTA represents a great value in providing a real-time snapshot of the
business process metrics, the use of triggers can directly affect the performance of the BAM
applications. Well-designed BAM architectures limit the use of RTA to the absolute key real-
time metrics and use scheduled aggregations for more complex aggregation procedures
that are not completely time-sensitive.
Scheduled aggregationsAs discussed in the previous section, the use of relational structures constrains the type of
aggregations that can be implemented by using RTAs. To address more complex scenarios,
BAM provides a second type of aggregation technique that takes full advantage of
multidimensional structures like OLAP cubes. These types of aggregations are known as
scheduled aggregations and can serve as a foundation for implementing complex business
intelligence procedures using BAM activity data.
The scheduled aggregation data model is based on a set of multidimensional structures
stored in the BAMAnalysis database, which are also based on a star schema relational
topology stored in the BAMStarSchema database. When deploying an activity model, the
BAM infrastructure adds an OLAP cube to the BAMAnalysis database for each activity view
that contains multidimensional structures like dimensions or measures. The dimensions of
that cube are based on the dimensions included in the activity views while the cube cells are
based on the measures.
Following best practices of multidimensional database modeling, BAM uses a third database
to adapt the pure relational structure of the BAMPrimaryImport database to the
multidimensional structure of the BAMAnalysis database. This is precisely the role of the
BAMStarSchema database, which is based on a classic star schema model that represents a
set of aggregated views of the information stored in the BAMPrimaryImport database. From
a single-view perspective, the conceptual model of the BAMStarSchema database looks like
the following:
Figure: Conceptual model of the BAMStarSchema database
The most important component of the BAMStarSchema database is the bam_<View
Name>_Facts table, which stores the activity instances that need to be processed into the
OLAP cube. That table keeps relationships with different dimension tables that are
generated based on the dimensions included in the activity model. The BAM infrastructure
creates a dimension table for each dimension declared as part of an activity view. For
instance, if you declared a time dimension named MonitoredTime, a dimension table name
bam_<View Name>_Dim_MonitoredTime will be created in the BAMStarSchema database.
Additionally, two views are created to filter the completed and active facts based on the
state of a specific activity.
This model represents a classic star schema structure with the fact views at the center and
the dimension tables at the edges. BAM leverages this structure as the foundation of the
OLAP cubes included in the BAMAnalysis database. The dimensions of the cubes are directly
based on the dimension tables of the BAMStarSchema database and the data used for the
measures is based on the fact views. The following figure illustrates the OLAP cube
corresponding to the activity view used in our example.
Figure: Transaction OLAP cube view
Although the previous figure is specific to our scenario, it reflects the conceptual structure of
the OLAP cubes stored in the BAM database. In that structure, the fact data is directly
extracted from the Active and Completed fact views in the BAMStarSchema database while
the dimension tables are modeled based on the dimension tables of the same database.
To populate the corresponding cubes, the activity data needs to be transferred from the
BAMPrimaryImport database into the BAMStarSchema database. After that, the OLAP cubes
can be processed to incorporate the new BAM activity information. The data exchange
process is executed by a SQL Server Integration Services (SSIS) package that is created
when the activity model is deployed. The BAM infrastructure creates specific SSIS packages
for each activity view that includes multidimensional artifacts and is not marked for real-
time aggregations. The name of the package is AN_<Activity View> and it contains all the
logic for moving activity data between the BAMPrimaryImport and BAMStarSchema
databases as well as for populating the OLAP cubes associated with a specific activity view.
Figure: BAM analysis SSIS package
The first part (highlighted green) of the SSIS package extracts the BAM activity data from
the bam_<View Name>_CubingSnapshot view in the BAMPrimaryImport database and
places it into the bam_<View Name>_Staging table in the BAMStarSchema database. After
that, the second part of the package (highlighted blue) inserts the data into the bam_<View
Name>_Facts tables from where it is directly reflected in the bam_<View
Name>_ActiveFacts and bam_<View Name>_CompleteFacts tables. Finally, the cube
dimensions are reprocessed with the new information.
From the infrastructure perspective, there is no doubt that scheduled aggregations are more
complex than real-time aggregations. On the other hand, scheduled aggregations can be a
great enabler for more sophisticated business intelligence procedures such as the
multidimensional BAM data that can be used as the source for data mining algorithms or
complex MDX reports.
The BAM Event Bus The BAM Event Bus service, also known as the Tracking Data Decode Service (TDDS), is a
fundamental component of the BAM infrastructure that processes data streams stored in the
BizTalk Message Box database (BizTalkMsgBoxDb) and inserts them into other databases for
easy processing. Specifically, TDDS is responsible for processing business monitoring data
into the BAMPrimaryImport database and health monitoring data into the BizTalkDTADb
database. From a BAM perspective, TDDS is the main enabler of the asynchronous event
streaming mechanisms used by objects like BufferedEventStream,
OrchestrationEventStream, and MessagingEventStream. When developers update activities
using those APIs the data gets inserted into a set of tracking tables within the
BizTalkMsgBoxDb database. After that, one of the TDDS instances associated with that
database will move the data into the BAMPrimaryImport database.
The process of transporting the tracking data is fairly sophisticated because it involves
algorithms that leverage a bookmarking scheme for TDDS resource coordination and
transaction integrity. On large BizTalk Server topologies it is common to find multiple BAM
Event Buses that move data between various BizTalkMsgBoxDb and BAMPrimaryImport
databases. For more information about BizTalk Server architectures see
http://go.microsoft.com/fwlink/?LinkId=134550.
Figure: BAM Event Bus sample topology
The configuration of the BAM Event Bus topology is stored centrally in the BizTalk
Management database (BizTalkMgmtDb). Each BAM Event Bus in the topology periodically
sends a “heartbeat” message, containing information about its current state, to the
management database. As a result the event bus will obtain information about the data that
needs to be moved and if possible it will take ownership of the task.
The BAM activity data is stored in the BizTalkMsgBoxDb in a partition structured across a set
of tables prefixed with dbo.TrackingData_<>. The data is stored in a binary format and
partitioned to optimize the performance of read/write operations. The BAM Event Bus will
process the different partitions of the activity data within the scope of a session. During this
process the data is extracted from the tracking tables in the BizTalkMsgBoxDb database, de-
serialized, and inserted into the activity tables in the BAMPrimaryImport database. The
entire process runs in isolation from other BizTalk Server processes and is highly optimized
for performance.
Querying BAM Data
OverviewThe primary benefit of Business Activity Monitoring (BAM) is its ability to provide intelligent
domain-specific metrics that reflect run-time behavior of business processes. These benefits
are ultimately reflected in the way applications and business users consume BAM activity
data in order to make more intelligent decisions about their business processes. Given the
universal nature of BAM, the scenarios for consuming BAM data can encompass a large
variety of products and technologies, but fundamentally we will use the same principles and
technologies we use for querying relational and multidimensional data sources. This is
mostly due to the fact that BAM does not introduce any new mechanisms for querying
activity models and, instead, relies on traditional data access technologies.
As we explored in the previous section, the BAM activity data is stored in two fundamental
models: relational and multidimensional. Both models offer different perspectives of the
information described by BAM activities and, at the first level, determine the type of
technologies developers can use to interact with BAM data. For instance, an application that
needs a real-time view of a business process will, most likely, rely on relational data access
technologies like ADO.NET and T-SQL to query elements such as activity or real-time
aggregation views stored in the BAMPrimaryImport database. However, if the application
requires a historic view of a business process it will, most likely, leverage multidimensional
data access mechanisms such as ADOMD.NET, Multidimensional Expressions (MDX), or XML
for Analysis (XMLA).
Although using the right data access technique is a fundamental step for implementing
applications that consume BAM data, it is typically an enabler for leveraging BAM data into
business intelligence applications that include capabilities like reporting, collaboration, or
performance analysis. To incorporate those features, developers need to combine BAM with
technologies such as SQL Server Reporting Services, Office SharePoint Server, or Office
Performance Point Server, which can complement BAM activity models with a richer set of
domain-specific capabilities such as KPI modeling, data composition, collaboration, etc. In
both cases, adopting the right technology for the correct scenario can be a key aspect of the
success or failure of a BAM implementation.
In this section we are going to explore some of the most common techniques, products, and
best practices that enable developers to consume BAM activity models.
Using BAM with data access librariesAs we explained in the previous section, the supported mechanism for querying BAM activity
models is interacting directly with the BAMPrimaryImport views and the BAMAnalysis cubes.
As in any other data-driven application, developers can use their data access programming
models for querying the BAM relational and multidimensional databases. In .NET
environments, developers can use ADO.NET-related technologies for querying the data
reflected in the BAM activity views stored in the BAMPrimaryImport database. To query the
BAM aggregations stored in the BAMAnalysis database, a developer can leverage
technologies such as the ADOMD.NET client and server programming models. The use of
ADOMD.NET is particularly interesting because it can leverage multiple query languages
such as Multidimensional Expressions (MDX), Data Mining Extensions (DMX), or even a
limited syntax of SQL. The following code fragment shows how to query our
multidimensional BAM model by using ADOMD.NET and MDX.
AdomdConnection connection = new AdomdConnection("Data Source=localhost");connection.Open();AdomdCommand command= connection.CreateCommand();command.CommandText = @" SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim]. [Day].ALLMEMBERS * DESCENDANTS ([TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE";
CellSet bamCellSet= command.ExecuteCellSet();TupleCollection tuplesOnColumns = bamCellSet.Axes[0].Set.Tuples;TupleCollection tuplesOnRows = bamCellSet.Axes[1].Set.Tuples;for (int row = 0; row < tuplesOnRows.Count; row++){ for (int col = 0; col < tuplesOnColumns.Count; col++) Console.WriteLine("Cell Column {0}, Row {1}, : {2}", tuplesOnColumns[col].Members[0].Caption, tuplesOnRows[row].Members[0].Caption, bamCellSet.Cells[col, row].FormattedValue + "\t"); } connection.Close();
Figure: Querying BAM using ADOMD.NET
The capabilities of both ADO.NET and ADOMD.NET are very well documented and there is no
reason to deep dive into their details in this article. It is important to notice that although
the use of technologies like ADO.NET and ADOMD.NET represents a powerful vehicle for
interacting with the BAM databases, their use is often an enabler for more sophisticated BAM
applications like the ones built by using Microsoft Office SharePoint Server or SQL Server
Reporting Services.
Using BAM with SQL Server Reporting ServicesSQL Server Reporting Services (SSRS) represents a natural choice to create comprehensive
reports based on a BAM activity model. By using SSRS, developers can create reports based
on the relational activity data stored in the BAMPrimaryImport database as well as reports
that reflect the multidimensional activity aggregations stored in the BAMAnalysis database.
Specifically, in the case of multidimensional databases, SSRS can reflect the result of
advanced business intelligence (BI) techniques such as data mining or key performance
indicators executed over the BAM activity data. Additionally, SSRS is commonly used to
create reports that combine the BAM activity data with external data sources to provide a
more comprehensive view of the business models.
Creating SSRS reports based on BAM activity models is no different than interacting with
other data sources. Typically, developers will extract the relational information (T-SQL) from
the activity views in the BAMPrimaryImport database and the multidimensional information
(MDX) from the cubes in the BAMAnalysis database.
To create an MDX-based report based on our activity model we can use the query builder
tool included in the SSRS project template. In this case we are segmenting the
TRANSACTION metrics based on the TRANSACTION_STG_Dim and TRANSACTION_TIME_Dim
dimensions.
Figure: SSRS query builder
Alternatively we can create custom MDX queries like the following.
SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON
COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim].
[TRANSACTIONS_TRANSACTION_TIME_Dim].[Day].ALLMEMBERS *
DESCENDANTS([TRANSACTIONS_TRANSACTION_STG_Dim].
[TRANSACTIONS_TRANSACTION_STG_Dim].[Level 02].ALLMEMBERS) ) } DIMENSION
PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME,
LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE, BACK_COLOR,
FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS
Figure: BAM MDX query
Once deployed, the report can be accessed using an SSRS report server URL as illustrated in
the following figure.
Figure: Using BAM from SQL Server Reporting Service
Although the use of the SSRS report server is a great fit in some scenarios, developers often
find themselves integrating SSRS reports into other Web applications. To enable those
scenarios, SSRS includes APIs and technologies that allow heterogeneous applications to
consume the data exposed by SSRS reports. For instance, ASP.NET applications can interact
with SSRS reports by using the Report Viewer controls. SSRS reports can also be accessed
through Microsoft Office SharePoint Server (MOSS) applications by using the Reporting
Services Add-in for SharePoint Technologies.
Using SSRS and other reporting technologies provides powerful capabilities to extract
meaningful information from a BAM activity model and enhance it with other artifacts such
as charts, delivery mechanisms, etc. Although the features of reporting technologies can be
used ubiquitously across various scenarios, there are several cases that require the use of
domain-specific technologies in areas like data mining or KPI analysis. In this case,
performance management scenarios for the combination of BAM and PerformancePoint
Server opens the door for many creative solutions.
Using BAM with Microsoft Office PerformancePoint Server
One of the most common uses of BAM in real-world scenarios is to provide metrics to
evaluate the run-time behavior of different business processes. Quite often, the evaluation
criteria are defined by business experts using techniques like Key Performance Indicators
(KPI) or other value segmentation techniques. The purpose of these procedures is to provide
business analysts or executives with a visually intuitive way to compare the real-time
behavior of a business process against strategic and operational performance goals. Generally,
this technique is known as performance management and is one of the most widely
established business intelligence (BI) disciplines among large corporations. Within the
Microsoft technology stack, Office PerformancePoint Server is a suite of tools and
technologies that allows customers to complement enterprise data with performance
management artifacts such as scorecards, dashboards, management reporting, or analytics.
PerformancePoint Server, and especially Monitoring Server, is a great vehicle to analyze and
report the BAM activity data against well-established performance management criteria.
Given that PerformancePoint Server interacts naturally with SQL Server Analysis Services
(SSAS), it can leverage the BAM data stored in the BAM Analysis database. Although it is not
an objective of this paper to provide a complete tutorial of PerformancePoint Server, the
following section will describe some of the fundamental techniques developers can use to
integrate BAM data into PerformancePoint Server.
To make PerformancePoint Server aware of the multidimensional structures of a specific
BAM activity model, we can import the specific activity view cube into PerformancePoint
Server Dashboard Designer. We can easily accomplish that by using the Create a Scorecard
wizard against the specific cubes we are interested in. As part of that process we can
generate PerformancePoint Server KPIs based on the measures included in the BAM activity
model. The following figure presents a snapshot of that process for our sample activity
model.
Figure: Creating PerformancePoint Server KPIs based on BAM activity data
Once imported, the KPIs can be adjusted to the specific criteria relevant to the business
process. When all the KPIs are correctly configured, we can combine them in a scorecard.
The following figure represents a snapshot of the design process of a scorecard that uses
the measures provided on our sample BAM activities.
Figure: Designing a PerformancePoint Server scorecard using BAM activity data
At this point we can include the scorecard in a PerformancePoint Server dashboard and
publish it to PerformancePoint Server Monitoring Server. This will make the dashboard
accessible to various applications including office applications like Microsoft Office Outlook®
or Web applications like Microsoft Office SharePoint Server (MOSS). The use of MOSS is
especially interesting in scenarios where multiple users are collaborating on the business
processes that influence the PerformancePoint Server scorecards. The integration between
MOSS and PerformancePoint Server is based on the PerformancePoint Server Dashboard
Viewer Web part that allows you to render a dashboard or a specific scorecard included in it.
Figure: Configuring the BAM PerformancePoint Server dashboard by using MOSS
Once configured properly the PerformancePoint Server dashboard hosting the BAM activity
will be accessible by any Web client without the need to install any components.
Figure: BAM PerformancePoint Server dashboard hosted in a MOSS Web site
MOSS is a particularly interesting technology for exposing BAM data in scenarios where
multiple users should interact and collaborate based on the information provided by BAM
activities. MOSS includes various technologies that can be used to interact with BAM activity
data. The following section will explore some of those technologies and common scenarios
for integrating BAM and MOSS.
Using BAM with Microsoft Office SharePoint ServerMicrosoft Office SharePoint Server (MOSS) provides a rich set of capabilities around
enterprise portals and collaboration that makes it a phenomenal technology for exposing
BAM data. In its current release, MOSS includes a series of technologies such as Excel
Services and Business Data Catalog (BDC) that can be used for exposing BAM activity data
to enterprise users in different scenarios. For instance, a developer looking to reflect the
BAM relational data in a MOSS site might be inclined to use a custom Web part when a
technology like the BDC can be more appealing in scenarios where the BAM data needs to
be complemented with Office documents or enterprise search. Additionally, as we explored
in the previous sections, MOSS Web parts can be used to consume BAM data from other
technologies like SSRS and PerformancePoint Server.
Exposing BAM data using Web partsThe Web Part Framework provided with ASP.NET 2.0 is the foundation of MOSS Web Parts.
Using that framework, developers can build custom Web parts that query BAM activity data
stored in the BAMPrimaryImport and BAMAnalysis databases. Out of the box, MOSS provides
some Web parts that can be used to accomplish this task. For instance, we can use the Data
View Web Part from SharePoint Designer (see the BAM data highlighted in red) to query the
activity views stored in the BAMPrimaryImport database.
Figure: Customizing the Data View Web Part from SharePoint Designer
Once deployed the Web part will render the BAM activity instances as illustrated in the
following figure.
Figure: MOSS site using the Data View Web Part to access the BAM activity data
Business Data Catalog
The use of Web parts, including the Data View Web Part, represents a great initial
alternative for querying BAM data from MOSS. However, quite often, the BAM activity data
needs to be complemented with other MOSS artifacts like document management,
enterprise search, or workflows. For those scenarios, the use of the Business Data Catalog
(BDC) might be a more adequate solution. The BDC provides a completely declarative
mechanism that allows MOSS to surface line-of-business (LOB) system data. Specifically, the
BDC provides a strong integration with SOAP-based Web services and databases that we can
use to integrate BAM activities into MOSS. One of the features that make the BDC so
appealing to enterprise applications is its capability to integrate business data into other
MOSS technologies like document management or enterprise search. From the BAM
standpoint, we can use the BDC not only to query BAM data but also to associate BAM
activities with Office documents, incorporating BAM activities in MOSS workflows or adding
search capabilities to the BAM data. The process of integrating BAM data into MOSS by using
the BDC is no different from the process followed for other relational databases.
The first step required to use the BDC with BAM is to create a BDC metadata model. This is
an XML-based declarative language that represents the different entities that will be used by
MOSS. Also, given that the entities are stored in the BAMPrimaryImport database, this model
should express the queries required to select the BAM activity instances that are relevant to
the MOSS applications. The following code illustrates sections of the BDC metadata model
associated with the Transaction BAM activity. You can see that this model declares the fields
that will be rendered in MOSS as well as the T-SQL command used to extract the data.
<LobSystem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog BDCMetadata.xsd" Type="Database" Version="1.0.0.0" Name="TransactionActivity" xmlns="http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog"> <LobSystemInstances> <LobSystemInstance Name="TransactionActivity_Instance"> <Properties> …Connection properties… </Properties> </LobSystemInstance> </LobSystemInstances> <Entities> <Entity EstimatedInstanceCount="10000" Name="bam_TRANSACTION_Instances"> <Properties> <Property Name="DefaultAction" Type="System.String">View Profile</Property> </Properties> <Identifiers> <Identifier TypeName="System.String" Name="ActivityID" /> </Identifiers>
<Methods> <Method Name="Find_bam_TRANSACTION_Instances"> <Properties> <Property Name="RdbCommandType" Type="System.Data.CommandType, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">Text</Property> <Property Name="RdbCommandText" Type="System.String">SELECT "ActivityID", "TRANSACTIONID", "CREDITCARD", "AMOUNT","TRANSACTION_CREATED","TRANSACTION_FAILED", "TRANSACTION_FAILED_ALERT_TRIGGERED","TRANSACTION_APPROVED", "CUSTOMER_NAME","CUSTOMER_AGE" FROM bam_TRANSACTION_AllInstances </Property> </Properties> <Parameters> <Parameter Direction="Return" Name="@bam_TRANSACTION_Instances"> <TypeDescriptor TypeName="System.Data.IDataReader, …" IsCollection="true" Name="Reader"> <TypeDescriptors> <TypeDescriptor TypeName="System.Data.IDataRecord,…”, Name="Record"> <TypeDescriptors> <TypeDescriptor TypeName="System.String, …" Name="ActivityID" IdentifierName="ActivityID"/> <TypeDescriptor TypeName="System.String, …"
Name="TRANSACTIONID" /> <TypeDescriptor TypeName="System.String, …" Name="CREDITCARD" /> <TypeDescriptor TypeName="System.Double, …" Name="AMOUNT" /> <TypeDescriptor TypeName="System.DateTime…" Name="TRANSACTION_CREATED" /> <TypeDescriptor TypeName="System.DateTime…" Name="TRANSACTION_FAILED" /> <TypeDescriptor TypeName="System.DateTime…" Name="TRANSACTION_FAILED_ALERT_TRIGGERED" /> <TypeDescriptor TypeName="System.DateTime…" Name="TRANSACTION_APPROVED" /> <TypeDescriptor TypeName="System.String…" Name="CUSTOMER_NAME" /> <TypeDescriptor TypeName="System.Int32…" Name="CUSTOMER_AGE" /> </TypeDescriptors> </TypeDescriptor> </TypeDescriptors> </TypeDescriptor> </Parameter> </Parameters> <MethodInstances> <MethodInstance Type="Finder" ReturnParameterName="@bam_TRANSACTION_Instances" ReturnTypeDescriptorName="Reader" ReturnTypeDescriptorLevel="0" Name="Find_bam_TRANSACTION_Instances">
…Rest of the elements have been omitted for brevity….
</MethodInstance> </MethodInstances> </Method> </Methods> </Entity> </Entities>
…Rest of the elements have been omitted for brevity….
</LobSystem>
Figure: BDC model for the Transaction BAM activity
Once created, the BAM metadata model can be deployed by using the MOSS administration
console. After the BDC model is deployed the BAM entities can be used from any MOSS Web
site with the proper privileges. To render the BAM activity records we can use some of the
BDC Web parts included in MOSS. In our scenario we will use the Business Data List Web
Part to render all the instances of the transaction activity and the Business Data Item Web
Part to render the details of a specific activity instance (see the following figure).
Figure: Configuring BAM-BDC Web parts
Once configured, the BDC Web part will provide real-time access to the information stored in
the BAMPrimaryImport database activity views.
Figure: Accessing BAM data by using BDC
As we explained at the beginning of this section, one of the benefits of integrating BAM with the BDC is the capability of exposing BAM data to other MOSS technologies. For instance, we can associate BAM activity instances with Office documents that provide additional information about the business processes.
Figure: Using BAM activities in a MOSS document library
Accessing BAM by using Web servicesIn the first part of this section we explored different Microsoft products and technologies that
can complement BAM with enterprise capabilities such as reporting, collaboration, or KPI
analysis. Undoubtedly, the combination of BAM with technologies like SSRS,
PerformancePoint Server, and MOSS is the correct answer to a large variety of the most
common BAM scenarios in the enterprise. However, these technologies provide the most
dividends in scenarios like end-user reports that have certain design-time knowledge of how
the BAM activity data needs to be processed. Looking back to our previous examples, all of
them consume the BAM data by using artifacts created at design time like reports or KPIs.
However, these types of techniques are not always the best fit in scenarios that require a
more dynamic interaction with the BAM activities.
Querying the BAMAnalysis database by using the XML for Analysis Web serviceXML for Analysis (XMLA) is an open standard for querying multidimensional data sources by
using Web services. The last two editions of SQL Server Analysis Services (SSAS 2005 and
SSAS 2008) support XMLA 1.1 as one of the programming models used by client applications
to interact with an SSAS instance. Even more importantly, XMLA is the fundamental protocol
underneath other multidimensional programming technologies like ADOMD.NET and Analysis
Management Objects (AMO). Essentially, XMLA proposes a model for encoding commands
such as MDX queries in a SOAP envelope that can be processed by an XMLA-compatible Web
service. In its current version, the XMLA specification describes two generally accessible
operations: Discover and Execute. The Discover operation is used to query metadata about
multidimensional data sources like cubes, dimensions, etc. For instance, by using that
operation an application can browse the list of BAM cubes, dimensions, and measures
deployed in the BAMAnalysis database.
<Discover xmlns="urn:schemas-microsoft-com:xml-analysis"> <RequestType>MDSCHEMA_CUBES</RequestType> <Restrictions> <RestrictionList> <CATALOG_NAME>BAManalysis</CATALOG_NAME> </RestrictionList> </Restrictions> <Properties> <PropertyList> <DataSourceInfo>Provider=MSOLAP;Data Source=local;</DataSourceInfo> <Catalog>BAMAnalysis</Catalog> <Format>Multidimensional</Format> </PropertyList> </Properties></Discover>
Figure: XMLA Discover operation to browse the cubes deployed in the BAMAnalysis database
The Discover operation is used exclusively for browsing metadata. To execute queries
against a multidimensional data source a developer needs to use the Execute operation. In
the BAM context, an application can invoke the XMLA Execute operation to run MDX queries
against some of the activity views stored as cubes in the BAMAnalysis database (see next
figure).
<Execute xmlns="urn:schemas-microsoft-com:xml-analysis"> <Command> <Statement> SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim]. [Day].ALLMEMBERS * DESCENDANTS ([TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE </Statement> </Command>
<Properties> <PropertyList> <DataSourceInfo>Provider=MSOLAP;Data Source=local;</DataSourceInfo> <Catalog>BAMAnalysis</Catalog> <Format>Multidimensional</Format> </PropertyList> </Properties></Execute>
Figure: Using the XMLA Execute operation to query the BAMAnalysis database
In SSAS 2005 and SSAS 2008, developers can configure HTTP-SOAP XMLA access by
configuring the ISAPI extension deployed in the Web directory. Then, developers can
generate an XMLA proxy using the technology of choice. The following code illustrates a
WCF client that invokes the Execute operation to run an MDX query against the
TRANSACTIONS OLAP cube deployed as part of our sample BAM activity model.
MsXmlAnalysisSoapClient xmlaService = new MsXmlAnalysisSoapClient(new BasicHttpBinding(), new EndpointAddress("http://localhost:8080/olap/msmdpump.dll"));XmlDocument xdoc= new XmlDocument();
XmlElement command= xdoc.CreateElement("", "Statement", " urn:schemas-microsoft-com:xml-analysis");
XmlElement properties = xdoc.CreateElement("", "PropertyList", " urn:schemas-microsoft-com:xml-analysis"); properties.InnerXml = "<DataSourceInfo xmlns=\"urn:schemas-microsoft-com:xm-analysis\"> Provider=MSOLAP;Data Source=btsr2; </DataSourceInfo> <Catalog xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> BAMAnalysis </Catalog> <Format xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> MultiDimensional </Format> <AxisFormat xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> ClusterFormat </AxisFormat>"; command.InnerXml = @"SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim].[Day].ALLMEMBERS * DESCENDANTS( [TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) }
DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE";
XmlElement response= xmlaService.Execute(command, properties);
Figure: Using the XMLA Web service to query the BAMAnalysis database
Although powerful, the use of the XMLA Web service is restricted to the BAMAnalysis
database. If an application is required to use Web services to access the activity fact data
stored in the BAMPrimaryImport database there is another option available: the BAM Query
Web service.
Querying the BAMPrimaryImport database by using the BAM Query Web serviceAlthough not officially supported, the BAM Query Web service provides an interesting
solution to applications that need to query the BAM activity data stored in the
BAMPrimaryImport database. The BAM Query Web service provides a single operation,
GetInstanceData, that abstracts a T-SQL query to a BAM activity view. The following code
shows a sample client that uses the BAM Query Web service to query our sample activity
model.
BamQueryServiceSoapClient queryService = new BamQueryServiceSoapClient( new BasicHttpBinding(), new EndpointAddress( "<server url>/BamQueryService.asmx"));queryService.ClientCredentials.Windows.ClientCredential = new System.Net.NetworkCredential(<username>, <password>);InstanceQuery query= new InstanceQuery();query.SelectClauses = new string[] { "ActivityID, TRANSACTIONID, AMOUNT, TRANSACTION_CREATED, CREDITCARD" };Column[][] columns= queryService.GetInstanceData("dbo.bam_TRANSACTION_AllInstances", "TRANSACTION", query, 0);
Figure: Using the BAM Query Web service
The use of both XMLA and the BAM Query Web service considerably improves the BAM
implementations in distributed environments and also facilitates the interoperability of the
BAM infrastructure with non-Microsoft technologies such as J2EE or dynamic languages.
However, it is important to highlight that developers should use these Web services only as
a very specific mechanism to query BAM data and never as a general-purpose API to
abstract the interactions with the BAM infrastructure. Neither the BAM Query Web service
nor XMLA provide any abstraction of the BAM data model. Instead, both of them require the
consumer application to have intrinsic knowledge of the schema of the BAM databases.
As we’ve explored in the different sections of this paper, BizTalk Server BAM provides a
generic infrastructure that does not impose any restrictions on the type of client applications
that can interact with an activity model. However, from an API standpoint, the use of BAM is
currently reduced to .NET Framework-based applications such as the ones developed in
BizTalk Server, WCF, or WF. Certainly, abstracting the BAM infrastructure with an open-
standard API can enable the adoption of BAM across heterogeneous technologies. Although
currently there is no “supported” API that serves this purpose, developers can leverage
technologies like WCF and emerging architecture styles like Representational State Transfer
(REST) as a vehicle to service enable their BAM infrastructure.
Building a BAM RESTful API
Why do we need a new API?Throughout this paper we have explored different mechanisms developers can use to
interact with BizTalk Server BAM. Although the current technology provides developers with
powerful alternatives to instrument their applications using BAM, there are a few
outstanding challenges in order to ubiquitously adopt BAM in heterogeneous enterprise
environments.
One of the most attractive capabilities of BizTalk Server BAM is that its infrastructure is not
dependent on BizTalk Server. This means that, theoretically, non-.NET Framework-based
applications can benefit from the use of BAM. However, even though this is true from the
BAM infrastructure standpoint, the APIs required to interact with BAM are only available
to .NET Framework-based applications.
Another challenge that developers face when implementing BAM applications is the lack of a
consistent programming model for querying and executing CRUD (Create-Update-Delete)
operations on BAM activities. In that sense, developers need to interact with BAM stream-
based APIs to populate elements of an activity model and with a completely different model
when it comes to querying BAM data. This contrasts with other data-driven programming
models such as ADO.NET that abstract both functionalities through a consistent API.
To improve the consistency and interoperability of the BAM programming model, developers
often find themselves creating Web service interfaces that abstract the BAM stream APIs.
Typically, those Web service interfaces are based on the SOAP-WSDL-WS-* model and
expose the specific BAM capability that is initially required by the applications using them.
Extending these APIs to include the broad set of BAM capabilities will introduce the
traditional versioning challenges that large SOAP-WSDL APIs face in the industry today.
Another limitation of this model is the constraint on adoption to technologies that don’t have
a powerful SOAP-based engine. That list includes some of the most powerful technologies for
building rich Web user interfaces that consume BAM data, like dynamic or scripting
languages and AJAX toolkits.
Being aware of all these challenges, this section will propose a slightly different model that
leverages the principles of a set of emerging service-oriented architecture styles.
Specifically, this section will explore the design of BAM Web service APIs based on the
Representational State Transfer (REST) model.
Embracing the Web programming principlesAlthough this section is not intended to discuss the pros and cons of Representational State
Transfer (REST) vs. traditional SOAP/WS-* services, we think is important to highlight why
REST is a better choice for designing a consistent BAM Web services API.
The term REST was first enunciated by Roy Thomas Fielding as part of his PhD dissertation.
At a high level, REST intended to formulate the principles that make the Web work in a
scalable and performant way. Fielding’s arguments highly resonated in the Web service
community in which REST started to be positioned as an alternative to the traditional
SOAP/WS-* model for Web services, firing off one of the most fascinating debates in the
history of distributed computing. In spite of all those arguments, the fact is that REST, as an
architecture style, is a great solution for implementing resource/entity-based services like
the ones required for a BAM API. Essentially, RESTful services follow a set of well-established
principles that differentiate them from other alternatives. The following list summarizes
some of the most important characteristics of RESTful services:
Give every resource an ID: Fundamentally, RESTful services abstract interactions
with resources (BAM activities, database entities, documents, etc). This principle
states that those resources should be addressable and uniquely identified. The
typical way to satisfy both requirements is to use URIs as the unique resource
identifier. For instance, a BAM activity instance can be identified by the URI, where
activity ID is the unique BAM identifier for that activity instance.
Link things together: In a typical REST system, resources can be related to each
other. For instance, two BAM activities can have a reference relationship or a
customer resource can have a relationship with the account resource. As resources
are identified by URI, one resource will include the links to its related resources as
part of its representation so that they can be individually addressed.
Use standard methods: Web architectures are based on HTTP. REST enunciates
how the different HTTP verbs should be associated with resource operations. In
essence, resources can be created using an HTTP POST, retrieved using a HTTP
GET, updated by an HTTP PUT, and deleted using an HTTP DELETE. Other
operations like metadata retrieval can be abstracted by other HTTP verbs such as
HEAD or OPTIONS.
Resources with multiple representations: When we design Web services using
the traditional SOAP/WS-* techniques, we are using SOAP as the single
representation format. In RESTful services, resources are conceptual entities that are
independent of the representation format used on the wire. For instance, a BAM
activity instance can be represented by using XML, JavaScript Object Notation (JSON),
or Atom formats.
Communicate statelessly: Web architectures are stateless by definition and this is
one of the characteristics that makes them highly scalable. By stateless, we mean
that any state of the interaction should be carried in the resource itself and not in a
“server session.”
Designing RESTful services in the Microsoft platform has come a long way and it is safe to
say that the current release of Windows Communication Foundation (WCF) can be
considered one of the most solid REST development platforms in the industry today. The use
of WCF to build a BAM RESTful API will be illustrated in the following sections of this paper.
Following the REST principles, we can start formulating what a BAM RESTful API looks like.
The following section will explore some of the design alternatives to expose BAM capabilities
using RESTful services.
Brainstorming a BAM RESTful APIThe first aspect to consider is that the BAM conceptual model can naturally be modeled by
using the principles of Resource Oriented Architecture (ROA). We emphasize the term
natural because, even though we can also model a generic BAM API by using RPC-based
concepts like operations or parameters, ROA transparently reflects the concepts behind the
BAM programming model. In that sense, the entire BAM infrastructure can be atomically
represented as a set of resources such as activities, activity views, cubes, or activity models.
These BAM resources can be related to each other; for instance, a BAM activity can have a
reference relationship with other BAM activities. When adopting this architecture style, we
can abstract the entire BAM development process as a series of query and CRUD operations
against BAM resources.
Having represented the BAM programming model as resources, it is time to think of how to
uniquely identify instances of BAM resources. To embrace the addressability principle of
REST we can use URI as the unique identifier of a BAM resource. For instance, an activity
instance can be identified by the URI http://<my BAM server>/activities/{activity
name}/{activityid}.
At this point, our next challenge is how to adapt the different HTTP verbs to represent
queries (HTTP GET) or CRUD (HTTP POST, PUT, and DELETE respectively) operations against
BAM resources. This process is highly dependent on the aforementioned URI model and
should correctly represent the BAM conceptual model. The following list illustrates some
examples of how to use the HTTP verbs to model BAM operations following the REST
principles:
An HTTP GET against http://<my BAM server>/activities/{activity name} will
return the list of instances of the activity identified by {activity name}. Each instance
will be identified by a URI in the form of http://<my BAM
server>/activities/{activity name}/{activity id}.
An HTTP POST against http://<my BAM server>/activities/{activity name} will
create a new instance of the {activity name} BAM activity. The new instance will be
identified by a URI in the form of http://<my BAM server>/activities/{activity
name}/{activity id}.
An HTTP GET against http://<my BAM server>/activities/{parent activity
name}/{activity id}/{child activity name} will return the instances of the {child
activity name} BAM activity related to the {activity id} instance of the {parent
activity name} BAM activity.
An HTTP PUT against http://<my BAM server>/activities/{activity
name}/{activity id} will update the {activity id} instance of the {activity name}
BAM activity.
Following the same RESTful principles, you can think of similar models to apply to other BAM
elements such as views, alerts, or cubes.
An aspect to consider when designing a BAM RESTful API is that some BAM resources
represent metadata associated with other BAM resources. For instance, an activity model
resource will contain the definition of the different activities and views included in that
model. In that sense, the activity model resource contains metadata associated with the
activity resource.
At this point, we already know how to model the BAM elements as resources and how to
associate the HTTP verbs with operations against those resources. That brings us to one of
the most important aspects of a RESTful API, which is to select a resource representation
format. This is a decision that we don’t normally face when designing SOAP/WS-* APIs
because, in that case, SOAP is the only representation format. Going back to the previous
section, one of the principles of REST states that resources are representation agnostic. In
that sense, we could represent the same resource by using an XML dialect, JSON, or a
syndication format such as RSS or Atom. Although, at first glance, plain XML might seem the
perfect choice, the fact is that creating a resource representation language entails a lot of
details that ultimately can result in interoperability challenges. This is the reason why many
successful RESTful APIs use well-established languages like RDF or Atom. The case of
syndication formats such as Atom or RSS is particularly popular in scenarios such as event
processing in which resources are modified at a high frequency. Additionally, AtomPub is a
well-established standard with a large support on heterogeneous platforms. Finally, the
extensibility of AtomPub makes it an ideal language for representing untyped structures
such as BAM activities. Considering all those factors, we decided to leverage Atom and the
Atom Publishing Protocol (AtomPub) as the resource representation language for our sample
BAM API. As you might already suspect, a complete BAM RESTful API will include a large
variety of operations that will be very difficult to cover in this paper. However, in order to
give a pragmatic view of how BAM can take advantage of the REST architecture style to
extend its functionality, the following section will explore the implementation of some of the
operations of a BAM RESTFul API using WCF 3.5.
A sample BAM RESTful APIAs we explained previously, this section does not intend to describe an entire BAM RESTful
API. Instead, we intend to explore the implementation of a few operations highlighting the
fundamental principles that can be applied to the implementation of other BAM RESTful
operations. For the simplicity of the examples, let’s start with the following WCF contract.
[ServiceContract][ServiceKnownType(typeof(Atom10FeedFormatter))]public interface ISampleBAMService { [OperationContract] Atom10FeedFormatter GetActivities();
[OperationContract] Atom10FeedFormatter GetActivityInstances(string activityname, string instanceid);
[OperationContract] Atom10FeedFormatter CreateActivityInstace(string activityname, bool isend, Stream feed);
[OperationContract] Atom10FeedFormatter UpdateActivityInstance(string activityname, string activityid, bool isend, Stream feed);
}
The first two operations are intended to navigate through the BAM activities deployed in a
BAMPrimaryImport database while the last three operations expose the capabilities required
for creating and updating BAM activity instances. As you can see all the operations return an
instance of Atom10FeedFormatter, which represents an Atom feed.
GetActivities
This operation executes based on an HTTP GET request and returns an Atom feed with the
definition of the BAM activities deployed in a specific BAMPrimaryImport database. The
following code highlights a section of the implementation of this operation (the entire
implementation can be found with the samples that accompany this paper).
public class SampleBAMService: ISampleBAMService{ [WebGet(UriTemplate="/activities")] public Atom10FeedFormatter GetActivities() { /*Retrieves the list of activities from the BAMPrimaryImportDB database. Details omitted for brevity…see the attached samples for the details */
BAMActivity[] activities= bamdb.GetBAMActivities();
List<SyndicationItem> items= new List<SyndicationItem>(); foreach (BAMActivity activity in activities) items.Add(BAM activity Atom representation); feed.Items = items; return new Atom10FeedFormatter(feed); } }
The WebGet attribute of this operation indicates that this operation is invoked based on an
HTTP GET with the form of http://<service endpoint>/activities. In a nutshell, this operation
retrieves the list of BAM activities from the bam_Metadata_Activities table in the
BAMPrimaryImportDB database and serializes the results into an Atom 1.0 feed. The
AtomPub response contains the BAM activity definitions as illustrated in the following figure.
<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-
ea47-4fe9-98cd-c5d1c319c475;id=1</id><updated>2008-09-30T18:43:03Z</
updated><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=2</id><title
type="text">ALERT</title><updated>2008-09-30T18:43:03Z</updated><link
rel="alternate" href="http://localhost:8090/bamservice/activities/ALERT/*"/><content
type="" Name="ALERT"
ID="IDD832C36CE39041529B431984828B4D76">...</content></entry><entry><id>uuid:
23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=16</id><title
type="text">TRANSACTION</title><updated>2008-09-30T18:43:03Z</updated><link
rel="alternate"
href="http://localhost:8090/bamservice/activities/TRANSACTION/*"/><content type=""
Name="TRANSACTION"
ID="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC">...</content></entry></feed>
Figure: GetActivities AtomPub response
Notice that, following the principles of REST, each activity is individually addressable by a
URI in the form of http://<service endpoint>/activities/<activity name>. Given that the
response is encoded using Atom it can be rendered in a Web browser as illustrated in the
following figure.
Figure: GetActivities response rendered in Internet Explorer®
Selecting a specific activity will invoke the GetActivity operation to retrieve its instances.
GetActivity
This operation returns an Atom feed that includes either all instances or a specific instance
of a BAM activity and is activated by executing an HTTP GET to a URL in the form
http://<service endpoint>/activities/<activity name>/*|<activity instance id>. The following
code illustrates fragments of the implementation of the GetActivity operation.
[WebGet(UriTemplate="/activities/{activityname}/{instanceid}")]public Atom10FeedFormatter GetActivityInstances(string activityname, string instanceid){ /*Retrieves the list of activities instances from a specific BAM activity Details omitted for brevity…see the attached samples for the details */
BAMActivityInstance[] instances=
bamdb.GetBAMActivityInstances(activityname, instanceid); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); foreach (BAMActivityInstance instance in instances) items.Add(instance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed);}
Figure: GetActivity implementation
If the instanceid parameter is * this operation returns all the instances of a BAM activity. The
following code shows a response that lists the instances of the TRANSACTION activity used
throughout this article.
<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=35</id><updated>2008-09-30T19:42:00Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=36</id><title type="text">125ca0b4-c896-4848-864e-b83287e1c539</title><updated>2008-09-30T19:42:00Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/125ca0b4-c896-4848-864e-b83287e1c539"/><content type=""> <ActivityID xmlns="">125ca0b4-c896-4848-864e-b83287e1c539</ActivityID> <TRANSACTION_CREATED xmlns="">9/30/2008 2:07:35 AM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns=""> </TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <CUSTOMER_AGE xmlns=""> </CUSTOMER_AGE> <AMOUNT xmlns="">25001</AMOUNT> <TRANSACTIONID xmlns=""> </TRANSACTIONID> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <LastModified xmlns="">9/30/2008 2:18:02 AM</LastModified></content></entry><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=50</id><title type="text">6052b24f-e863-420b-94bc-fa7edaf7eb7c</title><updated>2008-09-30T19:42:00Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/6052b24f-e863-420b-94bc-fa7edaf7eb7c"/><content type=""> <ActivityID xmlns="">6052b24f-e863-420b-94bc-fa7edaf7eb7c</ActivityID> <TRANSACTION_CREATED xmlns="">9/29/2008 10:07:35 PM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns="">9/30/2008 5:02:07 AM</TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">CustomerName_0</CUSTOMER_NAME> <CUSTOMER_AGE xmlns="">
</CUSTOMER_AGE> <AMOUNT xmlns="">10</AMOUNT> <TRANSACTIONID xmlns="">234567</TRANSACTIONID> <CREDITCARD xmlns="">2222444455557777</CREDITCARD> <LastModified xmlns="">9/30/2008 5:02:14 AM</LastModified></content></entry>…Other entries…</feed>
Figure: GetActivity Atom response
The results of this operation can also be rendered in a Web browser as illustrated in the
following figure.
Figure: GetActivity Atom response rendered in Internet Explorer
Selecting a specific activity will again invoke the GetActivities operation with a specific
activity ID producing the following response.
<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=69</id><updated>2008-09-30T20:16:59Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=70</id><title
type="text">125ca0b4-c896-4848-864e-b83287e1c539</title><updated>2008-09-30T20:16:59Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/125ca0b4-c896-4848-864e-b83287e1c539"/><content type=""> <ActivityID xmlns="">125ca0b4-c896-4848-864e-b83287e1c539</ActivityID> <TRANSACTION_CREATED xmlns="">9/30/2008 2:07:35 AM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns=""> </TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <CUSTOMER_AGE xmlns=""> </CUSTOMER_AGE> <AMOUNT xmlns="">25001</AMOUNT> <TRANSACTIONID xmlns=""> </TRANSACTIONID> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <LastModified xmlns="">9/30/2008 2:18:02 AM</LastModified></content></entry></feed>
Figure: GetActivity Atom response for an individual activity
Combining these two operations we can navigate through the BAM activity using a
syndication client. To create new instances of a BAM activity we need to execute the
CreateActivityInstance operation via an HTTP POST.
CreateActivityInstance
This operation creates an instance of a BAM activity based on an AtomPub request. The
following code illustrates sections of the implementation of this operation.
[WebInvoke(UriTemplate= "/activities/{activityname}?end={isend}", Method="POST")] public Atom10FeedFormatter CreateActivityInstace(string activityname, bool isend, Stream activityfeed) { /*Creates a new instance of a BAM activity using the Event Stream API. Details omitted for brevity…see the attached samples for the details */
BAMActivityInstance activityInstance= bamdb.CreateActivityInstance(activityname, activityFields, true, isend, activityContent); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); items.Add(activityInstance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed); }
As you can see in the code, this operation is activated by an AtomPub request to a URI in the
form http://<service endpoint>/activities/<activity name>?end=true|false. The following
code shows a sample AtomPub request that creates an instance of the Transaction activity
used in our example.
Host: localhost:8090Content-Type: application/atom+xmlContent-Length: 142
<entry xmlns="http://www.w3.org/2005/Atom"><id>1</id><content type=""> <AMOUNT xmlns="">45001</AMOUNT> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <TRANSACTION_CREATED>9/30/2008 2:07:35 AM</TRANSACTION_CREATED> </content></entry>
Figure: AtomPub CreateActivity request
The produced response is also an Atom message that details the new activity.
<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=75</id><updated>2008-09-30T20:59:56Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=76</id><title type="text">e5abe557-c3c4-420f-81c5-f2db4b113a10</title><updated>2008-09-30T20:59:56Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/a7ac8da1-624e-4d72-bc73-c7fdbd7d9a0a"/><content type=""> <AMOUNT xmlns="">45001</AMOUNT> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <TRANSACTION_CREATED>9/30/2008 2:07:35 AM</TRANSACTION_CREATED></content></entry></feed>
Figure: AtomPub CreateActivity response
The activities created by this operation can be completed depending on the value of the
“end” query parameter. If the activity is not completed it can be updated later by using the
UpdateActivity operation.
UpdateActivity
This operation updates an instance of a BAM activity based on an AtomPub request. The
implementation of this operation shares many similarities with the CreateActivity explained
in the previous section.
[WebInvoke(UriTemplate = "/activities/{activityname}/{activityid}?end={isend}", Method
= "PUT")]
public Atom10FeedFormatter UpdateActivityInstance(string activityname, string activityid,
bool isend, Stream activityfeed)
{
/*Updates a new instance of a BAM activity using the Event Stream API. Details omitted for brevity…see the attached samples for the details */
BAMActivityInstance activityInstance= bamdb.UpdateActivityInstance(activityname, activityid, activityFields, true, isend, activityContent); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); items.Add(activityInstance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed);}
As highlighted in the code, this operation is activated by an HTTP PUT message sent to a
URL with the form http://<service endpoint>/activities/<activity name>/<activity instance
id>?end=true|false. The following code illustrates a sample AtomPub request that updates
an instance of the BAM transaction activity used in our example.
PUT /bamservice/activities/transaction/a24b1028-8c08-430e-b2d1-e25f36c09a59?end=true Host: localhost:8090Content-Type: application/atom+xmlContent-Length: 142
<entry xmlns="http://www.w3.org/2005/Atom"><id>1</id><content type=""> <CUSTOMER_AGE xmlns="">32</CUSTOMER_AGE> </content></entry>
Figure: UpdateActivity AtomPub request
The UpdateActivity also produces an Atom response detailing the updated elements.
<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=73</id><updated>2008-09-30T20:42:24Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=74</id><title type="text">a24b1028-8c08-430e-b2d1-e25f36c09a59</title><updated>2008-09-30T20:42:24Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/a24b1028-8c08-430e-b2d1-e25f36c09a59"/><content type=""> <CUSTOMER_AGE xmlns="">32</CUSTOMER_AGE></content></entry></feed> Figure: UpdateActivity AtomPub response
In a similar fashion to CreateActivity, the UpdateActivity operation can end a BAM activity
instance based on the end query parameter.
The combination of the CreateActivity and UpdateActivity operations facilitates populating a
BAM activity model from heterogeneous technologies using well-established mechanisms
such as HTTP and AtomPub.
As we explained in the previous section, a complete BAM RESTful API will include many other
operations that will be impossible to cover in this paper. However, the principles and
techniques used to implement the operations explained in this section can serve as the
foundation for implementing other interesting operations of a BAM RESTful API.
The use of REST can be one of the fundamental vehicles to extend the use of BAM across
more heterogeneous environments including those developed by using emerging
programming paradigms such as dynamic languages, AJAX applications, etc.
SummaryThe intention of this paper is to provide a deeper insight into the world of BizTalk Server
Business Activity Monitoring, from the basic concepts and how BAM data is collected to how
that data is propagated across databases and the multiple methods by which it can be
accessed. The first half of this paper dives into the complex details of the BAM
infrastructure. We attempt to clarify those details that are critically important, so that
developers and administrators using BAM can maintain a high-performance BAM
implementation.
As described in the second half of this paper, BizTalk Server Business Activity Monitoring is a
very extensible technology that reaches beyond BizTalk Server into both Microsoft and non-
Microsoft technologies. Some of the Microsoft technologies include Windows Communication
Foundation, Windows Workflow Foundation, PerformancePoint Server, SharePoint Server,
and SQL Server Integration, Reporting, and Analysis Servers. BAM can be extended to non-
Microsoft technologies as well, through the use of Web services and programming models
such as Representational State Transfer (REST).
Using the information presented in this paper, developers and administrators can effectively
create BAM solutions while providing their business analysts a wide variety of tools for
accessing and analyzing the BAM data.
AuthorsJesus Rodriguez : Jesus Rodriguez is the Chief Architect of Tellago, Inc. He is also a Microsoft BizTalk
Server MVP, an Oracle ACE, and one of a few Architects worldwide to be a member of the Microsoft
Connected Systems Advisor team. As a member, Jesus has been selected to participate in a variety of
software design reviews with Microsoft product teams including Windows Communication Foundation,
Windows Workflow Foundation, and BizTalk Server.
Jesus derived his extensive experience with business process integration and messaging through
numerous implementations of disparate systems founded on the principles of SOA and BPM. Jesus is an
active contributor to the .NET and J2EE communities and an internationally recognized speaker and
author with contributions that include several articles for various publications including MSDN
Magazine, Microsoft Architecture Journal, SOAWorld, and Web Services Journal as well as speaking
engagements at top industry conferences such as Microsoft TechEd, SOAWorld, Microsoft SOA and BPM
Conference, Oracle Open World, Web Services Security Conference, and the Microsoft MVP Summit.
Additionally, Jesus has conducted a number of webcasts on varying SOA technologies.
Jesus is a prolific blogger on all subjects related to SOA and has a true passion for technology. You can
gain valuable insight on leading-edge technologies through his blog at http://weblogs.asp.net/gsusx.
Joe Klug : Joe Klug is the Chief Technical Officer at Tellago, Inc. Joe has an extensive background in
enterprise application integration and business process management, which he derived though practical
consulting engagements and enterprise-scale software development.
Prior to Tellago, he worked as a Product Manager in the BizTalk Server team at Microsoft Corporation.
During his six years at Microsoft, Joe was a key contributor on several releases of BizTalk Server. Some of
his key areas of responsibility included Web services integration and adapter development, and he was
one of the founders of the software development kit for developing Windows Communication
Foundation-based adapters.
While at Microsoft, Joe became a nationally recognized speaker through engagements at Microsoft
conferences, including TechEd, SOA & BPM Conference, Office Developers Conference, and SharePoint
Conference. He also worked as an interoperability consultant at J.D. Edwards, focusing on integrating
J.D. Edwards OneWorld software with other third-party applications, mainframe solutions, and e-
commerce gateways. Joe received a Bachelor of Science in Computer Science from Michigan
Technological University and an M.B.A. from the University of Colorado.
Joe shares his insights on technology through his blog at http://geekswithblogs.net/jklug/Default.aspx.
