Business Activity Monitoring in Depth for Developers

Published: January 2009 Authors: Jesus Rodriguez and Joe Klug, Tellago, Inc. Technical Reviewers: Ofer Ashkenazi, Microsoft Marcel Fernee, Microsoft

Stephen Kaufman, Microsoft Brian Loesgen, Neudesic LLC Jonathan Moons, Microsoft Allan Naim, Microsoft Paolo Salvatori, Microsoft Andy Shen, Microsoft

Applies to: BizTalk Server 2006 R2 and BizTalk Server 2009 To download a copy of this document and the sample code, go to

http://go.microsoft.com/fwlink/?LinkId=139086.

Summary
Microsoft added Business Activity Monitoring (BAM) to Microsoft BizTalk Server as a way for business analysts to track information that is important to them. This paper provides a deep, low-level review of how BAM works and how BAM can be extended. It reviews BAM basics, shows ways in which BAM data can be collected, discusses the BAM infrastructure, demonstrates methods for collecting BAM data, and shows how BAM can be extended to non-Microsoft technologies.

Copyright
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2009 Microsoft Corporation. All rights reserved. Microsoft, BizTalk, Excel, Internet Explorer, MSDN, Outlook, PerformancePoint, PivotTable, SharePoint, and Windows are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.

Table of Contents
Executive Summary....................................................................................................................................... 5 Business Activity Monitoring Overview ........................................................................................................ 6 BizTalk Server BAM: Exploring the activity model .................................................................................... 7 Defining a BAM sample activity model ..................................................................................................... 9 How BAM differs from data warehousing .............................................................................................. 16 Instrumenting Applications by Using BAM ................................................................................................. 17 Using BAM declaratively ......................................................................................................................... 17 Using BAM from BizTalk Server .............................................................................................................. 18 Using BAM from Windows Communication Foundation and Windows Workflow Foundation ............ 22 BAM WF interceptor ............................................................................................................................... 28 Using BAM programmatically ................................................................................................................. 33 BAM event stream API ............................................................................................................................ 33 Using the BAM interceptor API ............................................................................................................... 36 Inside the BAM Infrastructure .................................................................................................................... 39 Storing BAM data: BAMPrimaryImport database ................................................................................... 39 Archiving BAM Data: BAM Archive database ......................................................................................... 41 Partitioning BAM data............................................................................................................................. 41 Aggregating BAM data ............................................................................................................................ 44 The BAM Event Bus ................................................................................................................................. 50 Querying BAM Data .................................................................................................................................... 52 Overview ................................................................................................................................................. 52 Using BAM with data access libraries ..................................................................................................... 53 Using BAM with SQL Server Reporting Services ..................................................................................... 54 Using BAM with Microsoft Office PerformancePoint Server .................................................................. 56 Using BAM with Microsoft Office SharePoint Server ............................................................................. 61 Accessing BAM by using Web services ................................................................................................... 68 Building a BAM RESTful API ........................................................................................................................ 72 Why do we need a new API? .................................................................................................................. 72 Embracing the Web programming principles ......................................................................................... 73 Brainstorming a BAM RESTful API........................................................................................................... 74

A sample BAM RESTful API...................................................................................................................... 76 Summary ..................................................................................................................................................... 85 Authors........................................................................................................................................................ 85

Executive Summary
Microsoft added Business Activity Monitoring (BAM) to Microsoft BizTalk Server as a means for business analysts to track information that is important to them. One common example of this is tracking the dollar amounts of every transaction processed (i.e., total sales from a particular region). Since BAM was first introduced in 2004, there have been a number of white papers, webcasts, and examples about the basics of BAM. Unfortunately, whats been missing is a deep, low-level review of how BAM works and how BAM can be extended. In this white paper we will take you beneath the covers of BizTalk Server Business Activity Monitoring. After reviewing the basics, we will show you the various methods by which BAM data can be collected, from the pre-built interceptors to the APIs that can be used in custom solutions. The second part of the paper digs into the BAM infrastructure, explaining the BAM databases and what happens to them when BAM models are deployed, and the role of SQL Server Integration Services in BAM. In the third part of this paper we will demonstrate a number of methods for querying BAM data, covering the use of data access libraries, SQL Server Reporting Services, Microsoft Office PerformancePoint Server and Microsoft Office SharePoint Server, and Web services. Finally, this paper covers how BAM can be extended to non-Microsoft technologies through the use of Web services and programming models such as Representational State Transfer (REST). The four sections of this paper are relatively independent of each other. You dont have to read the section on the BAM infrastructure to understand how to extend BAM with REST. Our only suggestion is that you cover the basics at the beginning of this paper. We have also included with the white paper the samples that contain the code that is referenced in this paper. These examples can be compiled and deployed against a basic BizTalk Server installation.

Business Activity Monitoring Overview

Business Activity Monitoring (BAM) is one of the rapidly evolving areas of Business Process Management (BPM). Conceptually, BAM focuses on providing nearly real-time monitoring and analysis of business process activities. We can think of BAM as a bridge between integration and business intelligence (BI) techniques. Ultimately, the goal of BAM is to enable sophisticated BI techniques based on the information collected from business processes. In order to achieve this goal, BAM uses BI models based on the information from different business processes. These models are typically known as event or activity models and are normally translated into relational and multidimensional databases. BAM technologies populate those models using events that describe important milestones of a business process. BAM technologies are responsible for triggering those business events and mapping them to the activity models. The following figure conceptually illustrates a typical BAM architecture.

Business Process

Business Process

Business Process

Information we are interested in

Event

Event

Event

Event

Event

APIs

Activity Models

Event Processors

BAM DB

BAM DB

BAM DB

BAM DB

BAM DB

Figure: BAM concepts As a technology, BAM technology stacks have emerged as a fundamental component of integration and BPM suites such as Microsoft BizTalk Server, IBM WebSphere Process Server, Oracle SOA Suite, etc. This is mostly a strategic decision given that many of the business processes in the industry today are implemented by using those technology suites. In the case of BizTalk Server, the first version of BAM was introduced with the 2004 release and followed by subsequent versions in 2006 and 2006/R2.

BizTalk Server BAM: Exploring the activity model

Although BAM is released as part of BizTalk Server, it provides a flexible framework that can be used to instrument any .NET Framework-based application. Additionally, as we will explore in the last section of this paper, BizTalk Server BAM can be extended to interoperate with non-.NET technologies such as J2EE or dynamic languages. BizTalk Server BAM is built around the concept of an activity model. This type of model defines relevant business information about a business process. The fundamental element of a BizTalk Server BAM activity model is an activity. Activities represent atomic information units that need to be monitored. As an analogy in the database world, we can think of activities as entities of an entity relationship model. In that sense, activities are described by a collection of fields that represent business data or important business milestones. For instance, a banking transaction activity can contain a data field that indicates the monetary amount of the transaction and another milestone field that represents the time in which the transaction was processed. Milestones are a key concept of BAM that, as explained in the previous example, capture important states of a business process. In addition to business milestones (DateTime), activity fields can be of type Text, Float, and Integer. Activities can have relationships with other activities. For instance, the transaction activity can have a relationship with the customer activity. The data included in a set of activities can serve to provide relevant information to different domains. For instance, a bank executive might use a BAM activity set to obtain some financial analysis metrics while an IT manager might be more interested in seeing the average number of failed and successful transactions. To address these scenarios BizTalk Server BAM introduces the concept of an activity view, which projects a specific representation of a set of BAM activities.

One of the main advantages of activity views is that they can combine the information of various BAM activities, providing different real-time perspectives of a business process. Additionally, activity views can capture historical metrics incorporating multidimensional elements such as measures and dimensions. These concepts are the equivalent of the measures and dimensions included in most OLAP engines. Measures are numeric aggregations of an activity field; for instance, a measure AVG(Transaction Amount) will calculate the average of the financial amount of the transactions represented by the banking transaction activity. Measures are typically segmented by dimensions, which are information hierarchies that represent areas of interest in the information; for instance, we can use a Time dimension with three levels (year, month, and day) to analyze the information provided by the measure AVG(Transaction Amount) across different time intervals. The current release of BizTalk Server BAM includes the concept of time, data, numeric range, and progress dimensions. Progress dimension: Represents the creation of aggregations with respect to the progress of activities that are still in process. For instance, a progress dimension can represent the status of a financial transaction with the levels (RECEIVED, PROCESSING, APPROVED, DENIED). Data dimension: Used to categorize an aggregation. Data dimensions are based on the value of string-formatted data items in the BAM activity. Time dimension: Used to create aggregations across defined time segments. For instance, a time dimension with levels Year, Month, Day can be used to segment the measures of the banking transaction activity. Numeric range dimension: Used to categorize aggregations based on friendly names of given numeric ranges. For instance, a numeric range dimension can categorize the amount of a banking transaction into levels Small ($1-$10000), Medium ($10001-$100000), and Large ($100001-$1000000). Dimensions and measures can be used to define real-time or scheduled aggregations. A scheduled aggregation represents a time snapshot of an activity view. In order to execute all the calculations required for the measures and dimensions, this type of aggregation is scheduled to run based on specific time intervals. There are some scenarios in which different parts of an aggregation need to be available nearly in real time. For those scenarios, BizTalk Server BAM introduces the concept of a real-time aggregation (RTA), which executes some of the required calculations as the activity data is being populated. The Inside the BAM Infrastructure section drills down into the aggregation model.

BizTalk Server BAM represents the activity model using an XML-based language known as the BAM definition language. This language is described by the BAM definition schema. The BAM definition schema defines the constraints of the elements of a BAM activity model including activities, views, cubes, etc. Developers can create BAM activity models by using their favorite XML editor or specialized tools like the BAM Add-in for Microsoft Office Excel illustrated in the following section. It is important to notice that the process of defining a BAM activity model is completely independent of BizTalk Server. Developers or analysts can define activity models to instrument applications implemented using heterogeneous technologies. Once created, an activity model is deployed to BizTalk Server so that it can be used at run time. In that sense, the elements of the BAM activity model are translated into relational artifacts deployed in different BizTalk Server databases. Throughout this paper we are going to explore in detail the intricacies of BAM activity models. Specifically, the Inside the BAM Infrastructure section of this paper drills down into the different components of a BAM activity model and how they are represented in the BAM infrastructure. In order to get a better understanding of the process of creating a BAM activity model, the next section will walk us through the process of creating a sample activity model that we will use in the remaining sections of this paper.

Defining a BAM sample activity model

A traditional process of some financial institutions like banks is to process credit card transactions. As part of this process a transaction needs to be approved or denied based on certain criteria like the debt or credit limits associated with the account. Additionally, if the characteristic of a specific transaction matches some well-defined fraud pattern, an alert is triggered in order to take the appropriate actions. In this section, we are going to define an oversimplified transaction activity model that will serve as the basis for all the examples included in this paper. Conceptually, the fundamental elements of our model are the Transaction and Alert activities detailed in the following tables. Field name Amount Transaction_Approved Description Decimal: Amount of the intended transaction Milestone: Time at which the transaction was

approved Transaction_Created Transaction_Failed Transaction_Failed_Alert_Triggered Milestone: Time at which the transaction was created Milestone: Time at which the transaction failed Milestone: Time at which the transaction failed and an alert was fired based on a fraud criteria Customer_Age Customer_Name Integer: The age of the customer initiating the transaction Text: The name of the customer initiating the transaction Figure: Transaction activity Field name Code Description Alert_Created Alert_Processed Description Text: Code associated with the alert Text: Description of the alert Milestone: Time at which the alert was created Milestone: Time at which the alert was processed Figure: Alert activity Using the BAM Excel Add-in we can create the activity definition that contains the previous elements as illustrated in the following figure.

Figure: BAM activity definition At this point, we can extend the information of the TRANSACTION activity with some multidimensional elements that facilitate more sophisticated historical analysis over the activity data. To achieve that, we will create the following dimensions and measures. Element Count_Transaction Avg_Transaction Total_Transaction Count_Alert Transaction_Time_Dim Type Measure Measure Measure Measure Dimension Description Number of processed transactions Transaction amount average Total amount of the processed transactions Number of triggered alerts A time dimension to segment the transaction information based on Year, Month, Day, Hour levels Transaction_Stg_Dim Dimension A process dimension to segment the transaction information based on its different milestones Amount_Dim Dimension A numeric range dimension

categorizing the amount of a transaction into Small, Medium, or Large Alert_Time_Dim Dimension A time dimension segmenting the alert time into Year, Month, Day, Hour levels

Figure: Transaction dimensions and measures Using the Business Activity Monitoring View Wizard of the Excel Add-in we can define those elements as illustrated in the following figure.

Figure: BAM activity view aggregation elements Each of the multidimensional elements is defined separately.

Figure: Defining measures and dimensions After defining the multidimensional elements, we can add the final touches to our BAM activity view by creating specific PivotTable reportsusing the multidimensional elements of the activity model. The following figure illustrates this step.

Figure: Customizing a BAM activity view After exporting the Excel definition to XML we obtain a representation of the TRANSACTION activity model compatible with the BAM definition schema. Although the BAM Excel Add-in is, undoubtedly, the preferred tool for creating BAM activity models, developers can build custom tools that provide a more sophisticated user experience to help domain experts to create specific activity models. Similarly to the Excel Add-in, these would translate the graphical representation into the XML that can be processed by BizTalk Server. The following code illustrates sections of the XML representation of our sample activity model. <BAMDefinition xmlns="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM"> <Activity Name="TRANSACTION" ID="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC"> Sections have been omitted for brevity.. </Activity> Sections have been omitted for brevity.. <View Name="TRANSACTION" ID="ID12CBC52A28684F3BAE8F742DCF545A18"> <ActivityView Name="ViewTRANSACTION" ID="ID94EBCE7C807F48BBA5BA96F3F65B9619" ActivityRef="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC"> <Alias Name="AMMOUNT" ID="ID0399A8CD04CF49E0AF981322275908F9"> <CheckpointRef>ID7691FFDDA6074E2F88597ED5EF3967F5</CheckpointRef> </Alias> <Alias Name="CUSTOMER_AGE" ID="IDD82D0B7C03E14DAAAC7F245195F248EE"> <CheckpointRef>ID44ED38B3A28A4D04A674F9D8CE3238A6</CheckpointRef> </Alias> <Alias Name="CUSTOMER_NAME" ID="IDBF7F4BABE42A4CD0BC1B7F5B5BB941E9"> <CheckpointRef>ID949D996AC41F4E499944F8C388E2C439</CheckpointRef> </Alias> <Alias Name="TRANSACTION_APPROVED" ID="IDB74B11B4122A4C4D8523DFA1BB607D5C"> <CheckpointRef>ID604617A21C82456AB8BE76DBA9B2DCA8</CheckpointRef> </Alias> </ActivityView> </View> Sections have been omitted for brevity.. <Cube Name="TRANSACTION1" ID="IDB9050B5EB8BB4661ABE1CDA4BA729D55" CreateOlapCube="true" ActivityViewRef="ID94EBCE7C807F48BBA5BA96F3F65B9619"> <Measure Name="AVG_AMMOUNT" ID="ID0A47D6F650FA49AC8FDAC18D7F5E78EE" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Avg"/> <Measure Name="TOTAL" ID="IDD191BAAA7E964CE0B16710AC932C7343" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Sum"/> <Measure Name="COUNT_TRANSACTION" ID="IDB3E446EFDFFC4AC6BD364FBAA34EBA44" AliasRef="ID0399A8CD04CF49E0AF981322275908F9" AggregationFunction="Count"/> <TimeDimension Name="TRANSACTION_TIME_Dim" ID="IDC2CAA2E2B68C4C429CD8DEC19E12745B" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"> <TimeLevel>Year</TimeLevel> <TimeLevel>Week</TimeLevel> <TimeLevel>Day</TimeLevel>

<TimeLevel>Hour</TimeLevel> <TimeLevel>Minute</TimeLevel> </TimeDimension> <NumericRangeDimension Name="AMMOUNT_Dim" ID="IDE79FC5B4EBE848BCA78F67FD886E0A5D" NumericAliasRef="ID0399A8CD04CF49E0AF981322275908F9"> <Range Name="SMALL" ID="ID3853D9BC675A436CB2A1C7ACDB4E873D" From="5" To="100"/> <Range Name="MEDIUM" ID="IDB6567ACBFEF14C91BCD2073D569AD85E" From="101" To="1000"/> <Range Name="LARGE" ID="IDBFCC85280801407AB692DD7AFD76A4D2" From="1001" To="10000"/> </NumericRangeDimension> <ProgressDimension Name="TRANSACTION_STG_Dim" ID="ID7BD612325E5C4845B8F716469958AFC4"> <ProgressStage Name="TRANSACTION_CREATED" ID="IDFC5CBDAE550F4F5EBA6C3BBB6C864E3B" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"> <ProgressStage Name="CREATED" ID="ID1DD5128DC0B14CDFBE049A514666A782" TimeStampAliasRef="IDA03C947A157747CCA6369BADE07098D4"/> <ProgressStage Name="FAILED" ID="ID0788A06545854B3298FC410DAE1AF04D" TimeStampAliasRef="IDAC79D047E6134D3E922B23FC650D13D5"/> <ProgressStage Name="FAILED_ALERT" ID="IDDD2D200E51744439A65DA2CB4C2950B4" TimeStampAliasRef="ID18F90EE60D894A739CD13D081AB63387"/> <ProgressStage Name="APPROVED" ID="IDEB527B961B864E7C96E770AEB0D2AD68" TimeStampAliasRef="IDB74B11B4122A4C4D8523DFA1BB607D5C"/> </ProgressStage> </ProgressDimension> </Cube> Sections have been omitted for brevity.. </BAMDefinition> Figure: BAM XML definition After the activity model is in a form compatible with the BAM definition schema, it can be deployed to BizTalk Server by using the BAM deployment utility with the following syntax: >bm.exe deploy-all DefinitionFile:<BAM activity model file>. Once deployed, the BAM activity model can be updated and redeployed by using a similar mechanism. At this point, the BizTalk Server BAM runtime will create the necessary artifacts to enable the activity model so that it can be used by different applications. Technologies like BizTalk Server, Windows Communication Foundation, and Windows Workflow Foundation have embraced BAM into their programming model, providing tools and technologies that facilitate the instrumentation of applications using purely declarative mechanisms. More sophisticated scenarios or applications developed using other frameworks can also populate the BAM activities by taking advantage of the BAM APIs. The following sections explore the different alternatives that developers can use to instrument their applications using BAM.

How BAM differs from data warehousing

The use of BAM enables near real-time, business-centric visibility of enterprise processes. In that sense, BAM extrapolates meaningful business metrics based on the events produced by business processes. This level of visibility enables executives or other domain experts to monitor business processes, correlate metrics and analytics down to the actual business process, and react to and predict business scenarios based on real-time analysis. It is important to highlight a distinction between BAM and traditional data warehousing techniques. Although both techniques are enabled by relational and multidimensional database infrastructures, BAM is centered on capturing, processing, and analyzing real-time business process events, while data warehousing techniques are traditionally more focused on enabling historical aggregation and analysis of a variety of data sources. Along those lines, BAM implementations typically handle a very specific set of information related to a series of business processes. Data warehouses, on the other hand, are commonly used for representing aggregated data views containing a broader range of information than what is normally associated with a business process. Following that thinking, data warehouses can be seen as a complement to BAM technologies in order to enrich the information included in the BAM model. For instance, a BAM process might keep track of the business activities associated with one or more specific accounts. This information could then be complemented with elements from an ERP data warehouse that details information about the opportunities associated with those accounts as well as metrics about their historical behavior.

Instrumenting Applications by Using BAM

As weve explored in the previous section, BizTalk Server Business Activity Monitoring (BAM) provides very flexible mechanisms to express activity models that describe the information that needs to be monitored. However, the BizTalk Server BAM model does not impose any constraints on how the monitored information is collected and how the activity model is populated. This flexibility allows multiple technologies to leverage the benefits of BizTalk Server BAM in an optimal way. In order to make that integration even more flexible, some of those technologies have adopted domain-specific declarative mechanisms to indicate what information needs to be collected and how that information is mapped to a BAM activity model. BizTalk Server, Windows Communication Foundation (WCF), and Windows Workflow Foundation (WF) rank among the Microsoft technologies that have adopted declarative mechanisms to interact with BAM. However, BAM scenarios can be arbitrarily complex and some of them might require the use of programmatic interfaces to interact with a BAM activity model. This section drills down into the details of the declarative and programmatic technologies that developers can leverage to instrument their applications by using BAM.

Using BAM declaratively

Instrumenting an application by using BAM is a combination of two basic principles: knowing when to capture the application data and knowing how to map that data to BAM activities. Those two principles are easily applicable when we are talking about data or messagedriven applications such as those implemented with technologies like BizTalk Server, WCF, or WF. What makes these types of applications special? First, the data processed in those applications is represented by using a queryable format from which, at any point, the information can be extracted. Examples of that format are SOAP envelopes, BizTalk messages, or context properties. Secondly, applications implemented by using these technologies follow a series of execution points at which the flow can be intercepted and consequently the information can be extracted. For instance, WCF services flow messages through a dispatcher pipeline on which they can be intercepted by using components like channels or message inspectors. Following these two principles, we can think of developing generic technology components that can be configured to intercept the application data at different points, query the data for specific information, and finally map that information to a BAM activity model. Ideally, the applications will have no dependencies on BAM and the whole process will be driven

declaratively through configuration. Obviously, the complete picture is more complex but it follows the same basic concepts. As you might already know, technologies like BizTalk Server, WCF, and WF have taken the approach explained previously, providing declarative mechanisms that accelerate the instrumentation of applications by using BAM. The following section takes a deep look at the internals of those mechanisms and how developers can benefit from using them.

Using BAM from BizTalk Server

As the technology hosting the core components of the BAM infrastructure, BizTalk Server provides broad support for BAM in terms of tools, APIs, and implementation scenarios. The BizTalk Server BAM runtime provides the components for populating BAM activity instances from orchestration persistence points or pipeline components. Essentially, BizTalk Server will populate a BAM activity model using the data contained in BizTalk messages or context properties. Lets start with the following orchestration that processes a financial transaction. (The entire source code is included in the samples that accompany this paper.)

Figure: Financial transaction business process Our sample orchestration receives a message with the transaction information and applies some simple rules to determine whether the transaction can be approved. Luckily for us, the financial transaction processes in the real world are more complex. Having this orchestration, we can start thinking about how to populate our sample BAM activity model using the data contained in the BizTalk messages. For most of the scenarios, the easiest way to accomplish that is by using the BizTalk Server Tracking Profile Editor (TPE). This tool allows developers to create sophisticated mappings between BizTalk artifacts and a BAM activity model. The following figure illustrates the use of the TPE for populating our sample BAM activity model. (The entire tracking profile can be found with the samples that accompany this paper.) You can find more details about TPE in the online documentation.

Figure: Using TPE with the transaction activity model The mappings created by using the TPE are serialized into an XML file that can be interpreted by the BAM components. The following code illustrates a segment of our sample TPE file. <TrackingProfile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" VersionGuid="00000000-0000-00000000-000000000000" Name="TRANSACTION"> <Dimension Name="ActivityID" DataType="TraceID" /> <Dimension Name="TRANSACTION_CREATED" DataType="DATETIME"> <DataLevel Name="Created" SourceTypeSelected="Orchestration Payload" TargetAssemblyName="BAM-BTS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=79de3c67dc9393f3" OrchestrationReference="BAM_BTS.FinancialTransactionOrchestration" ShapeID="401586f7-f645-4936-a471-64d825edf85d" MessageName="InMsg" MessagePart="part" SchemaName="BAM_BTS.TransactionRequest" MessageDirection="Out" SomXPath="/*[local-name()='&lt;Schema&gt;' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[local-name()='Request' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[localname()='Created' and namespace-uri()='']" XPath="/*[local-name()='Request'

and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[localname()='Created' and namespace-uri()='']" /> </Dimension>

Other elements...

</TrackingProfile> Looking at the code we can notice that it describes how the TRANSACTION_CREATED activity field is populated from a BizTalk message payload. The highlighted sections of the tracking profile XML detail the orchestration shape, message, and XPath sentence that are used to populate the TRANSATION_CREATED activity. At this point, the TPE profile can be deployed instructing the BAM runtime where and how to populate the different elements of the BAM activity model.

Inside the BAM BizTalk interceptor

The architecture of the BizTalk Server BAM runtime is complex enough to require a separate paper. We will try to cover some of the most important components in this section. After its deployed, a BAM tracking profile is stored in the bam_Metadata_TrackingProfiles table in the BAMPrimaryImport database, with a reference to it in the bam_TrackingProfiles table of the BizTalk Management database (BizTalkMgmtDb). By default the BizTalk Server runtime uses the different interceptors that can be found in the Trackinginterceptor table of the BizTalk Management database. Fundamentally, the BizTalk engine uses the Microsoft.XLANGs.Mozart.NativeInterceptor and Microsoft.XLANGs.Mozart.BIWrapperInterceptor, which inherit from the Microsoft.XLANGs.RuntimeTypes.BaseInterceptor class. The BaseInterceptor class is the main interface that the BizTalk Server runtime components use to interact with the BAM infrastructure.

BTXEvents

BaseInterceptor

BIWrapperInterceptor

NativeInterceptor

BAM APIs

BAMPrimary Import

Figure: BizTalk Server BAM interceptor The complete architecture of the BizTalk Server BAM runtime is far more complex than what has been explained in this section, but we intended to illustrate some of its fundamental components.

Using BAM from Windows Communication Foundation and Windows Workflow Foundation
With the release of BizTalk Server 2006 R2, the use of BAM has been extended to Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF). Both technologies introduced a declarative mechanism to express the element that should be monitored by BAM. This mechanism is based on an interceptor configuration file that declares how the BAM activities are populated from WCF or WF elements. The interceptor configuration files for both WCF and WF share the same structure based on the CommonInterceptorConfiguration.xsd XML schema, which describes generic events that need to be intercepted and mapped to BAM activities. The common configuration file provides a schema that can be reused across the WCF and WF BAM interceptors. However, given the differences between the WCF and WF programming models, that common schema needs to be extended to incorporate the

specific interception mechanisms for collecting the BAM data. For instance, developers using the BAM WCF interceptor will, most likely, extract the data from a SOAP or XML message intercepted at different stages of the client or dispatcher runtimes. On the other hand, developers using the WF interceptor will extract the activity data from WF-specific components such as workflow properties or user data types. Essentially, in order to provide a complete declarative mechanism for populating a BAM activity model, the WCF and WF interceptors need to extend the common configuration schema with specific elements of their programming model.

BAM WCF interceptor

The BAM WCF interceptor is a flexible technology that allows developers to populate BAM activity models based on the information exchanged by WCF services and clients. In order to model the interactions with the BAM model, the BAM WCF interceptor extends the basic configuration model with WCF-specific elements. Essentially, the interceptor configuration file models how WCF messages are mapped to BAM activities throughout the different stages of the client or dispatcher runtimes. Lets take the following WCF service that processes a financial transaction. public class FinancialTransactionService: IFinancialTransactionService { public TransactionResponse ProcessTransaction(TransactionRequest transactioninfo) { //Implementation omitted for brevity.... } } [ServiceContract] public interface IFinancialTransactionService { [OperationContract] TransactionResponse ProcessTransaction(TransactionRequest transactioninfo); } [DataContract] public class TransactionRequest { private string transactionId; private double amount; private string customerName; private int customerAge; [DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; } }

[DataMember] public double Amount { get { return amount; } set { amount= value; } } [DataMember] public string CustomerName { get { return customerName; } set { customerName= value; } } [DataMember] public int CustomerAge { get { return customerAge; } set { customerAge = value; } } } [DataContract] public class TransactionResponse { private string transactionId; private bool success; private bool alertFired; private string alertDescription; [DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; } } [DataMember] public bool Success { get { return success; } set { success= value; } } [DataMember] public bool AlertFired { get { return alertFired; } set { alertFired= value; } } [DataMember] public string AlertDescription { get { return alertDescription; } set { alertDescription= value; } } } Figure: Sample WCF service (the complete implementation can be found in the samples included with this paper)

At this point we can create a WCF interceptor configuration file that maps the WCF messages to the sample activity model created in the previous section. The following figure highlights sections of the interceptor configuration file that indicate how the TRANSACTIONID element is populated based on the contents of the upcoming message. <ic:InterceptorConfiguration xmlns:ic="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorC onfiguration" xmlns:wcf="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WcfIntercep torConfiguration"> <ic:EventSource Name="TransactionEventSource" Technology="WCF" Manifest="BAM_WCF.IFinancialTransactionService, BAM-WCF, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"> <wcf:NamespaceMappings> <wcf:Namespace Prefix="soap" Uri="http://schemas.xmlsoap.org/wsdl/soap/" /> <wcf:Namespace Prefix="cs" Uri="http://tempuri.org/"/> </wcf:NamespaceMappings> </ic:EventSource> <ic:BamActivity Name="TRANSACTION"> <ic:OnEvent IsBegin="true" IsEnd="false" Name="TransactionRequest" Source="TransactionEventSource"> <ic:Filter> <ic:Expression> <wcf:Operation Name="GetServiceContractCallPoint" /> <ic:Operation Name="Constant"> <ic:Argument>ServiceRequest</ic:Argument> </ic:Operation> <ic:Operation Name="Equals" /> <wcf:Operation Name="GetOperationName" /> <ic:Operation Name="Constant"> <ic:Argument>ProcessTransaction</ic:Argument> </ic:Operation> <ic:Operation Name="Equals" /> <ic:Operation Name="And" /> </ic:Expression> </ic:Filter>

Rest of the configuration omitted for brevity

<ic:Update DataItemName="TRANSACTIONID" Type="NVARCHAR"> <ic:Expression> <wcf:Operation Name="XPath"> <wcf:Argument> //*[local-name()= 'TransactionId']/text() </wcf:Argument> </wcf:Operation> </ic:Expression> </ic:Update> </ic:OnEvent> </ic:BamActivity> </ic:InterceptorConfiguration> Figure: BAM WCF interceptor configuration file (the complete file is included in the samples provided with this paper) Looking at the preceding interceptor configuration file, we can notice that it is using a combination of elements from the namespaces

http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorConfiguration (ic: prefix) and http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WcfInterceptorConfiguration (wcf: prefix). The elements prefixed by ic: are part of the common configuration schema, which includes the instructions that are common for the WCF and WF interceptors like updating an activity. Additionally, the elements prefixed by wcf: refer to the elements of the WCF BAM configuration schema, which states WCF-specific instructions like detailing the operation name that needs to be intercepted. In that sense, the <wcf:Operation> element is the fundamental extension of the BAM WCF interceptor schema and encloses the necessary logic to extract the activity data from WCF messages. The interception logic can be applied at different levels of the client and service message processing pipeline such as ServiceRequest, ServiceReply, ClientRequest, ClientReply, etc. More details about how to use the WCF and WF interceptors can be found on MSDN at (http://go.microsoft.com/fwlink/?LinkId=134485) The interceptor configuration file can be deployed by using the BAM deployment utility with the following arguments: >bm.exe deploy-interceptor filename:<Interceptor configuration file> The BAM interceptor configuration is added to a WCF service by using an endpoint behavior that can be configured declaratively as the following: <configuration> <system.serviceModel>

<extensions> <behaviorExtensions> <add name="baminterceptor" type="Microsoft.BizTalk.Bam.Interceptors.Wcf.BamEndpointBehavior, Microsoft.BizTalk.Bam.Interceptors, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/> </behaviorExtensions> </extensions>
<services> <service name="BAM_WCF.FinancialTransactionService" behaviorConfiguration="ExposeContract"> <endpoint behaviorConfiguration="BAMTracking" /> </service> </services> <behaviors> <endpointBehaviors> <behavior name="BAMTracking"> <baminterceptor ConnectionString="Initial Catalog=BAMPrimaryImport;Data Source=.;Integrated Security=SSPI;" /> </behavior> </endpointBehaviors> </behaviors> </system.serviceModel>

</configuration> Additionally, the BAM interceptor can be added programmatically. private static void StartHost() { using (ServiceHost host = new ServiceHost(typeof(FinancialTransactionService))) { WCFBAMProgExtension bamExtension= new WCFBAMProgExtension(); bamExtension.PrimaryImportConnectionString = "Connection String"; bamExtension.PollingIntervalSec= 1500; foreach(ServiceEndpoint endpoint in host.Description.Endpoints) endpoint.Behaviors.Add(bamExtension.CurrentBAMBehavior); host.Open(); Console.ReadLine(); host.Close(); } } internal class WCFBAMProgExtension : BamBehaviorExtension { public IEndpointBehavior CurrentBAMBehavior { get { return (IEndpointBehavior)base.CreateBehavior(); } } } As illustrated in the preceding code, adding the BAM WCF interceptor programmatically is no different from adding any other WCF endpoint behavior to the hosts endpoint collection. Given that the actual endpoint behavior is implemented by the BamEndpointBehavior internal class, we need to access it through the BamBehaviorExtension class. To achieve that, we need to create a class that inherits from BamBehaviorExtension to access the protected CreateBehavior method that returns an instance of the BamEndpointBehavior class. You can find more details about this in the next section. From a developer perspective, the BAM endpoint behavior is the mechanism used to instruct the WCF runtime that a particular service will be monitored by using BAM. The specific instructions about how the service will be monitored are abstracted through the BAM interceptor configuration file. To get a more detailed understanding of how all these components fit together we should take a look at the BAM WCF interceptor architecture.

Inside the BAM WCF interceptor architecture

As we explained in the previous section, the BAM WCF interceptor leverages endpoint behaviors as the extensibility point used to insert the BAM interception mechanisms into the WCF client and dispatcher runtimes. When the endpoint behavior is loaded by the client or service host, it plugs in other WCF components that execute the BAM monitoring logic. The following figure illustrates a high-level view of the BAM WCF interceptor.

BAMBehaviorExtension

BAMEndpointBehavior

BAMMessageInspector

BAMInspector

BAMWcfInterceptor

BAM APIs

BAMPrimary Import

Figure: BAM WCF interceptor high-level architecture

Like any other WCF behavior, the BAMEndpointBehavior is configured by a behavior extension element (BAMBehaviorExtension). Once instantiated, the BAMEndpointBehavior behavior inserts a WCF message inspector (BAMMessageInspector, which inherits from the BAMInspector class) into the client or dispatcher runtime. At run time, the message inspector intercepts a WCF message and creates a BAM interceptor (BAMWcfInterceptor), which interprets the interception configuration file in order to log the data to the BAMPrimaryImport database. As you might imagine, there are other components involved in the BAM WCF interceptor architecture, and it is not our goal to cover them all in detail. From a developer standpoint, the complexity of this architecture is completely abstracted behind the BAMBehaviorExtension behavior extension element.

BAM WF interceptor
Similar to the BAM WCF interceptor, the BAM WF interceptor extends the BAM programming model into WF. Using this interceptor, developers can populate BAM activity models based on the information processed by WF workflows. The mechanism to declare how a BAM

activity model will be populated is also based on an interceptor configuration file, which extends the basic interceptor configuration model with specific WF elements. To better understand how to use the BAM WF interceptor lets start with the following workflow that processes a financial transaction.

Figure: Sample transaction workflow

This workflow receives the financial transaction information and executes a very simple logic to approve or reject the transaction. For this example, the fields that we would like to monitor are declared as properties of the workflow class as illustrated in the following code. public sealed partial class FinancialTransactionWorkflow: SequentialWorkflowActivity { private double transactionID;

...rest of the implementation has been omitted for brevity

public double TransactionID { get { return transactionID; } set { transactionID = value; } }

private void TransactionApproved_ExecuteCode(object sender, EventArgs e) { transactionID= sample transaction id.;

...rest of the implementation has been omitted for brevity

} } Figure: Transaction workflow implementation (you can find the complete implementation as part of the samples included with this paper) In order to populate a BAM activity model based on the state of the WF workflow we need to create a BAM WF interceptor configuration file like the one in the following figure. <ic:InterceptorConfiguration xmlns:ic="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/InterceptorC onfiguration" xmlns:wf="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM/WorkflowInte rceptorConfiguration"> <ic:EventSource Name="FinancialTransactionWorkflow" Technology="WF" Manifest="BAM_WF.FinancialTransactionWorkflow, BAM-WF, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/> <ic:BamActivity Name="TRANSACTION"> <ic:OnEvent Name="MyWorkflowEventEnd1" Source="FinancialTransactionWorkflow" IsBegin="false" IsEnd="true"> <ic:Filter> <ic:Expression> <wf:Operation Name="GetActivityName"/> <ic:Operation Name="Constant"> <ic:Argument>TransactionApproved</ic:Argument> </ic:Operation> <ic:Operation Name="Equals"/> <wf:Operation Name="GetActivityEvent"/> <ic:Operation Name="Constant"> <ic:Argument>Closed</ic:Argument> </ic:Operation> <ic:Operation Name="Equals"/> <ic:Operation Name="And"/> </ic:Expression> </ic:Filter>

Rest of the configuration omitted for brevity

<ic:Update DataItemName="TRANSACTIONID" Type="NVARCHAR"> <ic:Expression> <wf:Operation Name="GetWorkflowProperty"> <wf:Argument>TransactionId</wf:Argument> </wf:Operation> </ic:Expression> </ic:Update> </ic:OnEvent> </ic:BamActivity> </ic:InterceptorConfiguration> Figure: BAM WF interceptor configuration file (the complete file is included in the samples provided with this paper) Although it is not an objective of this paper to explain in detail the different sections of the interceptor configuration file, there are a few aspects that are worth highlighting. The BAM

WF interceptor captures different events at the workflow, activity, and user levels. On each one of those tracking points, we can extract the data from the workflow by using some of the custom operations provided by the interceptor. For instance, in this case, we are interested in the Closed event of the TransactionApproved WF activity (see filter section). When that event occurs, the interceptor will extract the TransactionID field of the Transaction activity and map it to the TransactionID activity field. It is important to notice that there are many other events and functions that can be combined to address diverse BAM WF monitoring scenarios. After the interceptor configuration file is created, it can be deployed by using the BAM deployment utility. >bm.exe deploy-interceptor filename:<Interceptor configuration file> At this point, we can instruct the WF runtime to use the BAM WF interceptor. The interaction between the WF runtime and the BAM WF interceptor is abstracted by the BamTrackingService tracking service, which can be added declaratively as the following: <WorkflowServiceContainer> <Services> <add type="Microsoft.BizTalk.Bam.Interceptors.Workflow.BamTrackingService, Microsoft.BizTalk.Bam.Interceptors, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" ConnectionString="Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport" PollingIntervalSec="5"/> </Services> </WorkflowServiceContainer> Figure: Adding the BAM tracking service declaratively Similar to other WF tracking services, the BamTrackingService can be also added by using code. string connectionString = "My Connection String."; int pollingInterval = 5; WorkflowRuntime workflowRuntime = new WorkflowRuntime(); workflowRuntime.AddService(new BamTrackingService(connectionString, pollingInterval)); Figure: Adding the BAM tracking service imperatively After this, every time an instance of the WF workflow executes, the BAM WF interceptor will populate the BAM activity model based on the instructions declared in the interceptor configuration file. From a developer standpoint, adding the BAM WF interceptor is no

different from adding any other WF tracking service. This service abstracts the complexities of the BAM WF interceptor architecture that we will explore in the following section.

Inside the BAM WF interceptor architecture

As explained in the previous section, the BAM WF interceptor is based on the WF tracking services framework. This framework is designed to enable hosts to observe workflow instances during execution by capturing events that are raised during workflow execution. The framework is based on a pluggable design pattern that enables hosts to facilitate complex tracking scenarios by including different tracking services. At a high level, a tracking service uses two fundamental components: tracking profiles and tracking channels. Tracking profiles are used by the tracking service as a mechanism to communicate the events it would like to receive from the runtime. This guarantees that the tracking service will only be called for specific events instead of the entire event set produced by a workflow during its execution. The second component used by the tracking service is the tracking channel. These components are responsible for receiving the tracking events and data about the execution of a workflow instance. With this knowledge of a tracking service model it is not difficult to imagine what the highlevel architecture of the BAM WF interceptor looks like.

BamTrackingProfile Manager BamTrackingService BamTrackingChannel

TrackingProfile

WorkflowInspector

BAM APIs

BAMPrimary Import

Figure: High-level architecture of the BAM WF interceptor In the previous section we explained how applications need to add the BAM tracking service (BamTrackingService class) in order to indicate to the WF runtime that they want to monitor

certain events by using BAM. At this point, the WF runtime calls the GetProfile operation of the BamTrackingService, which uses the BamProfileManager class to instantiate a tracking profile based on the configuration declared in the interceptor configuration file. The WF runtime also invokes the GetTrackingChannel operation of the BamTrackingService in order to get the tracking channel that will be used to receive the events. During its construction, the BamTrackingChannel instantiates a BAM interceptor (WorkflowInterceptor class) object, which abstracts the interaction with the BAM event streaming APIs. At run time, the data is sent to the tracking channel, which uses the interceptor to efficiently populate the BAM activity model.

Using BAM programmatically

Undoubtedly, the use of declarative mechanisms like the ones provided by BizTalk Server, WCF, and WF eliminates a lot of the complexities in the implementation of BAM solutions. However, one of the tradeoffs with using those mechanisms is that it requires a static correlation between the application data and the BAM activity model. Often, applications need to interact with BAM in a more dynamic manner that cant be easily described in a declarative way. Those scenarios are a better fit for the use of BAM APIs as mechanisms to dynamically populate the BAM activity model. In the current BAM release there are two fundamental APIs that developers can use to programmatically populate activity models: streaming and interceptor. As you might imagine, both APIs are the foundation of the runtime mechanisms implemented by BizTalk Server, WF, and WCF.

BAM event stream API

The event stream API is the fundamental mechanism that developers use to programmatically interact with a BAM activity model. These APIs provide the interfaces to execute operations like those that modify the state of a BAM activity model. The following table summarizes some of the fundamental operations of the BAM EventStream API. Operation signature(C#) public virtual void BeginActivity ( string activityName, string activityInstance ) public virtual void EndActivity ( string activityName, string activityInstance ) Description Instantiates a new BAM activity with a specific activity identifier

Completes a specific instance of a BAM activity

public virtual string string params )

activityName, activityInstance, Object[] data

void UpdateActivity (

Updates fields of a BAM activity

public virtual void EnableContinuation ( string activityName, string activityInstance, string continuationToken ) public virtual void Flush ()

Enables continuation for a BAM activity instance

Flushes the changes to the BAM database

Although the operations listed in the preceding table are not the only ones included in the BAM event streaming API, they summarize some of the most common functionalities needed to interact with a BAM activity model. Those interactions can be notably different based on different factors such as the invoking application (orchestration, pipeline, component, etc.), synchronization model (synchronous vs. asynchronous), or the information spectrum (message, context properties, etc). This is why the current release of BizTalk Server 2006 R2 includes the following variations of the event stream API that are optimized for specific scenarios. DirectEventStream From an infrastructure perspective, the DirectEventStream class is the simplest way to interact with a BAM activity model. This class is used to log BAM activity data synchronously to the BAMPrimaryImport database. The following code illustrates a simple case of how to use this API. string activityID = Guid.NewGuid().ToString(); DirectEventStream eventStream = new DirectEventStream("BAM Primary Import DB connection string", 1); eventStream.BeginActivity("TRANSACTION", activityID); eventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", 123456, "TRANSACTION_CREATED",DateTime.UtcNow); eventStream.EndActivity("TRANSACTION", activityID); eventStream.Flush(); Figure: Using DirectEventStream The synchronous nature of this API can have a direct impact on the performance of the calling application. After updating a BAM activity, the calling application normally needs to wait until all the database artifacts, such as stored procedures and RTA triggers associated

with that activity, execute. This API is normally recommended for applications that need near real-time processing and minimum latency in their BAM infrastructure. BufferedEventStream The BufferedEventStream class is a great alternative in order to mitigate some of the performance problems that might occur when using the DirectEventStream class. Instead of logging the data directly to the BAMPrimaryImport database, the BufferedEventStream inserts the data asynchronously into the BizTalk MessageBox database. The activity data is stored in an optimized binary format into specific tables named Tracking_Data_<number>_<number>. After that, the BAM Event Bus service can extract the data and insert it into the appropriate tables in the BAMPrimaryImport database. The event bus keeps transaction consistency throughout the entire process, which guarantees the integrity of the data. The following code shows a basic example of how to use the BufferedEventStream class. string activityID = Guid.NewGuid().ToString(); BufferedEventStream eventStream = new BufferedEventStream ("BAM Message Box DB connection string", 1); eventStream.BeginActivity("TRANSACTION", activityID); eventStream.UpdateActivity("TRANSACTION ", activityID, "TRANSACTIONID", 123456, "TRANSACTION_CREATED",DateTime.UtcNow); eventStream.EndActivity("TRANSACTION", activityID); Figure: UsingBufferedEventStream The BufferedEventStream API can be really advantageous for applications that need to maximize the performance of the interactions with the BAM API. As a tradeoff, the use of asynchronous mechanisms introduces an extra level of latency on the processing of the BAM activity data. OrchestrationEventStream The OrchestrationEventStream (OES) is one of the artifacts developers should consider in order to programmatically populate a BAM activity model from within a BizTalk orchestration. The OrchestrationEventStream (OES) class provides similar mechanisms to the ones implemented by the BufferedEventStream (BES) class. In that sense, the OES class also uses an asynchronous model to first log the data to the BizTalk MessageBox database so that an external process can move it to the BAMPrimaryImport database. OrchestrationEventStream.BeginActivity("TRANSACTION", activityID); OrchestrationEventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", 123456,

"TRANSACTION_CREATED",DateTime.UtcNow); OrchestrationEventStream.EndActivity("TRANSACTION", activityID); Figure: Using OrchestrationEventStream The fundamental difference between the OES and the BufferedEventStream is that the former is optimized to work in the context of an orchestration. In that sense, the activity data created by using the OES class will be persisted when the orchestration reaches a persistent point. For instance, assume we call the OES class within an Expression shape and that is followed by a Send shape (persistent point). If an exception occurs before we reach the Send shape, the BAM data wont be persisted because the orchestration will be rehydrated from the previous persistent point. This behavior is different from the BES, which persists the data as soon as the Flush operation is called. MessagingEventStream The MessagingEventStream (MES) API follows a similar approach to the OrchestrationEventStream (OES) and BufferedEventStream (BES) with the difference that MES is optimized for processing the BAM activity within a pipeline context. In that sense, the BAM activity data is persisted within a pipeline transaction. The following code illustrates the use of MES. string activityID = Guid.NewGuid().ToString(); IPipelineContext context; EventStream eventStream = context.GetEventStream(); eventStream.BeginActivity("TRANSACTION", activityID); eventStream.UpdateActivity("TRANSACTION", activityID, "TRANSACTIONID", 123456, "TRANSACTION_CREATED",DateTime.UtcNow); eventStream.EndActivity("TRANSACTION", activityID); Figure: Using MessagingEventStream

Using the BAM interceptor API

As weve explored in the previous section, the BAM event stream APIs abstract different interaction models used by applications to populate BAM activities. When using those APIs, applications still need to implement the logic that captures the data of interest and map it to the BAM activity model. BAM streamlines this process by using the BAM interceptor API, which allows developers to model how an application will be instrumented by using BAM. The first task in instrumenting an application with a BAM interceptor is to create a BAM configuration (ActivityInterceptorConfiguration) that declares a set of relevant steps that need to be monitored. Each step also expresses what data needs to be captured and what

element of the BAM activity model should be populated. The following code illustrates a sample configuration for our activity model. ActivityInterceptorConfiguration configuration = new ActivityInterceptorConfiguration("TRANSACTION"); configuration.RegisterStartNew("Start", ""); configuration.RegisterDataExtraction("TRANSACTIONID", "LogInfo", "//*[localname()= 'TransactionID']"); configuration.RegisterDataExtraction("TRANSACTION_CREATED", "LogInfo", "//*[local-name()= 'Transaction_Created']"); configuration.RegisterEnd("End"); BAMInterceptor interceptor= new BAMInterceptor(); configuration.UpdateInterceptor(interceptor); Figure: Configuring a BAM interceptor In the preceding code, you can see that the configuration does not express how the data should be extracted from the application data structures (XML, objects, etc.) in order to populate the corresponding BAM activities. A BAM interceptor typically delegates this process to an implementation of the IBAMDataExtractor interface provided by the application. The following code shows a sample BAM data extractor that uses XPath to extract the relevant information from an XML document. public class SampleDataExtractor : IBAMDataExtractor { public object GetValue(object trackItem, object location, object data) { if (location == "Start") return Guid.NewGuid().ToString(); string xpath = (string)trackItem; if (xpath.Length > 0) { XPathNavigator xn = (XPathNavigator)data; XPathNavigator dataItem = xn.SelectSingleNode(xpath); if (null == dataItem) return null; else return dataItem.InnerXml; } else return DateTime.UtcNow; } } Figure: Sample IBAMDataExtractor implementation At run time the application calls the OnStep operation of the BAM interceptor for each relevant step, passing the BAM data extractor as a parameter. At that point the interceptor will call the GetValue method of the BAM data extractor and map the results to the specific BAM activity fields declared in the ActivityInterceptorConfiguration object. MemoryStream stream = new MemoryStream();

byte[] buffer= Encoding.UTF8.GetBytes(@"<Transaction> <amount>10000</amount> <customer>John Doe 2</customer> <creditcard>8829749823748</creditcard> </Transaction>"); stream.Write(buffer, 0, buffer.Length); stream.Seek(0, SeekOrigin.Begin); XPathDocument msg = new XPathDocument(stream); XPathNavigator navigator= msg.CreateNavigator(); BufferedEventStream eventStream= new BufferedEventStream("MsgBox DB connection string", 1); ActivityInterceptorConfiguration configuration = new ActivityInterceptorConfiguration("TRANSACTION"); configuration.RegisterStartNew("Start", ""); configuration.RegisterDataExtraction("AMOUNT", "LogInfo", "//*[local-name()= 'amount']"); configuration.RegisterDataExtraction("CUSTOMER_NAME", "LogInfo", "//*[local-name()= 'customer']"); configuration.RegisterDataExtraction("CREDITCARD", "LogInfo", "//*[local-name()= 'creditcard']"); configuration.RegisterEnd("End"); BAMInterceptor interceptor= new BAMInterceptor(); configuration.UpdateInterceptor(interceptor); SampleDataExtractor extractor= new SampleDataExtractor(); interceptor.OnStep(extractor, "Start", navigator, eventStream); interceptor.OnStep(extractor, "LogInfo", navigator, eventStream); interceptor.OnStep(extractor, "End", navigator, eventStream); Figure: Using the BAM interceptor The combination of the BAM interceptor, configuration, and data extraction interfaces serves as the foundation for the declarative models provided by BizTalk Server, WCF, and WF. Those technologies provide specific implementations of the BAM interceptor API that facilitate the instrumentation of applications using complete declarative models.

Inside the BAM Infrastructure

Monitoring is by nature a heterogeneous process. Different applications deployed on the same infrastructure can have completely different business semantics, and consequently the BAM models for those applications will be completely different. To address this heterogeneity of BAM models, the BAM infrastructure creates a series of model-specific components that keeps a level of isolation between the data infrastructure of the different activity models. In that sense, each activity model will interact with specific artifacts such as tables, views, stored procedures, SSIS packages, dimensions, cubes, etc. that are created when an activity model is deployed. These artifacts are deployed across different parts of the BAM infrastructure such as the BAMArchive, BAMPrimaryImport, and BAMStarSchema relational databases, BAMAnalysis multidimensional database, and SQL Server Integration Services. In this section, we will take an in depth look at some of those components and the relationships among them.

Storing BAM data: BAMPrimaryImport database

The BAMPrimaryImport database is the main data store that contains the data structures to store BAM activities. The most important characteristic of this database is that it is partitioned by a set of tables, views, and stored procedures that are specific to a BAM activity. If you think in database terms, you can think of the BAMPrimaryImport database as a first-level multitenant model in which each tenant is represented by a BAM activity. In that sense, when we deploy an activity model a set of new tables, views, and stored procedures are created that keep the interaction with different activities in complete isolation.

Activity tables
This set of tables form the fundamental component of the BAM activity infrastructure because they store the records related to a BAM activity model. When deploying an activity model, multiple tables are created to store the instances of the BAM activities and, at the same time, provide a level of isolation between different activities. Some of the fundamental tables are summarized in the following list: bam_<Activity Name>_Active: This table stores the active instances of a specific BAM activity. bam_<Activity Name>_Completed: This table stores the completed instances of a specific BAM activity. bam_<Activity Name>_ActiveRelationships: This table stores the active relationship references of a specific BAM activity.

bam_<Activity Name>_CompletedRelationships: This table stores the completed relationship references of a specific activity instance. bam_<Activity Name>_Continuations: This table stores the continuations of a specific activity.

Activity views
The BAM activity views are also created when an activity model is deployed as a mechanism to abstract the interaction with the activity tables. These views enable two key functionalities in the BAM infrastructure: At the first level, activity views are the fundamental mechanism to expose BAM data to other applications, including other components of the BAM infrastructure like the aggregation process. Additionally, these views include some of the calculations required by elements such as measures that are not directly included in the activity tables. Some of the most relevant BAM activity views are summarized in the following list: bam_<Activity Name> _ActiveInstances: Returns the list of the active activity instances based on the records stored in the active activity table. bam_<Activity Name> _CompletedInstances: Returns the list of the completed activity instances based on the records stored in the completed activity table. bam_<Activity Name>_AllInstances: Joins the active and complete activity views, returning a list of all the BAM activity instances. bam_<Activity Name> _AllRelationships: Joins the active and complete activity relationship tables, returning a list of all the relationships of the BAM activity instances. The views listed above are directly derived from the deployment of a BAM activity. Additionally, a new set of views are created for each activity view that we include in the BAM activity model. Some of the most important views are summarized below:

bam_<ActivityName>_<ActivityView Name>_ActiveView: Returns the list of active

activity instances including the calculations of the required dimension levels.

bam_<ActivityName>_<ActivityView Name>_CompletedView: Returns the list of

completed activity instances including the calculations of the required dimension levels.

bam_<ActivityName>_<ActivityView Name>_View: Joins the two previous views,

returning the list of all activity instances included the calculation of the required dimension levels.

The BAMPrimaryImport database serves as the first host for the activity instance data. As those instances complete, the data needs to be archived and aggregated in order to keepup the performance of the BAM infrastructure.

Archiving BAM Data: BAM Archive database

The BAM Archive database (BAMArchive) stores the historical business data that is recycled from the BAM Primary Import database. This database is used mostly as a mechanism for controlling the performance of the applications interacting with the BAM Primary Import database without losing the historic business data. From the activity schema standpoint, the table structure of the BAM Archive database is relatively similar to the BAM Primary Import database. The main differences rely on the fact that the BAM Archive database does not keep a separate set of tables per state (Active, Completed) because all the activities are supposed to be completed at this stage. We can find the following tables in the BAM Archive database; note that each run of the SSIS package generates additional tables by suffixing the base

activity table with the time stamp of the run. These tables contain the data archived from the latest run:
bam_<Activity Name>_Instances: Stores all the recycled activity instances. bam_<Activity Name>_Relationships: Stores all the recycled relationships. bam_<Activity Name>_Active_Uncompleted: Stores all the uncompleted activity instances. bam_<Activity Name>_ActiveRelationships_Uncompleted: Stores all the uncompleted activity relationships. The process of archiving BAM data is executed by the BAM_DM_<Activity Name> SQL Server Integration Services (SSIS) package that is also generated as part of the activity model deployment process. We will explore the details in the next section.

Partitioning BAM data

Although the recycling of data between the BAM Primary Import and BAM Archive databases is a key process of the BAM infrastructure, it is not the only mechanism used to partition the BAM data in order to control performance. Consider a high-throughput scenario in which a large number of activities are being logged to the BAM system. In that scenario, the number of completed activities can rapidly increase, affecting the performance of the applications. To address this, the BAM infrastructure uses an algorithm to partition the completed activities based on its time stamp. In essence, when that algorithm executes it will swap the completed instances table with an empty table that has an identical structure. This mechanism allows controlling the number of records stored in the completed instances

table, which directly impacts the performance of BAM applications. The following diagram illustrates that concept.

Figure: Conceptual view of the BAM partition process Given that the BAM infrastructure uses specific tables and stored procedures per activity, it is obvious that the process of recycling and partitioning that data needs to also be specific per activity because they depend on activity-specific database artifacts. The time window that specifies how often the data needs to be partitioned can be specified by using the bm.exe tool. In the current release of BAM, these processes are controlled by a SQL Server Integration Services (SSIS) package that is generated when an activity is deployed. This SSIS package controls both the partition of the activity instances within the BAMPrimaryImport database and the recycle of the activity instances to the BAMArchive database. The following figure illustrates the SSIS package generated for our sample BAM activity.

Figure: BAM activity partition and archiving SSIS package In the previous figure, weve highlighted a section of the SSIS package. The first section executes the bam_Metadata_SpawnPartition stored procedure, which creates a new set of tables to store the completed activity instances and relationships. To determine the name of the new tables, this stored procedure appends a GUID to the name of the original tables. For instance, if the original set of tables are bam_<Activity Name>_Completed and bam_<Activity Name>_CompletedRelationships, every time we run the BAM_DM_<Activity Name> SSIS package it will create a new set of tables with the names bam_<Activity Name>_GUID and bam_<Activity Name>_GUID_Relationships. Additionally, the SSIS package will modify other artifacts of the BAM databases in order to incorporate these new tables. For instance, the bam_<Activity Name>_CompletedInstances and bam_<Activity Name>_CompletedRelationships views need to be reconstructed to reflect the records stored in the new tables. The following code illustrates the SQL script of the completed

instances view after the SSIS package executes. As you can see, the new SQL statement will query all the records in both tables, keeping the consistency from the client perspective. ALTER VIEW [dbo].[bam_<Activity Name>_CompletedInstances] AS SELECT * FROM dbo.[bam_<Activity Name>_Completed] WITH (NOLOCK) UNION ALL SELECT * FROM [dbo].[bam_<Activity Name>_GUID] WITH (NOLOCK)

The second part of the SSIS package (red) is responsible for archiving the records stored in the BAMPrimaryImport database into the BAMArchive database. This process will select the records from the bam_<Activity Name>_InstancesForArchive and bam_<Activity Name>_RelationshipsForArchive views and store them into the bam_<Activity Name>_Instances and bam_<Activity Name>_Relationships tables in the BAMArchive database respectively. The BAM infrastructure uses the mechanisms described in this section to partition and archive the activity instances in order to control the performance of BAM applications. It is the responsibility of the BizTalk Server administrator to correctly schedule the execution of the SSIS packages by implementing the corresponding SQL jobs. Among the direct consumers of the BAM relational data stored in the BAMPrimaryImport database are infrastructure mechanisms that aggregate that data into multidimensional formats in order to provide the foundation for more complex business intelligence processes.

Aggregating BAM data

The purpose of BAM data is to monitor real-time information that describes the behavior of different business processes. Over time, that real-time information can be aggregated to enable more sophisticated business intelligence mechanisms that analyze the historical behavior of the business processes. Along those lines, BAM provides the ability of aggregating activity data in multidimensional formats such as OLAP cubes. There are two fundamental mechanisms for implementing those aggregations: scheduled and real-time.

Real-time aggregations
Real-time aggregations (RTA) are a simpler type of aggregation that provides relational views of the multidimensional model. This type of aggregation is typically used on timesensitive scenarios in which the information should be aggregated in real time. Obviously, the fact that the aggregated information is stored in a relational (and not multidimensional) representation limits the complexity of the business intelligence procedures that can be

applied to the BAM data. When designing a BAM activity model, you can declare a real-time aggregation based on a BAM view as illustrated in the following figure.

Figure: Creating a real-time-aggregation using the BAM Excel Add-in When the activity model is deployed, the BAM infrastructure creates a separate table and view, named bam_<View Name>_<Pivot Table Name>_RTATable and bam_<View Name>_<Pivot Table Name>_RTAView respectively, in the BAMPrimaryImport database to store the aggregated information. The purpose of the RTA table is to store the activity instance data partitioned by the different dimension levels. Specifically, the RTA table will contain a column for each level of the dimensions included in the BAM view. For instance, if as a part of a BAM view you are using a time dimension with levels Year, Month, and Week the RTA table will include the following columns: <Dimension Name>_Year, <Dimension Name>_Month, and <Dimension Name>_Week. The RTA table does not calculate the measures needed to aggregate the data; that part is done by the RTA views, which include the T-SQL statements needed to calculate all the measures of the BAM view. Additionally, the BAM infrastructure adds SQL Server triggers to the activity Completed and Active tables in order to aggregate the information into the RTA table. These triggers

execute the T-SQL statements that extract the information from the activity tables, calculate the different dimension levels, and insert the records in the RTA table. The following figure illustrates the RTA aggregation process.
SQL Server triggers extract the new activity records and calculate the corresponding dimension levels such as Year, Month, etc

Queries the RTA table and calculates measures such as Avg, Sum, or Count

Active Instances Table

RTA Table Completed Instances Table

RTA View

Figure: Real-time aggregation process The aggregated data is stored in the RTA table for a predefined period of time before it is deleted. This time interval can be configured by modifying the bam_Metadata_RealTimeAggregations table in the BAMPrimaryImport database. Although the use of RTA represents a great value in providing a real-time snapshot of the business process metrics, the use of triggers can directly affect the performance of the BAM applications. Well-designed BAM architectures limit the use of RTA to the absolute key realtime metrics and use scheduled aggregations for more complex aggregation procedures that are not completely time-sensitive.

Scheduled aggregations
As discussed in the previous section, the use of relational structures constrains the type of aggregations that can be implemented by using RTAs. To address more complex scenarios, BAM provides a second type of aggregation technique that takes full advantage of multidimensional structures like OLAP cubes. These types of aggregations are known as scheduled aggregations and can serve as a foundation for implementing complex business intelligence procedures using BAM activity data. The scheduled aggregation data model is based on a set of multidimensional structures stored in the BAMAnalysis database, which are also based on a star schema relational topology stored in the BAMStarSchema database. When deploying an activity model, the BAM infrastructure adds an OLAP cube to the BAMAnalysis database for each activity view

that contains multidimensional structures like dimensions or measures. The dimensions of that cube are based on the dimensions included in the activity views while the cube cells are based on the measures. Following best practices of multidimensional database modeling, BAM uses a third database to adapt the pure relational structure of the BAMPrimaryImport database to the multidimensional structure of the BAMAnalysis database. This is precisely the role of the BAMStarSchema database, which is based on a classic star schema model that represents a set of aggregated views of the information stored in the BAMPrimaryImport database. From a single-view perspective, the conceptual model of the BAMStarSchema database looks like the following:

bam_<View Name>_ActiveFacts View

bam_<View Name>_CompletedFacts View

Dim1 Table

bam_<View Name>_Facts Table

Dim2 Table

. . .
Other dimension tables..

Figure: Conceptual model of the BAMStarSchema database The most important component of the BAMStarSchema database is the bam_<View Name>_Facts table, which stores the activity instances that need to be processed into the OLAP cube. That table keeps relationships with different dimension tables that are generated based on the dimensions included in the activity model. The BAM infrastructure creates a dimension table for each dimension declared as part of an activity view. For instance, if you declared a time dimension named MonitoredTime, a dimension table name bam_<View Name>_Dim_MonitoredTime will be created in the BAMStarSchema database. Additionally, two views are created to filter the completed and active facts based on the state of a specific activity.

This model represents a classic star schema structure with the fact views at the center and the dimension tables at the edges. BAM leverages this structure as the foundation of the OLAP cubes included in the BAMAnalysis database. The dimensions of the cubes are directly based on the dimension tables of the BAMStarSchema database and the data used for the measures is based on the fact views. The following figure illustrates the OLAP cube corresponding to the activity view used in our example.

Figure: Transaction OLAP cube view Although the previous figure is specific to our scenario, it reflects the conceptual structure of the OLAP cubes stored in the BAM database. In that structure, the fact data is directly extracted from the Active and Completed fact views in the BAMStarSchema database while the dimension tables are modeled based on the dimension tables of the same database. To populate the corresponding cubes, the activity data needs to be transferred from the BAMPrimaryImport database into the BAMStarSchema database. After that, the OLAP cubes

can be processed to incorporate the new BAM activity information. The data exchange process is executed by a SQL Server Integration Services (SSIS) package that is created when the activity model is deployed. The BAM infrastructure creates specific SSIS packages for each activity view that includes multidimensional artifacts and is not marked for realtime aggregations. The name of the package is AN_<Activity View> and it contains all the logic for moving activity data between the BAMPrimaryImport and BAMStarSchema databases as well as for populating the OLAP cubes associated with a specific activity view.

Figure: BAM analysis SSIS package The first part (highlighted green) of the SSIS package extracts the BAM activity data from the bam_<View Name>_CubingSnapshot view in the BAMPrimaryImport database and places it into the bam_<View Name>_Staging table in the BAMStarSchema database. After that, the second part of the package (highlighted blue) inserts the data into the bam_<View Name>_Facts tables from where it is directly reflected in the bam_<View Name>_ActiveFacts and bam_<View Name>_CompleteFacts tables. Finally, the cube dimensions are reprocessed with the new information.

From the infrastructure perspective, there is no doubt that scheduled aggregations are more complex than real-time aggregations. On the other hand, scheduled aggregations can be a great enabler for more sophisticated business intelligence procedures such as the multidimensional BAM data that can be used as the source for data mining algorithms or complex MDX reports.

The BAM Event Bus

The BAM Event Bus service, also known as the Tracking Data Decode Service (TDDS), is a fundamental component of the BAM infrastructure that processes data streams stored in the BizTalk Message Box database (BizTalkMsgBoxDb) and inserts them into other databases for easy processing. Specifically, TDDS is responsible for processing business monitoring data into the BAMPrimaryImport database and health monitoring data into the BizTalkDTADb database. From a BAM perspective, TDDS is the main enabler of the asynchronous event streaming mechanisms used by objects like BufferedEventStream, OrchestrationEventStream, and MessagingEventStream. When developers update activities using those APIs the data gets inserted into a set of tracking tables within the BizTalkMsgBoxDb database. After that, one of the TDDS instances associated with that database will move the data into the BAMPrimaryImport database. The process of transporting the tracking data is fairly sophisticated because it involves algorithms that leverage a bookmarking scheme for TDDS resource coordination and transaction integrity. On large BizTalk Server topologies it is common to find multiple BAM Event Buses that move data between various BizTalkMsgBoxDb and BAMPrimaryImport databases. For more information about BizTalk Server architectures see http://go.microsoft.com/fwlink/?LinkId=134550.

BizTalkMsg BoxDB

Event Bus

Event Bus BAMPrimary Import

BizTalkMsg BoxDB

Event Bus

Event Bus

BAMPrimary Import BizTalkMsg BoxDB Event Bus Event Bus

Figure: BAM Event Bus sample topology The configuration of the BAM Event Bus topology is stored centrally in the BizTalk Management database (BizTalkMgmtDb). Each BAM Event Bus in the topology periodically sends a heartbeat message, containing information about its current state, to the management database. As a result the event bus will obtain information about the data that needs to be moved and if possible it will take ownership of the task. The BAM activity data is stored in the BizTalkMsgBoxDb in a partition structured across a set of tables prefixed with dbo.TrackingData_<>. The data is stored in a binary format and partitioned to optimize the performance of read/write operations. The BAM Event Bus will process the different partitions of the activity data within the scope of a session. During this process the data is extracted from the tracking tables in the BizTalkMsgBoxDb database, deserialized, and inserted into the activity tables in the BAMPrimaryImport database. The entire process runs in isolation from other BizTalk Server processes and is highly optimized for performance.

Querying BAM Data

Overview
The primary benefit of Business Activity Monitoring (BAM) is its ability to provide intelligent domain-specific metrics that reflect run-time behavior of business processes. These benefits are ultimately reflected in the way applications and business users consume BAM activity data in order to make more intelligent decisions about their business processes. Given the universal nature of BAM, the scenarios for consuming BAM data can encompass a large variety of products and technologies, but fundamentally we will use the same principles and technologies we use for querying relational and multidimensional data sources. This is mostly due to the fact that BAM does not introduce any new mechanisms for querying activity models and, instead, relies on traditional data access technologies. As we explored in the previous section, the BAM activity data is stored in two fundamental models: relational and multidimensional. Both models offer different perspectives of the information described by BAM activities and, at the first level, determine the type of technologies developers can use to interact with BAM data. For instance, an application that needs a real-time view of a business process will, most likely, rely on relational data access technologies like ADO.NET and T-SQL to query elements such as activity or real-time aggregation views stored in the BAMPrimaryImport database. However, if the application requires a historic view of a business process it will, most likely, leverage multidimensional data access mechanisms such as ADOMD.NET, Multidimensional Expressions (MDX), or XML for Analysis (XMLA). Although using the right data access technique is a fundamental step for implementing applications that consume BAM data, it is typically an enabler for leveraging BAM data into business intelligence applications that include capabilities like reporting, collaboration, or performance analysis. To incorporate those features, developers need to combine BAM with technologies such as SQL Server Reporting Services, Office SharePoint Server, or Office Performance Point Server, which can complement BAM activity models with a richer set of domain-specific capabilities such as KPI modeling, data composition, collaboration, etc. In both cases, adopting the right technology for the correct scenario can be a key aspect of the success or failure of a BAM implementation. In this section we are going to explore some of the most common techniques, products, and best practices that enable developers to consume BAM activity models.

Using BAM with data access libraries

As we explained in the previous section, the supported mechanism for querying BAM activity models is interacting directly with the BAMPrimaryImport views and the BAMAnalysis cubes. As in any other data-driven application, developers can use their data access programming models for querying the BAM relational and multidimensional databases. In .NET environments, developers can use ADO.NET-related technologies for querying the data reflected in the BAM activity views stored in the BAMPrimaryImport database. To query the BAM aggregations stored in the BAMAnalysis database, a developer can leverage technologies such as the ADOMD.NET client and server programming models. The use of ADOMD.NET is particularly interesting because it can leverage multiple query languages such as Multidimensional Expressions (MDX), Data Mining Extensions (DMX), or even a limited syntax of SQL. The following code fragment shows how to query our multidimensional BAM model by using ADOMD.NET and MDX. AdomdConnection connection = new AdomdConnection("Data Source=localhost"); connection.Open(); AdomdCommand command= connection.CreateCommand(); command.CommandText = @" SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim]. [Day].ALLMEMBERS * DESCENDANTS ([TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE"; CellSet bamCellSet= command.ExecuteCellSet(); TupleCollection tuplesOnColumns = bamCellSet.Axes[0].Set.Tuples; TupleCollection tuplesOnRows = bamCellSet.Axes[1].Set.Tuples; for (int row = 0; row < tuplesOnRows.Count; row++) { for (int col = 0; col < tuplesOnColumns.Count; col++) Console.WriteLine("Cell Column {0}, Row {1}, : {2}", tuplesOnColumns[col].Members[0].Caption, tuplesOnRows[row].Members[0].Caption, bamCellSet.Cells[col, row].FormattedValue + "\t"); } connection.Close(); Figure: Querying BAM using ADOMD.NET The capabilities of both ADO.NET and ADOMD.NET are very well documented and there is no reason to deep dive into their details in this article. It is important to notice that although the use of technologies like ADO.NET and ADOMD.NET represents a powerful vehicle for interacting with the BAM databases, their use is often an enabler for more sophisticated

BAM applications like the ones built by using Microsoft Office SharePoint Server or SQL Server Reporting Services.

Using BAM with SQL Server Reporting Services

SQL Server Reporting Services (SSRS) represents a natural choice to create comprehensive reports based on a BAM activity model. By using SSRS, developers can create reports based on the relational activity data stored in the BAMPrimaryImport database as well as reports that reflect the multidimensional activity aggregations stored in the BAMAnalysis database. Specifically, in the case of multidimensional databases, SSRS can reflect the result of advanced business intelligence (BI) techniques such as data mining or key performance indicators executed over the BAM activity data. Additionally, SSRS is commonly used to create reports that combine the BAM activity data with external data sources to provide a more comprehensive view of the business models. Creating SSRS reports based on BAM activity models is no different than interacting with other data sources. Typically, developers will extract the relational information (T-SQL) from the activity views in the BAMPrimaryImport database and the multidimensional information (MDX) from the cubes in the BAMAnalysis database. To create an MDX-based report based on our activity model we can use the query builder tool included in the SSRS project template. In this case we are segmenting the TRANSACTION metrics based on the TRANSACTION_STG_Dim and TRANSACTION_TIME_Dim dimensions.

Figure: SSRS query builder Alternatively we can create custom MDX queries like the following. SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim].[TRANSACTIONS_TRANSACTION_TIME_Dim].[Day ].ALLMEMBERS * DESCENDANTS([TRANSACTIONS_TRANSACTION_STG_Dim].[TRANSACTIONS_TRANSACTION_STG_ Dim].[Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS Figure: BAM MDX query Once deployed, the report can be accessed using an SSRS report server URL as illustrated in the following figure.

Figure: Using BAM from SQL Server Reporting Service Although the use of the SSRS report server is a great fit in some scenarios, developers often find themselves integrating SSRS reports into other Web applications. To enable those scenarios, SSRS includes APIs and technologies that allow heterogeneous applications to consume the data exposed by SSRS reports. For instance, ASP.NET applications can interact with SSRS reports by using the Report Viewer controls. SSRS reports can also be accessed through Microsoft Office SharePoint Server (MOSS) applications by using the Reporting Services Add-in for SharePoint Technologies. Using SSRS and other reporting technologies provides powerful capabilities to extract meaningful information from a BAM activity model and enhance it with other artifacts such as charts, delivery mechanisms, etc. Although the features of reporting technologies can be used ubiquitously across various scenarios, there are several cases that require the use of domain-specific technologies in areas like data mining or KPI analysis. In this case, performance management scenarios for the combination of BAM and PerformancePoint Server opens the door for many creative solutions.

Using BAM with Microsoft Office PerformancePoint Server

One of the most common uses of BAM in real-world scenarios is to provide metrics to evaluate the run-time behavior of different business processes. Quite often, the evaluation criteria are defined by business experts using techniques like Key Performance Indicators (KPI) or other value segmentation techniques. The purpose of these procedures is to

provide business analysts or executives with a visually intuitive way to compare the realtime behavior of a business process against strategic and operational performance goals. Generally, this technique is known as performance management and is one of the most widely established business intelligence (BI) disciplines among large corporations. Within the Microsoft technology stack, Office PerformancePoint Server is a suite of tools and technologies that allows customers to complement enterprise data with performance management artifacts such as scorecards, dashboards, management reporting, or analytics. PerformancePoint Server, and especially Monitoring Server, is a great vehicle to analyze and report the BAM activity data against well-established performance management criteria. Given that PerformancePoint Server interacts naturally with SQL Server Analysis Services (SSAS), it can leverage the BAM data stored in the BAM Analysis database. Although it is not an objective of this paper to provide a complete tutorial of PerformancePoint Server, the following section will describe some of the fundamental techniques developers can use to integrate BAM data into PerformancePoint Server. To make PerformancePoint Server aware of the multidimensional structures of a specific BAM activity model, we can import the specific activity view cube into PerformancePoint Server Dashboard Designer. We can easily accomplish that by using the Create a Scorecard wizard against the specific cubes we are interested in. As part of that process we can generate PerformancePoint Server KPIs based on the measures included in the BAM activity model. The following figure presents a snapshot of that process for our sample activity model.

Figure: Creating PerformancePoint Server KPIs based on BAM activity data Once imported, the KPIs can be adjusted to the specific criteria relevant to the business process. When all the KPIs are correctly configured, we can combine them in a scorecard. The following figure represents a snapshot of the design process of a scorecard that uses the measures provided on our sample BAM activities.

Figure: Designing a PerformancePoint Server scorecard using BAM activity data At this point we can include the scorecard in a PerformancePoint Server dashboard and publish it to PerformancePoint Server Monitoring Server. This will make the dashboard accessible to various applications including office applications like Microsoft Office Outlook or Web applications like Microsoft Office SharePoint Server (MOSS). The use of MOSS is especially interesting in scenarios where multiple users are collaborating on the business processes that influence the PerformancePoint Server scorecards. The integration between MOSS and PerformancePoint Server is based on the PerformancePoint Server Dashboard Viewer Web part that allows you to render a dashboard or a specific scorecard included in it.

Figure: Configuring the BAM PerformancePoint Server dashboard by using MOSS Once configured properly the PerformancePoint Server dashboard hosting the BAM activity will be accessible by any Web client without the need to install any components.

Figure: BAM PerformancePoint Server dashboard hosted in a MOSS Web site

MOSS is a particularly interesting technology for exposing BAM data in scenarios where multiple users should interact and collaborate based on the information provided by BAM activities. MOSS includes various technologies that can be used to interact with BAM activity data. The following section will explore some of those technologies and common scenarios for integrating BAM and MOSS.

Using BAM with Microsoft Office SharePoint Server

Microsoft Office SharePoint Server (MOSS) provides a rich set of capabilities around enterprise portals and collaboration that makes it a phenomenal technology for exposing BAM data. In its current release, MOSS includes a series of technologies such as Excel Services and Business Data Catalog (BDC) that can be used for exposing BAM activity data to enterprise users in different scenarios. For instance, a developer looking to reflect the

BAM relational data in a MOSS site might be inclined to use a custom Web part when a technology like the BDC can be more appealing in scenarios where the BAM data needs to be complemented with Office documents or enterprise search. Additionally, as we explored in the previous sections, MOSS Web parts can be used to consume BAM data from other technologies like SSRS and PerformancePoint Server. Exposing BAM data using Web parts The Web Part Framework provided with ASP.NET 2.0 is the foundation of MOSS Web Parts. Using that framework, developers can build custom Web parts that query BAM activity data stored in the BAMPrimaryImport and BAMAnalysis databases. Out of the box, MOSS provides some Web parts that can be used to accomplish this task. For instance, we can use the Data View Web Part from SharePoint Designer (see the BAM data highlighted in red) to query the activity views stored in the BAMPrimaryImport database.

Figure: Customizing the Data View Web Part from SharePoint Designer Once deployed the Web part will render the BAM activity instances as illustrated in the following figure.

Figure: MOSS site using the Data View Web Part to access the BAM activity data Business Data Catalog The use of Web parts, including the Data View Web Part, represents a great initial alternative for querying BAM data from MOSS. However, quite often, the BAM activity data needs to be complemented with other MOSS artifacts like document management, enterprise search, or workflows. For those scenarios, the use of the Business Data Catalog (BDC) might be a more adequate solution. The BDC provides a completely declarative mechanism that allows MOSS to surface line-of-business (LOB) system data. Specifically, the BDC provides a strong integration with SOAP-based Web services and databases that we can use to integrate BAM activities into MOSS. One of the features that make the BDC so appealing to enterprise applications is its capability to integrate business data into other MOSS technologies like document management or enterprise search. From the BAM standpoint, we can use the BDC not only to query BAM data but also to associate BAM activities with Office documents, incorporating BAM activities in MOSS workflows or adding search capabilities to the BAM data. The process of integrating BAM data into MOSS by using the BDC is no different from the process followed for other relational databases. The first step required to use the BDC with BAM is to create a BDC metadata model. This is an XML-based declarative language that represents the different entities that will be used by

MOSS. Also, given that the entities are stored in the BAMPrimaryImport database, this model should express the queries required to select the BAM activity instances that are relevant to the MOSS applications. The following code illustrates sections of the BDC metadata model associated with the Transaction BAM activity. You can see that this model declares the fields that will be rendered in MOSS as well as the T-SQL command used to extract the data. <LobSystem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.microsoft.com/office/2006/03/BusinessDataC atalog BDCMetadata.xsd" Type="Database" Version="1.0.0.0" Name="TransactionActivity" xmlns="http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog"> <LobSystemInstances> <LobSystemInstance Name="TransactionActivity_Instance"> <Properties> </Properties> </LobSystemInstance> </LobSystemInstances> <Entities> <Entity EstimatedInstanceCount="10000" Name="bam_TRANSACTION_Instances"> <Properties> <Property Name="DefaultAction" Type="System.String">View Profile</Property> </Properties> <Identifiers> <Identifier TypeName="System.String" Name="ActivityID" /> </Identifiers> <Methods> <Method Name="Find_bam_TRANSACTION_Instances"> <Properties> <Property Name="RdbCommandType" Type="System.Data.CommandType, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">Text</Property> <Property Name="RdbCommandText" Type="System.String">SELECT "ActivityID", "TRANSACTIONID", "CREDITCARD", "AMOUNT","TRANSACTION_CREATED","TRANSACTION_FAILED", "TRANSACTION_FAILED_ALERT_TRIGGERED","TRANSACTION_APPROVED", "CUSTOMER_NAME","CUSTOMER_AGE" FROM bam_TRANSACTION_AllInstances </Property> </Properties> <Parameters> <Parameter Direction="Return" Name="@bam_TRANSACTION_Instances"> <TypeDescriptor TypeName="System.Data.IDataReader, " IsCollection="true" Name="Reader"> <TypeDescriptors> <TypeDescriptor TypeName="System.Data.IDataRecord,, Name="Record"> <TypeDescriptors> <TypeDescriptor TypeName="System.String, " Name="ActivityID" IdentifierName="ActivityID"/> <TypeDescriptor TypeName="System.String, " Name="TRANSACTIONID" /> <TypeDescriptor TypeName="System.String, "

Connection properties

Name="CREDITCARD" /> <TypeDescriptor TypeName="System.Double, " Name="AMOUNT" /> <TypeDescriptor TypeName="System.DateTime" Name="TRANSACTION_CREATED" /> <TypeDescriptor TypeName="System.DateTime" Name="TRANSACTION_FAILED" /> <TypeDescriptor TypeName="System.DateTime" Name="TRANSACTION_FAILED_ALERT_TRIGGERED" /> <TypeDescriptor TypeName="System.DateTime" Name="TRANSACTION_APPROVED" /> <TypeDescriptor TypeName="System.String" Name="CUSTOMER_NAME" /> <TypeDescriptor TypeName="System.Int32" Name="CUSTOMER_AGE" /> </TypeDescriptors> </TypeDescriptor> </TypeDescriptors> </TypeDescriptor> </Parameter> </Parameters> <MethodInstances> <MethodInstance Type="Finder" ReturnParameterName="@bam_TRANSACTION_Instances" ReturnTypeDescriptorName="Reader" ReturnTypeDescriptorLevel="0" Name="Find_bam_TRANSACTION_Instances">

Rest of the elements have been omitted for brevity.

</MethodInstance> </MethodInstances> </Method> </Methods> </Entity> </Entities>

Rest of the elements have been omitted for brevity.

</LobSystem> Figure: BDC model for the Transaction BAM activity Once created, the BAM metadata model can be deployed by using the MOSS administration console. After the BDC model is deployed the BAM entities can be used from any MOSS Web site with the proper privileges. To render the BAM activity records we can use some of the BDC Web parts included in MOSS. In our scenario we will use the Business Data List Web Part to render all the instances of the transaction activity and the Business Data Item Web Part to render the details of a specific activity instance (see the following figure).

Figure: Configuring BAM-BDC Web parts Once configured, the BDC Web part will provide real-time access to the information stored in the BAMPrimaryImport database activity views.

Figure: Accessing BAM data by using BDC As we explained at the beginning of this section, one of the benefits of integrating BAM with the BDC is the capability of exposing BAM data to other MOSS technologies. For instance, we can associate BAM activity instances with Office documents that provide additional information about the business processes.

Figure: Using BAM activities in a MOSS document library

Accessing BAM by using Web services

In the first part of this section we explored different Microsoft products and technologies that can complement BAM with enterprise capabilities such as reporting, collaboration, or KPI analysis. Undoubtedly, the combination of BAM with technologies like SSRS, PerformancePoint Server, and MOSS is the correct answer to a large variety of the most common BAM scenarios in the enterprise. However, these technologies provide the most dividends in scenarios like end-user reports that have certain design-time knowledge of how the BAM activity data needs to be processed. Looking back to our previous examples, all of them consume the BAM data by using artifacts created at design time like reports or KPIs. However, these types of techniques are not always the best fit in scenarios that require a more dynamic interaction with the BAM activities. Querying the BAMAnalysis database by using the XML for Analysis Web service XML for Analysis (XMLA) is an open standard for querying multidimensional data sources by using Web services. The last two editions of SQL Server Analysis Services (SSAS 2005 and SSAS 2008) support XMLA 1.1 as one of the programming models used by client applications to interact with an SSAS instance. Even more importantly, XMLA is the fundamental protocol underneath other multidimensional programming technologies like

ADOMD.NET and Analysis Management Objects (AMO). Essentially, XMLA proposes a model for encoding commands such as MDX queries in a SOAP envelope that can be processed by an XMLA-compatible Web service. In its current version, the XMLA specification describes two generally accessible operations: Discover and Execute. The Discover operation is used to query metadata about multidimensional data sources like cubes, dimensions, etc. For instance, by using that operation an application can browse the list of BAM cubes, dimensions, and measures deployed in the BAMAnalysis database. <Discover xmlns="urn:schemas-microsoft-com:xml-analysis"> <RequestType>MDSCHEMA_CUBES</RequestType> <Restrictions> <RestrictionList> <CATALOG_NAME>BAManalysis</CATALOG_NAME> </RestrictionList> </Restrictions> <Properties> <PropertyList> <DataSourceInfo>Provider=MSOLAP;Data Source=local;</DataSourceInfo> <Catalog>BAMAnalysis</Catalog> <Format>Multidimensional</Format> </PropertyList> </Properties> </Discover> Figure: XMLA Discover operation to browse the cubes deployed in the BAMAnalysis database The Discover operation is used exclusively for browsing metadata. To execute queries against a multidimensional data source a developer needs to use the Execute operation. In the BAM context, an application can invoke the XMLA Execute operation to run MDX queries against some of the activity views stored as cubes in the BAMAnalysis database (see next figure). <Execute xmlns="urn:schemas-microsoft-com:xml-analysis"> <Command> <Statement> SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim]. [Day].ALLMEMBERS * DESCENDANTS ([TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE </Statement> </Command> <Properties> <PropertyList> <DataSourceInfo>Provider=MSOLAP;Data Source=local;</DataSourceInfo> <Catalog>BAMAnalysis</Catalog>

<Format>Multidimensional</Format> </PropertyList> </Properties> </Execute> Figure: Using the XMLA Execute operation to query the BAMAnalysis database In SSAS 2005 and SSAS 2008, developers can configure HTTP-SOAP XMLA access by configuring the ISAPI extension deployed in the Web directory. Then, developers can generate an XMLA proxy using the technology of choice. The following code illustrates a WCF client that invokes the Execute operation to run an MDX query against the TRANSACTIONS OLAP cube deployed as part of our sample BAM activity model. MsXmlAnalysisSoapClient xmlaService = new MsXmlAnalysisSoapClient(new BasicHttpBinding(), new EndpointAddress("http://localhost:8080/olap/msmdpump.dll")); XmlDocument xdoc= new XmlDocument(); XmlElement command= xdoc.CreateElement("", "Statement", " urn:schemas-microsoft-com:xml-analysis"); XmlElement properties = xdoc.CreateElement("", "PropertyList", " urn:schemas-microsoft-com:xml-analysis"); properties.InnerXml = "<DataSourceInfo xmlns=\"urn:schemas-microsoft-com:xm-analysis\"> Provider=MSOLAP;Data Source=btsr2; </DataSourceInfo> <Catalog xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> BAMAnalysis </Catalog> <Format xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> MultiDimensional </Format> <AxisFormat xmlns=\"urn:schemas-microsoft-com:xml-analysis\"> ClusterFormat </AxisFormat>"; command.InnerXml = @"SELECT NON EMPTY { [Measures].[AVG_AMOUNT], [Measures].[COUNT_TRANSACTION] } ON COLUMNS, NON EMPTY { ([TRANSACTIONS_TRANSACTION_TIME_Dim]. [TRANSACTIONS_TRANSACTION_TIME_Dim].[Day].ALLMEMBERS * DESCENDANTS( [TRANSACTIONS_TRANSACTION_STG_Dim]. [TRANSACTIONS_TRANSACTION_STG_Dim]. [Level 02].ALLMEMBERS) ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME, PARENT_UNIQUE_NAME, LEVEL_NUMBER ON ROWS FROM [TRANSACTIONS] CELL PROPERTIES VALUE"; XmlElement response= xmlaService.Execute(command, properties); Figure: Using the XMLA Web service to query the BAMAnalysis database Although powerful, the use of the XMLA Web service is restricted to the BAMAnalysis database. If an application is required to use Web services to access the activity fact data

stored in the BAMPrimaryImport database there is another option available: the BAM Query Web service. Querying the BAMPrimaryImport database by using the BAM Query Web service Although not officially supported, the BAM Query Web service provides an interesting solution to applications that need to query the BAM activity data stored in the BAMPrimaryImport database. The BAM Query Web service provides a single operation, GetInstanceData, that abstracts a T-SQL query to a BAM activity view. The following code shows a sample client that uses the BAM Query Web service to query our sample activity model. BamQueryServiceSoapClient queryService = new BamQueryServiceSoapClient( new BasicHttpBinding(), new EndpointAddress( "<server url>/BamQueryService.asmx")); queryService.ClientCredentials.Windows.ClientCredential = new System.Net.NetworkCredential(<username>, <password>); InstanceQuery query= new InstanceQuery(); query.SelectClauses = new string[] { "ActivityID, TRANSACTIONID, AMOUNT, TRANSACTION_CREATED, CREDITCARD" }; Column[][] columns= queryService.GetInstanceData("dbo.bam_TRANSACTION_AllInstances", "TRANSACTION", query, 0); Figure: Using the BAM Query Web service The use of both XMLA and the BAM Query Web service considerably improves the BAM implementations in distributed environments and also facilitates the interoperability of the BAM infrastructure with non-Microsoft technologies such as J2EE or dynamic languages. However, it is important to highlight that developers should use these Web services only as a very specific mechanism to query BAM data and never as a general-purpose API to abstract the interactions with the BAM infrastructure. Neither the BAM Query Web service nor XMLA provide any abstraction of the BAM data model. Instead, both of them require the consumer application to have intrinsic knowledge of the schema of the BAM databases. As weve explored in the different sections of this paper, BizTalk Server BAM provides a generic infrastructure that does not impose any restrictions on the type of client applications that can interact with an activity model. However, from an API standpoint, the use of BAM is currently reduced to .NET Framework-based applications such as the ones developed in BizTalk Server, WCF, or WF. Certainly, abstracting the BAM infrastructure with an openstandard API can enable the adoption of BAM across heterogeneous technologies. Although currently there is no supported API that serves this purpose, developers can leverage technologies like WCF and emerging architecture styles like Representational State Transfer (REST) as a vehicle to service enable their BAM infrastructure.

Building a BAM RESTful API

Why do we need a new API?
Throughout this paper we have explored different mechanisms developers can use to interact with BizTalk Server BAM. Although the current technology provides developers with powerful alternatives to instrument their applications using BAM, there are a few outstanding challenges in order to ubiquitously adopt BAM in heterogeneous enterprise environments. One of the most attractive capabilities of BizTalk Server BAM is that its infrastructure is not dependent on BizTalk Server. This means that, theoretically, non-.NET Framework-based applications can benefit from the use of BAM. However, even though this is true from the BAM infrastructure standpoint, the APIs required to interact with BAM are only available to .NET Framework-based applications. Another challenge that developers face when implementing BAM applications is the lack of a consistent programming model for querying and executing CRUD (Create-Update-Delete) operations on BAM activities. In that sense, developers need to interact with BAM streambased APIs to populate elements of an activity model and with a completely different model when it comes to querying BAM data. This contrasts with other data-driven programming models such as ADO.NET that abstract both functionalities through a consistent API. To improve the consistency and interoperability of the BAM programming model, developers often find themselves creating Web service interfaces that abstract the BAM stream APIs. Typically, those Web service interfaces are based on the SOAP-WSDL-WS-* model and expose the specific BAM capability that is initially required by the applications using them. Extending these APIs to include the broad set of BAM capabilities will introduce the traditional versioning challenges that large SOAP-WSDL APIs face in the industry today. Another limitation of this model is the constraint on adoption to technologies that dont have a powerful SOAP-based engine. That list includes some of the most powerful technologies for building rich Web user interfaces that consume BAM data, like dynamic or scripting languages and AJAX toolkits. Being aware of all these challenges, this section will propose a slightly different model that leverages the principles of a set of emerging service-oriented architecture styles. Specifically, this section will explore the design of BAM Web service APIs based on the Representational State Transfer (REST) model.

Embracing the Web programming principles

Although this section is not intended to discuss the pros and cons of Representational State Transfer (REST) vs. traditional SOAP/WS-* services, we think is important to highlight why REST is a better choice for designing a consistent BAM Web services API. The term REST was first enunciated by Roy Thomas Fielding as part of his PhD dissertation. At a high level, REST intended to formulate the principles that make the Web work in a scalable and performant way. Fieldings arguments highly resonated in the Web service community in which REST started to be positioned as an alternative to the traditional SOAP/WS-* model for Web services, firing off one of the most fascinating debates in the history of distributed computing. In spite of all those arguments, the fact is that REST, as an architecture style, is a great solution for implementing resource/entity-based services like the ones required for a BAM API. Essentially, RESTful services follow a set of wellestablished principles that differentiate them from other alternatives. The following list summarizes some of the most important characteristics of RESTful services: Give every resource an ID: Fundamentally, RESTful services abstract interactions with resources (BAM activities, database entities, documents, etc). This principle states that those resources should be addressable and uniquely identified. The typical way to satisfy both requirements is to use URIs as the unique resource identifier. For instance, a BAM activity instance can be identified by the URI, where activity ID is the unique BAM identifier for that activity instance. Link things together: In a typical REST system, resources can be related to each other. For instance, two BAM activities can have a reference relationship or a customer resource can have a relationship with the account resource. As resources are identified by URI, one resource will include the links to its related resources as part of its representation so that they can be individually addressed. Use standard methods: Web architectures are based on HTTP. REST enunciates how the different HTTP verbs should be associated with resource operations. In essence, resources can be created using an HTTP POST, retrieved using a HTTP GET, updated by an HTTP PUT, and deleted using an HTTP DELETE. Other operations like metadata retrieval can be abstracted by other HTTP verbs such as HEAD or OPTIONS. Resources with multiple representations: When we design Web services using the traditional SOAP/WS-* techniques, we are using SOAP as the single representation format. In RESTful services, resources are conceptual entities that are

independent of the representation format used on the wire. For instance, a BAM activity instance can be represented by using XML, JavaScript Object Notation (JSON), or Atom formats. Communicate statelessly: Web architectures are stateless by definition and this is one of the characteristics that makes them highly scalable. By stateless, we mean that any state of the interaction should be carried in the resource itself and not in a server session. Designing RESTful services in the Microsoft platform has come a long way and it is safe to say that the current release of Windows Communication Foundation (WCF) can be considered one of the most solid REST development platforms in the industry today. The use of WCF to build a BAM RESTful API will be illustrated in the following sections of this paper. Following the REST principles, we can start formulating what a BAM RESTful API looks like. The following section will explore some of the design alternatives to expose BAM capabilities using RESTful services.

Brainstorming a BAM RESTful API

The first aspect to consider is that the BAM conceptual model can naturally be modeled by using the principles of Resource Oriented Architecture (ROA). We emphasize the term natural because, even though we can also model a generic BAM API by using RPC-based concepts like operations or parameters, ROA transparently reflects the concepts behind the BAM programming model. In that sense, the entire BAM infrastructure can be atomically represented as a set of resources such as activities, activity views, cubes, or activity models. These BAM resources can be related to each other; for instance, a BAM activity can have a reference relationship with other BAM activities. When adopting this architecture style, we can abstract the entire BAM development process as a series of query and CRUD operations against BAM resources. Having represented the BAM programming model as resources, it is time to think of how to uniquely identify instances of BAM resources. To embrace the addressability principle of REST we can use URI as the unique identifier of a BAM resource. For instance, an activity instance can be identified by the URI http://<my BAM server>/activities/{activity name}/{activityid}.

At this point, our next challenge is how to adapt the different HTTP verbs to represent queries (HTTP GET) or CRUD (HTTP POST, PUT, and DELETE respectively) operations against BAM resources. This process is highly dependent on the aforementioned URI model and should correctly represent the BAM conceptual model. The following list illustrates some examples of how to use the HTTP verbs to model BAM operations following the REST principles: An HTTP GET against http://<my BAM server>/activities/{activity name} will return the list of instances of the activity identified by {activity name}. Each instance will be identified by a URI in the form of http://<my BAM server>/activities/{activity name}/{activity id}. An HTTP POST against http://<my BAM server>/activities/{activity name} will create a new instance of the {activity name} BAM activity. The new instance will be identified by a URI in the form of http://<my BAM server>/activities/{activity name}/{activity id}. An HTTP GET against http://<my BAM server>/activities/{parent activity name}/{activity id}/{child activity name} will return the instances of the {child activity name} BAM activity related to the {activity id} instance of the {parent activity name} BAM activity. An HTTP PUT against http://<my BAM server>/activities/{activity name}/{activity id} will update the {activity id} instance of the {activity name} BAM activity. Following the same RESTful principles, you can think of similar models to apply to other BAM elements such as views, alerts, or cubes. An aspect to consider when designing a BAM RESTful API is that some BAM resources represent metadata associated with other BAM resources. For instance, an activity model resource will contain the definition of the different activities and views included in that model. In that sense, the activity model resource contains metadata associated with the activity resource. At this point, we already know how to model the BAM elements as resources and how to associate the HTTP verbs with operations against those resources. That brings us to one of the most important aspects of a RESTful API, which is to select a resource representation format. This is a decision that we dont normally face when designing SOAP/WS-* APIs because, in that case, SOAP is the only representation format. Going back to the previous

section, one of the principles of REST states that resources are representation agnostic. In that sense, we could represent the same resource by using an XML dialect, JSON, or a syndication format such as RSS or Atom. Although, at first glance, plain XML might seem the perfect choice, the fact is that creating a resource representation language entails a lot of details that ultimately can result in interoperability challenges. This is the reason why many successful RESTful APIs use well-established languages like RDF or Atom. The case of syndication formats such as Atom or RSS is particularly popular in scenarios such as event processing in which resources are modified at a high frequency. Additionally, AtomPub is a well-established standard with a large support on heterogeneous platforms. Finally, the extensibility of AtomPub makes it an ideal language for representing untyped structures such as BAM activities. Considering all those factors, we decided to leverage Atom and the Atom Publishing Protocol (AtomPub) as the resource representation language for our sample BAM API. As you might already suspect, a complete BAM RESTful API will include a large variety of operations that will be very difficult to cover in this paper. However, in order to give a pragmatic view of how BAM can take advantage of the REST architecture style to extend its functionality, the following section will explore the implementation of some of the operations of a BAM RESTFul API using WCF 3.5.

A sample BAM RESTful API

As we explained previously, this section does not intend to describe an entire BAM RESTful API. Instead, we intend to explore the implementation of a few operations highlighting the fundamental principles that can be applied to the implementation of other BAM RESTful operations. For the simplicity of the examples, lets start with the following WCF contract. [ServiceContract] [ServiceKnownType(typeof(Atom10FeedFormatter))] public interface ISampleBAMService { [OperationContract] Atom10FeedFormatter GetActivities(); [OperationContract] Atom10FeedFormatter GetActivityInstances(string activityname, string instanceid); [OperationContract] Atom10FeedFormatter CreateActivityInstace(string activityname, bool isend, Stream feed); [OperationContract] Atom10FeedFormatter UpdateActivityInstance(string activityname, string activityid, bool isend, Stream feed); }

The first two operations are intended to navigate through the BAM activities deployed in a BAMPrimaryImport database while the last three operations expose the capabilities required for creating and updating BAM activity instances. As you can see all the operations return an instance of Atom10FeedFormatter, which represents an Atom feed. GetActivities This operation executes based on an HTTP GET request and returns an Atom feed with the definition of the BAM activities deployed in a specific BAMPrimaryImport database. The following code highlights a section of the implementation of this operation (the entire implementation can be found with the samples that accompany this paper).

public class SampleBAMService: ISampleBAMService { [WebGet(UriTemplate="/activities")] public Atom10FeedFormatter GetActivities() { /*Retrieves the list of activities from the BAMPrimaryImportDB database. Details omitted for brevitysee the attached samples for the details */ BAMActivity[] activities= bamdb.GetBAMActivities();

List<SyndicationItem> items= new List<SyndicationItem>(); foreach (BAMActivity activity in activities) items.Add(BAM activity Atom representation); feed.Items = items; return new Atom10FeedFormatter(feed); } }

The WebGet attribute of this operation indicates that this operation is invoked based on an HTTP GET with the form of http://<service endpoint>/activities. In a nutshell, this operation retrieves the list of BAM activities from the bam_Metadata_Activities table in the BAMPrimaryImportDB database and serializes the results into an Atom 1.0 feed. The AtomPub response contains the BAM activity definitions as illustrated in the following figure.

<feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=1</id><updated>2008-0930T18:43:03Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=2</id><title type="text">ALERT</title><updated>2008-0930T18:43:03Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/ALERT/*"/><content type="" Name="ALERT"

ID="IDD832C36CE39041529B431984828B4D76">...</content></entry><entry><id>uuid: 23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=16</id><title type="text">TRANSACTION</title><updated>2008-09-30T18:43:03Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/TRANSACTION/*"/><content type="" Name="TRANSACTION" ID="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC">...</content></entry></feed> Figure: GetActivities AtomPub response Notice that, following the principles of REST, each activity is individually addressable by a URI in the form of http://<service endpoint>/activities/<activity name>. Given that the response is encoded using Atom it can be rendered in a Web browser as illustrated in the following figure.

Figure: GetActivities response rendered in Internet Explorer

Selecting a specific activity will invoke the GetActivity operation to retrieve its instances. GetActivity This operation returns an Atom feed that includes either all instances or a specific instance of a BAM activity and is activated by executing an HTTP GET to a URL in the form http://<service endpoint>/activities/<activity name>/*|<activity instance id>. The following code illustrates fragments of the implementation of the GetActivity operation. [WebGet(UriTemplate="/activities/{activityname}/{instanceid}")] public Atom10FeedFormatter GetActivityInstances(string activityname, string instanceid) { /*Retrieves the list of activities instances from a specific BAM activity Details omitted for brevitysee the attached samples for the details */ BAMActivityInstance[] instances= bamdb.GetBAMActivityInstances(activityname, instanceid); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); foreach (BAMActivityInstance instance in instances) items.Add(instance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed); } Figure: GetActivity implementation If the instanceid parameter is * this operation returns all the instances of a BAM activity. The following code shows a response that lists the instances of the TRANSACTION activity used throughout this article. <feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=35</id><updated>2008-0930T19:42:00Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=36</id><title type="text">125ca0b4-c896-4848-864eb83287e1c539</title><updated>2008-09-30T19:42:00Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/125ca0b4-c8964848-864e-b83287e1c539"/><content type=""> <ActivityID xmlns="">125ca0b4-c896-4848-864e-b83287e1c539</ActivityID> <TRANSACTION_CREATED xmlns="">9/30/2008 2:07:35 AM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns=""> </TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <CUSTOMER_AGE xmlns=""> </CUSTOMER_AGE> <AMOUNT xmlns="">25001</AMOUNT> <TRANSACTIONID xmlns=""> </TRANSACTIONID> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <LastModified xmlns="">9/30/2008 2:18:02 AM</LastModified>

</content></entry> <entry><id>uuid:23d61d50-ea47-4fe9-98cd-c5d1c319c475;id=50</id><title type="text">6052b24f-e863-420b-94bc-fa7edaf7eb7c</title><updated>2008-0930T19:42:00Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/6052b24f-e863420b-94bc-fa7edaf7eb7c"/><content type=""> <ActivityID xmlns="">6052b24f-e863-420b-94bc-fa7edaf7eb7c</ActivityID> <TRANSACTION_CREATED xmlns="">9/29/2008 10:07:35 PM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns="">9/30/2008 5:02:07 AM</TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">CustomerName_0</CUSTOMER_NAME> <CUSTOMER_AGE xmlns=""> </CUSTOMER_AGE> <AMOUNT xmlns="">10</AMOUNT> <TRANSACTIONID xmlns="">234567</TRANSACTIONID> <CREDITCARD xmlns="">2222444455557777</CREDITCARD> <LastModified xmlns="">9/30/2008 5:02:14 AM</LastModified> </content></entry>

Other entries
</feed>

Figure: GetActivity Atom response The results of this operation can also be rendered in a Web browser as illustrated in the following figure.

Figure: GetActivity Atom response rendered in Internet Explorer Selecting a specific activity will again invoke the GetActivities operation with a specific activity ID producing the following response. <feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=69</id><updated>2008-0930T20:16:59Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=70</id><title type="text">125ca0b4-c896-4848-864eb83287e1c539</title><updated>2008-09-30T20:16:59Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/125ca0b4-c8964848-864e-b83287e1c539"/><content type=""> <ActivityID xmlns="">125ca0b4-c896-4848-864e-b83287e1c539</ActivityID> <TRANSACTION_CREATED xmlns="">9/30/2008 2:07:35 AM</TRANSACTION_CREATED> <TRANSACTION_FAILED xmlns=""> </TRANSACTION_FAILED> <TRANSACTION_FAILED_ALERT_TRIGGERED xmlns=""> </TRANSACTION_FAILED_ALERT_TRIGGERED> <TRANSACTION_APPROVED xmlns=""> </TRANSACTION_APPROVED> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <CUSTOMER_AGE xmlns="">

</CUSTOMER_AGE> <AMOUNT xmlns="">25001</AMOUNT> <TRANSACTIONID xmlns=""> </TRANSACTIONID> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <LastModified xmlns="">9/30/2008 2:18:02 AM</LastModified> </content></entry></feed> Figure: GetActivity Atom response for an individual activity Combining these two operations we can navigate through the BAM activity using a syndication client. To create new instances of a BAM activity we need to execute the CreateActivityInstance operation via an HTTP POST. CreateActivityInstance This operation creates an instance of a BAM activity based on an AtomPub request. The following code illustrates sections of the implementation of this operation. [WebInvoke(UriTemplate= "/activities/{activityname}?end={isend}", Method="POST")] public Atom10FeedFormatter CreateActivityInstace(string activityname, bool isend, Stream activityfeed) { /*Creates a new instance of a BAM activity using the Event Stream API. Details omitted for brevitysee the attached samples for the details */ BAMActivityInstance activityInstance= bamdb.CreateActivityInstance(activityname, activityFields, true, isend, activityContent); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); items.Add(activityInstance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed); } As you can see in the code, this operation is activated by an AtomPub request to a URI in the form http://<service endpoint>/activities/<activity name>?end=true|false. The following code shows a sample AtomPub request that creates an instance of the Transaction activity used in our example. Host: localhost:8090 Content-Type: application/atom+xml Content-Length: 142 <entry xmlns="http://www.w3.org/2005/Atom"> <id>1</id> <content type=""> <AMOUNT xmlns="">45001</AMOUNT> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <TRANSACTION_CREATED>9/30/2008 2:07:35 AM</TRANSACTION_CREATED> </content> </entry>

Figure: AtomPub CreateActivity request The produced response is also an Atom message that details the new activity. <feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=75</id><updated>2008-0930T20:59:56Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=76</id><title type="text">e5abe557-c3c4-420f-81c5f2db4b113a10</title><updated>2008-09-30T20:59:56Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/a7ac8da1-624e4d72-bc73-c7fdbd7d9a0a"/><content type=""> <AMOUNT xmlns="">45001</AMOUNT> <CREDITCARD xmlns="">4444333322221111</CREDITCARD> <CUSTOMER_NAME xmlns="">John Doe</CUSTOMER_NAME> <TRANSACTION_CREATED>9/30/2008 2:07:35 AM</TRANSACTION_CREATED> </content></entry></feed> Figure: AtomPub CreateActivity response The activities created by this operation can be completed depending on the value of the end query parameter. If the activity is not completed it can be updated later by using the UpdateActivity operation. UpdateActivity This operation updates an instance of a BAM activity based on an AtomPub request. The implementation of this operation shares many similarities with the CreateActivity explained in the previous section. [WebInvoke(UriTemplate = "/activities/{activityname}/{activityid}?end={isend}", Method = "PUT")] public Atom10FeedFormatter UpdateActivityInstance(string activityname, string activityid, bool isend, Stream activityfeed) { /*Updates a new instance of a BAM activity using the Event Stream API. Details omitted for brevitysee the attached samples for the details */ BAMActivityInstance activityInstance= bamdb.UpdateActivityInstance(activityname, activityid, activityFields, true, isend, activityContent); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); items.Add(activityInstance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed);

As highlighted in the code, this operation is activated by an HTTP PUT message sent to a URL with the form http://<service endpoint>/activities/<activity name>/<activity instance id>?end=true|false. The following code illustrates a sample AtomPub request that updates an instance of the BAM transaction activity used in our example. PUT /bamservice/activities/transaction/a24b1028-8c08-430e-b2d1e25f36c09a59?end=true Host: localhost:8090 Content-Type: application/atom+xml Content-Length: 142 <entry xmlns="http://www.w3.org/2005/Atom"> <id>1</id> <content type=""> <CUSTOMER_AGE xmlns="">32</CUSTOMER_AGE> </content> </entry> Figure: UpdateActivity AtomPub request The UpdateActivity also produces an Atom response detailing the updated elements. <feed xmlns="http://www.w3.org/2005/Atom"><title type="text"/><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=73</id><updated>2008-0930T20:42:24Z</updated><entry><id>uuid:23d61d50-ea47-4fe9-98cdc5d1c319c475;id=74</id><title type="text">a24b1028-8c08-430e-b2d1e25f36c09a59</title><updated>2008-09-30T20:42:24Z</updated><link rel="alternate" href="http://localhost:8090/bamservice/activities/transaction/a24b1028-8c08430e-b2d1-e25f36c09a59"/><content type=""> <CUSTOMER_AGE xmlns="">32</CUSTOMER_AGE> </content></entry></feed> Figure: UpdateActivity AtomPub response In a similar fashion to CreateActivity, the UpdateActivity operation can end a BAM activity instance based on the end query parameter. The combination of the CreateActivity and UpdateActivity operations facilitates populating a BAM activity model from heterogeneous technologies using well-established mechanisms such as HTTP and AtomPub. As we explained in the previous section, a complete BAM RESTful API will include many other operations that will be impossible to cover in this paper. However, the principles and techniques used to implement the operations explained in this section can serve as the foundation for implementing other interesting operations of a BAM RESTful API. The use of REST can be one of the fundamental vehicles to extend the use of BAM across more heterogeneous environments including those developed by using emerging programming paradigms such as dynamic languages, AJAX applications, etc.

Summary
The intention of this paper is to provide a deeper insight into the world of BizTalk Server Business Activity Monitoring, from the basic concepts and how BAM data is collected to how that data is propagated across databases and the multiple methods by which it can be accessed. The first half of this paper dives into the complex details of the BAM infrastructure. We attempt to clarify those details that are critically important, so that developers and administrators using BAM can maintain a high-performance BAM implementation. As described in the second half of this paper, BizTalk Server Business Activity Monitoring is a very extensible technology that reaches beyond BizTalk Server into both Microsoft and non-Microsoft technologies. Some of the Microsoft technologies include Windows Communication Foundation, Windows Workflow Foundation, PerformancePoint Server, SharePoint Server, and SQL Server Integration, Reporting, and Analysis Servers. BAM can be extended to non-Microsoft technologies as well, through the use of Web services and programming models such as Representational State Transfer (REST). Using the information presented in this paper, developers and administrators can effectively create BAM solutions while providing their business analysts a wide variety of tools for accessing and analyzing the BAM data.

Authors
Jesus Rodriguez : Jesus Rodriguez is the Chief Architect of Tellago, Inc. He is also a Microsoft BizTalk Server MVP, an Oracle ACE, and one of a few Architects worldwide to be a member of the Microsoft Connected Systems Advisor team. As a member, Jesus has been selected to participate in a variety of software design reviews with Microsoft product teams including Windows Communication Foundation, Windows Workflow Foundation, and BizTalk Server. Jesus derived his extensive experience with business process integration and messaging through numerous implementations of disparate systems founded on the principles of SOA and BPM. Jesus is an active contributor to the .NET and J2EE communities and an internationally recognized speaker and author with contributions that include several articles for various publications including MSDN Magazine, Microsoft Architecture Journal, SOAWorld, and Web Services Journal as well as speaking engagements at top industry conferences such as Microsoft TechEd, SOAWorld, Microsoft SOA and BPM

Conference, Oracle Open World, Web Services Security Conference, and the Microsoft MVP Summit. Additionally, Jesus has conducted a number of webcasts on varying SOA technologies. Jesus is a prolific blogger on all subjects related to SOA and has a true passion for technology. You can gain valuable insight on leading-edge technologies through his blog at http://weblogs.asp.net/gsusx. Joe Klug : Joe Klug is the Chief Technical Officer at Tellago, Inc. Joe has an extensive background in enterprise application integration and business process management, which he derived though practical consulting engagements and enterprise-scale software development. Prior to Tellago, he worked as a Product Manager in the BizTalk Server team at Microsoft Corporation. During his six years at Microsoft, Joe was a key contributor on several releases of BizTalk Server. Some of his key areas of responsibility included Web services integration and adapter development, and he was one of the founders of the software development kit for developing Windows Communication Foundation-based adapters. While at Microsoft, Joe became a nationally recognized speaker through engagements at Microsoft conferences, including TechEd, SOA & BPM Conference, Office Developers Conference, and SharePoint Conference. He also worked as an interoperability consultant at J.D. Edwards, focusing on integrating J.D. Edwards OneWorld software with other third-party applications, mainframe solutions, and ecommerce gateways. Joe received a Bachelor of Science in Computer Science from Michigan Technological University and an M.B.A. from the University of Colorado. Joe shares his insights on technology through his blog at http://geekswithblogs.net/jklug/Default.aspx.