Atrium Integration Engine: What Is It? How Do You Use It?

Atrium Integration Engine (AIE) Overview

What is it? Overview of latest features

Tell me how to do it
Under the hood details of creating CMDB Integration using AIE

4/9/2010

BMC Atrium Integration Engine

What is it?
A mechanism to manage data transfers between external data sources and CMDB (also AR Forms) Bi-directional and multi-threaded data transfer engine An Enterprise Application Integration tool with data transformation capability It supports event-based or scheduled information flow Ideal for bulk, incremental, data synchronization transfers Provides interfaces to external data source and the CMDB Provides an intuitive graphical interface for field mapping configuration
2 Copyright 4/9/2010 BMC Software, Inc.

BMC Atrium Integration Engine

How is it used?
Transform data to the CMDB data model Efficiently load data in bulk Can create relationships between instances as they are inserted in the CMDB. Designed for complex integrations Easily configurable Can be used against a variety of applications Extensible

Advantages

When Is AIE a Fit?

Complex transfers requiring data transformation Handling large amount of data transfers Multiple integration points
3 Copyright 4/9/2010 BMC Software, Inc.

BMC Atrium Integration Engine 7.1

New Name and Repackaging for EIE Released in late August 2007 Broad Enhancement Areas
Performance and Scalability Usability Improvements
Improved performance of Relationship exchanges Better memory management for all update operations Lot of usability improvements (please see the release notes for details on new features)

Packaging

Ships with Atrium CMDB 2.1 to integrate with Atrium CMDB One product, multiple adapters
Oracle SQL DB2 XML New! New!

Flat file

XML adapter only supports class based relationships (not vendor)

Improved!

Copyright 4/9/2010 BMC Software, Inc.

How to Create AIE Based Integrations to CMDB: The Ingredients To Success

Data Structure Knowledge External Data from Discovery/Inventory Atrium Integration Engine Atrium CMDB Reconciliation Rules Workflow Filters if Needed

Copyright 4/9/2010 BMC Software, Inc.

AIE Terminology

Data Exchange Application (DEA) AIE Service

AR System based application at the heart of AIE DEA passes the service instructions on how to interact with the adaptor CMDB Class/AR Form mapped to External Table CMDB Attributes (or fields of AR Form) mapped to Columns of External Table Values mapped to External Table Columns Data Collection of similar Mappings Definition of when to run Pipe that manages the flow The Plug at the end of the wire
6 Copyright 4/9/2010 BMC Software, Inc.

Data Mapping

Exchange

AIE Service

Exchange Application Data Exchanges Data Mappings

Adapter

Adapter

Configuration SQL Example

Discovery Environment
Discovery AIE Service Data Exchange Application Data Exchanges Data Mappings
Data Cleansing Workflow CMDB Reconciliation Rules Remedy Asset Management

Remedy Environment

AR System
CMDB Discovery Dataset CMDB Reconciliation Engine CMDB BMC.ASSET Dataset

Adapter SQL Database

(can be on either server)

Database for AR System

Copyright 4/9/2010 BMC Software, Inc.

External Data Source

Database data XML data Flat file data Source Relationships can be stored as
foreign key relationship tables

Copyright 4/9/2010 BMC Software, Inc.

Atrium CMDB

Configuration Management Database

Store multiple datasets Reconcile datasets Store Relationships Base data for OOTB Applications

Data Sets

Data value in a field... but so much more Data Sets you should use
BMC.ASSET The Gold Standard BMC.SANDBOX Temp location for GUI changes to Assets VENDOR.PURPOSE.PRODUCT custom datasets
Examples: BMC.IMPORT.MARIMBA, MS.IMPORT.SMS, ACME.IMPORT.LEGACY

Copyright 4/9/2010 BMC Software, Inc.

AIE Main Console

10

Copyright 4/9/2010 BMC Software, Inc.

Configuration > Database Field Menus

Populates the vendor table and column menus Run before you start any mapping work

1. Load

3. Create Menus

2. Add Desired Tables

11 Copyright 4/9/2010 BMC Software, Inc.

How To Data Mapping

12

Mapping Name = name for new mapping Direction = External Data into CMDB Namespace = BMC.CORE in most cases Class = Destination for data, pick from the list External Data Store = SqlServer, Flatfile or Oracle Table Name = Source of data (pick from list) if not in the list, run rulehelper.exe to create the menu items. Row Element = (XML only) the element that represents a row Click Create to save
Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Data Mapping

If the Table Name does not show in the drop down list, it is because you have not configured the Database Field Menus from the main console

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

13

Copyright 4/9/2010 BMC Software, Inc.

How To Data Mapping

Used to map source and destination side primary/foreign keys Keys must be unique on both sides
If multiple keys, stored as AND to define Unique

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

14

Copyright 4/9/2010 BMC Software, Inc.

How To Data Mapping

Map appropriate CDMB Class.Attribute to a Data Store Attribute Make sure to specify CMDB mandatory fields such as

15

Fields use in Key mappings Data Set ID (automatic if Use Dataset is checked) Short Description Submitter Name = anything that identifies the entry AssetLifeCycleStatus = 3 (if Asset Management is installed)
Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Data Mapping

Rules
Works like Function key in Remedy User and Administrator Tools Select by highlighting the row and click Add Rule is added Refer to the Rules box for hints on how to complete the rule Available Rules

16

Copyright 4/9/2010 BMC Software, Inc.

How To Data Mapping

Only works if you can write to the Vendor / Source Database Can be any value Usually is completion status
0 = transfer with no error N = error # - corresponds to ARERROR
17 Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Data Mapping

If you Update Row to Indicate Delete i.e. Soft Delete If you choose to Permanently Delete Hard Delete
No field to set. AIE will do trigger the deletion
18 Copyright 4/9/2010 BMC Software, Inc.

Set MarkAsDeleted to 1 (Yes) Other attributes may be set as desired

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Data Mapping

- Use to limit amount of data in a given 19

transfer. Important if you are experiencing performance or memory issues If you specified Restrict to Dataset on the Main Tab, you dont need to include it here.
Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Data Mapping

1 2 3 4 5 6

1. 2. 3. 4. 5. 6.

If set to all data that can be transferred will be. This could result in partial data records. Transfer as NULL cleans up any data that might be mistaken for null in a field If trailing blanks are possible in your key values, choose to consider Stores change history in a diary field you designate. Attribute must exist in CMDB. If Yes, updates are only made when a change is detected in the source data You must select a field that can store a negative integer to store the checksum Data Exchange Allows you to retrieve new or change data since last transfer You must select a date/time field to store last transfer time Relationship Exchange Data Mapping
Relationship Mapping

20

Copyright 4/9/2010 BMC Software, Inc.

How To - Data Exchange

Exchange Name Name you create to identify the exchange Instance Name Specify the AIE thread to use Data Exchange Type Specify the type of mapping contained in the exchange External Data Store Must match the mapping(s) Data Exchange Execute Exchange As Relationship Exchange Data Mapping Schedule Only
Use Schedule tab

Event Driven Only

run eiexfer.exe from command line, filter or active link
21 Copyright 4/9/2010 BMC Software, Inc.

Relationship Mapping

How To - Data Exchange

Select the mapping(s) to include in the Exchange container. Multiple Mappings

Cuts down on queries for source or target Main Mapping Need to designate if multiple mappings included Key values only come from Main Mapping
Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping

22

How To - Data Exchange

Interval On Completion Section

Works like the interval function in escalation workflow
Data Exchange Data Mapping Relationship Exchange Relationship Mapping

When this exchange is done, you can configure another exchange to run
23 Copyright 4/9/2010 BMC Software, Inc.

How To - Data Exchange

Time
Works like the time function in escalation workflow Set day and time to run
24 Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To - Data Exchange

To Be Triggered
Use this option when you want to exchange to run on completion of some other exchange Must list the exchange in the On Completion section on the other exchange
25 Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To - Data Exchange

Use if the AR Server involved in the exchange is NOT the server that DEA is installed on set to No and fill in the info. Advantage 1 license/DEA can service Dev, Test & Production

You can either set the external connection information on each exchange or on the Connections link from the Data Exchange Console

Alternate Method

26

Copyright 4/9/2010 BMC Software, Inc.

How To - Data Exchange

While Testing In Production

Limit records so you have less to undo Set Enable Debug to No
27 Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To Relationship Mapping

V_Computer BMC_ComputerSystem

Data Mapping
V_DiskDrive BMC_DiskDrive

Data Mapping

Vendor Relationship Table

BMC_HostedSystemComponent

Vendor Relationship Mapping

Map using External Data Store Mapping. Trigger on Complete of Data Exchange.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

28

Copyright 4/9/2010 BMC Software, Inc.

How To Relationship Mapping

BMC_ComputerSystem

V_Computer

Data Mapping
V_DiskDrive BMC_DiskDrive

Data Mapping Class Relationship Mapping Map using CI Class Relationship Mapping Put in an Event Driven Only Exchange*
*In this case, the relationship data exchange only runs after the data transfer for the two configuration item classes has completed.
BMC_HostedSystemComponent

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

29

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

This is where you define which 2 classes to relate and which relationship Must Select Relationship Source on Create External Data into CMDB is the only valid Direction If you choose Relationship Source of External Data Store
Data Exchange Data Mapping

It is best to Restrict to DatasetID and specify

30 Copyright 4/9/2010 BMC Software, Inc.

must specify DB2, FlatFile, Oracle, SqlServer Table Name

Relationship Exchange Relationship Mapping

How To - Relationship Mapping

Define mapping criteria that defines the relationship Works just like creating Relationship in CMDB
Source on Left, Target on Right

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

31

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

Used to define values for a relationship instance Mandatory attributes shown above must be mapped To get a list of Attributes of each CI Class choose class from drop down.
Attributes list will then appear
32 Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To - Relationship Mapping

Used to define values for a relationship instance Mandatory attributes shown above must be mapped To get a list of Attributes of each CI Class choose class from drop down.
Attributes list will then appear
33 Copyright 4/9/2010 BMC Software, Inc.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

How To - Relationship Mapping

If you Update Row to Indicate Delete i.e. Soft Delete

34

Set MarkAsDeleted to 1 (Yes) Other attributes may be set as desired If you choose to Permanently Delete Hard Delete No field to set. AIE will do trigger the deletion
Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

Use if you are doing class based mapping Should match query used on class mapping If you specified Restrict to Dataset on the Main Tab, you
dont need to include it here.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

35

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

Use if you are doing class based mapping Should match query used on class mapping If you specified Restrict to Dataset on the Main Tab, you
dont need to include it here.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

36

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

Use if you are doing class based mapping Should match query used on class mapping If you specified Restrict to Dataset on the Main Tab, you
dont need to include it here.

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

37

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Mapping

Used only if a External Data Store mapping This is used to limit the external records
returned

Data Exchange Data Mapping Relationship Exchange Relationship Mapping

38

Copyright 4/9/2010 BMC Software, Inc.

How To - Relationship Exchange

Works the same as a normal Data Exchange Data Exchange Type = Relationship Class
Data Exchange Data Mapping Relationship Exchange Relationship Mapping

39

Copyright 4/9/2010 BMC Software, Inc.

General AIE Hints

When testing an exchange

Test one exchange at a time (when done chain them to be triggered) Use Run Now button to trigger an exchange immediately Set Limit = 1 (so that you only run once) Be sure you know your debug path

When writing a Vendor Query

Test it with Direct SQL If it doesnt work there, it wont work here Use Indexed fields if possible
This wont work MacAddress != NULL This will MacAddress IS NOT NULL

40

Copyright 4/9/2010 BMC Software, Inc.

General AIE Hints

When setting an AR System field in a mapping

Doesnt work like a setfield or pushfield operation Must Use Rules to convert functions, keywords, etc.
Example of an Integer Field This wont work 0 This will Constant|0 Example of Concatenation This wont work $Manufacturer0$+-+$MediaType0$ This will function|concat($Manufacturer0$,-,$MediaType0$)

41

Copyright 4/9/2010 BMC Software, Inc.

Atrium CMDB

Reconciliation Rules
Identify
Look for potential duplicates

CMDB 3rd Party Dataset

CMDB Master Dataset

Compare Merge Cleanup / Purge

Can delete entire data set from all tables Can delete selected items
Configuration Item Set field MarkedForDelete = Yes Create Reconciliation Rule to Purge if field is set Define how to handle related items Run as a Reconciliation Job

CMDB Reconciliation Engine

CMDB Reconciliation Rules

42

Copyright 4/9/2010 BMC Software, Inc.