PowerCenter Developer: Tips & Tricks for Mapping Designer

Lingaraju Ramasamy (Raju), Technical Architecture Manager Informatica Professional Services

Agenda
Introduction Architecture Best Practices

Mapping Tips & Tricks

Transformation Techniques

Use of Metadata Repository maintenance Q&A

Introduction

Presenter Contact
Lingaraju Ramasamy (Raju) lramasamy@informatica.com

408-368-2475 (Mobile)
Technical Architecture Manager, Informatica Professional Services

Architecture Best Practices

Architecture Best Practices

Consistency
Applying consistent standards reduces long term complications Naming Conventions (Velocity) Descriptions Environments Documentation (Hyperlink to SharePoint)

Modularity
Develop according to a modular design Common Error Handling Reprocessing Mapping Assistants

Reusability
Focus on reuse to make quick and universal modifications Mapplets, Worklets, Transformations, reusable functions
7

Architecture Best Practices

Scalability
Keep volumes in mind in order to create efficient mappings Caching Queries Partitioning Initial vs. Incremental Loads

Simplicity
Multiple simple processes are often better than few complex processes Multiple mappings Simple Queries Staging Tables Advantages Easy to develop, debug, maintain and debug
8

Sample Complex Mapping

SC_T_STR_AT TR_OUTL_ORG _WK (Oracle) SQ_SC_T_STR _ATTR_OUTL_ ORG_WK SEQ_ORG EXP_OUTL_OR G_BOOKEND SC_T_STR_AT TR_OUTL_HOL _FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ HOL_FN SEQ_HOL EXP_HOL_BOO KEND SEQ_OTHERS

SEQ_OUTLET

SC_T_STR_AT TR_OUTL_FN1 (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ FN1

EXP_OUTL_BO OKEND

JAVA_GENERA TE_MSGID

JAVA_GENERA TE_SESSID

SEQ_PK_FK_O UTLET

EXP_CREATE_ HDR_ELEMEN TS

SEQ_HR_HDR

SC_T_STR_AT TR_OUTL_HRS _WK1 (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ HRS_WK1

EXP_HRS_HDR

SC_T_STR_AT TR_OUTL_HRS _WK (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ HRS_WK

SEQ_HRS

EXP_HRS_BOO KEND

JNR_HRS_HDR

JNR_OUTL_HR S

JNR_OUTL_HR _HOL

Multiple Sources > WebServices

JNR_OUTL_LA NG_HOL_HRS JNR_TEAM_ME MBER JNR_ORGANIZ ATION EXP_PASS_TH ROUGH SRT EXP_CHK_NE W_RECS TC_TRANSACT ION_RECS RTR_HDR_DE T_DATA EXP_DETAIL_H OURS

SEQ_LANG

SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ LANG_FN

EXP_LANG_BO OKEND

SEQ_TEAM_ME MBER

SC_T_STR_AT TR_OUTL_TM_ FN (Oracle)

SQ_TEAM_ME MBER

EXP_TEAM_ME MBER

EXP_SEQ_SER VICE_REC_3

EXP_SEQ_DET AIL_HIERARCH Y_RECS

EXP_SEQ_HEA DER_HIERARC HY_RECS11

EXP_HEADER_ HOURS

EXP_SEQ_REC _BASIC

EXP_SEQ_REC _TM_ZONE

EXP_SEQ_DET AIL_HOLIDAY_ RECS

EXP_SEQ_REC _GEO_TYPE

EXP_SEQ_HEA DER_TEAM_M EMBER_RECS

EXP_ACCOUNT ING_REC_6

EXP_SEQ_DAY _OF_WEEK

EXP_SEQ_HEA DER_HOLIDAY _RECS

EXP_SEQ_IDE NTIFICATION_ REC_4

EXP_SEQ_STA TUS_REC_5

EXP_SEQ_DET AIL_TEAM_ME MBER_RECS

UNI_OUTLET

EXP_HDR_BOO KEND

WSC_STR_OU Tl_SV_OUTL

HOL_LEVEL3 (F lat File)

Simplified Complex Mapping

SEQ_LANG

SC_T_STR_AT TR_OUTL_CHG _WK1 (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ CHG_WK

EXP_SRC_BOO KEND

SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ LANG_FN

EXP_LANG_BO OKEND

SEQ_OUTLET

SEQ_OUTLET

SEQ_OTHERS

SC_LKP_GET_ MSGID

SC_T_STR_AT TR_OUTL_CHG _WK (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ CHG_WK

EXP_OUTL_BO OKEND

SC_EXP_CREA TE_HDR_ELEM ENTS

SEQ_OTHERS

SC_LKP_GET_ MSGID

SC_T_STR_AT TR_OUTL_HRS _WK (Oracle)

SC_EXP_CREA TE_HDR_ELEM ENTS JNR_OUTL_LA NG_HOL_HRS EXP_PASS_TH ROUGH SRT_INCM_RE CS EXP_CHK_NE W_RECS RTR_HDR_DE T_DATA EXP_HEADER_ LANG LANG_HEADER _SC_T_STR_A TTR_OUTL_W K (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ HRS_WK

SEQ_HRS

EXP_HRS_BOO KEND

JNR_OUTL_HR S

EXP_PASS_TH ROUGH

SRT_INCM_RE CS

EXP_CHK_NE W_RECS

RTR_HDR_DE T_DATA

EXP_SEQ_DAY _OF_WEEK

DAY_OF_WEEK _SC_T_STR_A TTR_OUTL_W K2 (Oracle)

Staging 1
SEQ_LANG SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ LANG_FN EXP_LANG_BO OKEND

EXP_DETAIL_L ANG

LANG_DETAIL_ STR_ATTR_OU TL_WK (Oracle)

EXP_HEADER_ HOURS

Staging 2
SC_T_STR_AT TR_SITE_FN (O racle) SQ_SC_T_STR _ATTR_SITE_F N EXP_SRC_BOO KEND JAVA_GEN_MS G_ID

HOURS_HDR_ SC_T_STR_AT TR_OUTL_WK ( Oracle)

EXP_DETAIL_H OURS

DETAIL_HOUR S_SC_T_STR_ ATTR_OUTL_ WK1 (Oracle)

SEQ_OUTLET

SC_T_STR_AT TR_OUTL_CHG _WK (Oracle)

SQ_SC_T_STR _ATTR_OUTL_ CHG_WK

EXP_OUTL_BO OKEND

JAVA_GEN_SE SSID

SEQ_OTHERS

SC_LKP_GET_ MSGID

SEQ_ID
SC_EXP_CREA TE_HDR_ELEM ENTS JNR_OUTL_LA NG_HOL_HRS EXP_PASS_TH ROUGH SRT_INCM_RE CS EXP_CHK_NE W_RECS RTR_HDR_DE T_DATA EXP_HEADER_ LANG LANG_HEADER _SC_T_STR_A TTR_OUTL_W K (Oracle)

EXP_GET_SEQ _NUM

WSC_STR_ATT R_SAVE_SITE

SC_T_STR_AT TR_SITE_FN1 ( Oracle)

EXP_DETAIL_L ANG

Staging 3

LANG_DETAIL_ STR_ATTR_OU TL_WK (Oracle)

Staging > WebServices

10

Mapping Tips & Tricks

11

Mapping Tips
Sources and Targets
Use shortcuts from shared folders Extract only what is necessary

Limit reads on source Distinguish between similar sources and targets

Example DIM_CUSTOMER1 = DIM_CUSTOMER_insert DIM_CUSTOMER2 = DIM_CUSTOMER_update

12

Mapping Tricks
Parameters & Variables
Reduce overhead of creating multiple mappings Replace hard coded values Use to incrementally extract data
Example

UpdateDateTime >= TO_DATE ($$PREV_RUN_TS)

: SetVariable ($$CURR_RUN_TS, SESSSTARTTIME)

13

Mapping Tricks
Parameters & Variables
Assign Parameter/Variable values in a Session

Pass values from one session to a subsequent session in same workflow/worklet On Components Tab in Session Properties Use user-defined workflow/worklet variables Non-reusable Sessions only

14

Mapping Tricks
Built-in Mapping Variables
Mapping Name Workflow Name Session Name

Integration Service Name Repository Service Name Repository User Name Folder Name Session Run Mode Source Table Names Target Table Names

15

Mapping Tricks
Group Expression (Anchor transformation)
Add expression transformation after a source qualifier and before a target

If source or target definition changes, reconnecting ports is much easier

16

Mass Update
pmrep massupdate
Session properties Session config attributes

Transformation instance attributes Session instance run time options

17

Mapping Assistants
Preview Data
View Data
Accommodate anomalies early Verification of extraction/loading strategies

Type of Data
Source/Targets Relational, Flat file XML Files

For further analysis, use Informatica Analyst

Analyze the content, quality and structure of source data Involves separate Profiling warehouse, client and reports

18

Mapping Assistants
Mapping Wizard
Pass-Through Slowly Changing Dimension Type 1 Dimension (No History) Type 2 Dimension (All History) Version Data Flag Current Effective Date Range Type 3 Dimension (Previous Versions)

Slowly Changing Dimension Template

19

Mapping Assistants
Mapping Analyst for Excel (MAE)
Standardize specifications Enhance collaboration between analyst and developer Improve documentation & audit ability of business logic
Data Analyst Defines Business Terms Specifies Transformation Rules Standardize Excel format DI Developer Augments, Tunes Generated Mappings from Specifications

Generate Specification

Generate Mapping

20

Mapping Assistants
Mapping Architect for Visio (MAV)
Define consistent methodology & structure for data integration projects

Build custom wizard based on pattern without coding Generate multiple mappings at one time Document data flow
DI Architect Creates & Publishes mapping template
Template File

DI Developer Augments, Tunes Generated Mappings

Informatica Toolbar

Informatica Stencil

Publish Template
Drawing Window

Generate Mappings

Parameter File

21

Mapping Assistants
Mapping Architect for Visio (MAV)
Case Study #1 7 templates were used across 2 projects to generate 600 mappings 97% of mappings were automatically generated and required no additional changes 3% needed to be manually modified or custom developed
Case Study #2 1 template was used to create 150 mappings for a data migration project along with PowerCenter sessions and workflows Total effort was less than one day Equivalent effort to create 150 mappings manually would have been 2 weeks (10x effort)

22

Transformation Techniques

23

Transformation Tips
Source Qualifier
Apply Default Query when possible
Utilize SQ Attributes (i.e., User Defined Join, Source Filter)

Understand advantages and limitations of the SQL override

PROS Utilize database optimizers (i.e., indexes, hints) Can accommodate complex queries CONS Processing impacts database resources Lose transformation logic in metadata searched Unable to utilize Partitioning or Pushdown Optimization options

Minimize complicated queries Add the SQL Override Query to the Description
24

Transformation Tips
Expressions
Understand Port process order
INs or IN/OUTs VARIABLEs OUTPUTs

Reduce code complexity

Use local variables
Redundant calculations Check previous record

Provide comments (-- or //) in expressions

Optimize Expressions
Numeric operations are faster than string operations Operators are faster than functions Un-Nest complicated logic (use IIF or DECODE)

25

Transformation Tips
User-Defined Functions
Build complex expressions and reuse them within repository
Two Types:
Public: Callable from any transformation expression Private: Only callable from another userdefined function

Include any valid function except aggregate functions

Can export to XML Files

26

Transformation Tips
Filters/Routers
Consider Source Qualifier with a filter to limit rows within relational sources Filter as close to the source as possible

Replace multiple filters with a router Pertaining to routers, rows will go to each path where the criteria is TRUE

27

Transformation Tips
Aggregators
Use sorted input to decrease use of aggregate caches
Limit connected input/output or output port Filter data before aggregating Use as early as possible

Joiners
Perform joins in Source Qualifier when possible
Limit use to heterogeneous and flat file sources

Perform normal joins when possible Join sorted input when possible Designate the master source as the source with fewer rows
28

Transformation Tips
Lookups
Using SQL Override in Lookup
Similar to Source Qualifier, avoid when possible Can apply Parameters and Variables Can query against multiple tables in same database Suppress ORDER BY statement by appending two dashes (--)

Add indexes to database columns Replace large lookup tables with joins in the Source Qualifier when possible

Relational Lookups should only return ports that meet the condition Remove all ports not used downstream or in the SQL Override

29

Transformation Tips
Lookup Caches
Lookup Cache Types Persistent Caches Save lookup cache files for reuse Dynamic Caches Retains the latest changes to data as rows pass through the mapping Updating a master table Real-time sessions Slowly changing dimension Cache Sizes Eliminate Paging Stores condition values in index, .idx, files Stores output values in data, .dat, files Apply the Cache Calculator in Session
30

Transformation Tips
Lookups
Cache Updates Update the dynamic lookup cache with results of an expression
Use Case: Update QTY on hand for new timestamp Add WHERE incoming row timestamp > cached timestamp

SQL Overrides for Uncached Lookups You must choose the Use Any Value on Lookup Policy on Multiple Match condition Multiple Rows Return
Use Case: Aggregate customer orders and store the total value

Database Deadlock Resilience NumOfDeadlockRetries DeadlockSleep

31

Transformation Tricks
Pipeline Lookup
Perform a lookup on an application source that is not a relational table or file
Partial pipeline contains Source & Source Qualifier but no target Integration Service reads source data and passes to Lookup Transformation to create cache Create partitions to improve performance

32

Transformation Tips
Transaction Control Transformation
Transaction in PowerCenter is a set of rows bound by commit or rollback Control commit and rollback transactions based on a row or set of rows that pass through the transformation
Use Case: Each invoice number is committed to the target database as a single transaction

Change Tracing Level to Terse

At higher tracing levels, every flush of the write buffers is logged
33

Transformation Tips
Associated Source Qualifier
Use ASQ when MQ data is flat file or COBOL ASQ is specific to the format of the MQ data

34

Transformation Tips
Sequence
Non-Reusable the counter is 0 Performance will be affected if cached is low Increase of caching will improve the performance This doesnt involve any database operation The caching allows to reserve number of rows in the memory

35

File Source and Target Commands

36

File Source and Target Commands

Commands for File Sources
Use a command to generate flat file source data input rows or file list or a session
Unix any valid UNIX command or shell script

Windows any valid DOS or batch file.

Service process variables ($PMSourceFileDir) can be used in the command.

37

File Source Command

Generating a File List

Input Type: Command (default: file) Command Type: Command Generating File List
Command writes list of file names to stdout PowerCenter interprets this as a file list.

38

File Source Command

Generating Source Data

Input Type: Command (default: file) Command Type: Command Generating Data
Command generates rows to stdout Flat file reader reads directly from stdout Removes need for staging data

Example use, reading compressed files

uncompress c $PMSourceFileDir/myCompressedFile.Z
39

File Target Command

Processing Target Data

Output Type: Command (default: file)

Flat file writer writes to the command

Writing compressed files

compress -c - > $PMTargetFileDir/myCompressedFile.Z

Sorting output data

40

Filename Port
Source Filename
Input Filename can be processed and passed to target

41

Filename Port
Target Filename
Write records to a dynamically named flat file

42

Change data detection

43

Change Detection for Updates

MD5 or CRC32
Challenge: a record with a lot of columns needs to be checked for changes
Solution: calculate an MD5 checksum on the columns and use a lookup to compare the value with any existing record

44

Sample Change data detection

Calculate MD5 for all columns except key Create lookup for primary key and MD5 value
Perform insert/update, store MD5 value in target

45

Use of Metadata

46

Querying the PowerCenter repository

Query in designer Limit querying on OPB tables

Use the MX views instead

Utilize Reporting Service

Use Meta Query tool Use Batch Web Services

47

Reporting Service Dashboard

48

Repository Maintenance

49

Repository Maintenance
Purge repository versions
Define version strategy for Dev, QA and Prod Archieve if required for future analysis Purge unwanted versions Run the purge in regular interval daily, weekly or monthly
pmrep connect -r $REPOSITORY_NAME -d $DOMAIN_NAME -n $ADMIN_USER -X INFA_ENCRYPTED_PASSWD

pmrep purgeversion -n $VERSIONS_TO_KEEP -o $FILE_NAME

50

Repository Maintenance
Purge repository logs
Define log strategy for Dev, QA and Prod Archieve if required for future analysis Purge unwanted logs Run the purge in regular interval daily, weekly or monthly
pmrep connect -r $REPOSITORY_NAME -d $DOMAIN_NAME -n $ADMIN_USER -X INFA_ENCRYPTED_PASSWD pmrep truncatelog -t $DAYS_TO_KEEP

Compute statistics on metadata tables

pmrep updatestatistics

51

Additional Informatica Resources

Refer the following...
http://mysupport.informatica.com http://velocity.informatica.com/

http://marketplace.informatica.com Product manuals Informatica Professional Services

52

Questions?

53