Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Abstract
This article describes how to implement PowerExchange 8.6.1 Oracle LogMiner change data capture (CDC) in a nonRAC Oracle environment on UNIX. It identifies the configuration steps that you perform in Oracle, PowerExchange, and PowerCenter to create a working CDC environment. It includes example configuration files, tips, and best practices.
Table of Contents
Overview ........................................................................................................................................................................... 3 Implementation Scenario .................................................................................................................................................. 3 Before You Begin.............................................................................................................................................................. 4 PowerExchange Installation ......................................................................................................................................... 4 Information Required for CDC Configuration................................................................................................................ 5 Oracle CDC Restrictions and Requirements ................................................................................................................ 7 Create Target Tables.................................................................................................................................................... 7 Oracle Configuration Tasks .............................................................................................................................................. 8 Step 1. Specify an Archive Log Destination ................................................................................................................. 8 Step 2. Set the Oracle transaction_auditing Parameter ............................................................................................... 8 Step 3. Enable ARCHIVELOG Mode ........................................................................................................................... 8 Step 4. Stop and Restart the Oracle Database ............................................................................................................ 8 Step 5. Grant the User Privileges Required for CDC ................................................................................................... 8 Step 6. Configure Oracle Minimal Global Supplemental Logging ................................................................................ 9 Step 7. Create a Table Space Exclusively for Oracle LogMiner Use (Optional)........................................................... 9 Step 8. Copy the Oracle Catalog to the Archived Logs .............................................................................................. 10 PowerExchange Configuration Tasks............................................................................................................................. 10 Step 1. Customize the dbmover.cfg File on UNIX ...................................................................................................... 11 Step 2. Customize the PowerExchange Logger pwxccl.cfg File ................................................................................ 16 Step 3. Start the PowerExchange Listener................................................................................................................. 19 Step 4. Customize the dbmover.cfg Files on the Windows and PowerCenter Integration Service Machines............ 20 Step 5. Create Capture Registrations......................................................................................................................... 20 Step 6. Add Change Indicator or Before Image Columns to Extraction Maps (Optional)........................................... 22 Step 7. Perform a Database Row Test of the Extraction Maps .................................................................................. 23 Step 8. Materialize the Target Tables......................................................................................................................... 23 Step 9. Start the PowerExchange Logger .................................................................................................................. 23 Step 10. Allow Changes to Be Written to the Source Tables ..................................................................................... 24 PowerCenter Configuration Tasks .................................................................................................................................. 24 Step 1. Import Extraction Maps for Source Tables..................................................................................................... 24
Step 2. Import Target Table Definitions...................................................................................................................... 25 Step 3. Create a Mapping........................................................................................................................................... 25 Step 4. Configure an Application Connection for the Oracle Sources........................................................................ 25 Step 5. Create a Workflow and Session..................................................................................................................... 26 Step 6. Configure the Restart Token File ................................................................................................................... 29 Step 7. Cold Start the CDC Workflow......................................................................................................................... 29 Additional Information ..................................................................................................................................................... 29
Overview
This article summarizes the steps that are necessary to successfully implement PowerExchange 8.6.1 Oracle LogMiner CDC for multiple Oracle source tables in a non-RAC environment on UNIX. It includes example configuration files and best practices. PowerExchange Oracle LogMiner CDC uses Oracle LogMiner to read change data from Oracle redo logs. To move the change data to target tables, PowerExchange uses the PowerExchange Client for PowerCenter (PWXPC) in conjunction with PowerCenter. To implement Oracle CDC, you need to perform configuration tasks in Oracle, PowerExchange, and PowerCenter. Often, these tasks are performed by more than one person, for example, the Oracle DBA, target RDBMS DBA, system administrator, PowerExchange administrator, and PowerCenter developer. This implementation uses the PowerExchange Logger for Linux, UNIX, and Windows, which is optional for Oracle LogMiner CDC. The PowerExchange Logger captures change data from Oracle redo logs and writes only the successful units of work (UOWs), in chronological order based on commit time, to PowerExchange Logger log files. The change data is extracted from the PowerExchange Logger log files in continuous extraction mode to provide nearreal-time access to the data on the target machine. The benefits of using the PowerExchange Logger include fewer database accesses, faster CDC restart, and no need to prolong retention of Oracle redo files for change capture. Note: Informatica strongly recommends that you use the PowerExchange Logger for Oracle LogMiner CDC. If you use real-time extraction mode without the PowerExchange Logger, PowerExchange starts a separate Oracle LogMiner session for each extraction session. Running multiple, concurrent LogMiner sessions can significantly degrade performance of the system where LogMiner runs. Although this article focuses on UNIX, the information herein also applies to Oracle sources on Linux and Windows.
Implementation Scenario
Scenario: You need to keep a data warehouse synchronized with an Oracle production database in near real time. This scenario makes the following assumptions about your CDC environment and requirements: y y y y y You are using PowerExchange CDC 8.6.1 or later with Oracle 10g or 11g. You need to capture changes from about 50 Oracle source tables that are on a UNIX machine. The UNIX machine is remote from the PowerCenter Integration Service machine. The PowerExchange Navigator and PowerCenter Client reside on the same Windows machine, which is remote from the PowerCenter Integration Service machine. The Oracle source tables are not in an Oracle Real Application Cluster (RAC) environment. You must move change data in near real time to multiple target tables on a remote machine.
y y
You run the PowerExchange Logger for Linux, UNIX, and Windows on the Oracle source machine to capture change data continuously to PowerExchange Logger log files. You extract change data from the PowerExchange Logger log files in continuous extraction mode. Continuous extraction mode provides near real time access to change data but is much less resource intensive than real-time extraction mode. The Oracle source machine has sufficient memory and resources to handle CDC efficiently without offloading PowerExchange Logger processing. You use PWXPC connections and PowerCenter workflows to extract the change data from the PowerExchange Logger log files and to write the data to multiple target tables. Note: PWXPC is a native plug-in that is installed with PowerCenter on the Windows and PowerCenter Integration Service machines.
y y
Gather the information about your Oracle source and environment that is required for CDC configuration. Review CDC limitations and restrictions. Create the target tables.
PowerExchange Installation
PowerExchange installation on UNIX and Windows is all-inclusive. You do not install PowerExchange components separately.
On the UNIX machine with the Oracle source tables, extract files from the 32-bit or 64-bit PowerExchange tar file for your UNIX operating system. Then run install.sh to perform the installation. After installation completes, set the following environment variables: y y y Set the PWX_HOME environment variable to point to the PowerExchange installation directory. Edit the PATH and LIBPATH environment variables to append the PowerExchange installation directory. Verify that the LANG and LC_ALL environment variables are set to C. If these environment variables are not set to C, ask your UNIX administrator to change them.
On the Windows machine, run the PowerExchange 32-bit executable. You must install the Windows 32-bit executable to use the PowerExchange Navigator, even on a Windows 64-bit machine. On the PowerCenter Integration Service machine, follow the PowerExchange installation instructions for the operating system type and bit-level. If the Integration Service runs on a 64-bit machine, use the PowerExchange 64-bit installation executable or tar file. If the Integration Service runs on a 32-bit machine, use the PowerExchange 32-bit executable or tar file. For more information about UNIX and Windows installation procedures, see the PowerExchange Installation Guide.
Table space for Oracle LogMiner Datafile for the Oracle LogMiner table space User-defined name for the LISTENER statement, which is used to start or stop the PowerExchange Listener on the UNIX machine Port number that the PowerExchange Listener uses on the UNIX machine User-defined name for the NODE statement that is used to connect to the UNIX machine
A NODE statement in the dbmover.cfg files on the following machines: Windows machine with the PowerExchange Navigator and PowerCenter Client PowerCenter Integration Service machine
Where Specified A NODE statement in the dbmover.cfg files on the following machines: Windows machine with the PowerExchange Navigator and PowerCenter Client PowerCenter Integration Service machine
Your Value
A NODE statement in the dbmover.cfg files on the following machines: Windows machine with the PowerExchange Navigator and PowerCenter Client PowerCenter Integration Service machine
Path to PowerExchange message log files on the UNIX machine Subdirectory of the LOGPATH directory for alternative message log files for the PowerExchange Logger Path to the directory that stores the CCT and CDCT files on the UNIX machine. These files store information about capture registrations and PowerExchange Logger log files. Path to the directory that stores extraction maps on the UNIX machine User-defined collection ID for an ORACLEID statement for the Oracle source instance. Maximum length is eight characters.
LOGPATH statement in dbmover.cfg on the UNIX machine PFX option of the TRACING statement in dbmover.cfg on the UNIX machine CAPT_PATH statement in dbmover.cfg on the UNIX machine
/pwx/logs altlogs
/pwx/capture
CAPT_XTRA statement in dbmover.cfg on the UNIX machine The following statements in dbmover.cfg on the UNIX machine: First positional parameter in the ORACLEID statement ORACOLL parameter in the ORCL CAPI_CONNECTION DFLTINST parameter in the CAPX CAPI_CONNECTION
/pwx/capture/xtramaps
ORACOL01
Also specified in: Collection ID field when you create a registration group in the PowerExchange Navigator, or Instance field when you view registration group properties. DBID parameter in pwxccl.cfg
Where Specified Second positional parameter in an ORACLEID statement in the dbmover.cfg files on the UNIX and Windows machines Third and fourth positional parameters in the ORACLEID statement in the dbmover.cfg files on the UNIX machine and on the Windows machine. This value is the connect string in the Oracle Client tnsnames.ora file on the UNIX machine. Capture registrations that you define from the PowerExchange Navigator EXT_CAPT_MASK in pwxccl.cfg file on the UNIX machine
Your Value
A database service name that PowerExchange uses to connect to the Oracle database
orcl01.world
Oracle schema.table names for the source tables, and the columns from which to capture change data Path and unique prefix to use for generating the PowerExchange Logger log files. To create the file names, the Logger appends the following:
.CND.CPyymmdd.Thhmmssnnn
pwx/pwxccl
Path and base name to use for generating the PowerExchange Logger checkpoint files Retention period, in days, for CDCT records and PowerExchange Logger log files Database user ID Database password or encrypted password
CHKPT_BASENAME in pwxccl.cfg on the UNIX machine COND_CDCT_RET_P statement in pwxccl.cfg on the UNIX machine CAPTURE_NODE_UID statement in pwxccl.cfg on the UNIX machine CAPTURE_NODE_PWD or CAPTURE_NODE_EPWD statement in pwxccl.cfg on the UNIX machine
pwx/pwxccl.chkpt
30
oracapt oracapt
PowerExchange does not capture changes from columns that have certain datatypes such as LOB or LONG. For more information about datatype restrictions, see the PowerExchange CDC Guide for Linux, UNIX, and Windows.
3.
Issue the following GRANT statements to provide the minimum privileges for running Oracle CDC to the user ORACAPT :
GRANT SELECT ON "PUBLIC"."V$INSTANCE" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$DATABASE" TO "ORACAPT"; GRANT SELECT ON "SYS"."DBA_LOG_GROUPS" TO "ORACAPT"; GRANT SELECT ON "SYS"."DBA_LOG_GROUP_COLUMNS" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$TRANSACTION" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$ARCHIVED_LOG" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$LOGMNR_CONTENTS" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$PARAMETER" TO "ORACAPT"; GRANT SELECT ON "PUBLIC"."V$NLS_PARAMETERS" TO "ORACAPT"; --- If you are using Oracle 10g or later, include the following statement: -- GRANT SELECT ANY TRANSACTION TO "ORACAPT"; -GRANT EXECUTE ON "SYS"."DBMS_FLASHBACK" TO "ORACAPT"; GRANT EXECUTE ON "SYS"."DBMS_LOGMNR_D" TO "ORACAPT"; GRANT EXECUTE ON "SYS"."DBMS_LOGMNR" TO "ORACAPT";
Note: The GRANT LOCK ANY TABLE statement is not required because this scenario uses GENRLOCK=N for the ORCL CAPI_CONNECTION. With this setting, PowerExchange does not lock Oracle tables while creating restart points. 4. Issue the following GRANT statement if you plan to use the Oracle user ID that you specified in the CREATE USER statement to create capture registrations and to automatically run the DDL that PowerExchange generates for creating supplemental log groups at registration completion:
GRANT ALTER ANY TABLE TO "ORACAPT";
For a description of the minimum required Oracle system privileges and object privileges, see the CDC Guide for Linux, UNIX, and Windows.
Note: You must also define a supplemental log group for each Oracle source table. When you register an Oracle table in the PowerExchange Navigator, PowerExchange generates DDL for adding a supplemental log group for the table.
Step 7. Create a Table Space Exclusively for Oracle LogMiner Use (Optional)
Create a table space exclusively for Oracle LogMiner use to prevent the SYSTEM table space (for Oracle version 9i) or SYSAUX table space (for Oracle 10g or later) from becoming full during CDC. This step is necessary only if you have not previously configured LogMiner for use with other Oracle features such as logical standby databases, Oracle Streams, or native Oracle change capture process. 1. To create the table space, issue the following DDL:
CREATE TABLESPACE "LOGMNRTS" NOLOGGING DATAFILE '/oracle_path/datafilename.ora' SIZE 50M REUSE AUTOEXTEND ON NEXT 10M MAXSIZE 100M EXTENT MANAGEMENT LOCAL;
Notes: y Specify NOLOGGING if you use Oracle LogMiner only for PowerExchange CDC and an occasional query. Change NOLOGGING to LOGGING if you use any of the following Oracle features: logical standby databases, Oracle Streams, or native Oracle change capture process. For the DATAFILE value, specify a file name based on your local Oracle database file naming standards for the data files that comprise this table space.
y 2.
If this statement fails with the ORA_01353 message, see the comments in oracapt.sql for more information. 3. To recompile the SYS.DBMS_LOGMNR_D package, enter the following command:
ALTER PACKAGE SYS.DBMS_LOGMNR_D COMPILE BODY;
Tip: Periodically, PowerExchange requests Oracle to recopy the catalog. You can use the CATBEGIN, CATEND, and CATINT parameters in the ORCL CAPI_CONNECTION statement of dbmover.cfg to control how often Oracle copies the catalog and the time period within which the copy occurs.
10
Key Parameter Descriptions The following table describes key parameters for Oracle CDC using the PowerExchange Logger:
Parameter CAPI_CONN_NAME Description Optional. Default CAPI_CONNECTION statement, across all source types.
11
Parameter CAPT_PATH
Description Required. Specifies a path to the local directory where the CCT and CDCT files reside. This directory must exist on the UNIX machine. The CCT file contains capture registrations. The CDCT file contains information about PowerExchange Logger log files. Required. Specifies a path to the local directory where extraction maps reside. This directory must exist on the UNIX machine. Optional. Specifies a service name and connection information for the PowerExchange Listener or PowerExchange Logger process to which you want to send pwxcmd commands from this machine. Note: In this scenario, you run the PowerExchange Listener and PowerExchange Logger in background mode. To issue commands to background processes, you must use the pwxcmd program. However, you can also use pwxcmd to issue commands in foreground mode or from a script file. For more information, see the PowerExchange Command Reference. For this scenario, define CMDNODE statements in the dbmover.cfg files on the UNIX and Windows machines so that you can issue pwxcmd commands from either machine. Syntax is:
CMDNODE=(service_name,service_type,host_name,port_number)
CAPT_XTRA CMDNODE
Where: service_name is a service name. This name can be any name you want to use. For consistency, you can match the service name in the corresponding SVCNODE statement in the dbmover.cfg file on this UNIX machine. service_type is either LISTENER for the PowerExchange Listener or CONDENSE for the PowerExchange Logger. host_name is the host name or IP address of the machine where the PowerExchange Listener or PowerExchange Logger runs. In this case, enter the host name or IP address of this UNIX machine. port_number is the port number that is specified in the corresponding SVCNODE statement of the dbmover.cfg file on this UNIX machine. This port is the one on which the PowerExchange Listener or PowerExchange Logger listens for pwxcmd commands.
If you specify a CMDNODE statement, also specify a corresponding SVCNODE statement. LOGPATH Recommended. Specifies a unique path for PowerExchange message log files. Use this parameter to create message log files in a directory that is separate from your current working directory so that you can find PowerExchange messages more easily. Required. Specifies Oracle SID and connection information for the Oracle source database. Syntax is:
ORACLEID=(collection_id,oracle_db,connect_string, capture_connect_string)
ORACLEID
Where: collection_id is a user-defined name for this ORACLEID statement. This value must match the ORACOLL value in the ORCL CAPI_CONNECTION statement, the collection ID that you specify in registration groups, and the DBID value in the PowerExchange Logger pwxccl.cfg file. Maximum length is eight characters. oracle_db is an Oracle database name. connect_string is a database service name that the PowerExchange Navigator uses to connect to the Oracle database. This value is the connect string in the Oracle Client tnsnames.ora file on the UNIX source machine, which is used to connect to the Oracle database. capture_connect_string is a database service name that the PowerExchange Logger uses to connect to the Oracle database. This value is the connect string in the Oracle Client tnsnames.ora file on the UNIX source machine, which is used to connect to the Oracle
12
Parameter
Description database. Tip: Informatica recommends that you set the following environment variables, whenever possible, to enable connection to the Oracle database without using the capture connect string and to bypass the use of Oracle SQL*Net: o o o o ORACLE_HOME ORACLE_SID PATH One of the following variables, depending on your UNIX platform type: LD_LIBRARY_PATH, LIBPATH, or SHLIB_PATH
If you can bypass the use of SQL*Net when connecting to your Oracle database, you can improve PowerExchange Logger performance, even when running on the same machine as the Oracle database. The capture string and capture connect string are required if you have multiple Oracle instances and you are capturing changes from an instance other than the default instance. You can specify up to 20 ORACLEID statements in a dbmover.cfg. CAPX CAPI_CONNECTION Required for continuous extraction mode. Named set of parameters for continuous extraction of change data from PowerExchange Logger log files. Syntax:
CAPI_CONNECTION=( NAME=name, [DLLTRACE=dll_trace_name], [TRACE=trace_name], TYPE=(CAPX, DFLTINST=collection_id, [FILEWAIT=seconds] ))
Where: NAME is a user-defined name for the CAPI_CONNECTION statement. TYPE must be CAPX for continuous extraction mode with the PowerExchange Logger. DFLTINST is the collection ID specified in the ORACLEID statement that is used by default when the Oracle instance is not specified for an extraction session. This value must also match the ORACOLL value in the ORCL CAPI_CONNECTION statement, the collection ID that you specify in registration groups, and the DBID value in the PowerExchange Logger pwxccl.cfg file. Maximum length is eight characters. Other parameters are optional. For more information, see the PowerExchange Reference Manual.
ORCL CAPI_CONNECTION
Required. Named set of parameters that the CAPI uses to connect to the Oracle change stream and control extraction processing. Syntax:
CAPI_CONNECTION=( NAME=name, [DLLTRACE=dll_trace_name], [TRACE=trace_name], TYPE=(ORCL, [ARRAYSIZE=nnnnnnnnnn], [BYPASSUF=Y|N], [CATBEGIN=hh:mm], [CATEND=hh:mm], [CATINT=minutes], [COMMITINT=minutes], [GENRLOCK=Y|N], [LOGDEST=dest],
13
Parameter
Description
[LGTHREAD=thread], [ONLINECAT=Y|N], ORACOLL=collection_id, [SELRETRY=nnnnnnnnnn], [SNGLINST=ora_instance] ) )
Where: NAME is a user-defined name for the CAPI_CONNECTION statement. TYPE must be ORCL for Oracle CDC. ARRAYSIZE is the size of the prefetch array that PowerExchange uses to read Oracle archived logs. Default is 100 rows. A value less than 100 can adversely affect Oracle capture performance. BYPASSUF indicates whether PowerExchange ends abnormally or issues a warning message whenever it receives an unformatted log record from Oracle LogMiner. Default is N. Enter Y only if your Oracle instance contains Global Temporary tables. CATBEGIN is the earliest 24-hour time of day, in hh:mm format, at which Oracle can attempt to write the Oracle catalog to the Oracle archive logs. Specify both CATBEGIN and CATEND with non-default values, or specify neither parameter. If you specify either CATBEGIN or CATEND, default of 00:00 is used. CATEND is the latest 24-hour time of day, in hh:mm format, at which Oracle can attempt to write the Oracle catalog to the Oracle archive logs. Specify both CATEND and CATBEGIN with non-default values, or specify neither parameter. If you specify neither parameter, default of 24:00 is used. If you specify either CATBEGIN or CATEND, default of 00:00 is used. CATINT is the interval, in minutes, between attempts to copy the Oracle catalog to the Oracle archive logs. Valid values are 1 through 1440. If the interval elapses outside of the CATBEGIN and CATEND timeframe, the catalog is copied during the next CATBEGIN and CATEND timeframe at the CATBEGIN time, if the PowerExchange Logger is running at that time. ORACOLL is the collection identifier that is specified in the ORACLEID statement. This value must also match the DFLTINST value in the CAPX CAPI_CONNECTION statement, the collection ID that you specify in registration groups, and the DBID value in the PowerExchange Logger pwxccl.cfg file. Maximum length is eight characters SELRETRY is the number of times that PowerExchange immediately loops back to the Oracle LogMiner call before implementing a graduated-scale wait loop. Valid values are 0 through 2147483647. Default is 1000. Informatica recommends a value of 0, which causes PowerExchange to not use non-blocking SQL. This setting improves CPU consumption but can prolong extraction session shutdown.
Tip: Include the CATBEGIN, CATEND, and CATINT parameters to control how often Oracle copies the Oracle catalog to the redo logs and the time period within which the copy occurs. If you do not have restrictions for this time period, do not specify CATBEGIN and CATEND. For more information about optional parameters, see the PowerExchange Reference Manual. SVCNODE Optional. Specifies the port on which the command-handling service for a PowerExchange Listener or PowerExchange Logger process listens for pwxcmd commands. Define a SVCNODE statement for each PowerExchange Listener or PowerExchange Logger service to which you want to send pwxcmd commands. Syntax is:
SVCNODE=(service_name,port_number)
14
Parameter
Description o o For the PowerExchange Listener: The node name that is specified in the LISTENER statement. For the PowerExchange Logger: The service name that is specified in the CONDENSENAME statement of the pwxccl.cfg file.
port_number is an available port number on which the PowerExchange Listener or PowerExchange Logger listens for pwxcmd commands. This port number must be different from the one specified in the LISTENER statement.
If you specify a SVCNODE statement, also specify a corresponding CMDNODE statement. TRACING Recommended. Enables PowerExchange to use alternative message log files for each PowerExchange process instead of the detail.log file for all PowerExchange processes. PowerExchange creates the alternative log files in a user-defined subdirectory of the LOGPATH directory. When an alternative log file becomes full, PowerExchange automatically switches to another log file. This parameter can help you find messages for a PowerExchange process more easily and prevent errors related to the message logs becoming full. Syntax:
TRACING=(PFX=subdirectory,[APPEND=Y|N],[BUFFERS=number], [FILENUM=number_of_files],[FLUSH=number_of_records], [RECLEN=record_length],[SIZE=log_size],[VIEW=Y|N])
Where: PFX is required and specifies the name of an existing subdirectory in which to store alternative log files. APPEND indicates how PowerExchange writes messages to existing log files after a restart. Enter Y to have PowerExchange append new log records to the log file that was last used. Enter N to have PowerExchange overwrite the oldest log file. Default is Y. FILENUM is the number of alternative log files. When a log file becomes full, PowerExchange switches to the oldest alternative log file and starts overwriting its contents. Valid values are 1 through 99. Default is 5. FLUSH is the number of records that PowerExchange collects before flushing them to the log file on disk. PowerExchange periodically flushes log records to recover from out-of-space conditions. Low flush values can cause more I/O on the log file. Valid values are 1 through 99. Default is 99. RECLEN is the record length that PowerExchange uses to write log records to the log file. PowerExchange writes the log record on multiple lines if the length of the message exceeds the record length. Valid values are 80 through 255 characters. Default is 80.
For more information about other optional parameters, see the PowerExchange Reference Manual. UOWC CAPI_CONNECTION Required. Named set of parameters for the UOW Cleanser. The UOW Cleanser reconstructs intermingled changes from multiple UOWs into complete and consecutive UOWs that are in chronological order by end time. This statement also points to the ORCL CAPI_CONNECTION. Syntax:
CAPI_CONNECTION=( NAME=name, [DLLTRACE=trace_id], [TRACE=trace], TYPE=(UOWC, CAPINAME=capiname, [BLKSIZE=block_size], [MEMCACHE=kilobytes] ) )
Where:
15
Parameter
Description NAME is a user-defined name for the CAPI_CONNECTION statement. TYPE must be UOWC for the UOW Cleanser. CAPINAME matches the NAME value specified in the ORCL CAPI_CONNECTION statement. MEMCACHE is the amount of memory cache, in kilobytes, that is allocated to the UOW Cleanser for reconstructing complete UOWs. Valid values are 1 through 519720. Default is 1024.
For more information about all dbmover.cfg parameters, see the PowerExchange Reference Manual.
Key Parameter Descriptions The following table describes key parameters for the PowerExchange Logger:
Parameter CAPT_IMAGE Description Data image type that the PowerExchange Logger captures to its log files. Enter AI to capture after images only, or enter BA to capture both before and after images of the data. Informatica recommends that you specify BA so that you have the flexibility to use either AI or BA for extraction and apply processing.
16
Description Optional. An encrypted password or clear text password that is associated with the user ID specified in the CAPTURE_NODE_UID parameter. This password, in conjunction with the CAPTURE_NODE_UID value, is used to control PowerExchange access to capture registrations and change data. Do not specify both CAPTURE_NODE_EPWD and CAPTURE_NODE_PWD. Optional. User ID that is used to control access to capture registrations and change data. Enter a database user ID that permits access to Oracle redo logs and Oracle LogMiner. Required. An existing directory path and a base file name for generated checkpoint files. Checkpoint files store information for properly resuming PowerExchange Logger processing after a warm start. The PowerExchange Logger writes information to the checkpoint files each time a log file switch or shutdown occurs. Maximum length is 256. Example:
/capture/pwxlogger.chkpt
CAPTURE_NODE_UID CHKPT_BASENAME
CHKPT_NUM COLL_END_LOG
Recommended. Number of checkpoint files to use. Enter a value from 2 through 999999. Default is 3. Required. PowerExchange Logger operational mode. Enter 0 for continuous mode. After each logging cycle, the PowerExchange Logger waits for the number of minutes specified in the NO_DATA_WAIT parameter before running another logging cycle. Recommended. Retention period, in days, for PowerExchange Logger log files and the CDCT records for the log files. Log files older than this period and their corresponding CDCT records are deleted automatically during PowerExchange Logger cleanup processing. Default is 60. Usually, a value from 7 to 30 is satisfactory. Tip: Set this parameter to minimize the size of the CDCT file while preserving log files with the earliest change data that you might need to access. When you use continuous extraction mode, PowerExchange reads the CDCT file each time the interval specified in the FILEWAIT parameter of the CAPX CAPI_CONNECTION statement elapses. If a CDCT file becomes large, this read activity can increase I/O, system resource use, and latency of change data extraction.
COND_CDCT_RET_P
CONDENSENAME
Optional. A name for the command-handling service for a PowerExchange Logger process to which you want to issue pwxcmd commands. Maximum length is 64 characters. Syntax is:
CONDENSENAME=service_name
This service name must also be specified in a SVCNODE statement of the dbmover.cfg file on the machine where the PowerExchange Logger runs. CONN_OVR Recommended. Name of the override UOWC CAPI_CONNECTION statement to use for the PowerExchange Logger. If you do not specify CONN_OVR, the PowerExchange Logger uses the default CAPI_CONNECTION if one is specified in dbmover.cfg. Note: In this scenario, the CONN_OVR statement is included for completeness but is not required because the default CAPI_CONNECTION is used. DBID Required. A source identifier, or instance name, that is defined in capture registrations. With DB_TYPE, it defines selection criteria for capture registrations in the CCT file. For Oracle, this value must match the first positional parameter in the ORACLEID statement in dbmover.cfg. This value should also match the ORACOLL value in the ORCL CAPI_CONNECTION statement and the collection ID in the registration group. Required. Source RDBMS type. For Oracle, this value must be ORA.
DB_TYPE
17
Parameter EXT_CAPT_MASK
Description Required. An existing directory path and a unique prefix to be used for generating the PowerExchange Logger log files. PowerExchange considers any file that matches this path and prefix to be a PowerExchange Logger log file. Maximum length is 256 characters. Example:
/capture/pwxlog
Warning: Do not use the same EXT_CAPT_MASK value for multiple PowerExchange Logger processes. Otherwise, a PowerExchange Logger process might corrupt log files that are used by another PowerExchange Logger process. FILE_FLUSH_VAL Recommended. File flush interval, in seconds, that the PowerExchange Logger waits before writing data to the current log file on disk. This parameter affects the latency of change data extractions that use continuous extraction mode. A file flush must occur before data can be read by extraction sessions. Enter a value from 0 through 86400. Default is -1, which causes the PowerExchange Logger process to not flush data to the current log file. Usually, the -1 value is used only when running extractions in batch extraction mode. Warning: A value of 0 can degrade PowerExchange Logger and file system performance. Enter a value this is appropriate for your CDC environment. Values that are too high can increase change extraction latency, and values that are too low can degrade PowerExchange Logger and system performance. For this scenario, enter a value of 10. FILE_SWITCH_CRIT FILE_SWITCH_MIN Optional. Units to use for the FILE_SWITCH_MIN and FILE_SWITCH_VAL parameter values. Enter M for minutes or R for records. Default is M. Use R for better control of file size. Optional. File-switch criteria that the PowerExchange Logger uses when it encounters change data for a new source. You can use this parameter to reduce change data latency when running extractions in continuous extraction mode. Syntax is:
FILE_SWITCH_MIN=(min_val,min_val_ign)
Where: min_val is the minimum number of FILE_SWITCH_CRIT units that must elapse after the PowerExchange Logger encounters a change record for a source that has no entry in the CDCT file, before a file switch can be performed. Valid values are -1 (which causes PowerExchange to ignore this parameter) through 2147483647. min_val_ign is the minimum number of FILE_SWITCH_CRIT units that must pass during a PowerExchange Logger cold start before the PowerExchange Logger uses the min_val value. Before the min_val_ign threshold is met, only FILE_SWITCH_VAL controls file switch activity. Valid values are 0 (which causes PowerExchange to use the min_val value) through 2147483647. This value is ignored if the PowerExchange Logger is warm started.
Default is (-1,0) Warning: A value of (0,0) can result in a large number of file switches when the PowerExchange Logger is cold started and you have registered many source tables for PowerExchange Logger processing.
18
Parameter FILE_SWITCH_VAL
Description Optional. Number of minutes or change records, as determined by FILE_SWITCH_CRIT, that must elapse before PowerExchange performs a file switch. If the PowerExchange Logger log files contain no data when this threshold is reached, the file switch does not occur. Default is 30. This value affects the size of the log files. Specify a value that results in log files of the appropriate size for your environment. Tip: When using continuous extraction mode, as in this scenario, set this parameter such that you have larger log files and a smaller CDCT file. When using batch extraction mode, set this parameter to a value that causes file switches to occur within a timeframe that meets your change propagation latency requirements.
NO_DATA_WAIT
Recommended. If you run the PowerExchange Logger in continuous mode, specify the number of minutes that the PowerExchange Logger must wait before starting the next logging cycle. Default is 60. For continuous extraction mode, enter the minimum value of 1 so that the next logging cycle starts shortly after the current one completes. If the value of FILE_SWITCH_CRIT is M and the value of FILE_SWITCH_VAL is less than the value of NO_DATA_WAIT, the PowerExchange Logger uses the FILE_SWITCH_VAL value instead. In this scenario, enter 1 to log changes on a near continuous basis.
NO_DATA_WAIT2
Recommended. Number of seconds that PowerExchange waits at the end-of-log for more change data before returning control to the PowerExchange Logger. If this wait period elapses and new change data has not been received, the PowerExchange Logger stops the current logging cycle. Default is 600. If you set FILE_SWITCH_CRIT to M, for minutes, Informatica recommends that you enter a NO_DATA_WAIT2 value that is 60 seconds less than the FILE_SWITCH_VAL value. In this scenario, this value is 840 seconds.
Optional. Parameters that define a restart point for starting change data processing when a PowerExchange Logger is cold started. If you enter 0s for both parameters, processing starts from the point in the redo logs where the last Oracle catalog write occurred. If you enter token values other than 0, processing starts at the specific point defined by these token values. If you do not specify these parameters, processing starts from the current end-of-log position.
Note: These parameters are ignored for a warm start. For this scenario, leave these parameters commented out. SIGNALLING Indicates whether the PowerExchange Logger attempts to take automatic action in the event of certain errors. Default is N. Enter Y to have the PowerExchange Logger automatically handle certain errors such as memory corruption. After the PowerExchange Logger handles an error, it attempts to shut down in a controlled manner.
For more information about dbmover.cfg parameters, see the PowerExchange Reference Manual.
19
Where: y y node_name is the first positional parameter in the LISTENER statement of your dbmover.cfg on the UNIX machine. config and license are options that are required only if you placed your dbmover.cfg and license.key files in a directory other than the PowerExchange installation directory, as recommended. In this case, mydirectory is the existing directory that contains your files.
nohup dtllst pwxlst config=mydir/dbmover.cfg license=mydir/license.key &
For the scenario covered in this article, enter: For more information, see the PowerExchange Command Reference.
Step 4. Customize the dbmover.cfg Files on the Windows and PowerCenter Integration Service Machines
You must also customize the dbmover.cfg files on the Windows machine that hosts the PowerExchange Navigator and PowerCenter Client and on the PowerCenter Integration Service machine to add NODE statements that point to the UNIX machine that contains the Oracle source tables. PowerExchange uses these NODE statements to connect to the UNIX machine to read Oracle change data or metadata. Note: You do not need to define a LISTENER statement in these dbmover.cfg files. The PowerExchange Listener runs only on the Oracle source machine to listen for SQL requests for data or metadata.
The CMDNODE statements are optional. Include them to be able to issue pwxcmd commands from the Windows machine to the PowerExchange Listener and PowerExchange Logger processes that run on the UNIX machine. For more information about issuing pwxcmd commands and the command syntax, see the PowerExchange Command Reference.
20
Field Location
Description Enter the user-defined node name for the UNIX machine that contains the Oracle source tables, as specified in a NODE statement of the local dbmover.cfg on the Windows machine. The PowerExchange Navigator uses this information to connect to the UNIX machine. Select ORACLE. Enter a database user ID that has sufficient privileges to access the Oracle source tables for registration. This value should match the CAPTURE_NODE_UID value in pwxccl.cfg. Enter the database password that is associated with the specified user ID. This value should match the CAPTURE_NODE_PWD or CAPTURE_NODE_EPWD value in pwxccl.cfg. Enter the name that you specified for the first parameter in the ORACLEID statement of dbmover.cfg. This value should also match the ORACOLL value in the ORCL CAPI_CONNECTION statement and the DBID value in pwxccl.cfg. Select this option to have PowerExchange display the Add Capture Registration Name and Table Filter page so that you can add capture registrations for Oracle source tables. Tip: Alternatively, to add the registrations later, open the registration group and click Add > Capture Registration.
Type User ID
ORACLE oracapt
Password
oracapt
Collection identifier
ORACOL01
Add Registration
N/A
3.
On the Add Capture Registration Name and Table Filter page, enter the following information: y y Required. In the Name field, enter a descriptive name for the registration in lowercase. This name can be up to eight alphanumeric characters in length and must begin with a letter. Optional. In the Schema Name and Table Name fields, enter schema name and table name masks to filter the list of tables to register. Use the asterisk (*) anywhere in one of these masks to represent one or more characters.
When you click Next, the Add Capture Registration Tables and Columns page appears and lists all tables that match your filter criteria. 4. 5. 6. In the Tables list, double-click a table that you want to register. The table columns appear in the Columns list. Select the columns for which to capture changes. Then click Next. On the Add Capture Registration Type page, enter the following information:
Field Type Status Condense Supplemental Log Group Name Description Select Synchronous. Select Active to set the registration status to active when it is created. Select Part to use the PowerExchange Logger. Enter a supplemental log group name, or accept the default name, which has the format schema_tablename. A supplemental log group must be defined for each table for which you want PowerExchange to capture Oracle change data. PowerExchange generates DDL for creating the supplemental log group for the table when you click Finish. You must save this DDL to a file. You can give the file to your Oracle DBA, or have PowerExchange run the DDL at registration completion.
21
Description Select this option to have PowerExchange run the DDL for creating the supplemental log group when you click Finish. Do not select this option if you do not have the privileges that are required to create supplement log groups on the Oracle table. Instead, give the file with generated DDL to your Oracle DBA.
7. 8.
Click Finish. The Save ALTER TABLE SQL file dialog box appears. Enter the name of the file to which to save the DDL for creating the supplemental log group, and click Save. PowerExchange saves the DDL file to the specified directory and writes the capture registration to the CCT file on the UNIX source machine. If you selected Execute DDL now, PowerExchange also runs the DDL. If a supplemental log group already exists for the table, an error occurs. Tip: For change capture to occur, you must create a supplemental log group for every table that has an active capture registration with Condense set to Part in the registration group. Each supplemental log group must include all of the columns from which PowerExchange is to capture changes. If a supplemental log group with the same name already exists for a table, an error occurs.
Step 6. Add Change Indicator or Before Image Columns to Extraction Maps (Optional)
In the PowerExchange Navigator, you can edit the extraction maps to add PowerExchange-generated change indicator (DTL_CI) and before image (DTL_BI) columns. A DTL_CI column indicates if UPDATEs caused data in the column to change. Based on this information, you can perform some processing, such as capture data only for the columns for which the indicator is Y. A DTL_BI column stores the before image of data in a column that was updated. If you do not add DTL_BI columns, only the after image is captured. 1. 2. 3. In the PowerExchange Navigator, open an extraction map for an Oracle source table. Right-click in the Extraction Definition window, and select Amend Change Indicator/Before Image Extensions. The Extract Definition - Amend Columns dialog box appears. To add DTL_CI or DTL_BI columns, perform the following actions: y y 4. On the Change Indicators tab, select columns in the Available Columns list to which to add a DTL_CI column, and click Add. On the Before Images tab, select the columns In the Available Columns list to which to add a DTL_BI column, and click Add.
Click OK. The DTL__CI_column_name and DTL_BI_column_name columns appear in the Extraction Definition window.
Tip: You can also remove columns from an extraction map. Open the extraction map and clear the column check boxes. Changes to the column data are still captured but not extracted. For more information, see the PowerExchange Navigator User Guide.
22
If you store the pwxccl.cfg file in a directory other than the PowerExchange installation directory, as recommended, you must include the cs option to point to the file:
pwxccl cs=directory\pwxccl.cfg
After the PowerExchange Logger has been cold started once, you can run it continuously in the background by entering the following command:
nohup pwxccl cs=directory\pwxccl.cfg 2>&1 < /dev/null &
23
4. 5.
Click Connect. The Select Datamaps box lists the extraction maps that match your criteria. Select one or more extraction maps to import.
24
6.
Click OK. In the Repository Navigator, the Oracle source table definitions appear under ORACLE_location.
25
Value Select AI, even though the CAPT_IMAGE parameter in the PowerExchange Logger pwxccl.cfg file specifies BA. In this scenario, both before images and after images are captured but only the after images are extracted and written to targets
Restart Token File Folder Number of Runs to Keep Restart Token File
Accept the default value of $PMRootDir/Restart as the restart token file directory. For this default only, PWXPC can automatically create the directory. Enter the maximum number of backup copies to keep of the restart token file. Default is 0. You should specify a value other than 0 or 1 option to retain more than a single pair of initialization and termination restart token files. In this scenario, enter 5.
For more information about all attributes, see PowerExchange Interfaces for PowerCenter. 5. 6. Click OK. Click Close.
Recovery Strategy
Properties
26
Tab Properties
Value Amount of memory allocated to the session from the Data Transformation Manager (DTM) process for reading, writing, and transforming data. By default, the Integration Service determines the DTM buffer size at run time. The minimum is 12 MB. Often, the default amount is too low for CDC processing. Increase this value if necessary. The Integration Service interprets your value as bytes unless you append KB, MB, or GB. Note: This value is related to the Default buffer block size on the Config Object tab. The minimum number of buffer blocks must be greater than the total number of sources and targets. The number of buffer blocks in a session = DTM buffer size / buffer block size. Default settings create enough buffer blocks for 83 sources and targets. If the session contains more than 83, you might need to increase DTM buffer size or decrease default buffer block size. For more information, see the PowerCenter Workflow Basics Guide.
Enter 1 to ensure target data and restart token integrity is maintained. Select the first Oracle source table in the left pane. Then, under Connections, select Application in the Type field. In the corresponding Value field, select the generic CDC Real Time connection object that you created. For Oracle source tables other than the first table, select None in the Type field. Note: In the next step, you customize connection attributes for the first source table and then apply the connection type and attributes to all of the other tables, which have a connection type of None.
5. 6.
On the Mapping tab, under Connections, click the pencil icon next to the Value field while the first Oracle source table is still selected in the left pane. In the Connection Object Definition dialog box, customize the Oracle source connection attributes to be used for the session. The following table describes the connection attributes that you should customize for a PowerExchange CDC session:
Attribute Application Name Value Enter a unique application name. Maximum length is 20 characters. Warning: If you use the same application name for different CDC sessions, data loss or damage to data integrity might occur. Restart Token File Name UOW Count Enter a unique name for the restart token file. Enter the number of units of work (UOWs) that PWXPC must process before it flushes the data buffer to commit the change data to the targets. Default is 1. A value of -1 disables this attribute. Note: If you enter a large number, the RDBMS might hold many locks on tables until the UOW count is reached and commit processing is done. For the commit to occur, the Minimum Rows Per commit threshold, if specified, must also be met. You can also control commits that occur at UOW boundaries based on time by specifying the Real-time Flush Latency attribute. You can specify UOW Count or Real-time Flush Latency or both.
27
Value Enter the number of seconds the PowerExchange Listener remains idle after reaching the end of the change log, as indicated by the message PWX-09967, before returning an end-offile (EOF). Default is -1, which causes an EOF to not be returned and the session to run continuously. A value of 0 causes an EOF to be returned at the end of log and the session to end successfully. Enter the number of milliseconds that must pass before PWXPC flushes the data buffer to commit the change data to the targets. When the real-time latency period expires, PWXPC continues to read the changes in the current UOW until the end of that UOW is reached. Then, PWXPC flushes the data buffer to commit the change data to the targets. Valid values are 0 to 86400. If you specify a value from 1 through 1999 or accept the default of 0, PWXPC uses 2000 milliseconds (or 2 seconds). A value of -1 disables this attribute. For the commit to occur, the Minimum Rows Per commit threshold, if specified, must also be met. You can also control commits that occur at UOW boundaries based on number of UOWs by specifying the UOW Count attribute. You can specify UOW Count or Real-time Flush Latency or both.
Enter the maximum number of change records that PWXPC must process before it can flush the data buffer to commit the change data to the targets. If necessary, PWXPC continues to process change records across UOW boundaries until the maximum rows value is met. PWXPC does not wait for a UOW boundary to commit the change data. Default is 0, which disables this attribute. For the commit to occur, the Minimum Rows Per commit threshold, if specified, must also be met. Tip: Usually, you set this attribute only if you have very large UOWs.
Enter the minimum number of change records that PowerExchange must read from the change stream before it can pass any commit record to PWXPC. Before reaching this minimum, PowerExchange passes only the change records, without any commit records, to PWXPC. Default is 0, which disables this attribute. Tip: This attribute is useful if you have a large number of small UOWs and want a consistent flow of same-sized UOWs.
Enter the name of the CAPX CAPI_CONNECTION statement that you defined in the dbmover.cfg on the UNIX machine with the Oracle source tables. In this scenario, the value is CAPX01.
For more information about Oracle connection attributes, see the PowerExchange Interfaces for PowerCenter guide. Tip: When you are done and return to the Edit Tasks dialog box, right-click the Application field and select Apply Connection Attributes > To all Instances to apply the connection object attributes to the other Oracle source tables. 7. On the Mapping tab, select the first target table in the left pane and customize attributes. The attributes that you set depend on your target type. You should at least set the following attributes: y y 8. 9. For the Target load type attribute, select Normal. For the Table Name Prefix attribute, enter the schema or owner name of the target table. Apply this attribute to all instances.
For more information about creating sessions and workflows, see the PowerCenter Workflow Basics Guide.
28
Additional Information
For more information about PowerExchange CDC for Oracle, see the following resources: y y y y y PowerExchange CDC Guide for Linux, UNIX, and Windows: For configuration of the PowerExchange Logger and of CDC for Oracle and other data sources on Linux, UNIX, or Windows. PowerExchange Interfaces for PowerCenter: For PWXPC information and PowerExchange CDC attributes and options in PowerCenter. PowerExchange Reference Manual: For descriptions of all dbmover.cfg parameters, pwxcmd configuration, and PowerExchange security configuration. How-To article: Configuring PowerExchange 8.6.1 Oracle LogMiner CDC Parameters Based on Environment Type Key Knowledge Base (KB) articles: KB 108325: Knowledge Link: Working with PowerExchange Oracle CDC sessions
29
KB 115883: FAQ: Where should PowerExchange CDC for Oracle components run?
You can access these publications and KB articles from my.informatica.com. New articles and KBs on Oracle CDC are added on a continuing basis. Search the Knowledge Base for additional articles and KBs periodically.
Author
Virginia Pfeifle Principal Technical Writer
30