Simplifying Data Conversion in Oracle Retail

Simplifying Oracle Retail Data Conversion

Nageshwar Mishra Nageshwar_mishra@mahindrasatyam.com Data Conversion from the legacy system is one of the key activities for any retailer implementing Oracle Retail. While many documents have been released by Oracle on Oracle Retail Data Conversion and also many good articles available on net for Data Migrations/Conversions, sometimes we wonder where to start from? Purpose of this document is the same, providing the one stop document for the understanding of high level approach, data conversion process and also functional areas which require the data conversion for Oracle Retail.

Conversion Approach
Data Migration should be treated as a separate small parallel project along with the complete implementation. Scope of this document is to discuss the high level approach for the end to end data conversion from the legacy system to Oracle Retail MOM suite. Oracle released Data Conversion Toolkit from RMS 12.0.3 for the automatic data conversion from the legacy system. Latest version of this tool is available in RMS 13.1.1 version. Though we have this Toolkit available, still we need to write lot of script using SQL Developer for the data migration as toolkit doesnt support migrating dynamic/transactional data. Data Conversion for Oracle Retail can be broadly categorized in 4 phases: a) Requirement Definition and Design Phase This is the most important and crucial phage. In this phase, the detailed Data Mappings (with legacy and current system) and the strategy will be required in order to begin the detailed design process and also to produce conversion detailed designs and document procedures (manual and automated data conversion) to source, clean, convert and validate the data to be converted from the legacy systems to RMS. Identifying the actual data requirements for the target applications along with all associated data mappings, assumptions, challenges, complete process flows, data volumes, any dependencies and data cleansing requirements are the main task in the Requirement Definition and Design phase. Data Cleansing is one of the key activities which require special focus in this phase. Following is the brief approach, should be followed:

Go Live Conversion and Cutover Phase Build & Test Phase

Req & Def Phase

1. Determine Data Conversion Scope based on the designed functionality of RMS 2. Determine the Configuration Options for all applications 3. Specify Target Data Requirement and the mode of conversion for each tables field in the functional area.

A Mahindra Satyam White Paper

Simplifying Data Conversion in Oracle Retail

4. Identify the data source (existing source systems) that will be used to populate the Target applications. 5. If data is not available in source systems, it should be created manually in flat files or in spreadsheets. 6. Identify and compare the Data Volume and Complexity of the data structures (source and target systems). 7. Using the target data requirements which are already identified, map the fields to the current source system. 8. Determine if any transformation required between source and target data systems. 9. Develop a data conversion functional design document for each identified functional area in scope. Document should specify complete conversion process for each functional area. Document specifies all the data extract and transform requirements based on the mapping exercise and also provides the solution approach to all conversion functional issues. 10. Document the data cleansing requirements available and identified at this stage 11. Identify all the pre and post dependency of each functional area conversion. 12. Understand data model of each functional area in source system. 13. Document all default values and any data validation requirements. 14. Create validation approach for conversion results done by Merchandise toolkit and manual conversion. 15. Plan all the validations which are not supported by toolkit with proper business acceptance. 16. Plan all the static/transaction tables which need to be reloaded each time in every environment. 17. Get the Sign Off on each functional design document before moving to second phase. b) Build & Test Phase: By now, the entire data requirement must have been frozen and functional design documents are created. As mentioned before, you should come up with the detail Data Conversion Approach for each client and since Oracle provides Conversion Utility Tool which takes care of the data conversion for almost all foundation data. Detailed conversion processes for all such functional areas handled by Oracle Utility can be explained Detailed Data Conversion Approach rather coming up with a separate Functional Design Document. But for all Transactional data (e.g. PO, Receipts, Transfers, Inventory etc...). It is strongly advisable to create functional design diagrams for each such area. Objective of this phase is to develop and test complete data conversion script based on these functional designs and conversion approach for other functional areas. Following approach should be followed for this phase: 1. Review the Functional Design Document and Data Conversion approach for all categories. 2. Create a Technical Design Document for the standard solution approach and get it signed off. 3. Develop Data Conversion Scripts based on the functional designs based on the technical design. 4. Develop Test Scripts and Business Acceptance Criteria for unite test and post data conversion validations.

A Mahindra Satyam White Paper

Simplifying Data Conversion in Oracle Retail

5. Prepare the target Oracle Retail environment and make sure that the space required is available and all tables are in place, then install and configure Merchandising Conversion Toolkit. 6. Make sure that data cleansing must be completed before starting the conversion process. 7. Make sure that all Seed data loaded in all merchandising applications. 8. Transform the data flat file into the desired format of Oracle Conversion Utility. 9. Build and unit test the script created for each category. 10. Makes sure that all Database Constraints are turned off before data conversion. 11. Execute Data Conversion Process. It can be done sequentially for each category or all in one go. Executing it sequentially for each category is always advisable. 12. Execute complete batch cycle to validate data integrity among all merchandising applications. 13. Validate the converted data end to end and identify any changes/fixes required. 14. Repeat the Trial after applying the enhancements/fixes. 15. Get the Sign Off on the final successful Build pack. 16. Key deliverables of this phase are final Build Pack along with Technical specifications. c) Conversion & Cutover Phase: Objective of Conversion & Cutover phase is to convert the legacy data into the final production environment prior to go live, in a phased manner and get the business acceptance on that. Following approach should be followed for this phase: 1. Define the Cutover Plan and identify the Cutover Window as short as it can be to avoid any business restrictions. Cutover Window is the time between the initiations of the first data conversion activity and go-live, for the data conversion. 2. Make sure that business restrictions have been imposed during cutover to maintain data accuracy. 3. Make sure that proper data backup has been taken. 4. Execute the data conversion process for each module based on the cutover plan. Another approach is bi Big Bang which is not advisable. 5. Make sure that conversion process is aligned to Deployment Plan 6. Validate the converted data for each functional area. Normally business users who will sign off the converted data should perform this. 7. Make sure that access to production environment is read only. 8. Test the application, batches and report modules as per the identified business acceptance criteria. 9. Make sure that Test scripts should not add/update any data. No transactions should be done before getting the sign of the converted data. 10. Get the Business Data Sign Off. d) Go Live: Once the business users have certified the conversion result meeting the pre-defined acceptance criteria, the new system can go into the production. It is recommended to go live in the beginning of financial year to have less impact of data migration in any reporting. Make sure that any manual post conversion process identified in phase one is done e.g. we might need to update the sock count at the end of the conversion.

Conversion Areas
As mentioned above, the entire data conversion for Oracle Retail MOM Suite is divided in 6 main functional areas based on foundation and transaction data. Below are the table level details which need to be populated for each functional area. There are tables (Prerequisites Tables) which should be populated before using the conversion toolkit for any functional area

A Mahindra Satyam White Paper

Simplifying Data Conversion in Oracle Retail

and even after converting the date in Main Tables, there are certain tables (Post-Loaded) which should be updated before moving to next functional area. Below is the list of all functional areas in its sequential order: 1. Core Data
Prerequisite Tables Required CALENDAR CURRENCIES HALF TSF_ENTITY BANNER CHANNEL DIFF_TYPE TSFZONE STORE_FORMAT Optional SEASONS PHASES BUYER Tables Loaded Using Oracle Conversion Toolkit CUSTOMER TERMS_HEAD TERMS_DETAIL FREIGHT_TYPE FREIGHT_TERMS FREIGHT_SIZE VAT_CODES VAT_CODE_RATES VAT_CODE_REGIONS UDA UDA_VALUES TICKET_TYPE_HEAD TICKET_TYPE_DETAIL DIFF_IDS Post-Loaded Tables Required DIFF_GROUP_HEAD DIFF_GROUP_DETAIL Optional DIFF_RANGE_HEAD DIFF_RANGE_DETAIL DIFF_RATIO_HEAD DIFF_RATIO_DETAIL SOURCE_DLVRY_SCHED SOURCE_DLVRY_SCHED_DAYS SOURCE_DLVRY_SCHED_EXC STOP_SHIP TRANSIT_TIMES

2. Merchandise Hierarchy
Prerequisite Tables Required DIVISION GROUPS COMPHEAD Tables Loaded Using Oracle Conversion Toolkit DEPS CLASS SUBCLASS STOCK_LEDGER_INSERTS VAT_DEPS UDA_ITEM_DEFAULTS RPM_DEPT_AGGREGATION Post-Loaded Tables Required RPM_MERCH_RETAIL_DEF HIERARCHY_PERMISSION (Retail Security Manager [RSM] table) All department UDA defaults must be set up manually where UDA_ITEM_DEFAULTS.REQUIRED _IND = Y

3. Organizational Hierarchy
Prerequisite Tables Required CHAIN AREA Tables Loaded Using Oracle Conversion Toolkit ADDR WH WH_ADD STOCK_LEDGER_INSERTS TRANSIT_TIMES COST_ZONE COST_ZONE_GROUP_LOC REGION DISTRICT STORE_ADD STORE_DEPT_AREA Post-Loaded Tables Required DEPT_CHRG_HEAD DEPT_CHRG_DETAIL STORE_HIERARCHY COST_ZONE_GROUP (zone level pricing) Location level COST_ZONE_GROUP should have been created by the seed data installation. COST_ZONE COST_ZONE_GROUP_LOC RPM_ZONE_GROUP_TYPE RPM_ZONE_GROUP RPM_ZONE RPM_ZONE_LOCATION Optional (after running Batch) STORE_ATTRIBUTES WH_ATTRIBUTES STORE_SHIP_DATE

A Mahindra Satyam White Paper

Simplifying Data Conversion in Oracle Retail

WH_DEPT WH_DEPT_EXPL WH_BLACKOUT WH_STORE_ASSIGN

4. Suppliers
Prerequisite Tables Required PARTNER (required types: AG=agents, BK=advising or issuing banks, FA=factory) OUTLOC (required types: DP=discharge ports, LP=lading ports) Tables Loaded Using Oracle Conversion Toolkit SUPS ADDR (for supplier addresses) SUP_IMPORT_ATTR Post-Loaded Tables Required SUP_BRACKET_COST Optional SUP_ATTRIBUTES SUP_INV_MGMT SUP_REPL_DAY SUPP_PREISSUE SUPS_MIN_FAIL

5. Items & Prices

Prerequisite Tables Required N/A Tables Loaded Using Oracle Conversion Toolkit ITEM_MASTER VAT_ITEM UDA_ITEM_LOV ITEM_CHRG_HEAD ITEM_CHRG_DETAIL PACKITEM PACKITEM_BREAKOUT PRICE_HIST RPM_ITEM_ZONE_PRICE ITEM_SUPPLIER ITEM_SUPP_COUNTRY ITEM_SUPP_COUNTRY_DIM ITEM_LOC ITEM_LOC_SOH RPM_FUTURE_RETAIL ITEM_SUPP_COUNTRY_LOC FUTURE_COST UDA_ITEM_DATE UDA_ITEM_FF ITEM_SEASONS ITEM_TICKET Post-Loaded Tables Optional PACK_TMPL_HEAD PACK_TMPL_DETAIL ITEM_ATTRIBUTES ITEM_SUPP_UOM ITEM_LOC_TRAITS SUB_ITEMS_HEAD SUB_ITEMS_DETAIL ITEM_FORECAST REPL_ITEM_LOC REPL_DAY MASTER_REPL_ATTR

6. Transactional Data If conversion of transaction data is in scope, they we should make sure that below tables in each functional category must be populated:
Purchase Orders PO_TYPE ORDHEAD ORDSKU ORDLOC Inventory ITEM_LOC_SOH REPL_DAY REPL_ITEM_LOC ITEM_LOC_SOH Transfers TSFHEAD VAT_HISTORY Sales TRAN_DATA_A DAILY_SALES_DISCOUNT VAT_HISTORY

A Mahindra Satyam White Paper

Simplifying Data Conversion in Oracle Retail

Conversion Process
Though there are multiple tools available for the data conversion including oracle retail data conversion toolkit. Its always better to prefer the toolkit for the main set of tables and SQL Loader of other tables required for pre and post loading and also for transaction tables. Ensure that Merchandise applications must be installed and all seed data loaded before conversion. Bellow is standard process for the data conversion: Capture the data in Excel Templates. Convert the template into CSV File. Convert CSV file into *.dat file extension. Convert *.dat file into Pipe (|) delimited flat *.dat file. Follow the Standard naming conventions e.g. Supplier flat file name: dc_sups.dat. 6. File should be encoded in UTF-8. Character fields should not contain carriage returns which indicate a new record. 7. Copy files (have a size greater than 0) in the data directory defined in the common configuration file from where it can be read from Conversion toolkit of SQL Loader. 8. Ensure that the script file has the proper permissions: rwxrwx-r-x 9. Ensure that conversion is done sequentially for each functional area. 10. Load Prerequisites tables first using SQL *Loader before converting to main tables for any functional area. 11. Ensure that post-loading tables populated before moving to next functional area. 12. Use Oracle conversion toolkit to convert all main tables of foundation data (i.e. first 5 functional areas defined above) 13. Delete the status (*.status), discard (*.dsc), and bad (*.bad) files before running conversion toolkit 14. Include the directory where the conversion scripts will be executed in the environment path variable (PATH), 15. Follow below process to run the Conversion Toolkit: a) Run the master script using the following syntax (where > represents the UNIX or Linux command line prompt): a. dc_load_main.ksh q <sequence-file-name> b) To run individual segment wrapper scripts, the q <sequence-file-name> portion of the command line is not required. For example: a. dc_load_core.ksh 16. Validate converted data and do hot-fix if required. 1. 2. 3. 4. 5.

References:
1. 2. 3. 4. 5. Oracle Retail Merchandising Data Conversion Planning Guide Oracle Retail Merchandising Data Conversion Operations Guide Oracle Retail Documentation Oracle Metalink ERP Data Conversion

Copyright 2010 by Mahindra Satyam Computer Services. All Rights Reserved. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. Reproduction of this publication in any form without prior written permission is forbidden

A Mahindra Satyam White Paper