DRAFT PRODUCTION SUPPORT STANDARDS AND PROCEDURES Daily Pager Rotation Responsibilities 1.

Respond to and resolve Informatica pages by 7 am. The Berkeley Reporting Portal is available to users Monday through Saturday from 7 am to midnight. 2. Notify users by 7 am of unresolved issues from item 1 above. 3. Notify users of reconciliation issues by 7 am. A page is sent at 6 am if there are any new reconciliation issues greater than 5 cents. 4. Resolve reconciliation issues. 5. Resolve email only notifications from Informatica. These are lower priority issues that do not issue a page but only send an email. 6. Review daily reconciliation email from Informatica. 7. Record production and reconciliation issues in DW Production Issues log on sharepoint. https://calshare.berkeley.edu/sites/RAPO/EDW/reporting/Lists/DW%20Produ ction%20Issues/AllItems.aspx 8. Verify that any jobs that were unscheduled due to production issues are rescheduled before the next days processing. 9. Respond to any Footprints tickets assigned to BAIRS or related to BAIRS. Email bairpthelp@lists.berkeley.edu if issue is related to the portal. Weekly Pager Rotation Responsibilities 10. Once, at the beginning of the support week, look at the recon tables, bairstg.RECON_* to review existing out-of-balance conditions. 11. Review Informatica scheduled jobs for correctness. As Needed Pager Rotation Responsibilities 12. When server is rebooted during patches, etc., verify Informatica production schedule has reloaded correctly. Schedule should reload automatically except for workflows that were in file wait mode. 13. After server or database upgrades that may affect connections between the server and databases, test db2 and oracle connections by running test sessions in folder Test_connections. CONTACTS etldoctor@ccsmail.berkeley.edu bairpthelp@lists.berkeley.edu dbdoctor@berkeley.edu db2doctor@berkeley.edu
unix-tickets@berkeley.edu unixteam@berkeley.edu

BAIRS back end team BAIRS front end team Oracle dba team DB2 dba team
Unix team (opens a Footprints problem ticket) Unix team for non-Footprints issues

win-trouble@berkeley.edu windoctor@berkeley.edu servicedesk@berkeley.edu

IST Windows team IST Windows team? IST Service desk (642-8500 phone)

CHECKLIST DETAIL DESCRIPTION Daily Pager Rotation Responsibilities 1. See below. 2. If issues are not resolved or data load is not complete by 7 am: a. Determine nature and extent of user impact. What subject areas are involved; what tables/reports are affected; when will the load complete. b. Contact Oracledoctor to bring database down, if required. This is for severe impact issues. (How do we determine this?) c. Contact Portal Administrator (Michael Leung) to disable report(s), if required. This is useful if one or a few reports will be severly impacted for more than a few hours (what time frame is appropriate?). d. Email message to Em Mandoriao-Furuya emfuruya@berkeley.edu , Ravi Dhurvas dhurvasr@berkeley.edu , Linda Fitzgerald fitzgerald@berkeley.edu , etldoctor etldoctor@ccsmail.berkeley.edu , front end group bairpthelp@lists.berkeley.edu . Linda will send out listserve message. e. Email message to IST Service Desk to post on IST System Status page. servicedesk@berkeley.edu . See sample messages in sharepoint? sharedrive? this doc? f. Post message in Berkeley Reporting Portal Broadcast Messages. Sample messages available in sharepoint? sharedrive? The portal is not available until 7 am. g. When issue is resolved, notify Em, Ravi, Linda, etldoctor and bairptlhelp. Email IST Service Desk to remove message from IST System Status page. Remove message from Portal. 3. Notify users of reconciliation issues by 7 am. A page is sent at 6 am if there are any new reconciliation issues greater than 5 cents. a. Portal message does not have to be posted for i. Problems where amount difference is less than 5 cents ii. Problems that existed the day before. b. A portal message should be posted for all other issues. See the doc PORTAL Messages and Instructions in this folder for instructions. c. Determine extent and impact of reconciliation issue. For example, subject area and reports impacted. For Financials, its useful to know the Ledger, Fiscal Year, Accounting Period and chartstring. Send out notifications below at 7 am with whatever information you have and send updates later with more information, if necessary. d. Post message in Berkeley Reporting Portal Broadcast Messages. Sample messages available in sharepoint? sharedrive? e. Email message to IST Service Desk to post on IST System Status page. servicedesk@berkeley.edu . See sample message below. f. Email Em Fururya, Ravi Dhurvas, Linda Fitzgerald, etldoctor, bairptlhelp with information about issue. Linda will send out listserve message.

g. When issue is resolved, notify Em, Ravi, Linda, etldoctor and bairptlhelp. Email IST Service Desk to remove message from IST System Status page. Remove message from Portal. 4. Resolve reconciliation issues. a. Check emails sent to etldoctor from Informatica. Subject lines are Daily Reconciliation -ucbdw1p and Daily Future Periods Reconciliation ucbdw1p. b. If status is Reconcilation failed., query the table listed in the email. E.g., for the message Reconcilation failed. Please check: BAIRSTG.RECON_FIN , run query select * from bairstg.recon_fin where match_flg =
'N'

c. For recon_type FD and SD, you can find the full chartstring by running
select * from bairstg.recon_fin_chartstring

d. Followup info should be sent to ESS (Em and Ravi) for resolution after researching potential cause of issue. 5. x 6. x 7. x 8. x 9. x 10. x 11. Take a snapshot of the Informatica schedule a close to before the server reboot as possible. Compare to schedule reloaded after reboot.

For after hours problems with EDW mainframe jobs (BISFIN10, BISFIN11), contact the Production Services oncall person. The session will not be launched until the trigger file appears. Once the session sees the trigger file, it will start initialization. When the session is initialized, the trigger file will be deleted. Financials 1. Dimension not complete. If dimension load is not started, leave yesterdays data and inform users. If dimension load has started but not completed by 7 am., inform dbas to keep bair in dbo mode. 24 Support desk: Operations 642-3043 How to start and stop the server. How to verify server is up. HRMS 1. Dont access hrsprod between 6:20 and 7:00 am. Doing so will cause critical HRMS processes to fail. Contact Lyuda Kerman lyudak@uclink4.berkeley.edu with any problems regarding this. 2. Message received while trying to connect to hrsprod

SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "". Communication function detecting the error: "connect". Protocol specific error code(s): "10060", "*", "*". SQLSTATE=08001

Solution: Contact db2doctor. Probable cause is MVS DDF (distributed data facility) came down for last night's backups but did not come back up. The DDF, which supports DB2 access, is normally stopped before and restarted after backups begin. Daily Recon Manual reports

BAIRS Financials Reconciliation The reconciliation strategy is to compare summary data in ATLD to detail data in JAD by Account. Summary Data Source: Bisstg.WK_PS_LEDGER (actuals amts) Bisstg.WK_PS_LEDGER_BUDG (tempbudg amt) Bisstg.WK_PS_LEDGER_KK (encumbrance, pre-encumbrance amts) Target: Bairfact.ACTUAL_TEMP_LEDGER_DETAIL Detail Data Source: Journal Data Bisstg.WK_PS_JRNL_HEADER Bisstg.WK_PS_JRNL_LN Accounting Data Bisstg.WK_PS_FS_JRNL_CARS_UC Bisstg.WK_PS_FS_JRNL_MAIN_UC Bisstg.WK_PS_FS_JRNL_PPS_UC Bisstg.WK_PS_UCOP_INTFC_UC Commitment Control Bisstg.WK_PS_TRANS_LOG Bisstg.WK_PS_ACTIVITY_LOG Bisstg.WK_PS_SOURCE_HDR Bisstg.WK_PS_SOURCE_LN

Bisstg.WK_PS_REFERENCED

Target: Bairfact.JOURNAL_ACCTG_DETAIL Bairfact.JRNL_DETAIL Bairfact.ENCUMBRANCE_DETAIL Bairfact.PRE_ENCUMBRANCE_DETAIL Bairfact.PPS_DETAIL Note that some source to stage copy tables (begin with WK_PS_ for PeopleSoft source tables) may not be complete copies when the source tables are extremely large. In particular, sessions in folder ETL_fin, workflow b_FIN_copy_source, worklet b_JAD_copy_source, b_JAD_copy_7_days contain the last seven days of data. When in doubt, you can check the source data directly. But keep in mind the source database is being updated after 7 am. Journal_Accounting_Detail contains Journal Line and Accounting entry records used by Standard Detail and Full Detail Reports as well as beginning balances for each period. Therefore, it contains duplicate data. In order to compare amounts in JAD with ATLD, limits need to be applied to the query to select only one set of data. For example, to select Standard Detail data in JAD,
SELECT * FROM bairfact.JOURNAL_ACCTG_DETAIL JAD WHERE transaction_type <> 0 AND report_type in ('A','S') AND fiscal_year = x AND accounting_period = x AND account_code = x

Standard Detail reports use the Standard Detail view.

SELECT * FROM bairfact.JOURNAL_ACCTG_DETAIL JAD WHERE report_type in ('A','S')

Customer Detail reports use the Full Detail view.

SELECT * FROM bairfact.JOURNAL_ACCTG_DETAIL JAD WHERE (account_code like '4%' OR account_code like '5%') AND JAD.report_type in ('A','F')

Expanded Detail reports use the Full Detail NoPPS view.

SELECT * FROM bairfact.JOURNAL_ACCTG_DETAIL WHERE ( (REPORT_TYPE = 'A') OR (REPORT_TYPE = 'F' and source_code <> 'PPS') OR (REPORT_TYPE = 'S' JAD

and source_code = 'PPS'))

Financials Reconciliation ETLs Financials Fact tables: ACTUAL_TEMP_LEDGER_DETAIL ENCUMBRANCE_DETAIL JOURNAL_ACCTG_DETAIL JRNL_DETAIL PRE_ENCUMBRANCE_DETAIL Process: 1. ACTUAL_TEMP_LEDGER_DETAIL (summary data) is compared to the Data Warehouse copy of the source data (PeopleSoft Ledger tables). 2. The detail fact tables are summed by Business Unit, Account and compared to summary data (ATLD) summed by Business Unit, Account. 3. Mismatches found for JAD data is expanded by chartstring for quicker problem analysis. Target Tables: 1. bairstg.RECON_FIN_PERIODS
FISCAL_YEAR ACCOUNTING_PERIOD DW_LOAD_DT NUMBER (5) NUMBER (5) DATE NOT NULL, NOT NULL,

2. bairstg.RECON_FIN
RECON_TYPE FISCAL_YEAR ACCOUNTING_PERIOD BUSINESS_UNIT_CODE ACCOUNT_CODE MATCH_FLG RECON_DESCR TEMPBUDG_AMOUNT_ATLD TEMPBUDG_AMOUNT TEMPBUDG_AMOUNT_DIFF ACTUALS_AMOUNT_ATLD ACTUALS_AMOUNT ACTUALS_AMOUNT_DIFF PRE_ENCUMBRANCE_AMOUNT_ATLD PRE_ENCUMBRANCE_AMOUNT PRE_ENCUMBRANCE_AMOUNT_DIFF ENCUMBRANCE_AMOUNT_ATLD ENCUMBRANCE_AMOUNT ENCUMBRANCE_AMOUNT_DIFF DW_LOAD_DT CHAR (3) NOT NUMBER (5) NOT NUMBER (5) NOT CHAR (1) NOT CHAR (5) NOT CHAR (1) NOT VARCHAR2 (30), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), DATE NULL, NULL, NULL, NULL, NULL, NULL,

3. bairstg.RECON_FIN_CHARTSTRING
RECON_TYPE FISCAL_YEAR ACCOUNTING_PERIOD BUSINESS_UNIT_CODE CHAR (3) NUMBER (5) NUMBER (5) CHAR (1) NOT NOT NOT NOT NULL, NULL, NULL, NULL,

ACCOUNT_CODE FUND_CODE ORG_CODE PROGRAM_CODE PROJECT_NO FLEXFIELD_CODE TEMPBUDG_AMOUNT_ATLD TEMPBUDG_AMOUNT TEMPBUDG_AMOUNT_DIFF ACTUALS_AMOUNT_ATLD ACTUALS_AMOUNT ACTUALS_AMOUNT_DIFF PRE_ENCUMBRANCE_AMOUNT_ATLD PRE_ENCUMBRANCE_AMOUNT PRE_ENCUMBRANCE_AMOUNT_DIFF ENCUMBRANCE_AMOUNT_ATLD ENCUMBRANCE_AMOUNT ENCUMBRANCE_AMOUNT_DIFF DW_LOAD_DT

CHAR (5) NOT NULL, CHAR (5) NOT NULL, CHAR (5) NOT NULL, CHAR (2), CHAR (6), CHAR (5), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), NUMBER (15,2), DATE

Recon Type/Descr Values RECON TYPE RECON DESCR FD ATLD vs JAD Full Detail SD ATLD vs JAD Standard Detail JD ATLD vs Jrnl Detail ED ATLD vs Encumbrance Detail PD ATLD vs Pre Encumbrance Detail LD ATLD vs WK PS LEDGERs Mappings/Sessions 1. Calculate periods to run reconciliations for. This is all open periods for the Actuals ledger plus one period previous. One row per Fiscal Year/Accounting Period. Mapping: Session: Workflow: Source: Target: m_RECON_FIN_PERIODS s_m_RECON_FIN_PERIODS wf_ bairdim.CALENDAR_PERIOD bisstg.WK_PS_FIN_OPEN_PERIOD_BK (in src qualifier) bairstg.RECON_FIN_PERIODS

Portal messages: how and what. message matrix.

Portal messages do not seem to refresh. I have to log off and back on to see changed message.

Where can docs be found? AP/PO Archive procedure separate doc. Post overview docs where they can be found. Should they be in standards and procedures. EDW FINANCIALS FACT TABLES QUICK FACTS KK to JAD data flow ETL summary docs Triggers BAIRS HRMS Overview

Troubleshooting reconciliation problems 1. Revise reconciliation reports to find account(s) out-of-balance: a. LdgrSumy section: i. Delete Master Account Tree table. ii. Limit: Add Account begins with xx where xx is level 4 accounts appearing out-of-balance in reports. iii. Request: Delete Account Level 4 node. Add Account Code. Delete Sums requested that are not out-of-balance. b. FullDtl section: i. Delete Account Tree table. ii. Limit: Add Account begins with xx where xx is level 4 accounts appearing out-of-balance in reports. iii. Request: Delete Account Level 4 node. Add Account Code. Delete Sums requested that are not out-of-balance. c. Join results on Account Code. Add Account Code and sums from two tables to Request line. 2. Revise reconciliation reports to find fund(s) out-of-balance: a. LdgrSumy and FullDtl sections: i. Limit: Change Account Code from begins with to actual Account ii. Request: Add Fund Code b. Join results on Account and Fund Code. Add Fund Code to Request line. 3. Revise reconciliation reports to find fund, org(s) out-of-balance: a. LdgrSumy and FullDtl sections: i. Limit: Add Fund Codes found above. ii. Request: Delete sum field. Add Org Code and Amount field. b. Outer Join results on Account, Fund and Org Code. Add match flag to results where if (Tempbudg_Amount == Tempbudg_Amount2) {'Y' } else { 'N'}. Limit on match flag = N. 4. Add more detail fields to narrow down exact rows with mismatch. Look for Match Verification flag of error on FullDtl (hint: remove from limit line). Narrow in on a specific case. They are often all related to a single Journal entry. 5. Compare JAD data with source data.