Scripting OBIEE Is UDML and XML all you need ?

Andreas Nobbmann Consultant Business Intelligence

Andreas.Nobbmann@trivadis.com

Brighton, May 14th, 2009

Basel

Baden

Bern

Lausanne

Zrich

Dsseldorf

Frankfurt/M.

Freiburg i. Br.

Hamburg

Mnchen

Stuttgart

Wien

Why scripting OBIEE ?

Scripting OBIEE Is UDML and XML all you need ?

2009

Agenda
What is UDML ? How to use UDML ? Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

2009

Agenda
What is UDML ? How to use UDML ? Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

2009

What is UDML ?
UDML = Universal Database Markup Language
Commonly known as: Scripting and definition language to describe databases or perform certain actions against databases E.g. DDL creation etc. In OBIEE: Undocumented feature in BI Server repository to manage mouseless administrative scripting Oracle recommends for example to merge Oracle BI repositories by using UDML

Scripting OBIEE Is UDML and XML all you need ?

2009

What to use it for

In general for scripting or describing everything stored in your Oracle BI repository file
Connection pools Users / groups, LDAP server definitions, Security settings, privileges Initialization blocks, variables physical, aliases, logical tables, views... Primary / secondary keys, Foreign key, complex joins Dimensions / hierarchies External name settings (DISPLAY NAME CN_.. / CD_..) .......
Scripting OBIEE Is UDML and XML all you need ? 6 2009

Agenda
What is UDML ? How to use UDML ? Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

2009

How to use UDML ? Ad Hoc

Simply do copy (CTRL+C) on the object you want to script Paste (CTRL+V) it to your favoured editor, edit script and copy back to repository

Scripting OBIEE Is UDML and XML all you need ?

2009

How to use UDML ? Ad Hoc

Attention: Copy is context sensitive, means place your cursor on the father of the object you want to insert the UDML If new Business Model (BM) do not click on any object and paste into the BM window If new table source Go to BM and paste it into And so on
Remark: users and groups cannot be copied this way, they can only be merged into repositories

Scripting OBIEE Is UDML and XML all you need ?

2009

How to use UDML ? UDML generation

Use nQUDMLGen.exe generates UDML from your repository

- N - Q - S

does not generate upgrade IDs (needed for versioning) without security objects only security objects (overwrites Q parameter)

-N use if you want to create a repository from the scratch, OBIA creates new object IDs

c:\OracleBI\server\Bin\nQUDMLgen.exe U Administrator P Administrator R c:\OracleBI\server\repository\paint.rpd O paint_udml

Scripting OBIEE Is UDML and XML all you need ? 10 2009

How to use UDML ? merging repositories

Use nQUDMLExec.exe merges UDML into repository and optionally backs it up

- I - B - O -8

input script name optional base repository name output repository name is for UTF-8

c:\OracleBI\server\Bin\nQUDMLexec.exe U Administrator P Administrator I c:\OracleBI\server\repository\ChangeRepo_2_prodDB_UDML.txt -B c:\OracleBI\server\repository\sales.rpd O c:\OracleBI\server\repository\sales_new.rpd Remark: always make a backup before working with nQUDMLExec.
Scripting OBIEE Is UDML and XML all you need ? 11 2009

How to use UDML ? update via data source

Use nQUDMLcli.exe adding UDML to OBIEE repository via Data Source

-D -w -P

Data Source name as in odbcad32.exe (ODBC Data Source Administrator) optional passphrase for SSL connections if you leave empty it uses the profile of the client you are connecting from

c:\OracleBI\server\Bin\nQUDMLcli.exe U Administrator P Administrator -D AnalyticsWeb I c:\OracleBI\server\repository\ChangeRepo_2_prodDB_UDML.txt Remark: sometimes this tool is not running because of lacking privileges. Copy it with all DLLs named NQSUDMLxx.dll into another directory and restart.

Scripting OBIEE Is UDML and XML all you need ?

12

2009

Agenda
What is UDML ? How to use UDML Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

13

2009

Good examples for UDML usage

Parsing your repository Development street Automating import of user groups Update row counts Add icons to your repository

To be continued .....

2009

Good UDML examples Parsing your repository

Download UDML parser jar-ball from http://code.google.com/p/udmlparser/. Copy it to a directory you want Use nQUDMLgen.exe with options N Q -8 for the repository Fire java jar udmlparser0_4.jar udml=YourUDMLfile.txt rpdxml=YourUDML_XML.xml against YourUDMLFile.txt. You get a well formed XML, which is of course editable and transformable via Style Sheets etc.

Thanks to Daniel Galassi Scripting OBIEE Is UDML and XML all you need ?

for the parser

15

2009

UDML examples Development street

Assume we have three environments, where DWHDEV, DWHTST and DWHPRD, are the data sources names defined within TNSNAMES.ora Export the UDML from your DEV repository with nQUDMLgen.exe c:\OracleBI\server\Bin\nQUDMLgen.exe U Administrator P Administrator R c:\OracleBI\server\repository\sales.rpd O sales_dev_udml search for the connection pool (DECLARE CONNECTION POOL) Copy the whole block of the connection pool into two text files

Scripting OBIEE Is UDML and XML all you need ?

16

2009

UDML examples Development street cont.

In the first file replace the text within the {} with DWHTST In the second file replace it with DWHPRD Adapt your user settings as required Save them as for example DBConnectInfos_UDML_DWHTST and DBConnectInfos_UDML_DWHPRD Generate the repository sales_tst.rpd by using nQUDMLExec c:\OracleBI\server\Bin\nQUDMLexec.exe U Administrator P Administrator I c:\OracleBI\server\repository\DBConnectInfos_UDML_ DWHTST -B c:\OracleBI\server\repository\sales.rpd O c:\OracleBI\server\repository\sales_tst.rpd
Thanks to Venkat, John and Mark for their blogs around that !

Scripting OBIEE Is UDML and XML all you need ?

17

2009

UDML examples Development street cont.

Another way is to copy this into the files

Scripting OBIEE Is UDML and XML all you need ?

18

2009

UDML examples Automating user group import

Get your user groups from your table with a simple SQL Select DECLARE SECURITY ROLE usergroup AS usergroup UPGRADE ID 78077 description{Data Security Group usergroup } PRIVILEGES (READ); from DWH_APP.usergroups Save the result of your SQL in a text file and update your repository with nQUDMLexec c:\OracleBI\server\Bin\nQUDMLexec.exe U Administrator P Administrator I c:\OracleBI\server\repository\UserGroups_UDML -B c:\OracleBI\server\repository\sales.rpd O c:\OracleBI\server\repository\sales_UserGroups.rpd

Thanks to Venkat for his blog around that !

Scripting OBIEE Is UDML and XML all you need ?

19

2009

UDML examples update row counts

2 general steps to do so normally Right click on table and do update row count Or use tools / update all row counts WAITING When using hints in Oracle or an event table for Cache you can get the updated row count from there and copy the results into a UDML file DECLARE TABLE "U".."SYSTEM"."S_ETL_DAY" AS "S_ETL_DAY" UPGRADE ID 80890 NO INTERSECTION ROW COUNT 13394.000000 LAST UPDATED '2009-04-29 18:22:43 PRIVILEGES ( READ);

Scripting OBIEE Is UDML and XML all you need ?

2009

UDML examples making the repository intuitive

Assign the dollar sign to all fact tables DECLARE TABLE "Paint"..."Fact" AS "Fact" UPGRADE ID 554 ICON INDEX 102 HAVING Assign the stop sign to the database connections when you work on them for a longer period of time DECLARE DATABASE "Forecast" AS "Forecast" UPGRADE ID 162 ICON INDEX 92 TYPE XMLDirect Other examples: Question mark (Index 90), globe (Index 81), work cone (index 81) etc.

Scripting OBIEE Is UDML and XML all you need ?

2009

Agenda
What is UDML ? How to use UDML Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

22

2009

XML
Is used for everything defined in the presentation catalog
Requests Prompts, filters Dashboards, dashboard pages User objects, group objects Privileges Descriptions etc.

Scripting OBIEE Is UDML and XML all you need ?

23

2009

XML
3 ways to insert XML into catalog Advanced tab in request Paste XML and Click Set XML

In catalog manager Edit and paste

Or directly in the file system under

<OracleBIData>\web\catalog\<YourCatalog>\path_to_your_report\YourReport

Open with editor Paste XML and save

Scripting OBIEE Is UDML and XML all you need ? 24 2009

Agenda
What is UDML ? How to use UDML Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

25

2009

Good examples for XML usage

Automatically create pivot-table label heading style Format all columns with Percent in name Format all columns depending on their data type Editing XML directly in your editor

To be continued .....

Scripting OBIEE Is UDML and XML all you need ?

26

2009

XML examples create pivot-table

Scripting OBIEE Is UDML and XML all you need ?

27

2009

XML examples create pivot-table

Here the XML source:

Scripting OBIEE Is UDML and XML all you need ?

28

2009

XML examples create pivot-table column formulas

Now open a good XML editor (for example XMLSpy) and create a XSLT stylesheet Add namespaces, start at / elements, then match all saw:report elements and copy them to target

Copy all saw:report elements @* = all attributes

With this part of the stylesheet you copy all the column formulas. <saw:column formula="&quot;D2 Market&quot;.&quot;M01 Market&quot;" columnID="c0"/> .....
Scripting OBIEE Is UDML and XML all you need ? 29 2009

XML examples create pivot-table parameters

The next part is related to parameters needed for formatting purpose You see the parameters aggRule hAlign fontStyle aggregation rule within Pivot Allignment Font Style
3 = # of views in compoundView parameters

Scripting OBIEE Is UDML and XML all you need ?

30

2009

XML examples create pivot-table compoundView

Next is the standard compoundView Here also the pivotTableView1 is added

Scripting OBIEE Is UDML and XML all you need ?

31

2009

XML examples create pivot-table makePivot

next part is creating the Pivot table in XML dynamically Parameters are used with { }

Parameter {aggRule}

Scripting OBIEE Is UDML and XML all you need ?

32

2009

XML examples format all Percent columns

Lets take a look on the pattern functionalities of XML We define the rule for the percent pattern on template saw:column <xsl:if test="contains(@formula, 'Percentage')"> <xsl:attribute name="percent" select="'true'"/> </xsl:if>

pattern Pretty easy, isnt it ?

Scripting OBIEE Is UDML and XML all you need ?

33

2009

XML examples format on data type

Lets derive the data type from the UDML-XML we parsed from the original sales sample

Goal:

define all columns with data type double like 1.000.000,00

34 2009

Scripting OBIEE Is UDML and XML all you need ?

XML examples format on data type

Get presentation column depending on your input column on Xpath level PresentationColumnMappingID

Get the logical column from your UDML data by searching the presentation column on Xpath level LogicalColumnCalculation

Scripting OBIEE Is UDML and XML all you need ?

35

2009

XML examples format on data type cont.

Get the physical column by searching for the logical column on Xpath level PhysicalColumnID

and derive the data type PhysicalColumnDatatype from physical column

Scripting OBIEE Is UDML and XML all you need ?

36

2009

XML examples format on data type

The test result looks like that of course it can be further extended by formatting per data type

Scripting OBIEE Is UDML and XML all you need ?

37

2009

XML examples editing directly in your editor

If you have a good editor (like UltraEdit etc.) installed on your PC then it probably has the column mode, you can put your cursor to a column and edit multiple columns at the same time

Scripting OBIEE Is UDML and XML all you need ?

38

2009

Agenda
What is UDML ? How to use UDML Good examples for UDML usage XML
Data is always part of the game.

Good examples for XML usage The last word

Scripting OBIEE Is UDML and XML all you need ?

39

2009

The last word

UDML and XML are really very useful for scripting Good news: everything can be automated by scripts Cache Seeding, Backup, Synchronizing styles & skins with OC4J, users & user groups, repository, presentation catalog ... UDML&XML provide a good approach for creating repositories and reports automatically from the metadata I am working on it Good news: new tool Content Accelerator Framework (CAF) published within 10.1.3.4.1: Venkat blogged already about it http://oraclebizint.wordpress.com/2009/05/05/oracle-bi-ee-101341content-accelerator-framework-caf-synchronizer-a-reviewpreview-xudml/ My resumee: Scripting with OBIEE is fun !
Scripting OBIEE Is UDML and XML all you need ? 40 2009

Thank you very much!

?
www.trivadis.com
Basel Baden Bern Lausanne Zrich Dsseldorf Frankfurt/M. Freiburg i. Br. Hamburg Mnchen Stuttgart Wien

If you have any questions or want the scripts

Please feel free to drop me a note or give me a call

Andreas.Nobbmann@trivadis.com Trivadis AG Basel Consultant Business Intelligence +41-79 264 8807

Scripting OBIEE Is UDML and XML all you need ?

42

2009

Appendix
Error Messages UDML parser

Data is always part of the game.

Scripting OBIEE Is UDML and XML all you need ?

43

2009

UDML Parser error messages

UDML Parser Messages This category contains message numbers 28xxx. 28001 Near Line @1%d, Near (@2%s): String must be in a single line. Cause. The named string must be on a single line. Response. Remove all the new lines within this string. 28002 Near Line @1%d, Near (@2%s): End of file has been reached. Cause. The singly or doubly quoted string does not have a matching quote. Response. Add single or double quote at the end. 28003 Near Line @1%d, near (@2%s): Syntax error @3%s. Cause. A keyword was misspelled or an unexpected identifier was encountered. Response. Verify the syntax and correct any errors before retrying the action. 28004 Near Line @1%d, (@2%s): Duplicate definition. Cause. The same object was defined more than once. Response. Change the name of one of the objects.

Scripting OBIEE Is UDML and XML all you need ?

44

2009

UDML Parser error messages cont.

28005 Near Line @1%d, (@2%s): Multiple matches found in the current repository. Cause. More than one match was found in the repository. The repository may be corrupted. Response. Repair the repository. Restore it from a back-up, or use the Check Consistency command in the Administration Tool to identify compilation errors. Correct any problems before retrying your actions. 28006 Near Line @1%d, (@2%s): Not defined. Cause. The named symbol is not defined either in the script or the repository. Response. Define the named symbol in the script or create an object in the repository. If necessary, contact Siebel Systems technical support for assistance. 28007 Near Line @1%d, (@2%s): This data type is not supported. Cause. The named data type is not supported. Response. Correct the data type and retry the action. If necessary, contact Siebel Systems technical support for assistance. 28008 Near Line @1%d, (@2%s): This database type is not supported. Cause. The named database type is not supported. Response. Correct the database type and retry the action. If necessary, contact Siebel Systems technical support for assistance.

Scripting OBIEE Is UDML and XML all you need ?

45

2009

UDML Parser error messages cont.

28009 Near Line @1%d, (@2%s): This connection type is not supported. Cause. The named connection type is not supported. Response. Correct the connect type and retry the action. If necessary, contact Siebel Systems technical support for assistance. 28010 Near Line @1%d, (@2%s): It cannot be a parent or container of itself. Cause. A circular definition was found. Response. Correct the appropriate definitions to break the circle. If necessary, contact Siebel Systems technical support for assistance. 28012 Near Line @1%d, (@2%s): This role is not defined. Cause. The required role is missing. Each relationship needs to have two roles. This is an internal error. Response. Contact Siebel Systems technical support for assistance. 28013 Near Line @1%d, (@2%s): This object privilege cannot be used for none set. Cause. This object privilege is wrong. This is an internal error. Response. Contact Siebel Systems technical support for assistance. 28014 Near Line @1%d, (@2%s): This object privilege cannot be used for read set. Cause. This object privilege is wrong. This is an internal error. Response. Contact Siebel Systems technical support for assistance.

Scripting OBIEE Is UDML and XML all you need ?

46

2009

UDML Parser error messages cont.

28009 Near Line @1%d, (@2%s): This connection type is not supported. Cause. The named connection type is not supported. Response. Correct the connect type and retry the action. If necessary, contact Siebel Systems technical support for assistance. 28010 Near Line @1%d, (@2%s): It cannot be a parent or container of itself. Cause. A circular definition was found. Response. Correct the appropriate definitions to break the circle. If necessary, contact Siebel Systems technical support for assistance. 28012 Near Line @1%d, (@2%s): This role is not defined. Cause. The required role is missing. Each relationship needs to have two roles. This is an internal error. Response. Contact Siebel Systems technical support for assistance. 28013 Near Line @1%d, (@2%s): This object privilege cannot be used for none set. Cause. This object privilege is wrong. This is an internal error. Response. Contact Siebel Systems technical support for assistance. 28014 Near Line @1%d, (@2%s): This object privilege cannot be used for read set. Cause. This object privilege is wrong. This is an internal error. Response. Contact Siebel Systems technical support for assistance.

Scripting OBIEE Is UDML and XML all you need ?

47

2009

UDML Parser error messages cont.

28015 Near Line @1%d, (@2%s): This object privilege cannot be used for read_write set. Cause. This object privilege is wrong. This is an internal error. Response. Contact Siebel Systems technical support for assistance. 28016 Near Line @1%d, (@2%s): This object privilege cannot be used for erase set. Cause. This object privilege is wrong. Response. Contact Siebel Systems technical support for assistance. 28017 Near Line @1%d, (@2%s): This object privilege cannot be used for grant set. Cause. This object privilege is wrong. Response. Contact Siebel Systems technical support for assistance. 28018 Near Line @1%d, (@2%s): This object privilege cannot be used for all set. Cause. This object privilege is wrong. Response. Contact Siebel Systems technical support for assistance. 28019 Near Line @1%d: In the metadata expression {@2%s ......}, the following error occurred: @3%s@4%s. Cause. The named expression has one or more errors. This can be caused by spelling errors or objects referenced in the expression that are missing. Response. Ignore and then reconstruct the problem expressions.

Scripting OBIEE Is UDML and XML all you need ?

48

2009

UDML Parser error messages cont.

28020 Near Line @1%d, (@2%s): is already in the repository and it's ID is different from the designated ID. Cause. An object of the same name exists in the repository but with a different ID. The repository may be corrupted. Response. Repair the repository. Restore it from a back-up, or use the Check Consistency command in the Administration Tool to identify compilation errors. Correct any problems before retrying your actions. 28021 Near Line @1%d, (@2%s): is an invalid feature. Cause. The system could not find the named feature. Response. Verify the spelling of the named feature. 28022 Near Line @1%d, (@2%s): is an invalid value for the feature (@3%s) Cause. The specified value is invalid for the named Boolean feature. Response. Valid values include words like YES, NO, ON or OFF. Make the appropriate changes before retrying the action. 28023 Near Line @1%d, (@2%s): is an invalid value repository property name. Cause. The property name is wrong. Response. Use 'CustomPresentationLayer'. 28024 Near Line @1%d, (@2%s): This alias appears more than once. Cause. The alias appeared twice in the repository, either as the name or alias of an object of the same type. Response. Find the duplicates in the repository using the Query tool and make the appropriate changes.

Scripting OBIEE Is UDML and XML all you need ?

49

2009

UDML Parser error messages cont.

28025 Near Line @1%d, (@2%s): This alias is the same as an object in the script. Cause. The alias has the same name as another object of the same type in the script. Response. Correct the script so the alias and the object use different names. 28026 Near Line @1%d, (@2%s): This alias is the same as an object in the repository. Cause. The alias has the same name as another object of the same type in the repository. Response. Correct the repository so the alias and the object use different names. 28027 Near Line @1%d: The Administrator security role does not inherit from any other roles. Cause. The Administrators role inherited from another role. Response. The Administrator security role can only be a root level role. 28028 Near Line @1%d: (@2%s): Invalid DateTime format. Cause. The format of the Datetime string is invalid. Response. Correct the format of the Datetime string and retry the action. 28029 Invalid level relationship to @1%s found. Please recreate the level(s) relationships. Cause. The level with the required ID does not exist in the repository. The repository may be corrupted. Response. Repair the repository. Restore it from a back-up, or use the Check Consistency command in the Administration Tool to identify compilation errors. Correct any problems before retrying your actions.

Scripting OBIEE Is UDML and XML all you need ?

50

2009

UDML Parser error messages cont.

28030 Error: The object @1%s(@2%s) does not exist. If it is on-line mode, please check in changes from the File menu. Cause. The object with the required ID does not exist in the repository. The object may not be checked in. Response. If the repository is in on-line mode, use the Check-in function from the File menu to make the object available. 28031 Error: Cannot open the file: @1%s. Cause. Cannot open the named UDML script file. The file may not exist, or may be in use. Response. Verify that the file exists and is not in use by another process. If the problem persists, contact Siebel Systems technical support for assistance. Source: http://download.oracle.com/docs/cd/E12100_01/books/AnyMsg/AnyMsgMessages10.html (Siebel 2004)

Scripting OBIEE Is UDML and XML all you need ?

51

2009

How to use UDML ? equalize repositories

Use equalizerpds.exe equalizes 2 repositories into a new one

-X -Y -O

treats name-wise similar facts as one (e.g facts Factxxxx as Fact) equals the string sets does not update directly, creates a new repository

c:\OracleBI\server\Bin\equalizerpds.exe A Administrator B Administrator -C c:\OracleBI\server\repository\Sales_Version1.rpd -D Administrator E Administrator -F c:\OracleBI\server\repository\Sales_Version2.rpd X -Y

Scripting OBIEE Is UDML and XML all you need ? 52 2009