the metadata tool - kent state university · cognos 10.2.2 ... 1 portal admin / web developer ......

The Metadata Tool Making ODS Metadata More Useful to End Users

Meredith Pritchard

BI Developer/Analyst, Denison University

About Denison

Founded in 1831 Formerly known as Granville College (until 1853) Liberal Arts college (38% acceptance rate) 2,250 students (2014-15 - 43% male, 57%

female, 36% multicultural, 20% First Generation) 226 Full Time Faculty Fall and Spring semesters Residential campus Located in Granville, OH – just 25 miles east of

Columbus Source: Wikipedia

ERP and Reporting Environment

Banner ◦ Live in 1997 ◦ Version: 8 ◦ Currently implementing XE in Test Environment

ODS ◦ 8.4.1 ◦ Materialized Views

Cognos ◦ 10.2.2 ◦ Report Studio = 10 IT ◦ Query Studio and CWA = 50 End Users ◦ Consumers = 200+ Faculty and Staff

Oracle 11.2.0.4 ◦ Linux (virtualized)

Enterprise Apps Team

1 Director

1 DBA

4 Programmer Analysts

2 BI Analysts

1 Portal Admin / Web Developer

1 Integration Specialist / Web Developer

Most Popular Scenarios for utilizing Delivered Metadata

End Users ◦ I want to create a Query in Cognos which includes these Banner fields. What package do I use?

Report Writers ◦ Translating a business spec into a report spec. Does this Cognos Data Item map to the Banner field that the customer needs?

Example Delivered Metadata

Delivered Metadata Downfalls & Project Goal

Delivered Metadata Downfalls ◦ Difficult for End Users to navigate

◦ Like drinking from a fire hose

◦ Incomplete - does not map to Cognos Packages

Goal ◦ Create useable metadata for End Users

Solution Overview

1. Scrape the Cognos FWM model xml (sghe_ods_bv_model.xml) for the lineage

2. Join scraped model data with WMT table data from IA_ADMIN schema

3. Create a table (DU_Metadata) with complete lineage data

4. Model the DU_Metadata table in FWM 5. Create a report that uses DU_Metadata table to

show lineage – mapping a Banner field to all possible Cognos Packages or vice versa

6. Publish Report to all Folders & Create Job Aid

Solution Detail – Step 1: Scrape the Cognos Model

This process uses the delivered FWM model (a large, ugly xml file - sghe_ods_bv_model.xml) and the WMT Tables from the IA_ADMIN schema.

Credit for this process goes to our DBA Ninja, Doug Davenport.

Solution Detail – Step 1: Scrape the Cognos Model

1. Used tool (xmllint --format) to reformat the xml file so he could inspect the model to determine patterns within model

2. Create an xslt query (query language for xml) to document the various patterns (heirarchies) of the xml data. Example of 1 of the patterns:

<xsl:text>"</xsl:text> <xsl:value-of select="../../../../cognos:name"/> <xsl:text>","</xsl:text> <!-- Package Name --> <xsl:value-of select="../../../cognos:name"/> <xsl:text>","</xsl:text> <!-- Query Subject --> <xsl:value-of select="../cognos:name"/> <xsl:text>","</xsl:text> <!-- Query Item --> <xsl:value-of select="cognos:name"/> <xsl:text>","</xsl:text> <!-- Expression --> <xsl:value-of select="normalize-space(cognos:expression)"/> <xsl:text>"</xsl:text> <!-- Output a new line --> <xsl:text>&#10;</xsl:text> </xsl:for-each>

Solution Detail – Step 1: Scrape the Cognos Model

3. Create 1st shell script that:

Checks out model xml from git repository

Runs the xslt query (from step 2) against the xml model using xsltproc (command line utility)

Creates du_query_item.csv which becomes the basis of an Oracle External table

Solution Detail – Step 1: Scrape the Cognos Model

DU_QUERY_ITEM

Solution Detail – Step 2: Join metadata to model data

3. Create 2nd shell script (du_metadata.shl) that:

Uses PL/SQL to loop through the du_query_item external table and supplement each row with data from the WMT tables from IA_ADMIN, then writes a row to an oracle table (du_metadata).

Solution Detail – Step 2: Join metadata to model data

Snippet of DU_METADATA.SHL: DECLARE l_rec1 du_ods.du_query_item%ROWTYPE; l_rec2 ia_admin.wmt_source_to_target_map%ROWTYPE; l_rec3 ia_admin.wmt_target_column%ROWTYPE; l_rec4 ia_admin.wmt_target%ROWTYPE; l_rec5 ia_admin.wmt_source_column%ROWTYPE; l_rec6 ia_admin.wmt_source%ROWTYPE; l_rec7 ia_admin.wmt_subject_area%ROWTYPE; l_tbl varchar(32); BEGIN FOR l_rec1 IN (SELECT * FROM du_ods.du_query_item) LOOP BEGIN l_tbl := 'wmt_source_to_target_map'; SELECT * INTO l_rec2 FROM ia_admin.wmt_source_to_target_map WHERE TARGET_TYPE = 'REPORTING VIEW' AND TARGET_NAME = UPPER(REPLACE(l_rec1.query_view, ' ', '_')) AND TARGET_COLUMN_NAME = UPPER(REPLACE(l_rec1.query_column, ' ', '_')) AND ROWNUM = 1; -- l_tbl := 'wmt_target_column'; SELECT * INTO l_rec3 FROM ia_admin.wmt_target_column WHERE TARGET_TYPE = 'REPORTING VIEW' AND TARGET_NAME = UPPER(REPLACE(l_rec1.query_view, ' ', '_')) AND TARGET_COLUMN_NAME = UPPER(REPLACE(l_rec1.query_column, ' ', '_')) AND ROWNUM = 1; …

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 3 : Create Table

DU_Metadata

Solution Detail – Step 4 : Model DU_Metadata in FWM

Solution Detail – Step 4 : Model DU_Metadata in FWM

Solution Detail – Step 5 : Create Report

Prompt Page

Solution Detail – Step 5 : Create Report

Results and Re-Prompt Page

Solution Detail – Step 5 : Create Report

Demo

Solution Detail – Step 6 : Publish Report & Create Job Aid

1. Report is stored in a limited access folder

2. Report views are published to all Functional Area Professional Reports folders

3. Promote this report to top of all Professional Reports folders

4. Create job aid for users on how to do a lookup on a Banner form to identify field name. Then how to use the Metadata Tool (report) to find location of that field in ODS

Solution Detail – Step 6 : Publish Report & Create Job Aid

Solution Detail – Step 6 : Publish Report & Create Job Aid

Solution Detail – Step 6 : Publish Report & Create Job Aid

Solution Detail – Step 6 : Publish Report & Create Job Aid

Solution Detail – Step 6 : Publish Report & Create Job Aid

Solution Detail – Step 6 : Publish Report & Create Job Aid

Caveats & Disclaimers

The join between the Cognos metadata and the ODS metadata is an inner join. Of the approximate 60,000 items in the model, roughly 4000 are being excluded and written to the exception logs

A sizeable portion are custom query subjects/reporting views (Examples: Constituent Spouse_DU, Advancement Custom Calcs_DU).

Another sizeable chunk are LOV and Presentation view (level of model) query subjects.

Some are delivered views that are created within the model (not in ODS) and therefore no metadata exists. (Example: Telephone Current – in ODS 8.3)

Lastly, there is one data item (ID) from Person_Address that is on the log that is referred to as 1 thing in the database, but another in the model (ID1). I attempted to resolve by adding an ID1 to the metadata, but then it threw an error on the Health Check saying there was an item in the metadata but missing from the view. So the solution would be to rename the field in the model, but at the risk of breaking reports that may be using that field. At this time, we choose not to address this.

Caveats & Disclaimers

Therefore, we expect and accept that there may be the occasion when this tool fails to provide the data requested. We believe it to be 90-something-ish% effective. However, we have not yet faced the scenario when an End User reported that they were unable to determine the lineage of an item using this tool.

The scraping process is somewhat fragile and not guaranteed to be 100% effective.

We have only used this with ODS versions 8.3 and 8.4.1. This is a run-on-demand job. Typically we run after an upgrade. We are still left with the question of how to incorporate our

custom view metadata into this tool. We haven’t yet spent the time/resources to determine how we can stuff the metadata from those custom views into the model so that we can scrape it along with the delivered metadata. Or if that is even the right approach.

Possible Enhancements

Access Banner Table and column descriptions (They are not currently coming through in the metadata, but could be accessed directly from Banner.)

Add sort order from Query Subject in Query Studio.

Stuff lineage back into the model under ‘Description’ in the properties – use 2nd language?

Add to daily refresh

If you would like us to share more…

Contact Me.

pritchardm@denison.edu

The End

Questions?