l/o/g/o metadata business intelligence erwin moeyaert
TRANSCRIPT
L/O/G/O
MetadataBusiness IntelligenceMetadataBusiness Intelligence
Erwin Moeyaert
OverviewOverview
• What is metadata?• Why is it needed?• Types of metadata• Metadata life cycle
3
What is Metadata?What is Metadata?
•Data ‘reporting’ – WHO created the data?
– WHAT is the content of the data?
– WHEN was it created?
– WHERE is it geographically?
– HOW was the data developed?
– WHY was the data developed?
The MetadataThe Metadata
• The name suggests some high-level technological concept, but it really is fairly simple. Metadata is “data about data”.
• With the emergence of the data warehouse as a decision support structure, the metadata are considered as much a resource as the business data they describe.
• Metadata are abstractions -- they are high level data that provide concise descriptions of lower-level data.
5
What is Metadata ?What is Metadata ?
time period
author
sources
(file) size
title
supplemental information
abstract
©2005 CSC Brands, L.P. All Rights Reserved
6
What is Metadata?What is Metadata?
©2005 CSC Brands, L.P. All Rights Reserved
•entity
•attributes
The Meta DataThe Meta Data
• Last component of DW environments.
• It is information that is kept about the warehouse rather than information kept within the warehouse.
• Legacy systems generally don’t keep a record of characteristics of the data (such as what pieces of data exist and where they are located).
• Better end user data access and analysis tools can help users figure out how to get information they need out of the warehouse
• only good, easily accessible metadata can help them figure out what is available in the data warehouse and how to ask for it.
Metadata RepositoriesMetadata Repositories
Users and Developers often need a way to find information on the data they use. Information can include:• Source System(s) of the Data, contact information• Related tables or subject areas• Programs or Processes which use the data• Population rules (Update or Insert and how often)• Status of the Data Warehouse’s processing and
condition
Metadata
General Metadata Issues Associated with Data Warehouse UseGeneral Metadata Issues Associated with Data Warehouse Use
What tables, attributes, and keys does the data warehouse contain?
Where did each set of data come from? What transformation logic was applied in loading the data? How has the metadata changed over time? What aliases exist and how are they related to each other? What are the cross-references between technical and business
terms? How often does the data get reloaded? How much data is there? (assists in avoiding the submissions
of unrealistic queries)
Typical Mapping MetadataTypical Mapping Metadata
Identification of original source fields. Simple attribute-to-attribute mapping. Attribute conversions. Physical characteristic conversions. Encoding/reference table conversions. Naming changes. Key changes. Defaults values. Logic to choose from among multiple sources. Algorithmic changes.
Copyright © 1997, Enterprise Group, Ltd.
Data Warehouse Process
Source OLTPSystems Data Marts
• Design• Mapping
• Design• Mapping
• Extract• Scrub
• Transform
• Extract• Scrub
• Transform
• Load• Index
• Aggregation
• Load• Index
• Aggregation
• Replication• Data Set Distribution
• Replication• Data Set Distribution
• Access & Analysis• Resource Scheduling & Distribution
• Access & Analysis• Resource Scheduling & Distribution
Meta DataMeta Data
System MonitoringSystem Monitoring
• Raw Detail• No/Minimal History
• Integrated• Scrubbed
• History• Summaries
• Targeted• Specialized (OLAP)
Data Characteristics
DataWarehouse
Meta Data DescriptionMeta Data Description
• Information about the data warehouse system– Content– Organizational– Structural– Management Information– Scheduling Information– Contact Information– Technical Information
Why Do You Need Meta Data?Why Do You Need Meta Data?
• Share resources– Users– Tools
• Document system• Without metadata
– Not Sustainable– Not able to fully utilize resource
Metadata Life CycleMetadata Life Cycle• Collection - Identify metadata and capture into
repository; automate
• Maintenance - Put in place processes to synchronize metadata automatically with changing data architecture; automate
• Deployment - Provide metadata to users in the right form and with the right tools; match metadata offered to specific needs of each audience
Metadata CollectionMetadata Collection• Right metadata at the right time• Variety of collection strategies• Sources
– potential sources of data for DW– external data– data structures
• Data Models - enterprise data model start point– import from CASE tool– correlate enterprise and warehouse models
Metadata CollectionMetadata Collection• Warehouse mappings
– map operational data into warehouse data structure– Need record of logical connection used for mapping
and transformation• Warehouse usage information
– After roll out– What tables accessed, by whom and for what– What queries written– Capture nature of business problem or query
Maintaining MetadataMaintaining Metadata
• Up to date with reality• Capture incremental changes
Metadata DeploymentMetadata Deployment• Warehouse developers need:
– physical structure info for data sources– enterprise data model– warehouse data model– concerned with accuracy, completeness and flexibility of
metadata– Need access to comprehensive impact analysis capabilities– Need to defend against accuracy & integrity questions
Meta DataMeta Data
• Types – Technical– Business / User
• Levels– Core– Basic– Deluxe
Core Technical Meta DataCore Technical Meta Data
• Source• Target• Algorithm
Basic Technical Meta DataBasic Technical Meta Data• History of transformation changes • Business rules • Source program / system name• Source program author / owner• Extract program name & version• Extract program author / owner• Extract JCL / Script name• Extract JCL / Script author / owner• Load JCL / Script name
Basic Technical Meta Data (con’t)Basic Technical Meta Data (con’t)• Load JCL / Script author / owner• Load frequency• Extract dependencies• Transformation dependencies• Load dependencies• Load completion date / time stamp• Load completion record count• Load status
Deluxe Technical Meta DataDeluxe Technical Meta Data
• Source system platform• Source system network address• Source system support contact• Source system support phone / beeper• Target system platform• Target system network address• Target system support contact• Target system support phone / beeper• Etc.
Core Business Meta DataCore Business Meta Data
• Field / object description• Confidence level• Frequency of update
Basic Business Meta DataBasic Business Meta Data
• Source system name• Valid entries (i.e. “There are three valid codes: A, B,
C”)• Formats (i.e. Contract Date: 82/4/30)• Business rules used to calculate or derive
the data• Changes in business rules over time
Deluxe Business Meta DataDeluxe Business Meta Data
• Data owner• Data owner contact information• Typical uses• Level of summarization• Related fields / objects• Existing queries / reports using this field /
object• Estimated size (tables / objects)
Amount of Meta DataAmount of Meta Data
• How much Meta Data do I need?• As much as you can support!
Meta Data Functions - TechnicalMeta Data Functions - Technical• Maintenance• Troubleshooting• Documentation• Logging / Metrics
Meta Data LocationMeta Data Location
• DB Resident– Almost always relational– C/S predominantly– Normalized design– OODB is popular option for proprietary
solutions
RepositoryRepository
• Specialized databases designed to maintain metadata, together with tools and interfaces that allow a company to collect and distribute its metadata
• Repository Requirements– Logically Common– Open– Extensible
Meta Data ProcessMeta Data Process
Copyright © 1997, Enterprise Group, Ltd.
• Integrated with entire process and data flow– Populated from beginning to end
– Begin population at design phase of project– Dedicated resources throughout
• Build• Maintain
• Design• Mapping
• Design• Mapping
• Extract• Scrub
• Transform
• Extract• Scrub
• Transform
• Load• Index
• Aggregation
• Load• Index
• Aggregation
• Replication• Data Set Distribution
• Replication• Data Set Distribution
• Access & Analysis• Resource Scheduling & Distribution
• Access & Analysis• Resource Scheduling & Distribution
Meta DataMeta Data
System MonitoringSystem Monitoring
General Metadata IssuesGeneral Metadata Issues
General metadata issues associated with Data Warehouse use:– What tables, attributes and keys does the DW
contain?– Where did each set of data come from?– What transformations were applied with cleansing?– How have the metadata changed over time?– How often do the data get reloaded?– Are there so many data elements that you need to be
careful what you ask for?
Components of the MetadataComponents of the Metadata
• Transformation maps – records that show what transformations were applied
• Extraction & relationship history – records that show what data was analyzed
• Algorithms for summarization – methods available for aggregating and summarizing
• Data ownership – records that show origin• Patterns of access – records that show
what data are accessed and how often
Typical Mapping MetadataTypical Mapping Metadata
Transformation mapping records include:– Identification of original source– Attribute conversions– Physical characteristic conversions– Encoding/reference table conversions– Naming changes– Key changes– Values of default attributes– Logic to choose from multiple sources– Algorithmic changes
32
MetadataMetadata
• The structure of metadata will differ between each process, because the purpose is different.
• This means that multiple copies of metadata describing the same data item are held within the data warehouse.
• Most vendor tools for copy management and end-user data access use their own versions of metadata.
33
MetadataMetadata
• Copy management tools use metadata to understand the mapping rules to apply in order to convert the source data into a common form.
• End-user access tools use metadata to understand how to build a query.
• The management of metadata within the data warehouse is a very complex task that should not be underestimated.
METADATA VIEWSMETADATA VIEWS
• BUSINESS USER’S VIEWFROM A BUSINESS USER’S VIEW, METADATA SHOULD CONTAIN THE FOLLOWING SIX ELEMENTS:
1. TABLE OF CONTENTS2. ORIGIN OF THE DATA FOR THE WAREHOUSE3. TRANSFORMATION SEQUENCE4. ACCESS LEVEL5. TIMELINE OF THE JOURNEY6. ACCESS ESTIMATES
METADATA VIEWSMETADATA VIEWS
• DSS (DECISION SUPPORT SYSTEM) DEVELOPER’S VIEW
1. TRANSFORMATION AND BUSINESS RULES2. DATA MODELS3. AVAILABLE OPERATION DATA
METADATA VIEWSMETADATA VIEWS
• CORPORATE VIEW
METADATA IS A LOGICAL COLLECTION OF METADATA FROM VARIOUS SOURCES, INCLUDING THE FOLLOWING SIX PLACES:
METADATA VIEWSMETADATA VIEWS
1. LEGACY SYSTEM METADATA CONSISTING OF A DATA DICTIONARY CONTAINING
INFORMATION ABOUT PROGRAM LIBRARIES, DATABASE CATALOGS AND FILE LAYOUTS.
2. OPERATIONAL CLIENT/SERVER SYSTEMS – CONSISTING OF DISTRIBUTED SOFTWARE COMPONENTS FROM A VARIETY OF VENDORS.
3. ENTERPRISE MODELS –THEY ARE THE FIRST STAGE IN THE ULTIMATE GOAL OF BUILDING CORPORATE METADATA.