turning bad habits into good practicesnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfturning bad...

54
© 2005 by Michael H. Brackett D RD R P.O. Box 151 Lilliwaup, Washington 98555 Voice 360 877 5507 Fax None [email protected] members.aol.com/mhbrackett DATA RESOURCE QUALITY TURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005 NEW ORLEANS MICHAEL H. BRACKETT CONSULTING DATA ARCHITECT DATA RESOURCE DESIGN & REMODELING

Upload: others

Post on 09-Jul-2020

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DRDRP.O. Box 151Lilliwaup, Washington 98555Voice 360 877 5507Fax [email protected]/mhbrackett

DATA RESOURCE QUALITY

TURNING BAD HABITSINTO GOOD PRACTICES

NATIONAL FORUM ON EDUCATION STATISTICSFEBRUARY 21, 2005

NEW ORLEANS

MICHAEL H. BRACKETT

CONSULTING DATA ARCHITECTDATA RESOURCE DESIGN & REMODELING

Page 2: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CONTENTS

CURRENT SITUATION

DATA RESOURCE CONCEPTS

HALTING DATA DISPARITY

RESOLVING DATA DISPARITY

CULTURAL CONSIDERATIONS

CONCLUSION

PRESENTER’S BIBLIOGRAPHY

Page 3: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

BUSINESS INFORMATION DEMANDAN ORGANIZATION’S CONTINUOUSLY INCREASING, CONSTANTLY CHANGING NEEDFOR CURRENT, ACCURATE, INTEGRATED INFORMATION, OFTEN ON SHORT NOTICE OR VERY SHORT NOTICE, TO SUPPORT ITS BUSINESS ACTIVITIES

CURRENT BUSINESS IMPACTTHIS BUSINESS INFORMATION DEMAND IS NOT BEING METPUBLIC & PRIVATE SECTOR BUSINESS IS IMPACTEDBUSINESS DECISIONS ARE LESS-INFORMEDCITIZENS & CUSTOMERS ARE IMPACTEDOPPORTUNITIES ARE MISSED

PROBLEMHIGH QUANTITIES OF DISPARATE DATA IN MOST ORGANIZATIONSDATA NOT THOROUGHLY UNDERSTOODDATA RESOURCE NOT FORMALLY MANAGEDDATA NOT READILY AVAILABLEDATA QUALITY LOW

BUSINESS INFORMATION DEMAND NOT BEING MET!

Page 4: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

DISPARATE DATA - A TRUISMDATA THAT ARE ESSENTIALLY NOT ALIKE, OR ARE DISTINCTLY DIFFERENT IN KIND, QUALITY, OR CHARACTER. THEY ARE UNEQUAL AND CANNOT BE READILY INTEGRATED TO ADEQUATELY MEET THE BUSINESS INFORMATION DEMAND.

BASIC DATA PROBLEMUNKNOWN DATA EXISTANCE

ORGANIZATION NOT AWARE OF ALL DATA AT ITS DISPOSALUSUALLY NOT EVEN INVENTORIED

UNKNOWN DATA MEANINGCONTENT AND MEANING OF DATA NOT FULLY KNOWNDATA NOT THOROUGHLY UNDERSTOOD

HIGH DATA REDUNDANCYDATA HIGHLY REDUNDANT AND INCONSISTENTAVERAGE REDUNDANCY FACTOR OF 10 FOR LARGE ORGANIZATIONS

HIGH DATA VARIABILITYDATA HIGHLY VARIABLE IN FORMAT AND CONTENTAVERAGE FACTOR OF 15 TO 20 FOR LARGE ORGANIZATIONS

THAT’S THE GOOD NEWS!

Page 5: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

DISPARATE DATA RESOURCEA DATA RESOURCE THAT IS SUBSTANTIALLY COMPOSED OF DISPARATE DATA THAT ARE DIS-INTEGRATED AND NOT SUBJECT ORIENTED. A STATE OF DISARRAY WHERE THE LOW QUALITY DOES NO, AND CANNOT, ADEQUATELY SUPPORT THE BUSINESS INFORMATION DEMAND.

DISPARATE DATA CYCLEA SELF-PERPETUATING CYCLE WHERE DISPARATE DATA CONTINUE TO BE PRODUCED AT AN EVER-INCREASING RATE BECAUSE PEOPLE DO NOT KNOW ABOUT EXISTING DATA OR DO NOT WANT TO USE EXISTING DATA.

THAT’S THE BAD NEWS!

Can't find dataDon't trust data

Can't access data

People uncertain about the data

People create their own data

Data not integratedData not

documented

Page 6: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

DISPARATE DATA SPIRALDISPARATE DATA INCREASING BY SEVERAL ORDERS OF MAGNITUDECREATING DISPARATE DATA FASTER THAN EVER BEFORESPREADING TO NON-TABULAR DATA AND DATA RESOURCE DATADISPARITY SPIRALING OUT OF CONTROLNO FORESEEABLE END IN SIGHT!

DATA RESOURCE DRIFTTHE NATURAL, STEADY DRIFT OF A DATA RESOURCE TOWARDS DISPARITY IF ITS DEVELOPMENT IS NOT PROPERLY MANAGED AND CONTROLLED.

THAT’S THE REALLY BAD NEWS!

Low QualityData Resource

High QualityData Resource

Data Resource

Diff

icul

t

Easiest

Natural Drift

Page 7: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

DATA DILEMMAORGANIZATIONS FACING REAL DILEMMA ABOUT THEIR DATA RESOURCEHIGH DEMAND FOR INTEGRATED DATA TO SUPPORT BUSINESS NEEDSCONTINUED RAPID PRODUCTION OF DISPARATE DATA

NO STATUS QUOTHERE IS NO STATUS QUO FOR DEVELOPING A HIGH-QUALITY DATA RESOURCENATURAL DRIFT TOWARD DISPARITY WILL CONTINUESITUATION WILL GET WORSE

A STATUS QUO LEADS TOORGANIZATIONAL FAILURE BY INFORMATION DEPRIVATION!

Page 8: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CURRENT SITUATION

MUST CONSCIOUSLY ALTER THE NATURAL DRIFTTOWARD HIGH-QUALITY DATA RESOURCEBREAK DISPARATE DATA CYCLESTOP THE SPIRALING DISPARITYCREATE A HIGH-QUALITY SHARABLE DATA RESOURCE

SILVER BULLETS JUST DO NOT EXISTOBJECTIVE: AN ATTEMPT TO ACHIEVE SOME GAIN WITHOUT ANY PAINRESULT: ENDURING CONSIDERABLE PAIN WITH MINIMAL GAIN

THERE ARE NO SILVER BULLETS!

Page 9: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

BUSINESS INTELLIGENCE VALUE CHAIN

I-ORGANIZATIONINTELLIGENT LEARNING ORGANIZATION - HUMAN RESOURCE REALMHUMAN RESOURCE REALM LINKS TECHNOLOGY REALM & BUSINESS REALMREQUIRES GOOD INFORMATION TO SUPPORT BUSINESS STRATEGIES & GOALSREQUIRES A HIGH-QUALITY SHARABLE DATA RESOURCE

MUST BEGIN WITH THE DATA RESOURCE!

DATA RESOURCEINFORMATION

KNOWLEDGE ENVIRONMENT

BUSINESS GOALS

BUSINESS INTELLIGENCEBUSINESS STRATEGIES

Page 10: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

MANY PEOPLE STILL CONFUSE ‘DATA’ - ‘INFORMATION’ - ‘KNOWLEDGE’DATA

INDIVIDUAL FACTS OUT OF CONTEXT WITH LITTLE MEANING123.45

DATA IN CONTEXTINDIVIDUAL FACTS WITH MEANINGACCOUNT BALANCE AT NOON ON JANUARY 12, 2001

INFORMATIONSET OF DATA IN CONTEXTRELEVANT TO ONE OR MORE PEOPLEAT A POINT IN TIME OR FOR A PERIOD OF TIME

KNOWLEDGEINFORMATION RETAINED BY INDIVIDUALSCOMBINED WITH EXPERIENCE

DATA ARE THE FOUNDATION!

Page 11: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

QUALITY DEFINEDDATA RESOURCE QUALITYA MEASURE OF HOW WELL THE DATA RESOURCE SUPPORTS THE CURRENT AND THE FUTURE BUSINESS INFORMATION DEMAND

DATA QUALITYA SUBSET OF DATA RESOURCE QUALITY DEALING WITH DATA VALUES

ULTIMATE DATA RESOURCE QUALITYA DATA RESOURCE THAT IS STABLE ACROSS CHANGING BUSINESS AND CHANGING TECHNOLOGY SO IT CONTINUES TO SUPPORT THE CURRENT AND FUTURE BUSINESS INFORMATION DEMAND

INFORMATION QUALITYA MEASURE OF THE ABILITY TO GET THE RIGHT DATA, TO THE RIGHT PEOPLE, IN THE RIGHT PLACE, AT THE RIGHT TIME, IN THE RIGHT FORM, AT THE RIGHT COST, SO THEY CAN MAKE THE RIGHT DECISIONS, AND TAKE THE RIGHT ACTIONS.

INTELLIGENT LEARNING ORGANIZATION NEEDS A HIGH-QUALITY STABLE DATA RESOURCE!

Page 12: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

COMPARATE DATA VISIONCOMPARATE DATADATA THAT ARE ALIKE, SIMILAR IN KIND, QUALITY, AND CHARACTER, ARE EASILY UNDERSTOOD, AND CAN BE READILY INTEGRATED

COMPARATE DATA RESOURCESUBJECT ORIENTED, INTEGRATED, HIGH QUALITY, SHARABLE DATA RESOURCE THAT FULLY SUPPORTS THE CURRENT AND THE FUTURE BUSINESS INFORMATION DEMAND

ComparateData

Resource

DataResource

Guide

InformationSystem

BusinessInformation

Demand

DisparateData Resource

COMMON DATA ARCHITECTURE

Page 13: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

COMPARATE DATA CYCLEA SELF-PERPETUATING CYCLE WHERE THE USE OF COMPARTE DATA IS CONTINUALLYREINFORCED BECAUSE PEOPLE UNDERSTAND AND TRUST THE DATA.

RESOLVE THE DATA DILEMMABRING STABILITY TO THE DATA RESOURCE TO TRANSCEND CHANGE

STOP THE NATURAL DRIFT OF A DATA RESOURCEBY HALTING THE DISPARATE DATA CYCLE

AND STARTING A COMPARATE DATA CYCLE!

People find, trust, and access data

Existing data resource readily

shared

New data created when necessary

New data integrated and documented

Page 14: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

COMPARATE DATA RESOURCESUBJECT ORIENTED

BASED ON BUSINESS OBJECTS AND EVENTS IN THE REAL WORLDDATA SUBJECTS REPRESENT BUSINESS OBJECTS AND EVENTSDATA RESOURCE STRUCTURED BY DATA SUBJECTSALL CHARACTERISTICS ABOUT A DATA SUBJECT STORED WITH THE DATA SUBJECT

INTEGRATEDFULLY INTEGRATED WITHIN A COMMON DATA ARCHITECTUREONE ARCHITECTURE FOR THE ENTIRE DATA RESOURCEALL DATA MANAGED WITHIN THAT SINGLE ARCHITECTUREDATA PROPERLY DEPLOYED TO MEET BUSINESS NEEDS

SUPPORTED BY DATA RESOURCE GUIDECOMPREHENSIVE INDEX TO DATA RESOURCEA COMPLETE INTEGRATED UNDERSTANDING ABOUT THE DATA RESOURCE

INTEGRATEDACROSS ORGANIZATION UNITSACROSS BUSINESS ACTIVITIES

AN INTEGRATED DATA RESOURCE IS ONE VERSION OF TRUTH ABOUT THE BUSINESS!

Page 15: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

DATA ARCHITECTURE DEFINITION 1THE METHOD OF DESIGN AND CONSTRUCTION OF A DATA RESOURCE THAT ISBUSINESS DRIVEN,BASED ON REAL-WORLD SUBJECTS PERCEIVED BY THE ENTERPRISE,AND IMPLEMENTED INTO APPROPRIATE OPERATING ENVIRONMENTS.IT CONSISTS OF COMPONENTS THATPROVIDE A CONSISTENT FOUNDATION ACROSS ORGANIZATIONAL BOUNDARIESTO PROVIDE EASILY IDENTIFIABLE, READILY AVAILABLE HIGH-QUALITY DATATO SUPPORT THE BUSINESS INFORMATION DEMAND

DATA ARCHITECTURE DEFINITION 2THE COMPONENT OF THE DATA RESOURCE FRAMEWORK THATCONTAINS ALL THE ACTIVITIES, AND THE PRODUCTS OF THOSE ACTIVITIES, RELATED TO THE IDENTIFICATION, NAMING, DEFINITION, STRUCTURING, INTEGRITYACCURACY, EFFECTIVENESS, AND DOCUMENTATION OF THE DATA RESOURCE

DATA INTEGRATION REQUIRES A FORMAL DEFINITIONOF DATA ARCHTIECTURE!

Page 16: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

COMMON DATA ARCHITECTURECOMMON CONTEXT FOR

INVENTORYING ALL DATAUNDERSTANDING THE CONTENT AND MEANING OF DATAIMPROVING DATA QUALITYTRANSFORMING AND INTEGRATING DISPARATE DATADEFINING NEW DATAMANAGING DYNAMIC DATA DEPLOYMENT

EINSTEIN’S PRINCIPLEA PROBLEM CANNOT BE SOLVED WITH THE SAME LEVEL OF TECHNOLOGY USED TO CREATE THAT PROBLEMA HIGHER LEVEL OF TECHNOLOGY IS NEEDEDTHE COMMON DATA ARCHITECTURE IS THE HIGHER LEVEL OF TECHNOLOGY FOR UNDERSTANDING AND MANAGING DATA

DEVELOP CUSTOMIZED METHODONE SIZE FITS ALL METHODS DO NOT WORKCOMMON DATA ARCHITECTURE HAS CONCEPTS - PRINCIPLES - TECHNIQUESORGANIZATION HAS PROBLEMS - NEEDS - ENVIRONMENTCUSTOMIZE A METHOD TO SUIT ORGANIZATION

CUSTOMIZED METHOD HELPS RESOLVE THE DATA DILEMMA!

Page 17: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

ENTERPRISE DATA ARCHITECTUREMANY DIFFERENT DEFINITIONSCONFLICTING AND CONFUSING

AN ENTERPRISE DATA ARCHITECTURE IS A SUBSET OF THE COMMON DATA ARCHITECTURE THAT REPRESENTS THE COMPARATE DATA RESOURCE.

IT REPRESENTS THE DESIRED ENTERPRISE-WIDE DATA ARCHITECTURE FOR AN ORGANIZATION.

IT IS THE PATTERN FOR BUILDING THE COMPARATE DATA RESOURCE FROM THE DISPARATE DATA

THE ENTERPRISE DATA ARCHITECTURE REPERESENTSTHE COMPARATE DATA RESOURCE!

COMMON DATA ARCHITECTURE

Disparate Data Resource Comparate Data Resource

Enterprise Data Architecture

Page 18: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE CONCEPTS

ACHIEVING AN INTEGRATED DATA RESOURCEPHASE 1 - HALT THE CREATION OF DISPARATE DATA

ESTABLISH A COMMON DATA ARCHITECTUREDEVELOP NEW DATA WITHIN THAT COMMON DATA ARCHITECTUREPROACTIVE PHASEPREVENTS FURTHER DATA DISPARITYMUST BE DONE FIRST

PHASE 2 - RESOLVE THE EXISTING DATA DISPARITYTRANSITION DISPARATE DATA TO A COMPARATE DATA RESOURCEFORMAL DATA TRANSFORMATION WITHIN A COMMON DATA ARCHITECTUREREACTIVE PHASERESOLVES DISPARATE DATAMUST BE DONE AFTER PHASE 1

DATA INTEGRATION BEGINS BYSTOPPING FURTHER DATA DISPARITY

AND THEN RESOLVING EXISTING DISPARITY!

Page 19: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA NAMES

FORMAL DATA NAMESDATA NAMES THAT READILY AND UNIQUELY IDENTIFY A FACT OR GROUP OF FACTSIN THE DATA RESOURCE. THEY ARE DEVELOPED WITHIN A FORMAL DATA NAMING TAXONOMY AND ARE ABBREVIATED WHEN NECESSARY WITH A FORMAL SET OFABBREVIATIONS AND A FORMAL ABBREVIATION ALGORITHM.

GOOD PRACTICESFORMAL DATA NAMING TAXONOMY

MAKES DATA NAMING EQUIVALENT TO ANIMALS, PLANTS, MINERALS, CHEMICALS

DATA NAMING VOCABULARYEXPAND CLASS WORD CONCEPT TO ALL COMPONENTS OF THE DATA NAMING TAXONOMYSET OF COMMON WORDS FOR EACH COMPONENT OF DATA NAMING TAXONOMYPROVIDE COMMON MEANING ACROSS DATA NAMES

PRIMARY DATA NAMETHE FORMAL DATA NAME THAT IS FULLY SPELLED OUT, REAL WORLD, UNABBREVIATED,UNTRUNCATED, BUSINESS NAME OF THE DATA THAT HAS NO SPECIAL CHARACTERSOR LENGTH LIMITATIONS.

MUST BE UNIQUE AND MEANINGFUL TO THE BUSINESSALL OTHER DATA NAMES ARE ALIASES OF THE PRIMARY DATA NAME

Page 20: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA NAMES

DATA NAME WORD ABBREVIATIONSPRIMARY DATA NAMES MUST BE FORMALLY ABBREVIATED TO MEET LENGTH RESTRICTIONSDATA NAME WORD ABBREVIATIONS

DATA NAME ABBREVIATION ALGORITHMA FORMAL PROCEDURE FOR ABBREVIATING PRIMARY DATA NAMES USING ABBREVIATIONS VARIETY OF DIFFERENT ALGORITHMSCAN HAVE MORE THAN ONE ABBREVIATION ALGORITHMTHE NUMBER OF ALGORITHMS IS NOT IMPORTANT - THE FORMALITY IS

Page 21: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA DEFINITIONS

COMPREHENSIVE DATA DEFINITIONSFORMAL DATA DEFINITIONS THAT PROVIDE A COMPLETE, MEANINGFUL, EASILY READ,READILY UNDERSTOOD DEFINITION THAT THOROUGHLY EXPLAINS THE CONTENT ANDMEANING OF THE DATA.

GOOD PRACTICESMEANINGFUL DATA DEFINITIONS

CONTENT AND MEANING WITH RESPECT TO THE BUSINESSNOT WHERE DATA STORED, HOW DATA USED, OR WHERE DATA CAPTUREDSIMILARITIES AND DIFFERENCES WITH CLOSELY RELATED DATAHISTORY IF PERTINENT TO THE MEANING - NOT AN AUDIT TRAIL OF DEFINITIONSWHAT IS AND IS NOT INCLUDEDINCLUDE BUSINESS CLIENTS

THOROUGH DATA DEFINITIONSNO LENGTH LIMITATIONSAPPROPRIATE BUSINESS TERMINOLOGYSTRONG DENOTATIVE MEANING - LIMIT CONNOTATIVE MEANING1 TO3 PARAGRAPHS OF 2 TO 3 SENTENCES EACH

CORRECT DATA DEFINITIONSCURRENT WITH THE BUSINESSROUTINELY REVIEWED AND ENHANCEDEVOLVE AS THE BUSINESS EVOLVES

Page 22: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA STRUCTURES

PROPER DATA STRUCTURESA DATA STRUCTURE THAT PROVIDES A SUITABLE REPRESENTATION OF THE BUSINESS,AND DATA SUPPORTING THE BUINESS, THAT IS RELEVANT TO THE INTENDED AUDIENCE.

GOOD PRACTICESDATA STRUCTURE COMPONENTS

ENTITY - RELATION DIAGRAMSHOWS THE DATA SUBJECTS AND THE RELATIONS BETWEEN THEMLIKE THE FLOOR PLAN OF A HOUSEDOES NOT SHOW DETAIL WITHIN EACH DATA SUBJECTSYMBOLS FOLLOW SEMIOTIC THEORY

DATA ATTRIBUTE STRUCTURESHOWS DATA ATTRIBUTES IN A DATA SUBJECTSHOWS ROLES DATA ATTRIBUTES PLAY

Page 23: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA STRUCTURES

PRINCIPLESTECHNICALLY CORRECT & CULTURALLY ACCEPTABLE

TECHNICALLY CORRECT IS LIKE BUILDING CODESCULTURALLY ACCEPTABLE IS LIKE AESTHETICS

STRUCTURALLY STABLE & BUSINESS FLEXIBLEARCHITECTURE REMAINS STABLE AS BUSINESS CHANGESSUPPORTS BUSINESS PROCESS IMPROVEMENTENCOURAGES BUSINESS IMPROVEMENT

TURNING POINT BETWEEN PREDOMINANTLY DISPARATE ANDPREDOMINANTLY COMPARATE DATA RESOURCE!

Page 24: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA INTEGRITY RULES

PRECISE DATA INTEGRITY RULESA DATA INTEGRITY RULE THAT PRECISELY SPECIFIES THE CRITERIA FORHIGH-QUALITY DATA VALUES AND REDUCES OR ELIMINATES DATA ERRORS.

GOOD PRACTICESDATA RULE CONCEPT

SUBSET OF BUSINESS RULES ACCORDING TO THE ZACHMAN FRAMEWORKDEALS WITH THE DATA COLUMN OF THE FRAMEWORKALLOWS PRECISE SPECIFICATION FOR QUALITY IMPROVEMENT

A DATA DEFINITION IS NOT A DATA RULEA DEFINITION IS, BY DEFINITION, A DEFINITION

PROBLEM MAKING DEFINITION A RULEIMPACTS A COMPREHENSIVE DEFINITIONSLEADS TO ONE- LINE PHRASES WITH ADDITIONAL EXPLANATIONADDITIONAL EXPLANATION OFTEN NOT PROVIDEDREVERSION TO LEGACY DATA DEFINITIONS

A DATA DEFINITION IS NOT A DATA RULE!

Page 25: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA INTEGRITY RULES

DATA INTEGRITY RULE NAMESFORMAL NAME ACCORDING TO DATA NAMING TAXONOMY & VOCABULARYUSES THE ! NOTATION CUSTOMER. NAME CHANGE!REQUIRES NORMALIZATION OF DATA RULES

DATA RULE IS NORMALIZEDTO THE DATA RESOURCE COMPONENT IT REPRESENTSOR ON WHICH IT TAKES ACTION

DATA INTEGRITY RULE NOTATIONACCEPTABLE AND UNDERSTANDABLE TO BUSINESS CLIENTSBASED ON MATHEMATICAL AND LOGIC NOTATION WHERE PRACTICALNOTATION READILY AVAILABLE ON STANDARD KEYBOARD

DATA INTEGRITY RULE TYPESDATA VALUECONDITIONAL DATA VALUEDATA STRUCTURECONDITIONAL DATA STRUCTUREDATE DERIVATIONDATE RETENTION

Page 26: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA INTEGRITY RULES

DATA INTEGRITY RULE ENFORCEMENTONE SET OF DATA INTEGRITY RULESAPPLY TO ALL DATA ENTERING DATA RESOURCEAPPLY AS CLOSE TO CAPTURE AS POSSIBLEVIOLATIONS ACTIONS DEFINEDVIOLATION NOTIFICATION NOTIFICATION DEFINEDDEFAULT VALUES SPECIFIED

PROACTIVE DATA QUALITY MANAGEMENT - SELF CORRECTING PROCESSRAPID DATA ERROR IDENTIFICATIONAPPLY TO ENTIRE DATA RESOURCEDEFAULT VALUES DOCUMENTED WHEN INSERTED

Page 27: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA DOCUMENTATION

ROBUST DATA DOCUMENTATIONDOCUMENTATION ABOUT THE DATA RESOURCE THAT IS COMPLETE, CORRECT, CURRENT, UNDERSTANDABLE, NON-REDUNDANT, READILY AVAILABLE, AND KNOWN TO EXIST.

GOOD PRACTICESDATA RESOURCE DATA CONCEPT

THE TERM META-DATA MISUSED AND ABUSED TO THE POINT THAT ITS MEANING IS UNCLEAR

META-DATA FIASCO UNFOLDINGMETA-META-DATA, META-META-META-DATA, ETC.

DATA RESOURCE DATA IS A MUCH BETTER TERMANY DATA NECESSARY FOR THOROUGHLY UNDERSTANDING, FORMALLY MANAGING, AND FULLY UTILIZING THE DATA RESOURCE TO SUPPORT THE BUSINESS INFORMATION DEMAND

DESIGNED AND MANAGED THE SAME AS ANY OTHER DATA - FINANCIAL DATA, CUSTOMER DATA, JUSTICE DATA, HUMAN RESOURCE DATA

SEGMENT OF THE ORGANIZATION’S DATA RESOURCE THE SAME AS ANY OTHER DATA

PART OF THE COMMON DATA ARCHITECTURE

DATA RESOURCE DATA DOCUMENT THE ENTIRE DATA RESOURCE!

Page 28: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA DOCUMENTATION

DATA RESOURCE DATA ASPECTSSEMANTIC DATA RESOURCE DATABUSINESS ASPECTTHE MEANING OF THE DATAHELP PEOPLE UNDERSTAND AND USE THE DATA RESOURCERELATIVELY DIFFICULT TO CAPTURE AND MAINTAIN

TECHNICAL DATA RESOURCE DATAPHYSICAL ASPECTPARAMETERS ABOUT THE DATA STORAGE AND DATA MANIPULATIONDATA ABOUT BUILDING, MANAGING, AND MAINTAINING DATABASESRELATIVELY EASY TO CAPTURE AND MAINTAIN

DATA RESOURCE DATA MUST INCLUDE BOTH SEMANTIC DATA AND TECHNICAL DATA!

Page 29: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA DOCUMENTATION

COMPLETE DATA DOCUMENTATIONINCLUDE SEMANTIC AND TECHNICAL DATA RESOURCE DATAINCLUDE ENTIRE SCOPE OF THE DATA RESOURCEINCLUDE ALL DATA RESOURCE COMPONENTS

CURRENT DATA DOCUMENTATIONDOCUMENTATION KEPT CURRENT WITH THE BUSINESSADEQUATELY REPRESENT HISTORICAL CHANGES IN THE BUSIENSS / DATA RESOURCE

UNDERSTANDABLE DATA DOCUMENTATIONUNDERSTANDABLE TO ALL AUDIENCESCORRECT DETAIL FOR INTENDED AUDIENCEAPPROPRIATE PRESENTATION

NON-REDUNDANT DATA DOCUMENTATIONONE SINGLE VERSION OF THE TRUTH FOR THE DATA RESOURCEDEVELOP WITHIN A COMMON DATA ARCHITECTURESAME CONCEPTS, PRINCIPLES, TECHNIQUES FOR DEVELOPING BUSINESS DATA

READILY AVAILABLEREADILY AVAILABLE TO ALL AUDIENCES - BUSINESS & TECHNICALEASILY ACCESSIBLE THROUGH BUSINESS APPLICATIONS

Page 30: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA DOCUMENTATION

KNOWN TO EXISTALL AUDIENCES MUST KNOW DATA DOCUMENTATION EXISTSCONSTANTLY MAKE PEOPLE AWARE THAT DATA DOCUMENTATION EXISTS

ANCILLARY DATA DOCUMENTATIONDATA SUBJECT THESAURUS - BUSINESS TERMS POINT TO DATA SUBJECTKEYWORD IN CONTEXT & KEY TERM OUT OF CONTEXT

BUSINESS GLOSSARYLIST OF TERMS AND ABBREVIATIONS USED IN THE BUSINESSBUSINSS TERMS AND ABBREVIATIONS OFTEN AS DISPARATE AS THE DATAHELPS PEOPLE UNDERSTAND THE BUSINESS AND FOCUS ON PREFERRED TERMS

Page 31: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA ORIENTATION

REASONABLE DATA ORIENTATIONA DATA ORIENTATION THAT IS PRIMARILY TOWARD THE BUSINESS AND SUPPORT OFBOTH THE CURRENT AND FUTURE BUSINESS INFORMATION DEMAND.

GOOD PRACTICESBUSINESS SUBJECT ORIENTATION

PRIMARY ORIENTATION TOWARD BUSINESS SUBJECTSBUSINESS OBJECTS AND BUSINESS EVENTS IN REAL WORLD

BUSINESS CLIENT ORIENTATIONDIRECT BUSINESS CLIENT INVOLVEMENT FOR BUSINESS KNOWLEDGECANNOT EXCLUDE BUSINESS KNOWLEDGECANNOT HAVE BUSINESS CLIENTS DRIVE DATABASE DESIGNFIND RIGHT MIX OF BUSINESS KNOWLEDGE AND TECHNICAL SKILLS

COOPERATIVE DEVELOPMENT OF A HIGH-QUALITY DATA RESOURCE!

FIVE-TIER CONCEPTBUILDS ON THE THREE-TIER CONCEPTADDS AN ANALYTICAL TIER AND A PREDICTIVE TIER

Page 32: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA ORIENTATION

DATA NORMALIZATIONEXPANDED PERSPECTIVEBRINGING DATA INTO NORMAL FORM FOR THE INTENDED PURPOSEOPERATIONAL DATA NORMALIZATIONANALYTICAL DATA NORMALIZATIONPREDICTIVE DATA NORMALIZATION

SINGLE ARCHITECTURE ORIENTATIONONE INTEGRATED ORGANIZATION-WIDE DATA ARCHITECTUREONE ARCHITECTURE FOR ALL FIVE TIERS AND ALL FIVE SCHEMACOMMON DATA ARCHITECTURE CONCEPT

SINGLE FACT ORIENTATIONSINGLE FACT DATA ATTRIBUTEBIRTH DATE & REASON FOR NO BIRTH DATE

SINGLE VALUE DATA ATTRIBUTESINGLE NAME PER DATA ATTRIBUTE

DATA ATTRIBUTE NORMALIZATIONSINGLE FACT OR GROUP OF CLOSELY RELATED FACTSSINGLE VALUE

Page 33: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA AVAILABILITY

ACCEPTABLE DATA AVAILABILITYENSURE THAT THE DATA ARE AVAILABLE TO MEET THE BUSINESS INFORMATION DEMANDWHILE PROPERLY PROTECTING AND SECURING THOSE DATA.

GOOD PRACTICESADEQUATE DATA ACCESSIBILITY

BALANCE BETWEEN TOO SECURE OR NOT SECURE ENOUGHPROTECTION FROM UNAUTHORIZED ACCESS, ALTERATION, OR DESTRUCTION

ADEQUATE DATA PROTECTIONREASONABLE PROTECTION AGAINST REASONABLE FAILURESREGULAR BACKUPS DEPENDING ON CRITICAL NATURE OF DATAPROCEDURES TO CONTINUE BUSINESS DURING FAILURE

ADEQUATE DATA RECOVERYRECOVER FROM FAILURE AS QUICKLY AND COMPLETELY AS POSSIBLEPROCEDURES TO BRING DATA RESOURCE CURRENT WITH THE BUSINESS

PROTECTED PRIVACY AND CONFIDENTIALITYPROTECT PERSON / ORGANIZATION’S RIGHT TO PRIVACY / CONFIDENTIALITYCONSTANTLY TRACK CURRENT LAWS AND REGULATIONS

APPROPRIATE DATA USEONGOING REVIEW OF THE USE OF DATA

Page 34: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESPONSIBILITY

ADEQUATE DATA RESPONSIBILITYDEFINE FORMAL RESPONSIBILITIES FOR MANAGING A SHARED DATA RESOURCE.

GOOD PRACTICESAUTHORIZED DATA STEWARDSHIP

DATA STEWARD - A PERSON WHO WATCHES OVER THE DATA AND ISRESPONSIBLE FOR THE WELFARE OF THE DATA RESOURCE AND ITS SUPPORTTO THE BUSINESS INFORMATION DEMAND

STRATEGIC LEVEL (EXECUTIVE LEVEL)LEGAL / FINANCIAL RESPONSIBILITY FOR A MAJOR SEGMENT OF THE DATA RESOURCESETS DIRECTIONS, ESTABLISHES PRIORITIES, COMMITS RESOURCESCOLLECTIVELY MANAGE THE ORGANIZATION’S DATA RESOURCE

DETAIL LEVEL (KNOWLEDGE WORKER)KNOWLEDGEABLE ABOUT THE DATA BY REASON OF INTIMIATE FAMILIARITY WITH DATAARCHITECTURAL RESPONSIBILITYDEVELOPS DATA ARCHITECTURE AND DATA RESOURCE DATANO DECISION MAKING AUTHORITY

TACTICAL LEVEL (LIAISON)LARGE 0R MULTI-NATIONAL ORGANIZATIONS FACE DATA GLOBALIZATION ISSUESNEED ADDITIONAL LEVEL OF STEWARDSHIPREPRESENT UNIQUENESS OF INDIVIDUAL DEPARTMENTS OR COUNTYS

Page 35: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESPONSIBILITY

REASONABLE MANAGEMENT PROCEDURESDEVELOPED BY DATA STEWARDSEQUIVALENT TO FINANCIAL, HUMAN RESOURCE, PROPERTY MANAGEMENTMUST BE REASONABLE - ENCOURAGE PARTICIPATIONMUST BE READILY AVAILABLE AND UNDERSTANDABLE

PROCEDURES YOU WOULD LIKE TO SEE IF YOU WERE ON THE RECEIVING END.

CENTRALIZED CONTROLRESULT FROM DATA STEWARDS AND MANAGEMENT PROCEDURESSINGLE COMMON DATA ARCHITECTURE WITH LOCAL IMPLEMENTATIONNOT A SINGLE ‘DATABASE IN THE SKY’

REVERSE OF TRADITIONAL APPROACHMANDATE CENTRALIZED CONTROLESTABLISH PROCEDURESSTEWARDS TO ENFORCE PROCEDURES

BEGIN THINKING GLOBALLY AND ACTING LOCALLY!

Page 36: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA VISION

EXPANDED DATA VISIONAN INTELLIGENT FORESIGHT ABOUT THE DATA RESOURCE THAT INCLUDES THE SCOPEOF THE DATA RESOURCE, ITS DEVELOPMENT DIRECTION, AND A PLANNIG HORIZON.

GOOD PRACTICESWIDER SCOPE

MUST ULTIMATELY INCLUDE ALL DATA AT AN ORGANIZATION’S DISPOSAL

NON-CRITICAL DATAHAVE SOME BUSINESS USEFULNESS - MUST ULTIMATELY BE MANAGED

NON-TABULAR DATADATA MEGATYPE - MAJOR GROUPING OF DATA BASED ON OVERALL STRUCTURE AND PHYSICAL MANGEMENT, SUCH AS SPATIAL, IMAGE, TEXTUAL, ETC.

BASE DATA TYPE - SPECIFIC FORM OF DATA WITHIN A DATA MEGATYPESUCH AS DATE, TEXT, INTEGER, ETC.

DISTINCT DATA TYPE - DATA UNIT OR STYLE WITHIN A BASE DATA TYPESUCH AS METERS / YARDS / INCHES OR POUNDS / MARKS / US DOLLARS

TABULAR / NON-TABULAR DATA OFTEN CONFUSED WITHSTRUCTURED / UNSTRUCTURED DATA!

Page 37: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA VISION

HISTORICAL DATAPROPERLY MANAGE TEMPORAL OR ‘TIME RELATIONAL’ DATA

OPERATIONAL DATA - SUBJECT ORIENTED, INTEGRATED, TIME CURRENT, VOLATILE DATAIN SUPPORT OF DAILY OPERATIONS AND OPERATIONAL DECISIONS.

EVALUATIONAL DATA - SUBJECT ORIENTED, INTEGRATED, TIME VARIANT, NON-VOLATILE DATAIN SUPPORT OF MANAGEMENT DECISION MAKING.

NON-AUTOMATED DATAINCLUDE IN SAME PRIORITIZATION AS AUTOMATED DATAOPENS UP WHOLE NEW HIDDEN DATA RESOURCE

REASONABLE DEVELOPMENT DIRECTIONPRIMARILY ON BUSINESS DIRECTION SECONDARILY ON DATABASE MANAGEMENT SYSTEM DIRECTION

REALISTIC PLANNING HORIZON

CHALLENGING YET ACHIEVABLESTRETCH THE IMAGINATION BUT NOT BE UNREALISTICMUST BE UNDERSTANDABLE AND ACHIEVABLEDIFFERENT HORIZONS FOR DIFFERENT AUDIENCES

COOPERATIVE ESTABLISHMENTESTABLISHED AND ACCEPTED COLLECTIVELY BY THE STAKEHOLDERS

Page 38: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RECOGNITION

APPROPRIATE DATA RECOGNITIONRECOGNIZE THAT DATA ARE A CRITICAL RESOURCE AND IMPROVE THE QUALITY OF THAT CRITICAL RESOURCE TO MEET THE BUSINESS INFORMATION DEMAND

GOOD PRACTICESTARGET VESTED INTEREST

HIGHEST LEVEL IN ORGANIZATION WITH VESTED INTEREST IN DATA RESOURCEWORK FROM THERE THROUGH THE ORGANIZATIONTHIS IS HIGHEST PROBABILITY OF SUCCESSLEVEL VARIES FROM ONE ORGANIZATION TO ANOTHER

DIRECT BUSINESS INVOLVEMENTDIRECT INVOLVEMENT OF KNOWLEDGEABLE PEOPLETHE GREATER THE LIMITATION OF BUSINESS -- THE GREATER THE CHANCE OF FAILURE

THE GREATER THE INVOLVEMENT OF BUSINESSTHE GREATER THE CHANCE OF SUCCESS!

TAP THE KNOWLEDGE BASEFIND PEOPLE WITH KNOWLEDGE OF THE DATA -- BUSINESS OR INFORMATION TECHNOLOGYUSE THEM AS DETAIL DATA STEWARDS OR CONTRIBUTORS

Page 39: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RECOGNITION

START WITHIN CURRENT BUDGETCURRENT BUDGET INITIATIVES GET RECOGNIZEDSTART SMALL AND GROW AS THE RECOGNITION GROWS

INCREMENTALLY COST EFFECTIVE APPROACHLONG-TERM BENEFITS HAS LOST ITS MEANINGSHORT-TERM BENEFITS LOSING ITS MEANING

INCREMENTALLY COST EFFECTIVE -- LAY A SUCCESS ON THE TABLE, GET MORE RESOURCESLAY ANOTHER SUCCESS ON THE TABLE, AND SO ON.

PROOF POSITIVE PERSPECTIVEGO TO MANAGEMENT WITH A PROOF-POSITIVE PERSPECTIVEGAIN RESOURCES TO CONTINUE SUCCESSFAR BETTER THAN A ‘TRUST ME’ REQUEST FOR RESOURCES

THERE IS A HIGHER RISK -- BUT THE REWARDS FAR OUTWEIGH THE RISKS!

BE OPPORTUNISTICSEIZE EVERY OPPORTUNITY TO DELIVER THE MESSAGESIZE OF OPPORTUNITY NOT IMPORTANT -- GETTING THE MESSAGE OUT IS IMPORTANTAVOID EXCESSIVE PILOT STUDIES AND PROOF OF CONCEPTSAVOID PRECISE RIGID PLANS

Page 40: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RECOGNITION

BUILD ON LESSONS LEARNEDEVERY PROJECT HAS SUCCESSES AND FAILURESSTOPPING DATA DISPARITY IS A DISCOVERY PROCESSMANAGEMENT MUST ACCEPT CONSTRUCTIVE FAILURES

NO BLAME - NO WHITEWASH ATTITUDETHE CURRENT SITUATION EXISTS -- THAT IS A FACTBLAME POLARIZES AND ALIENAGES PEOPLE -- DEFEATS THE INITIATIVECOVERING IT UP DELAYS ANY INITIATIVE FOR IMPROVEMENT

NO UNNECESSARY JUSTIFICATIONELIMINATE THE REQUIREMENT FOR EXTENSIVE JUSTIFICATION TO IMPROVE QUALITYTHE NEED FOR IMPROVED DATA RESOURCE QUALITY SHOULD BE A TRUISM

IF ANYTHNG -- A JUSTIFICATION SHOULD BE REQUIRED TO CONTINUERUINING THE QUALITY OF A CRITICAL RESOURCE!!

Page 41: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE TRANSITION

DATA RESOURCE TRANSITIONFORMALLY MOVING FROM A DISPARATE DATA RESOURCE TO A COMPARATE DATARESOURCE WITHIN THE COMMON DATA ARCHITECTURE

RESOLVES THE EXISTING DATA DISPARITYNOT A MIGRATION - IMPLIES WANDERING OR PERIODIC RETURN TO DISPARITYTRANSITION THROUGH FORMAL TRANSITION STATES

DATA RESOURCE TRANSITION PROCESSDISCOVERY PROCESSREQUIRES THOUGHT, ANALYSIS, INTUITION, PERCEPTION, AND SOME LUCKEVOLUTIONARY PROCESS DUE TO UNCERTAINTY

FORMALIZETHE UNDERSTANDING OF DISPARATE DATAINTEGRATE DISPARATE DATA

AVOID SUCK-AND-SQUIRT APPROACHESAVOID BRUTE-FORCE-PHYSICAL APPROACHES

DATA RESOURCE TRANSITION IS A FORMAL PROCESS!

Page 42: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA RESOURCE TRANSITION

DATA RESOURCE TRANSITION STATES

DISPARATE DATA RESOURCE - CURRENT STATEDATA NOT WELL UNDERSTOOD

FORMAL DATA RESOURCE - NECESSARY STATENECESSARY TO THOROUGHLY UNDERSTAND DISPARATE DATA

VIRTUAL DATA RESOURCE - DESIRED STATEREAL-TIME PRODUCTION TRANSFORMATION OF DISPARATE DATA

COMPARATE DATA RESOURCE - IDEAL STATEPERMANENT TRANSFORMATION OF DISPARATE DATA

Enterprise Data Resource

Common Data Architecture

DisparateData Resource

FormalData Resource

VirtualData Resource

ComparateData Resource

Understand DataWithin Common Context

Real-TimeTransformation

PermanentTransformation

Page 43: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

FORMAL DATA RESOURCE

FORMAL DATA RESOURCEDATA INVENTORY

INVENTORY EXISTING DISPARATE DATARAISES AWARENESS OF THE DATA THAT EXISTSOLVES THE FIRST PROBLEM WITH DISPARATE DATA

DATA CROSS REFERENCECROSS-REFERENCE DISPARATE DATA TO COMMON DATA ARCHITECTUREPROVIDES AN UNDERSTANDING OF DISPARATE DATA IN A COMMON CONTEXTSOLVES THE SECOND PROBLEM WITH DISPARATE DATA

DATA VARIABILITYIDENTIFY DATA VARIABILITYDESIGNATE PREFERRED DATA VARIATIONSDEVELOP DATA TRANSLATION SCHEMESPREPARATION TO SOLVE THE THIRD PROBLEM WITH DISPARATE DATA

DATA REDUNDANCYIDENTIFY DATA REDUNDANCYDESIGNATE PREFERRED DATA SOURCESPREPARATION TO SOLVE THE FOURTH PROBLEM WITH DISPARATE DATA

DEVELOPING A FORMAL DATA RESOURCE IS NON-DESTRUCTIVE!

Page 44: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

FORMAL DATA RESOURCE

ENTERPRISE DATA ARCHITECTUREREMEMBER - THE ENTERPRISE DATA ARCHITECTURE REPRESENTS THE COMPARATE DATA RESOURCE

THE PREFERRED DESIGNATIONS DEFINE THE COMPARATE DATA RESOURCEPREFERRED DATA SOURCESPREFERRED DATA VARIAITONSPREFERRED DATA INTEGRITY RULESDATA TRANSLATION SCHEMES

THESE SET THE STAGE FOR TRANSFORMING DISPARTE DATA TO COMPARATE DATA

THE PREFERRED DESIGNATIONS DEFINE THE ENTERPRISE DATA ARCHITECTURE!

Page 45: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA TRANSFORMATION

PROBLEMSMANY ORGANIZATIONS NOT FORMALLY INTEGRATING THEIR DATAUSUALLY INCOMPLETE / INCONSISTENT DATA INTEGRATIONIDENTIFYING A SINGLE SYSTEM OF REFERENCE USUALLY NOT VALIDUSING SUCK-AND-SQUIRT METHODS TO OBTAIN DATA USUALLY DON’T WORKUSING BRUTE-FORCE-PHYSICAL APPROACHESMORE THAN SIMPLE EXTRACT - TRANSFORM - LOADMORE THAN MINOR DATA CLEANISINGRESULT IS LOW-QUALITY DATA

APPROACHFORMAL DATA TRANSFORMATION WITHIN COMMON DATA ARCHITECTUREBASED ON FORMAL DATA RESOURCE - THOROUGH UNDERSTANDING OF DATAREQUIRES ROBUST DATA TRANSFORMATIONNOT A TRIVIAL TASK - BUT NOT IMPOSSIBLE EITHER IGNORED IN MANY METHODS

DATA INTEGRATION MUST BE BASED ON THE FORMAL DATA RESOURCE!

Page 46: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA TRANSFORMATION

VIRTUAL / COMPARATE DATA RESOURCEEXTRACT - THREE STEPS

IDENTIFYING AND EXTRACTING THE BEST DISPARATE DATA TO A DATA DEPOTISOLATED FROM SOURCE AND TARGET DATAPREPARATION FOR DATA TRANSFORMATION

TRANSFORM - FIVE STEPSTRANSFORMING THE DISPARATE DATA TO COMPARATE DATAUSING FORMAL DATA TRANSFORMATION RULESACTUALLY CHANGING THE DATA

LOAD - THREE STEPSEDITING THE COMPARATE DATALOADING THE COMPARATE DATA FROM A DATA DEPOTBUILDING THE COMPARATE DATA RESOURCE

Page 47: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

DATA TRANSFORMATION

DATA TRANSFORMATION PROCESS

IdentifyTarget Data

IdentifySource Data

ExtractData

Data Reconstruction

DataTranslation

DataRecasting

DataIntegrity

DataLoading

DataReview

DataRestructuring

DataDerivation

Data Extract

Data Transform

Data Load

Page 48: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CULTURAL CONSIDERATIONS

A SHARABLE DATA RESOURCEGETTING PEOPLE TO INTERACT AND SHARE

AWARE OF EXISTING DATA

THOROUGHLY UNDERSTAND DATA

USE DATA TO SUPPORT THEIR BUSINESS ACTIVITIES

REAL DATA SHARING INCENTIVESMAKE EASY TO MEET BUSINESS NEEDSMAKE EASY TO CROSS ORGANIZATIONAL BOUNDARIES

WHEN THE PAIN OF NOT MEETING BUSINESS NEEDSEXCEEDS THE PAIN OF CROSSING ORGANIZATIONAL BOUNDARIES

PEOPLE WILL INTERACT AND SHARE!

Page 49: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CULTURAL CONSIDERATIONS

DATA SHARING - DATA QUALITY - DATA INTEGRATION CYCLEWHEN PEOPLE UNDERSTAND DATA - THEY SHARE THOSE DATAWHEN PEOPLE SHARE DATA - THE DATA QUALITY IMPROVESWHEN DATA QUALITY IMPROVES - THE DATA ARE INTEGRATEDAND THE CYCLE KEEPS GOING

THIS IS NOT ROCKET SCIENCE!

Data QualityImproves

People GetInvolved

Data IntegrationProgresses

Data SharingIncreases

Data AreUnderstood

Page 50: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CULTURAL CONSIDERATIONS

A PEOPLE ORIENTATIONUNCERTAINTY

HELP RESOLVE UNCERTAINTYREDUCE RESISTANCE TO CHANGEMANAGE EXPECTATIONS

WILLINGNESS TO CHANGEMOST PEOPLE WILLING TO CHANGETHEY JUST DON’T KNOW HOWCONCERNED ABOUT IMPACT TO BUSINESS

RESISTANCE TO CHANGEPEOPLE UNSURE OF NEW PARADIGMCOMFORTABLE WITH EXISTING PARADIGMCONCERNED ABOUT PAST CURE-ALL FAILURESNO TIME TO STOP THE BUSINESS TO CHANGE

SURFACE DIFFERENCESDOCUMENT DIFFERENT DEFINITIONSDON’T ATTEMPT TO RESOLVE DIFFERENCES

PEOPLE DON’T MIND CHANGING!PEOPLE DO MIND BEING CHANGED!

Page 51: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CULTURAL CONSIDERATIONS

A BUSINESS ORIENTATIONFOCUS ON THE BUSINENSS

BASED ON BUSINESS PLAN AND STRATEGIES

BASED ON ENTERPRISE’S PERCEPTION OF BUSINESS WORLD

CRITICAL AREA APPROACHIDENTIFY CRITICAL BUSINESS AREAS

DETERMINE DATA NEEDED FOR THOSE AREAS

IDENTIFY DATA THAT ALREADY EXIST

IDENTIFY DATA THAT NEED TO BE ACQUIRED

DEVELOP COMPARATE DATA FOR THOSE CRITICAL AREAS

CLIENT FOCUSED APPROACHORIENT THE PROCESS TO THE CLIENT’S WAY OF THINKING

DRAW THE CLIENT INTO THE PROCESS

TAP THE CLIENT’S KNOWLEDGE BASE

DIRECT CLIENT INVOLVEMENTINVOLVEMENT LEADS TO COMMITMENT

COMMITMENT LEADS TO ACCEPTANCE

ACCEPTANCE LEADS TO SUCCESS

Page 52: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CULTURAL CONSIDERATIONS

SUCCESS MOTIVATIONSUCCESS IS CONTAGIOUS

ORIENT TOWARD CONTINUED SUCCESSESTHRIVE ON LESSONS LEARNED

INCREMENTALLY COST EFFECTIVELONG TERM BENEFITS HAS LOST ITS PUNCH

SHORT TERM BENEFITS BEGINNING TO LOSE ITS PUNCH

INCREMENTALLY COST EFFECTIVE IS A GOOD TERM

AVOID MANDATESMANDATES DO NOT WORK

IMPLY COMPLIANCE, ENFORCEMENT, PUNISHMENT

CREATES A WIN-LOSE SITUATION

ESTABLISH GUIDELINES GAIN CONSENSUS AND ACCEPTANCE

EVOLVE WITH TECHNOLOGY

DESIRED APPROACHEASIEST ROUTE TO FOLLOWASSURE SUCCESS

CREATE WIN - WIN SITUATION!

Page 53: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

CONCLUSION

ULTIMATE DATA RESOURCE QUALITY IS STABILITY ACROSSCHANGING BUSINESS STRATEGIES & INFORMATION NEEDSCHANGING TECHNOLOGY

STABILITY ACHIEVED BY ESTABLISHING A COMMON DATA ARCHITECTUREHALTING THE CONTINUED DATA DISPARITYRESOLVING EXISTING DATA DISPARITY

NO SILVER BULLETSATTEMPT TO ACHIEVE SOME GAIN WITHOUT ANY PAINUSUALLY ENDURE CONSIDERABLE PAIN WITH MINIMAL GAIN

DON’T DELAYNEED NO JUSTIFICATION TO IMPROVE DATA RESOURCE QUALITYCONCEPTS, PRINCIPLES, & TECHNIQUES ARE AVAILABLETHE SITUATION IS ONLY GETTING WORSE

Page 54: TURNING BAD HABITS INTO GOOD PRACTICESnces.ed.gov/forum/pdf/data_res_quality_ppt.pdfTURNING BAD HABITS INTO GOOD PRACTICES NATIONAL FORUM ON EDUCATION STATISTICS FEBRUARY 21, 2005

© 2005 by Michael H. Brackett

PRESENTER’S BIBLIOGRAPHY

DEVELOPING DATA STRUCTURED INFORMATION SYSTEMSKEN ORR & ASSOCIATES, 1984

DEVELOPING DATA STRUCTURED DATABASESPRENTICE HALL, 1987

PRACTICAL DATA DESIGNPRENTICE HALL, 1990

DATA SHARING USING A COMMON DATA ARCHITECTUREJOHN WILEY & SONS, 1994

THE DATA WAREHOUSE CHALLENGE: TAMING DATA CHAOSJOHN WILEY & SONS, 1996

DATA RESOURCE QUALITY: TURNING BAD HABITS INTO GOOD PRACTICESADDISON WESLEY, 2000

DATA RESOURCE INTEGRATION: UNDERSTANDING & RESOLVING DATA DISPARITYIN PROGRESS, NO PUBLISHER OR DUE DATE

DATA RULES: KEY TO DATA RESOURCE QUALITYIN PROGRESS, NO PUBLISHER OR DUE DATE

CONSULT PRESENTER’S WEB PAGE FOR ADDITIONAL DETAILS