Howard UniversityOctober 10,2012

The Integrity of Data ManagementResponsible Conduct of Research

Workshop

Peggy Carr, Ph.D.Associate Commissioner

National Center for Education Statistics

1

Overview of Statistical and Research Standards

2

OMB Guidelines for Federal Statistical Agencies

16 Federal Statistical Agencies adhere to Office of Management and Budget Guidelines to maintain high standards of performanceDevelopment of concepts and methodsPlanning and design of surveys and other

means of collecting dataCollection of dataProcessing and editing of data

OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, February 22, 2002, (67 FR 8752-8460) 3

Best Practices for the Conduct of Research

4

Best Practices for the Conduct of Research

1. Conceptualize Research Problem2. Operationalize Study Design3. Engage in Peer Review and Networking

Activities4. Publish and Publish Early/Often5. Be Ethical

5

1. Conceptualize Research Problem

6

Literature Review

Statement of Problem

Hypotheses Research Question

Definition of Terms, Concepts, and

Constructs

ConceptualOperational

2a. Operationalize Study DesignThe Concept of Study Design

Hypothesis 1

Explained Variance

Unexplained Variance

Sampling Error

Measurement Error

Validity

Reliability

Validation of ConstructsFace – Content validation across judgesCriterion – Predictive and concurrentConstruct – Methodological approaches to validation

8

Jealousy Envy

2b. Operationalize Study Design

2c. Operationalize Study DesignReliability of Measures

Test and Retest – stability over timeEquivalency – consistency across formsInternal – internal consistency

9

UNRELIABLE RELIABLE

2c. Operationalize Study DesignResearch Design

Internal Validity – refuting rival plausible hypothesesExternal Validity – generalizability of findingsChoice of Designs – experimental, quasi, survey,

correlational

10

Quality Assurances of DataData Checks (e.g., out of range variance, criterion

reference checks, replication)

Thoughtful and Appropriate Data AnalysisThe nature of your dataLimitations of statistical technique

Discuss Findings and Conclusions with ColleaguesRelate back to literature reviewImplications for future researchFlaws and limitations

11

NEXT

2c. Operationalize Study Design

Trend in fourth-grade NAEP mathematics average scores

12

2c. Operationalize Study Design

SOURCE: The Nation’s Report Card, 4th Grade NAEP: Mathematics, 1990-2011.

13

2c. Operationalize Study DesignTrend in fourth-grade NAEP mathematics percentile scores

SOURCE: The Nation’s Report Card, 4th Grade NAEP: Mathematics, 1990-2011.

Oprah discussing education and research

3. Engage in Peer Review and Networking Activities

Find a specialty and stick with it

Attend professional meetings and conferences – share your research

Keep an open mind and learn from others

15

4. Publish and Publish Early/OftenStart your publication record early in peer

reviewed journalsAs first, second, or third author

Publish and/or present oftenRegardless of professional focusAnnually

16

5. Be EthicalBe committed and follow through

Adhere toSubjects’ RightsInstitutional Review BoardConfidentiality and Privacy Rights

17

Questions?

18