ies.ed.gov

Connecting Research,Policy and Practice

84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions

84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions

(FY 2016 Grant Competition)

Robert Ochsendorf, Ph.D.National Center for Special Education Research

Phill Gagne, Ph.D.Allen Ruby, Ph.D.

National Center for Education Research

ies.ed.gov

Overview

• Overview of IES and its mission• General Requirements• Purpose and Products• Project Narrative

– Significance– Partnership– Research Plan– Personnel– Resources

• Other important sections of the application• Preparing an application

2

ies.ed.gov

Legislative Mission of IES

• Describe the condition and progress of education in the United States

• Identify education practices that improve academic achievement and access to education opportunities

• Evaluate the effectiveness of Federal and other education programs

3

ies.ed.gov

Organizational Structure of IES

4

National Board for Education

SciencesStandards &

Review Office

Office of the Director

National Center for Education Evaluation

National Center for Education Statistics

National Center for Education Research

National Center for

Special Education Research

ies.ed.gov

IES Grant Programs: Research Objectives

• Develop or identify education interventions (i.e., practices, programs, policies, and approaches) – that enhance academic achievement– that can be widely deployed

• Identify what does not work and thereby encourage innovation and further research

• Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

5

ies.ed.gov

Partnerships & IES Priorities

IES seeks to... • Encourage education researchers to develop

partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers

• Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies

(See http://ies.ed.gov/director/board/priorities.asp)

6

ies.ed.gov

Short Description of Low-Cost Evaluation Projects

• Carried out by Partnerships– New or established – Minimum: research institution and a state or local

education agency

• Purpose – Carry out rigorous evaluations of education interventions

implemented by state or local education agencies • High importance to the education agency• Use secondary data (e.g., administrative data)• Low-cost: maximum grant of $250,000• Short-duration: 2 years

ies.ed.gov

Impetus for Low-Cost Grant Program

• Take advantage of opportunities to use administrative data to do evaluations

• Provide useful information to education agencies in a more timely manner than traditional evaluations

• Create additional opportunities for research institutions and education agencies to work together

• Identify the strengths, weaknesses, and applicability of this type of evaluation

ies.ed.gov

General Requirements

• Focus on student education outcomes– 84:305L: For students from prekindergarten through

postsecondary and adult education– 84.324L: For students from prekindergarten through grade 12

with or at-risk for disability• Research occurs in an authentic education setting• Evaluate education interventions using secondary data• Partnership between research institutions and state and

local education agencies• Disseminate findings in ways useful to agency decision-

making

ies.ed.gov

Student Population

• 84:305L: Students from prekindergarten through postsecondary and adult education

• 84.324L: Students from prekindergarten through grade 12 with or at-risk for disability– A student with a disability is defined in Public Law

108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA)

– Additional requirements for identifying students at risk for developing a disability

– see http://ies.ed.gov/ncser/definition.asp 10

ies.ed.gov

Focus on Student Education Outcomes

• Research must address education outcomes of students. For both 305L and 324L these include– Academic outcomes– Social and behavioral competencies that support

student success in school• For 324L, these also include– Functional and transitional outcomes for students

with or at-risk for disability

11

ies.ed.gov

Student OutcomesGrade Outcome

Prekindergarten 305L & 324L

324L

School readiness (e.g., pre-reading, language, vocabulary, early math and science knowledge, social and behavioral competencies)Developmental outcomes

K - 12 305L & 324L

324L

Learning and achievement in reading, writing, mathematics, and science; Progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout);Social and behavioral competencies important to academic and post-academic success.Functional outcomes that improve educational results; Transitions to employment, independent living, and postsecondary education.

12

ies.ed.gov

Additional 305L Student Outcomes

Grade Outcome

Postsecondary(Grades 13 – 16)

Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics; success in gateway math and science courses, introductory English composition

Adult Education(Adult Basic Education, Adult Secondary Education, Adult ESL, and HS equivalency preparation)

Student achievement in reading, writing, English language proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs

13

ies.ed.gov

Education Interventions

• The wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the student, classroom, school, district, state, or federal level to improve student education outcomes

• High importance to state or local education agency• Expected to produce meaningful improvement in student

education outcomes within a short period (e.g., a semester or year)

• Administrative data (or other source) focused on intervention and outcomes (primary data collection not supported by grant).

ies.ed.gov

Applications Must be from a Partnership

• Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from a U.S. state or local education agency– PI from research institution: Must have the ability and

capacity to conduct scientifically valid research and expertise in the education issue to be addressed

– PI from State or local education agency: Must have decision-making authority for the intervention within his or her agency

15

ies.ed.gov

Partnership

• Partnership may be new or existing• Research institution has a broad definition– Ability and capacity to conduct scientifically valid research– Not restricted to types of institutions

16

ies.ed.gov

Partnership: SEAs

• State education agencies – Examples: education agencies, departments, boards,

commissions – Oversee early learning, elementary, secondary, and/or

postsecondary and adult education– Also includes education agencies in District of Columbia,

the Commonwealth of Puerto Rico, and each of the outlying areas.

ies.ed.gov

Partnership: LEAs

• Local education agencies are primarily public school districts• Community college districts• Tribal education agencies• State and city postsecondary systems

– If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner

– If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner

– A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project

18

ies.ed.gov

Additional Partners

• Partnerships may include more than one State or local education agency if they share similarities and interests

• Non-education state and local agencies may be partners as long as an education agency is a partner

• Partnerships may include more than one research institution if they have shared interests and will make unique contributions

• Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups, non-public organizations that oversee or manage schools) that will contribute to the partnership and its work

19

ies.ed.gov

Dissemination/Products

• Projects are to aid education agencies in decision-making • Required dissemination

– Oral briefing on results to education agency– Written brief, written for non-technical audience, made

available free to public• Recommended dissemination

– Partner presentations to academic and practitioner audiences– Partner publications in academic and practitioner journals– Toolkit or guide for other education agencies on how to

conduct a similar study

ies.ed.gov

Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant Program

• If you are not looking at student outcomes, then IES is not the appropriate funding agency

• If you need time and effort to build a partnership and prepare for an evaluation, consider:– Researcher-Practitioner Partnerships under 84.305H

• If the intervention you want to evaluate – Is not implemented by a state or local education agency,– Cannot be evaluated using secondary data, or– Is not expected to improve student outcomes within a short period.– Then consider the:

• Education Research Grants Program (84.305A) or • Special Education Research Grants Program (84.324A)

ies.ed.gov

Low-Cost Evaluation: Purpose

• Promote joint evaluation research by research institutions and state and local education agencies– On an education intervention identified as having great

importance by the education agency– That includes practitioner input into the research– That will provide timely rigorous evidence for the agency’s

decision-making regarding the intervention– And the results of which will be broadly disseminated in ways

easily accessible to researchers, practitioners, and the public

22

ies.ed.gov

What should the partnerships do during the grant?

• Identify an education intervention• Implemented by an SEA or LEA

• Of high priority to that agency

• Intended to improve student education outcomes

• Carry out an evaluation of that intervention – Using a RCT or an RDD design

– Using secondary data

– Obtain overall impacts

– Obtain subgroup impacts for subgroups identified in the secondary data and of interest to the education agency and/or literature

– Examine other moderators and mediators of interest, fidelity of implementation, and comparison group practice if such data are available 23

ies.ed.gov

Expected Products of the Grant

• Causal evidence of the impact of a clearly specified intervention implemented by an SEA or LEA– Overall impacts– Impacts for available subgroups of interest

• Advice for the SEA or LEA – Continuing and/or expanding the use of the intervention– Further research needs, e.g.,

• Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication

• Implementation

24

ies.ed.gov

The Project Narrative

Page Limit: 15•Significance•Partnership•Research Plan•Personnel•Resources

25

ies.ed.gov

Significance

• The education intervention to be evaluated– The education problem/issue the intervention is to address

within the SEA/LEA• Relevance to other SEAs or LEAs (secondary importance)

– Components of the intervention– Rationale for why the intervention can improve student

outcomes within a short period (e.g., 1 semester, 1 year)• May include theory of change• Difference from status quo• Related findings from previous studies and how this study will

improve upon past work

ies.ed.gov

Significance

• The implementation of the intervention– Who will implement it and how will it be implemented– That education agency will implement or will oversee

implementation– Adequate funding available for implementation– Implementation during Year 1 of the project at a level

expected to impact student outcomes• Sources of secondary data to be used in the evaluation

– How these data are collected– How these data will be obtained by researchers by 1st

quarter of Year 2 of project

ies.ed.gov

The Project Narrative

Page Limit: 15•Significance•Partnership•Research Plan•Personnel•Resources

28

ies.ed.gov

Partnership

• Describe the partners– The research institution and the education agency

• Offices or divisions within agency whose cooperation is necessary

– Any other members of the partnership– Common interest in and benefit from this evaluation– The process through which the partners determined the

specific intervention to evaluate– Data sharing agreement – the strategy to obtain the

secondary data and provide it for analysis by the 1st quarter of the second year

ies.ed.gov

The Project Narrative

Page Limit: 15•Significance•Partnership•Research Plan•Personnel•Resources

30

ies.ed.gov

Research Plan

• State research questions/hypotheses• Describe sample and setting– Define population and how your sample and sampling

procedures will allow inferences to the population– Exclusion and inclusion rules and their justification– How sample size set to address expected attrition– Strategies used to increase participation and reduce

attrition– Describe the setting and its implications for the

generalizability of your study

31

ies.ed.gov

Research Plan: Design

• Discuss how design will support causal inferences and identify potential threats to internal validity

• Discuss how degree of equivalence at baseline will be determined

• Discuss possibility of bias from overall and differential attrition

• Required use of RCT or RDD– Meet WWC evidence standards without reservations

32

ies.ed.gov

Research Plan: Design

Randomized Controlled Trial (RCT)• Note unit of randomization and justify choice• Describe process for random assignment and

maintaining its integrity• Different Approaches to RCTs - Potential Issues– Entire population: Treatment fidelity– Volunteers: Comparison group status– Lotteries: Attrition of non-accepted parties– Staggered roll out: Little time for true comparison– Variations of program/policy: Issue of overall significance

33

ies.ed.gov

Research Plan: Design

Regression Discontinuity Design (RDD)• Appropriateness of assignment variable• Show true discontinuity• Discuss possibility of manipulation of design variable

and analyses to determine such manipulation• Sensitivity analyses to assess influence of key

procedural or analytic decisions on results

34

ies.ed.gov

Research Plan: Statistical Power

• Detailed description of power analysis – Justify method used to calculate power– Justify parameters used and assumptions made

• Provide power for main analyses and important subgroup analyses

• Along with identifying minimum detectable effect for your analysis, justify its– Reasonableness– Practical meaning

• Reviewers should be able to check power calculations35

ies.ed.gov

Research Plan: Outcome Measures

• Student education outcome measures relevant to states, districts, and schools– Found in administrative data or other secondary data

• Discuss reliability, validity, and appropriateness• Must be collected during Year 1 of project– Additional data from previous years of intervention’s

implementation may also be used if appropriate to the evaluation design

• Clearly link measures to rationale for the intervention

36

ies.ed.gov

Research Plan: Optional Measures

• If available in secondary data, describe measures of– Moderators (subgroups expected)– Mediators (intermediate outcomes)– Fidelity of implementation– Comparison group practice

ies.ed.gov

Research Plan: Analysis

• Detail impact analyses– Make clear how analyses directly answer your

research questions – Show that analyses are based on the design– Address clustering of students in classrooms in

schools – Address missing data– If multiple datasets are to be linked, detail how this

will be done• Describe any other analyses to be done (e.g.,

subgroups, other moderators, mediators, and fidelity of implementation)

38

ies.ed.gov

The Project Narrative

Page Limit: 15•Significance•Partnership•Research Plan•Personnel•Resources

39

ies.ed.gov

Personnel

• Identify all key personnel on the project team – The PI from the research institution who has previous experience

carrying out the proposed evaluation design (RCT or RDD)– The PI from the education agency who makes program decisions– Other key personnel

• Roles and responsibilities on the project– Each individual’s roles and responsibilities on the project– Their qualifications (i.e., expertise and experience) for their role– Their % FTE on the project – Past success at working in similar partnerships

• PI qualifications for managing a grant of this type• Ensure objectivity of evaluation

40

ies.ed.gov

Resources: To Conduct the Project

• Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research– Institutional capacity to manage the grant– Resources available at the partner institutions to support

the project– Plans to acquire any major resources not yet in hand (e.g.,

secondary data)– Joint Letter of Agreement by partners (Appendix D)– Letter of Agreement to provide administrative data (App. D)

41

ies.ed.gov

Resources: Dissemination of Results

• Results expected to be useful to the SEA/LEA partner and, perhaps, other SEA/LEAs– Findings of both beneficial impacts or no impacts

• Describe your capacity and resources to disseminate findings– Required dissemination through an oral briefing for the

agency and a written brief freely available to the public– Dissemination to other audiences (e.g., researchers,

policymakers, practitioners, students and their families, public)

42

ies.ed.gov

Other Important Sections of the Application

• Appendix A• Appendix B• Appendix C• Appendix D• Budget & Budget Narrative

43

ies.ed.gov

Appendix A (Required for Resubmissions)

Page Limit: 3

• If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments

44

ies.ed.gov

Appendix B (Optional)

Page Limit: 5

• Figures, charts, or tables that supplement the project narrative

• Timelines for the project (very useful)• Examples of instruments used in the collection of the

administrative or other secondary sources of data• Do NOT include narrative text

ies.ed.gov

Appendix C (Optional)

Page Limit: 5

• Examples of materials used in the intervention:– curriculum materials– computer screen shots– training documents– assessment items– other materials

• Do NOT include narrative text

46

ies.ed.gov

Appendix D (Required)

No Page Limit• Required Letters of Agreement– Joint Letter from the research institution and the SEA/LEA

• Document participation and cooperation in the partnership• Set out each’s roles and responsibilities under the project

– Letter from the office in charge of the agency’s data• Project will have access to data required in time to do analysis

• Optional Letters of Agreement– Separate Letters from other organizations taking part– Letters from any consultants and schools taking part

47

ies.ed.gov

Budget & Budget Narrative

• Maximum project length is 2 years• Maximum award is $250,000– Funds must be used for evaluation only (e.g., cannot

be used for implementation of the intervention or primary data collection)

– Award size depends on project scope

• Include a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested

48

ies.ed.gov

Preparing Your Application

• Important dates• Information sources– Read the RFA– Talk with a program officer

• Review process

ies.ed.gov

Application Deadline

Letter of Intent Due

Date

Application Package Posted

Start Dates

January 12, 2016

4:30:00 PM DC Time

November 12, 2015

November 12, 2015

July 1 – September 1,

2016

Important Dates & Deadlines

50

ies.ed.gov

Information Sources

• Request for Applications– http://ies.ed.gov/funding/

• Abstracts of Projects– http://ies.ed.gov/funding/grantsearch/index.asp

• Application Package– www.grants.gov

• Program Officers

– Robert.Ochsendorf@ed.gov 84.324L– Phill.Gagne@ed.gov 84.305L– Allen.Ruby@ed.gov 84.305L

51

ies.ed.gov

Peer Review(Standards & Review Office)

• Compliance screening for format requirements• Responsiveness screening for program requirements• Assignment to review panel– 2 to 3 reviewers (substantive and methodological)– The most competitive proposals are reviewed by full panel

• Many panelists will be generalists to your topic• Panels contain experts in relevant methodologies

– Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources

52

ies.ed.gov

Notification

• All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS):• Status of award• Reviewer summary statement

• If you are not granted an award the first time, consider resubmitting and talking with the Program Officer

53

ies.ed.gov

For More Information

http://ies.ed.gov/funding

Robert.Ochsendorf@ed.gov 84.324LPhill.Gagne@ed.gov 84.305LAllen.Ruby@ed.gov 84.305L

54