planning and evaluation: portfolio assessment for csrees strategic objectives

United States Department of AgricultureUnited States Department of Agriculture

Cooperative State, Research, EducationCooperative State, Research, Educationand Extension Serviceand Extension Service

PLANNING and EVALUATION:PLANNING and EVALUATION:Portfolio Assessment for Portfolio Assessment for

CSREES Strategic ObjectivesCSREES Strategic Objectives

Jeffrey GilmoreDirector

Higher Education Programs

Collaborating to Solve National Collaborating to Solve National Problems in the U.S.Problems in the U.S.

•Hammons Products•American Forest and Paper Association•Forest Resources Association

•Professional societies (TWS, SRM,SAF)•University organizations (NASULGC, NAPFSC, NAUFWP)•Conservation organizations (TNC, NACD,AFF)

•Hispanic-Serving Institutions•Alaska Native and Native Hawaiian-Serving Institutions

•1862, 1890, 1994 Land Grant Universities•Non Land Grant colleges and universities

NGO’s

Research Institutes & Universities

Private Industry

Federal Government

USDA Collaborations with Other USDA Collaborations with Other Federal EntitiesFederal Entities

USDA

US EPACSREES and USEPA are collaborating on

indoor air quality

DOD

CSREES and FS are collaborating on the

National Agrogorestry Center at Alabama A&M

DOECSREES, NRCS, FS and DOE are collaborating in

biomass & fuel processing waste for energy and products

NASACSREES and NASA are

collaborating and funding the Geospatial

Extension Program

NSFCSREES and NSF are

collaborating on Microbial Observatories

USDA and Its Seven Mission USDA and Its Seven Mission AreasAreas

USDA

Food Safety

Farm and Foreign

Agricultural Service

Marketing and

Regulatory

Rural Developme

ntResearch, Education

& Economic

sFood,

Nutrition &

Consumer Service

Natural Resources

& Environme

nt

The Research, Education and The Research, Education and Economics Mission AreaEconomics Mission Area

National Agricultural

Library (NAL)

Economic

Research Service (ERS)

Agricultural Research Service (ARS)

National Agricultural

Statistics Service (NASS)

Cooperative State Research

Edu. & Ext. Service

(CSREES)

CSREES Emphasis AreasCSREES Emphasis Areas

Plants and Plant Products

Food, Nutrition &

Health

Technology and

EngineeringPest

Management

Natural Resources & Environment

Agricultural and Food

Biosecurity

Agricultural Systems

Animals and Animal

Products

Biotechnology & Genomics

Economics & Commerce

Families, Youth, &

Communities

The AgencyThe Agency

• CSREES is a federal assistance agency.

• CSREES - <400 people, $1.2 billion.

• University System, mainly land grant:• >130 colleges of agriculture/natural resources• 60 agricultural experiment stations• 58 cooperative extension services• 65 schools of forestry• eighteen 1890 institutions• thirty three 1994 institutions

Portfolio ReviewPortfolio Review

• Significant accomplishments have been achieved.

• The work of our partners = the work of CSREES.

• Impacts/Outcomes of accomplishments now being realized…short, medium & long term

Time PeriodTime Period

• The Portfolio timeframe (1999-2003) reflects work that occurred in 1997-2001.

• Now 6 months into FY 2005.

• FY2006 President’s Budget released on February 7, 2005.

• The budget planning process for the FY2007 President’s Budget is underway.

Current Priorities not in Current Priorities not in Review CycleReview Cycle

• Agro-security• Obesity• Nanotechnology• Water security• Air quality• e-Extension• Higher education

reporting into CRIS

CSREES Goal 5CSREES Goal 5

Portfolio5.1

Portfolio5.2

Goal 5:Goal 5:Protect & EnhanceProtect & Enhance

the Nation’sthe Nation’sNatural ResourceNatural Resource

BaseBase & Environment& Environment

Portfolio 5.1 Forests, Range, Fish Portfolio 5.1 Forests, Range, Fish and Wildlife by Problem Areasand Wildlife by Problem Areas

Portfolio5.1

PA 121: Mgmt of Range Resources

PA 122: Mgmt & Control of Forest & Range Fires

PA 123: Mgmt & Sustainability of Forest Resources

PA 124: Urban Forestry

PA 125: Agroforestry

PA 135: Aquatic & Terrestrial Wildlife

PA 136: Conservation of Biological Diversity

Portfolio 5.2: Management of Soil, Air, and Portfolio 5.2: Management of Soil, Air, and Water by Problem AreasWater by Problem Areas

Portfolio5.2

PA 101 – 104Soil resources

PA 111, 112, & 405Water

Resources

PA 131Alternate Uses

Of Land

PA 605Natural

ResourceEconomics

PA 133Pollution

Prevention &Mitigation

PA 141Air ResourceConservation

& Management

PA 403Waste Disposal,

Recycling,& Reuse

PA 132Weather and

Climate

CSREES Inputs to PortfolioCSREES Inputs to Portfolio

• Funding to partners– Formula funds– Competitively awarded funds– Congressionally directed

funds

CSREES Inputs to PortfolioCSREES Inputs to Portfolio

• Human resources– National Program Leaders– Program Specialists and

Analysts– Administrative Support Staff

CSREES NPL FunctionsCSREES NPL Functions

• Program leadership– Serve as a technical and disciplinary expert.– Network and collaborate to identify mission-

relevant goals.– Initiate and facilitate networks, liaisons, and

collaborations to identify mission-relevant goals– Promote a shared and integrated agenda.– Mobilize rapid response to immediate issues.– Conceptualize, formulate and execute programs

and activities to respond to emerging issues and opportunities.

CSREES NPL FunctionsCSREES NPL Functions

• Program management– Conceptualize, formulate and direct program

policy, technical standards, and guidelines.– Develop and oversee the implementation of

evaluation strategies.– Prepare requests for applications.– Remove barriers to program efficiency and

effectiveness.– Allocate scarce resources.



The Scope of the Portfolio:The Scope of the Portfolio:Partners, Integration, and ProgramsPartners, Integration, and Programs

Other CSREES Governmental Other CSREES Governmental PartnersPartners

CSREES

ARS

Forest Service

DOC

NRCS

DOIState

AgricultureDepts.

EPA

CountyConservation

Districts

County Extension Councils

State Environ.

ProtectionDept.

CSREES Non-government CSREES Non-government PartnersPartners

CSREES

Environ. & natural resources

orgs.Professional

SocietiesNASULGC

Commodity Groups

Industry

Small Business

For-Profit

Non-Profit

Who Benefits From Our Work?Who Benefits From Our Work?Our Stakeholders --Our Stakeholders --

• Historical focus on agricultural & rural landowners, citizens, businesses & communities.

• Today, all Americans who utilize or benefit from sound agricultural, natural resources & environmental management.

• Members of The Partnership.• Secondary, college & university

students.• Socially disadvantaged citizens.

CSREES Functions & IntegrationCSREES Functions & Integration

• Integration of Research, Education & Extension is an overall CSREES goal.

• Many funding lines are for specific mission elements.• Portfolio supported largely by formula and competitively

awarded funds.• Some units focus on single agency functions (e.g.

Competitive Programs).• Other program units (e.g., NRE, PAS, SERD) integrate

multiple functions (research, teaching, and extension). • Trend toward greater NPL functional fluidity & inter-Unit

cooperation.

CSREES Problem Areas (PAs)CSREES Problem Areas (PAs)

• 84 R, E & E “programs” described in 5 goals of CSREES 2004-2009 Strategic Plan.

• Derived from USDA Current Research Information System (CRIS): 80 numerical Research Problem Areas (RPAs).

• System used for data organizational & retrieval purposes.

• PA concept is an attempt to link CSREES strategic plan to activities.

Problem Areas (cont’d)Problem Areas (cont’d)

• PAs do not directly reflect organizational structure or staffing.

• CSREES doesn’t staff or plan using PAs.

• PAs (and CRIS RPAs) expected to “evolve” over time as issues change.

• PAs will stabilize and provide a consistent arrangement and connection to strategic plans and implementation of activities.



Logic Model, Accomplishments, Logic Model, Accomplishments, and and

OMB CriteriaOMB Criteria

Logic ModelLogic Model• A program leader’s model for the

program theory.• Illustrates the relationships

between the components of the strategy for the portfolio.

• Specifies potential cause and effect relationships.

• Serves as the framework for developing a program evaluation plan.

• Links the components with intended outcomes.

Ref: E. Norland, US EPA.

Conduct workshops, meetingsDeliver servicesDevelop products, curriculum, resourcesTrainProvide counselingAssessFacilitatePartnerWork with Media

Inputs Outputs

Activities Participation

Outcomes-Impact

Short term Medium Term Long Term

Assumptions External Factors

EvaluationFocus - Collect Data – Analyze and Interpret - Report

What we invest

Staff

Volunteers

Time

Money

Research base

Materials

Equipment

Technology

Partners

What we do Who we reach

Participants

Clients

Agencies

Decision-makers

Customers

Satisfaction

What the short term results are

Learning

Awareness

Knowledge

Attitudes

Skills

Opinions

Aspirations

Motivations

What the medium term results are

Action

Behavior

Practice

Decision-making

Policies

Social Action

What the ultimate impact(s) is

Conditions

Social

Economic

Civic

Environmental

Situation

Needs and Assets

Symptoms versus problems

Stakeholder engagement

Priorities:

Consider:

Mission

Vision

Values

Mandates

Resources

Local Dynamics

Collaborators

Competitors

Intended Outcomes

Program Action- Logic Model

Program Development Planning-Implementation-Evaluation

Generic Logic ModelGeneric Logic Model



Logic Model Integrating Research, Extension, and Education

Financial Resources

External Factors

Situation Inputs

Human Resources

Outputs

Outcomes

Short Medium Long

Source: Planning and Accountability, 2004

Target Audience

Assumptions

Proposals:

•Submitted•Reviewed

•Accepted forfunding

•Awarded•Implemented Educators

Paraprofessionals

Practitioners

Producers

Researchers

Extension

Education

OutcomesOutreach

Learning Action Change

Research

Portfolio Review –Portfolio Review –What It EncompassesWhat It Encompasses

All Functions– Research– Education– Extension

Across All Budget Lines– Smith-Lever Act– Hatch Act– Evans-Allen– RREA– McIntire-Stennis Cooperative Forestry Research– National Research Initiative– Small Business Innovation Research– Higher Education Programs– Congressionally-Directed Projects– Etc.

Portfolio AccomplishmentsPortfolio Accomplishments

– New knowledge and discoveries from research activities provide new management practices.

– Extension programs provide the technology transfer to enable adoption of new practices.

– Higher education programs accelerate the development of new curriculum and the training of new scientists and food, agriculture, and natural resources professionals.

What does this mean?What does this mean?

• The “work” of the agency is the work of the partners.

• CSREES facilitates that work through numerous funding mechanisms.

• Assessing the accomplishments of the portfolio involves examining the work of the partners and the agency leadership and guidance that accompanies the funding.

Assessing Programs for Assessing Programs for the Office of Management and Budgetthe Office of Management and Budget

Relevance (scope – focus – emerging issues – integration – balance):

– Are the problems being addressed the relevant issues of the day?

– Are they being addressed in a broad, interdisciplinary manner?


Quality (outputs and findings – assessment – alignment – methodologies):

– Are the outputs and findings significant?– Do stakeholders participate in assessing

the work and have opportunities to provide input?

– Are appropriate methodologies used in carrying out the research, education and extension work?


Performance (productivity – completeness – timeliness – guidelines – accountability)– Is productive work being accomplished:

• New discoveries?• Changes in attitudes, knowledge practices?• Competent faculty and workforce ready graduates?

– Does work get done in a timely manner?– Does the agency provide sufficient guidance in

order for productive accomplishments to be achieved?



Budget and Performance Budget and Performance IntegrationIntegration

New Directions and Expanded Efforts: New Directions and Expanded Efforts: Program ManagementProgram Management

• Strategic Planning/Directions - most programs have no strategic plans

• Program Accountability:– Budget and performance integration:

• Department and agency goals.• Performance criteria – measures, indicators,

baseline values and targets for out-years.• Logic model.



Logic Model Integrating Research, Extension and Education

Financial Resources

External Factors

Situation Inputs

Human Resources

Outputs

Outcomes

Short Medium Long

Source: Planning and Accountability, 2004

Target Audience

Assumptions

Proposals:

•Submitted•Reviewed

•Accepted forfunding

•Awarded•Implemented Educators

Paraprofessionals

Practitioners

Producers

Researchers

Extension

Education

OutcomesOutreach

Learning Action Change

Research

Example of Performance Integration

Goal

Inputs

Outputs Outcomes

USDA Strategic

Objective 5Agency Goal

Performance Criteria

Activities Participants Short Medium Long

Protect and Enhance the Nation's Natural Resource Base and Environment

Goal 5.1: Provide science-based knowledge and education to improve management of forest and rangelands

5.1.3 Urban Forestry (PA 124)

$$$ Quantify the volatiles emitted by pines which are not susceptible to attack by the Nantucket pine tip moth

Increased knowledge of patterns of infestation

Developed semichemical based monitoring and protective techniques to enhance integrated control strategy

Identified distinct differences in the olfactory look of loblolly and slash pines

Example of an Activity Data Matrix

Problem Area 124: Urban Forestry

Performance Baseline FY06 FY07 FY08

Project/Objective Measure Indicator Year Value Target Actual Target Actual Target Actual

Quantify the volatiles emitted by pines which are not susceptible to attack by the Nantucket pine tip moth

Measure emission of susceptible pines

# of units of measurement

Measure emission of non-susceptible pines

# of units of measurement



Summary and DiscussionSummary and Discussion

SummarySummary

• Process is useful and enlightening for National Program Leaders.– What we know.– What we don’t know.– What we thought we knew but had not

compiled the data.

SummarySummary

• Reviews have utility not only for the Office of Management and Budget but also for the Agency and National Program Leaders.– Program direction – RFA’s; strategic

planning; joint planning– Program integration – strategies for

which, when and how– Planning and reporting – what is

reasonable and affordable

SummarySummary

• Accountability will continue to be emphasized. – Which programs are making a difference and

which are not?– Budget decisions are being made based on

program performance.– Budget and program integration (BPI).

SummarySummary

• We need to negotiate and collaborate with our partners to identify performance indicators that are relevant, reportable and meaningful.

Thank-You!!!Thank-You!!!

• CSREES extends our thanks and appreciation for the work that you are undertaking and the outcomes that it will produce.

Questions???Questions???

planning and evaluation: portfolio assessment for csrees strategic objectives

Documents

work of csrees

missionrelevant goalspromote

education economicsfood

federal entitiesusdausda

problem areasportfolio5

fy2007 presidents budget

fy2006 presidents budget

federal assistance agency