IPSP Outcomes Reporting FrameworkWhat you need to know and what you need to do

Measuring and monitoring outcomes: Why are we doing this?Measuring and monitoring outcomes enables us to identify:• If/how the IPSP leads to changes in thinking and practice, and

the extent of this change• If/how program elements work together to produce outcomes• The types of support and professional development that

produce the best outcomes• Opportunities for a nationally consistent approach, while

retaining flexibility to be regionally responsive• Most importantly, opportunities for the continuous

improvement of the program.

Elements of the FrameworkThe IPSP Service Process Logic (ie: the “program logic”)

• The service process logic is drawn from IPSP guidelines and summarises how IPSP is theorised to work in terms of:• Inputs (forms of investment in IPSP)• Processes (accountability and compliance; internal

IPSP management; IPSP planning and delivery)• Outputs (IPSP services delivered)• Impacts (short-medium term changes)• Outcomes (longer-term changes)

NOTE: the logic also notes external variables that influence the achievement of outcomes

Elements of the FrameworkThe IPSP Service Process Logic (ie: the “program logic”)

• At an overarching level, understanding the service process logic allows us to ask questions about IPSP in terms of:

• the implementation of IPSP (with a focus on continuous improvement in service delivery)

• the logic of IPSP (with a focus on continuous improvement in the design of IPSP in the future)

Elements of the FrameworkThe IPSP Performance Indicator Framework

• The IPSP Performance Indicator Framework outlines the indicators of program equity, effectiveness and efficiency

• Indicators include:• Output indicators (indicators of accessibility,

appropriateness and quality of the services delivered)• Impact indicators • Outcome indicators

NOTE: Not all indicators can be easily measured

Elements of the FrameworkData Collection Instruments (across all IPSP elements – PSC, ISA, IPSU)

Layer Data Collection Instrument Timeline

Planning Needs Analysis Survey (All program elements) April-May 2014

Needs Analysis Consultation (All program elements) Undertaken by IPSP providers

Formative Feedback

Reflective Conversations (All program elements) On-going July 2014 – June 2016

Initial Feedback Survey: Inclusion Support (ISA) On-going July 2014 – June 2016

Initial Feedback Survey: Professional Development (PSC & IPSU)

On-going July 2014 – June 2016

Follow-up Feedback Survey (PSC & ISA) On-going July 2014 – June 2016

Summative Reporting

Annual Outcomes Focus Groups (All program elements) March 2014 & 2015

Annual Outcomes Survey (All program elements) September 2014 & 2015

IPSU Data collection

1. Initial feedback survey2. Reflective Conversations

ApproachOur focus here is to approach this in ways that:• are culturally appropriate (recognising

that surveys are a mainstream cultural instrument)• are practical and do not detract from

service delivery capacity• assist with building capacity for

engaging with the National Quality Frameworks

Your Outcomes Reporting ToolkitShift Consulting has developed a toolkit for providers in consultation with the IPSU Outcomes working group which includes:•Guidelines for implementation• This PowerPoint• Information for educators fact sheet• Reflective conversations resource for

educators•Hard copy survey• Reflective conversation data sheets

IPSU Feedback Surveys

• Initial feedback is collected through a short survey. • The survey can be administered:•On-line•Off-line (eg: via an IPSU ipad)•Paper-based (and answers entered by the IPSU into the survey platform)

IPSU feedback surveyIs to be undertaken as follows:• With services on your first visit after July 2014• With Coordinators/educators when they are new

Links to surveysOnline versionhttps://www.surveygizmo.com/s3/1700022/IPSUFeedback

Offline versionhttp://www.surveygizmo.com/s3/offline/index.php?__slug=www.surveygizmo.com/s3/1700022/IPSUFeedback&sLanguage=Auto

How to get the survey data into our system: Online modeSurvey undertaken by educators in online mode

• They can do it alone of with your assistance

• Requires internet connection

They are given a link

by you

They go online and fill out the

survey

Their completed survey is in the system

How to get the survey data into our system: Offline modeSurvey undertaken by educators with your assistance in offline mode

• They can do it alone with your assistance

• Does not require internet connection but does require laptop, ipad or smartphone

They are given a link by you or

you sit together

They follow the link

and fill out the survey

You later upload the survey into the system

How to get the survey data into our system: hard copySurvey undertaken by educators by filling out a hard copy form

• They can do it alone

• Is handed to you at the conclusion of the support visit

• Later, you enter the survey into the system (offline mode quickest)

You provide hard copy

survey

They fill out the survey and hand it back to you

You later enter the

survey into the system

Why Reflective Conversations?The data gathered through the ‘reflective conversation’ instrument provides us (IPSP) with a means to build knowledge and understanding of the process of change in the IPSP context.Obviously, in the course of facilitating a conversation, “change” becomes contextualized with reference to the specific service environment, children, families and educators.

Reflective ConversationsA reflective conversation is a guided conversation that encourages educators to: Reflect on

change

Identify barriers

Identify strategies

Plan for positive change

Implement plans & strategies

Reflective Conversations Documenting reflections (targeted at educators)• The key to conducting a good reflective conversation is active listening and responsively guiding the conversation in ways that encourage critical reflection, problem solving and planning. • Documenting key reflections and ideas along

the way provides the facilitator and the educator(s) with a framework for drawing links between ideas and critically reflecting on these over time.•  Documenting the conversation is modelling the

same process as an educator might use to document children’s learning

Reflective Conversations Where educators begin• How a conversation about change occurs and the

depth of reflection that will occur is influenced by where educators begin.• An educator’s “level of reflection” will influence

how the reflective conversation will take place.• The role of a professional facilitating a reflective

conversation is to assist the educator to reflect more deeply and grow their reflection and understanding

Bain’s Critical Reflection Framework

Level 1: REACTING• Focuses on own

feelings• Offers straight forward

description• Seeks basic

understanding

Level 2: ELABORATING• Compares self with

others• Considers alternatives• Seeks deeper

understanding

Level 3: RECONSTRUCTING• Draws new conclusions

about practice• Explores relationships

between theory and practice

• Drives own learning

(Miller, 2011, p. 6)

Reflective Conversations Levels of Reflection

Level 1: REACTING• Focuses on own feelings: Commenting on

feelings related to an event• Offers straight forward description: Providing a

straight description with no added observation or insights

• Seeks basic understanding of what occurred and why

Reflective ConversationsLevels of Reflection

Level 2: ELABORATING• Compares self with others: Comparing one’s reaction

to other people’s responses and perspectives• Considers alternatives: Analysing an event by asking

questions and considering alternatives• Seeks deeper understanding of an event by relating

it to current literature and theories

Reflective ConversationsLevels of Reflection

Level 3: RECONSTRUCTING• Draws new conclusions about practice: Drawing

revised and new conclusions about own practices and the practices of others

• Explores relationships between theory and practice: Ongoing exploration of relationships between practice, literature and theory

• Drives own learning: Planning further learning on the basis of your reflections.

Reflective ConversationsA frame of reference for facilitators• Conversations flow… sometimes people offer up

information without prompting… at other times they need significant support to draw out the details.

• There is a set of questions that can provide a facilitator with a frame of reference to support a holistic conversation.

• The key is that the facilitator must keep an ear out for the what information has been offered up and where they need to ask questions and encourage further reflection.

• The documentation tools provided can assist with this process.

Reflective ConversationsReference Questions – Initial Visit (from July 2104)• What are the most significant learnings you

have drawn from today?

• Are there any ways in which you have changed the way you think about [topic]?

• How do you think you will continue to use what you have learned from today?

Reflective ConversationsReference Questions – Follow-up visitWhat is the most significant change that has taken place as a result of receiving this support? Think about (encourage reflection on)• The way you think about your role/approach/practice?• Your relationships with others such as your team members,

children and families?• The practices that you use in relation to working with team

members, children and families?

• Have you experienced any barriers to implementing change in your practice?

• How have or could these be overcome?

• How do you think you will continue to use what you have learned in the future?

How to get the Reflective conversations data into our system:Utilise existing methods of engaging in reflective conversations with educators. This is NOT a structured interview and the questions are there to guide your conversation.

• Document the conversation in terms of the reflective questions and themes outlined in the datasheet

Gather data using

existing reflective

conversations strategies

Transfer information to reflective conversations data sheet

Forward to Shift

Consulting as required (3 monthly)

Got any questions?

Please forward any questions through the IPSU IPSP Outcomes working group – members are Chris, Kate and Peter.