a framework for measuring equity performance in australian ... · a framework for measuring equity...

Forward Plan for Data and Statistics

Make tomorrow better. ncsehe.edu.au

Tim Pitman and Paul Koshy, November 2015

A FRAMEWORK FOR MEASURING EQUITY PERFORMANCE IN AUSTRALIAN HIGHER EDUCATIONIdentification and Enhancement

curtin.edu.au

A Framework for Measuring Equity Performance in Australian Higher Education: Forward Plan for Data and Statistics Identification and Enhancement

National Centre for Student Equity in Higher Education (NCSEHE), November 2015 1

Contents

Introduction ................................................................................................................................... 2

The Forward Plan for Data and Statistics Identification and Enhancement ............................. 3

An Outline of the Equity Performance Framework for Australian Higher Education ............ 3

Issues from the Wider Consultation and Testing Phase ......................................................... 7

Outcomes of the Testing Phase ................................................................................................ 8

Key Findings from the Testing Phase..................................................................................... 10

Options for the Implementation of the Framework .................................................................. 13

Option 1 (Continue Reporting in the Current Context) ......................................................... 15

Option 2 (Separate Reporting of Equity Performance Data) ................................................ 18

Option 3 (Development of a Standalone ‘Equity Report’) ..................................................... 20

The Requirements of Implementation ....................................................................................... 26

Contact Details............................................................................................................................ 26



Introduction

In June 2013, the Australian Government Department of Education, now the Australian Government Department of Education and Training (“the Department of Education and Training”), commissioned the National Centre for Student Equity in Higher Education at Curtin University (NCSEHE) to provide further detail on the proposed Equity Performance Framework (“The Framework”), its structure, overall logic and potential data sources.

The Framework is intended to:

• provide a set of Indicators that will allow the measurement of institutional and system-wide achievement in the higher education sector against the Government’s commitments, targets and goals in relation to equity;

• inform policy through the provision of relevant information to support the development of evidence‐based policy;

• foster informed debate through the provision of key information; • provide a platform which will guide evaluation of programs by government and

activities by Institutions; and • inform practice within, and support equity in, the higher education system.

The Framework will report on specific institutional and system-wide performance among higher education providers in terms of the access, participation and performance of six groups of under-represented students (equity groups) in higher education:

• Low socio-economic status (LSES) students; • Students with disability; • Indigenous students; • Students from regional and remote areas; • Women in non-traditional areas of study (WINTA); and • Students from a non-English speaking background (NESB).



The Forward Plan for Data and Statistics Identification and Enhancement

This document, the Forward Plan for Data and Statistics Identification and Enhancement (the “Forward Plan”) for the Framework, outlines key issues in regards to:

• An outline of the final form of the Framework; • Issues from the Wider Consultation and Testing Phase; • Options for the rollout of the Framework; and • Requirements of Implementation.

This document provides implementation options in the context of the work undertaken on the Framework to date, as well as outlining key issues to be addressed during implementation planning.

An Outline of the Equity Performance Framework for Australian Higher Education

The Framework is outlined in the two project documents delivered to date:

• A Framework for Measuring Equity Performance in Australian Higher Education, Draft Framework Document, V1.8 (the “Framework Document”); and

• A Framework for Measuring Equity Performance in Australian Higher Education: Draft Data Indicator Dictionary, V1.8 (“Data Dictionary”).

The Framework document outlines the structure of the proposed Framework for Equity Performance, highlights existing gaps in the collection of data and outlines potential areas for collection of data and indictors to cover gaps. The Data Dictionary provides detail on the location and collection requirement for each identified Indicator.

The Framework reports a series of Indicators for equity performance in higher education. It is hierarchical in structure and is comprised of three Tiers:

1. Context (Pre-higher education) – covering pre-primary, primary and secondary education results;

2. Performance (Higher education) – covering all 129 Australian higher education providers (see Appendix B) and university students at all levels of study; and

3. Outcomes (Post-higher education) – covering graduates from higher education.

Each Tier has related Domains and within each Domain are specific Indicators that measure higher education equity performance. Underlying each Indicator are data which represent the measurement of that Indicator.

Figure A provides an overview of the structure of Tiers, Domains and Indicators.



Within and across the tiers and domains, a total of 28 indicators are developed to measure higher education equity performance. In addition, the Framework also reports an “Input Domain”, which includes two indicators, for funding and activity respectively, of inputs into equity in higher education.

The 28 final indicators largely rely on existing data sourced from key (primarily educational) stakeholders who have in place rigorous and systematic data reporting processes and protocols. More specifically:

• Fifteen indicators use existing data and data protocols; • Four indicators rely on existing data and protocols, however the suitability of the

data (for the purposes of the Framework) is not ideal; • Three indicators will require new data and protocols to be developed, principally

because of the transition to a new collection instrument – the Graduate Outcomes Survey (GOS); and

• Six indicators require the development of new data sources and protocols, in addition to relying on existing data with the development of new protocols.

One original indicator, 5.02 (Acceptances by students) as shown in Figure A, has been removed on the grounds that it replicated reporting of another, Indicator 5.03 (Commencements (new students)).

Of the 28 indictors, the 11 in Domains 1 to 3 of Tier 1 relate to broader measurements of preparedness for higher education, indicators in Domains 4 to 6 of Tier 2 report on access and performance in higher education, across all levels of study – sub-bachelor, bachelor, postgraduate coursework (postgraduate) and higher degree by research (HDR). Indicators in Tier 3’s Domain 7 will report data on graduate outcomes.

The Input Domain has two indicators: the first records identified Commonwealth resourcing for higher education equity programs; and the second reports on institutional program structures and activities, for instance, the number of schools in a given outreach program. It is envisaged that data for the Input Domain will be collected via the Single Equity Report (SER), to be managed by the Department of Education and Training.

The Framework will allow the Department of Education and Training to publish aggregated and disaggregated data. It is proposed that the tables will be made publicly available for researchers and other stakeholders to conduct bespoke analysis. Individual Domain reports should be released annually with a more comprehensive, comparative analysis released once every five years.



Figure A: Equity Performance Framework for Australian Higher Education

KEY Regular text = Indicator uses existing data and protocols Blue-coloured underlined text = Indicator uses existing data and protocols but data are potentially too weak for the purposes of the Framework Red text shaded grey = Combination: some requirement for data and protocols to be created but also uses existing data but data protocols do not currently exist Red-coloured italics text = Indicator requires data and protocols to be created Black italics text on grey = Removal of indicator from the Framework after testing.

TIER 3: OUTCOMES (Post-higher education)

Domain 7

Graduate outcomes

7.01 Graduate earnings (GDS) 7.02 Graduate employment (GDS) 7.03 Graduate further study(GDS) 7.04 Graduate satisfaction (GDS)

TIER 2: PERFORMANCE (Higher education)

Domain 4 Aspirations for higher education

Domain 5 Access to higher education

Domain 6 Achievement in higher education

4.01 Intentions to undertake higher education studies 4.02 Year 12 applications 4.03 Non Year 12 applications

5.01 Offers made to students 5.02 Acceptances by students 5.03 Commencements (new students) 5.04 Enrolments (all students) 5.05 Student Satisfaction (UES) 5.06 Course transitions

6.01 Student retention 6.02 Student success 6.03 Student completion

TIER 1: CONTEXT (Pre-higher education)

Domain 1 Early childhood development

Domain 2 Primary education

Domain 3 Secondary education

1.01 Early childhood development (physical, social and learning)

1.02 Participation in pre-school

2.01 Reading (NAPLAN) 2.02 Numeracy (NAPLAN) 2.03 School attendance

3.01 Reading (NAPLAN) 3.02 Numeracy (NAPLAN) 3.03 School attendance 3.04 Year 12 completion 3.05 ATAR 3.06 Achievement in advanced maths and science (PISA)

INPUTS INTO EQUITY

Domain I Equity Inputs

I.01 Higher education equity funding

I.02 Outreach activities (number of students and organisations reached)



Importantly, the Framework has been designed to take into account future expansions in collections in all three contexts:

• An increase in the number of equity groups identified in higher education: This topic will be the subject of an independent investigation project funded by the Department of Education and Training in 2016. As the Framework is drawn from HEIMS primarily, it is well placed to be adapted to include new groups in the context of higher education-related collections (e.g. access; participation; graduation).

• The expansion in data collection for a given Indicator: The Framework draws on several data sources which are either: being expanded in terms of their scope (HEIMS collections), being developed as of 2015-16 (GOS), or proposed from 2016 onwards (the Widening Participation Longitudinal Survey). It is flexible enough to accommodate these changes given its theoretical reporting structure and established Data Indicator Dictionary.

• Additions of new Indicator types: The Framework was expanded in 2015 to include reference to ‘equity inputs’ which include funding from the Department of Education and Training to institutions, as well as self-reported institutional activities such as equity-related staffing and levels of student engagement. It is possible that this could be extended to include ‘output’ measures such as equity Indicators among staff.



Issues from the Wider Consultation and Testing Phase

The Wider Consultation and Preliminary Testing phase of the Framework project was conducted between November 2014 and March 2015. The latest version of the Framework document (V1.8) reflects the current structure of the Framework in view of feedback on the initial design and the results of a preliminary test on the collection of data. The discussion below catalogues the outcomes of this process. In the first instance, the Testing Phase involved the interrogation of key identified data sources for equity in education in Australia. These are reported in Figure B below.

Figure B: Data Sources Investigated During the Testing of the Framework

Data Source Agency Reporting

Format Data Availability Reportin

g Tier

Australian Early Development Census (AEDC)

Social Research Centre (SRC)

.xls flat file/ Template

Three yearly; 2015 next year, report in 2016. Context

Report on Government Service (ROGS),

Productivity Commission (PC)

.xls flat file/ROGS website

Annual; ROGS released in March Context

National Assessment Program for Language and Numeracy (NAPLAN)

Australian Curriculum Assessment and Reporting Authority (ACARA)

.xls flat file/NAPLAN

website Annual; NAPLAN released in December Context

PISA Test

Australian Council for Educational Research (ACER); Reported in ROGS

.xls flat file/ROGS website

PISA – 15 year olds test – three yearly; 2015 next year Context

Longitudinal Study of Australian Youth (LSAY)

National Centre for Vocational Education and Research (NCVER) Request

LSAY last cohort in 2009 Context

Higher education applications data

Department of Education and Training (Department)


Annual; finalised in December Context

Student Enrolment and Performance Data Department


Annual; finalised in August

Performance

Graduate Destination Survey (GDS)

Graduate Careers Australia (GCA)


2015-16, last GDS survey Outcomes

Beyond Graduation Survey (BGS)

Graduate Careers Australia (GCA)


2015-16, last GDS survey Outcomes

University Experience Survey (UES) SRC

.xls flat file/ Template Annual; March Outcomes

Graduate Outcomes Survey (GOS) SRC


Annual; at design; Commences in 2016 Outcomes

Employer Satisfaction Survey (ESS) SRC

.xls flat file/ Template Unknown; at design Outcomes

Single Equity Report (SER) Department

.xls flat file/ Template Unknown; at design Outcomes



Outcomes of the Testing Phase

The outcomes from testing are reported in Figure C below, where this matches each of the identified Indicators with the above data source and provides information on the testing outcome in terms of data availability and protocols for collecting data. A number of observations can be drawn from the testing:

• For pre-higher education collections (“Context”), there is little or no comparability in terms of reporting for equity groups as identified in higher education, with the exception of disaggregated reporting on Indigeneity. The identification of socioeconomic status is generally not exactly consistent with that reported in higher education, while definitions of disability and NESB tend to be self-reported. As a result, Indicators in the Context Domain provide some indication of current trends in the system using definitions of equity which are broadly consistent with those in the Framework but which offer no chance of direct comparability.

• Nationally consistent data for Indicators 3.04 (Year 12 completion) and 3.05 (ATAR) are missing at the public reporting level. The first Indicator has been somewhat beset by a range of discrepancies in reporting between states and territories, brought about by specific differences across jurisdictions, be it differences in the linkages between secondary and introductory tertiary studies (TAFE) or a lack of coordination in collecting Year 12 completion data in a nationally consistent fashion. Similarly, ATAR data is not easily reported at the national level, except in the context of applications data which is collected from TACs and institutions by the Department of Education and Training. It is likely that both 3.04 and 3.05 could be reported upon under separate collection arrangements, ideally with uniform equity definitions applying at both secondary and higher education levels.

• The biggest gap in the data collection occurs in Domain 4 (Aspirations for higher education), specifically in relation to Indicator 4.01 (Intentions to undertake higher education studies), where the closest data source for this measure, the Longitudinal Study of Australian Youth (LSAY), was last undertaken in 2009 by the NCVER, with the 2012 cohort study not be undertaken and considerable doubt being cast on the 2015 study. This creates a gap in the data, which could best be served through the introduction of a new survey instrument to survey Year 10 and 12 students in relation to their aspirations for non-compulsory education and training.

• In regards to Domains 5 (Access to Higher Education) and 6 (Achievement in Higher Education), collection of data from the higher education system is administratively straight-forward from the Data Analysis Branch of the Department of Education and Training, with internal costing being the only issue (~$1,350 for the testing request).



Figure C: Outcomes from Testing of the Equity Performance Framework for Australian Higher Education

Indicator Data/Agency Data Equity Groups Protocols 1.01 Early childhood development (physical, social and learning) AEDC/SRC Yes All, except

Disability; WINTA AEDC – 3

year census

1.02 Participation in pre-school ROGS/PC Yes LSES; Indigenous; Special Needs. Yes

2.01 Reading (NAPLAN) (Yrs 3 & 5) NAPLAN/ACARA Yes All, except Disability, WINTA Yes

2.02 Numeracy (NAPLAN) (Yrs 3 & 5) NAPLAN/ACARA Yes All, except

Disability, WINTA Yes

2.03 School Attendance ROGS/PC Yes Indigenous Yes

3.01 Reading (NAPLAN) (Yrs 7 & 9) NAPLAN/ACARA Yes All, except Disability, WINTA Yes

3.02 Numeracy (NAPLAN) (Yrs 7 & 9) NAPLAN/ACARA Yes All, except

Disability, WINTA Yes

3.03 School Attendance ROGS/PC Yes Indigenous Yes 3.04 Year 12 completion -- No -- -- 3.05 ATAR -- No -- --

3.06 Achievement in advanced maths and science PISA/ACER Yes

LSES; Indigenous; Regions; language

background Yes

4.01 Intentions to undertake higher education studies LSAY/NCVER No -- --LSAY in

abeyance

4.02 Year 12 applications Department Yes LSES; Indigenous; Regions Yes

4.03 Non- Year 12 (non-school leaver) applications Department Yes LSES; Indigenous;

Regions Yes

5.01 Offers made to students Department Yes Yes Yes 5.02 Acceptances by students (removed from Framework) Department -- -- --

5.03 Commencements (new students)

Department Yes All Yes

5.04 Enrolments (all students) Department Yes All Yes 5.05 Student satisfaction (UES) – Yes Yes Yes 5.06 Course transitions Department No Possible Yes 6.01 Student retention Department Yes All Yes 6.02 Student success Department Yes All Yes 6.03 Student completion Department Yes All Yes

7.01 Graduate earnings GDS/GCA; GOS/SRC Yes Likely in future Likely

7.02 Graduate employment GDS/GCA; GOS/SRC Yes Likely in future Likely

7.03 Graduate further study GDS/GCA; GOS/SRC Yes Likely in future Likely

7.04 Graduate satisfaction UES/SRC Yes No; Likely in future Yes

I.01 Higher education equity funding SER/Department Yes No Likely


SER/Department Yes Not Likely Likely



KEY

Regular text = Indicator uses existing data and protocols Blue-coloured underlined text = Indicator uses existing data and protocols but data are potentially too weak for the purposes of the Framework Red text shaded grey = Combination: some requirement for data and protocols to be created but also uses existing data but data protocols do not currently exist Red-coloured italics text = Indicator requires data and protocols to be created Black italics text on grey = Removal of Indicator from the Framework after testing.

Key Findings from the Testing Phase

The key findings from the Testing Phase are: Scope for Institutional Reporting: The critical issue in Domains 5 and 6 is the extent to which reporting coverage can be ensured across the entire higher education sector – all 129 higher education providers and not just the 38 Table A providers (including the Batchelor Institute). For reasons of data record confidentiality, the Department of Education and Training does not report student number counts of less than 5, although it does report counts of 0. This means that at even the aggregated level (equity group performance by institution), Non-University Higher Education Institutions struggle to report meaningful figures. For instance, a simple count of students by equity group in a given institution (see Figure D below) , saw only 16 out of 86 Non-University Higher Education Institutions report figures for Indigenous students, implying a student count of less than five – in many cases a count of zero. At even lower levels of analysis, for instance, equity students by field of study by institution, the level of reporting approaches zero, as no institution has enough students to generate a publicly reportable result. Ultimately, many sub-reports for higher education enrolments will only be adequately populated for the 38 Table A providers.

Figure D: Equity Students by Institution, Number of Institutions Reporting More than 5, By Equity Group

Table A

Providers Table B

Providers Table C

Providers

Non-University

Higher Education

Institutions Total Total Number of Institutions 38 2 3 86 129 Number of Institutions reporting a student count greater than 5 for an Equity Group – Students from a Non English speaking background 37 2 1 40 80 Students with a disability 37 2 1 48 88 Women in Non-Traditional Area 37 1 2 44 84 Indigenous 38 1 1 16 56 Low SES (SA1 measure) 38 2 1 76 117 Regional (2011 ASGS) 38 2 1 66 107 Remote (2011 ASGS) 37 2 1 13 53



In addition, data on graduate earnings and satisfaction can be sourced from the Graduate Destination Survey (GDS) and University Experience Survey (UES) respectively. Testing of the GDS and UES data sets has found that institutional averages for starting salary and student satisfaction are easily reportable at the institutional (Table A) provider level. However, reporting for individual equity groups at each institution is unreliable due to small sample size, although reporting on these groups at the aggregate level is acceptable.

Recommendation: In the first instance, reporting on student outcomes remains restricted to Table A providers.

Scope for Equity Group Reporting: Similarly, as Figure D shows, reporting for two equity groups (Indigenous; Remote) was particularly patchy across the system, due to the relatively small number of students in both groups. For this reason, the Framework may be thought of as being particularly applicable to the Low SES and Regional groups in particular, with reporting for NESB, WINTA and Students with Disability seeing reasonable institutional response rates.

Recommendation: The Framework can continue to report across all equity groups but with some consideration required for more specific reporting on smaller equity groups (NESB, WINTA and Students with Disability). Reporting on Inputs: Input Indicators can be sourced from the proposed SER collected by the Department. This collection will allow institutions to report on funding (received and spent) and activities in relation to equity programs, although at this point it is focused on outlook programs, with no immediate indication that it will collect data on bridging and enabling programs once students enter the system. However, as this item is still at the design and feedback stages, it is difficult to confirm the availability of data for future collections. Current Departmental collections would only partially cover I.01 (Higher education equity funding) and then only for aggregate funding for equity programs, with no further breakdown of expenditure.

Recommendation: The Framework can report on input measures. Reporting Scope: The core of the Framework lies in the Indicators reported in Tiers 2 and 3. The Testing phase has indicated that reporting for all institutions may not be possible on a consistent basis because of the small number of enrolments in many Non-University Higher Education Institutions, which do not allow for disaggregation at the equity group levels at even the total enrolment level, much less the level required of any sub-reports, for instance by Field of Study or Level of Study. For this reason, some consideration will have to be given to the relative merits of a ‘one size fits all’ approach to reporting.



Recommendation: The core recommendation is that initial implementation focus on reporting for Table A institutions across all equity groups and for all institutions, only focussing on equity groups with larger populations: Low SES and Regional Students, with a separate reporting mechanism being developed for Indigenous Students; NESB; WINTA; Students with Disability; and Students from Remote Areas in view of the particular policy requirements and data collection limitations of each.

Template Format: The Testing phase found that the Framework will source two types of data:

• Pre-higher education data: These data sets provide context for the equity issues confronting higher education policymakers. However, they are generally not easily connected to one another or higher education data via a consistent reporting format.

• Higher education data: All proposed collection of higher education data can be

sourced directly from, or matched to, the HEIMS system. This will allow for consistency in the definition of equity status across these collections and for relatively easy reporting of data at the institutional level.

It is conceivable that the entire Framework could be reported as a flat file via a web interface, with sections of the institutional reporting being shared with other Department of Education and Training projects, namely the Quality Indicators for Learning and Teaching (QILT) and/or the Datamart project.

In addition, a more involved reporting template, for instance, an online ‘data dashboard’ could be developed in conjunction with the Datamart project. Examples of such an approach are included below in the discussion of options as part of this Forward Plan.

Recommendation: Discussed as part of Options 1 to 3 below.

Reporting Timeframe: This is largely determined by institutional reporting, with the previous year’s enrolment usually confirmed and released by August of the following year. For instance, the enrolment data for 2014 will be finalised by August 2015, which enables a compilation of the Framework annual reporting of 2014 by November 2015.

Recommendation: Implementation will establish data collection timelines and protocols for an annual collection and reporting of data.


National Centre for Student Equity in Higher Education, November 2015 13

Options for the Implementation of the Framework At present, equity performance data in higher education are prepared by the Data Analysis Section of the Department of Education and Training and reported in the annual release, Selected Higher Education Statistics – Student Data, under Appendix 5: Equity Performance Data (“Appendix 5”). This release takes place in a flat file format (.xls) for viewing in Excel.

Figure E below outlines the Indicators in the proposed Framework, and identifies those which are currently reported in Appendix 5. These variables are also largely used to calculate and report related Indicators, such as participation and performance ratios. These Indicators are largely restricted to access and participation (enrolment) data across equity groups and form the basis for the new Framework.

In addition to these Indictors, the next column (‘Available Within the Higher Education Section”) indicates those data sources which are readily available in the Higher Education Section of the Department of Education and Training, including application data and graduate outcomes data. This excludes the entire ‘Context’ tier which draws on collections in other parts of the Department and which are not reported at the institutional level. and can be reported as part of the Framework.

The right hand column (“Can be Reported in Current Format”) subsets all internally available data for those that can be readily included in current formatting, effectively excluding input sources on funding and activities which typically are not reported in the context of student statistics.

The rows shaded grey encompass those Indicators which can readily be included in a reporting of the Framework as part of the current Appendix 5 reports each year. All other Indicators are either not directly related to higher education participation at the institutional level (Indicators 1.01 to 4.03) or require a ‘mixed’ reporting of financial or activity data (the input Indicators I.01 and I.02) which are not readily included in student statistics data.



Figure E: Data Requirements of the Equity Performance Framework for Australian Higher Education

Indicator Current

(Appendix 5)

Available Within Higher

Education Section

Can be Reported in

Current Format 1.01 Early childhood development (physical, social and learning) – – –

1.02 Participation in pre-school – – – 2.01 Reading (NAPLAN) (Yrs 3 & 5) – – – 2.02 Numeracy (NAPLAN) (Yrs 3 & 5) – – – 2.03 School Attendance – – – 3.01 Reading (NAPLAN) (Yrs 7 & 9) – – – 3.02 Numeracy (NAPLAN) (Yrs 7 & 9) – – – 3.03 School Attendance – – – 3.04 Year 12 completion – – – 3.05 ATAR – Yes – 3.06 Achievement in advanced maths and science – – –

4.01 Intentions to undertake higher education studies – – –

4.02 Year 12 applications – Yes – 4.03 Non- Year 12 (non-school leaver) applications – Yes –

5.01 Offers made to students – Yes Yes 5.02 Acceptances by students - -- Yes removed 5.03 Commencements (new students) Yes Yes Yes

5.04 Enrolments (all students) Yes Yes Yes 5.05 Student satisfaction (UES) – Yes Yes 5.06 Course transitions – Yes Yes 6.01 Student retention Yes Yes Yes 6.02 Student success Yes Yes Yes 6.03 Student completion Yes Yes Yes 7.01 Graduate earnings (GOS) – Yes Yes 7.02 Graduate employment (GOS) – Yes Yes

7.03 Graduate further study (GOS) – Yes Yes

7.04 Graduate satisfaction (GOS) – Yes Yes

I.01 Higher education equity funding – Yes –


– Yes –



Given the findings from the Testing Phase and the above example, three potential models for implementation of the Framework can be offered:

• Option 1 (Continue Reporting in the Current Context); • Option 2 (Separate Reporting of Equity Performance Data); and • Option 3 (Development of a Standalone ‘Equity Report’).

Option 1 (Continue Reporting in the Current Context)

The most immediate implementation option is for the Department to continue with its current reporting of equity outcomes as part of Selected Higher Education Statistics – Student Data, but with the addition of new Indicators on application data at the institutional level to provide more information on access issues and the inclusion of graduate outcomes (GOS data) to provide evidence for graduate outcomes, with Indicators for equity groups being reported.

The requirements of this option are as follows:

Coverage: Data Indicator coverage under this option is as follows:

Tier 1: Excluded, but with a separate reporting mechanism to be developed in collaboration with sections of Department of Education and Training with responsibility for schools, development and training as well as the Australian Bureau of Statistics.

Tier 2: Includes all current collections under Appendix 2 and 5, as well as new collections in relation to the following:

• Applications data by identifiable equity group by institution; and • Student Satisfaction (UES questionnaire outcomes).

Tier 3: Includes graduate outcomes data – employment, further study and earnings, sourced from the Graduate Outcomes Survey (GOS).

Under this proposal, Tier 1 reporting will not take place as part of the existing release, while input data collected by the Department will be reported separately by the Equity Section.

Format: Framework data will be prepared by the Data Analysis Section of the Department of Education and Training and reported in the annual release of existing spreadsheets as part of the Selected Higher Education Statistics – Student Data series:

1. Appendix 2: Equity Groups: For the 2014 release, please see: https://docs.education.gov.au/documents/2014-appendix-2-equity-groups

https://docs.education.gov.au/documents/2014-appendix-2-equity-groups



2. Appendix 5: Equity Performance Data: For the 2014 release, please see:

https://docs.education.gov.au/documents/2014-appendix-5-equity-performance-data

This approach retains the release’s focus on Table A providers, but with aggregate reporting for other institutions.

This release takes place in a flat file format (.xls) for viewing in Excel and includes institutional reporting of equity group Indicators in the following format:

The advantage of this format is that it uses an existing reporting mechanism within the Department though the Data Analysis Section.

However, there are several disadvantages of reporting in the Excel format. Chief among these is that Excel is not a database so accesses, parsing and matching data is difficult, with users having to do considerable work with the data to build even a simple analysis.

Resourcing: The additional reporting requirements under the proposed Framework are minimal in the context of current reporting and primarily involve additional data requests from two sources:

• Social Research Centre – Graduate Outcomes Survey data; and




• Applications data – sourced internally from the Department of Education and Training.

Stakeholders: The implementation of the Framework, within the context of the current reporting mechanism, will allow stakeholders (institutions, policymakers, and practitioners) to make comparisons at the institutional level while also assessing trends at the system-wide horizon in the context of a significantly more encompassing set of Indicators of student enrolment which will now provide evidence on access (application data) and student outcomes.



Option 2 (Separate Reporting of Equity Performance Data)

A second option, builds on Option 1 but also includes input data for institutions, allowing for the development of performance Indicators at the institutional level which link policy resourcing and equity enrolment outcomes. This option retains reporting of equity outcomes as part of Selected Higher Education Statistics – Student Data, but with the potential for the more policy-orientated report to be established separately from general student collections.

The requirements of this option are as follows:

Coverage: As per Option 1, Reporting in this context will necessitate an expansion in the number of Indicators being reported to include:

• Applications data by identifiable equity group by institution; • Graduate outcomes data (employment; further study; earnings) by institution; and • Input activity data.

In this proposal, Tier 1 reporting will still not take place, while input data collected by the Department will be reported in addition to other institutional-specific data.

Format: As with Option 1, Framework data will be prepared by the Data Analysis Section of the Department of Education and Training. The reporting format will be separated from, Selected Higher Education Statistics – Student Data, under Appendix 5: Equity Performance Data, and will include all Indicators covered in the Framework.

A. Current reporting in Appendix 2 and Appendix 5 of the Selected Higher Education Statistics – Student Data: Please see the following for details:

This release takes place in a flat file format (.xls) for viewing in Excel, with as part of the overall student data release or as a special data release with an exclusive focus on equity policy. As per Option (1), the best examples of this format can be found in the equity group releases under Selected Higher Education Statistics – Student Data:

• Appendix 2: Equity Groups: https://docs.education.gov.au/documents/2014-appendix-2-equity-groups

• Appendix 5: Equity Performance Data: https://docs.education.gov.au/documents/2014-appendix-5-equity-performance-data







Resourcing: The additional reporting requirements under the proposed Framework are minimal in the context of current reporting and primarily involve additional data requests:

• Social Research Centre – Graduate Outcomes Survey data; • Applications data – sourced internally from the Department; and • Input data – sourced internally from the Department of Education and Training.

Stakeholders: Option 2 retains the significant expansion in Indicators of student performance seen in Option 1, with the addition of data on inputs (financial and activity) which allows users to generate comparative assessments of performance in view of overall resourcing.



Option 3 (Development of a Standalone ‘Equity Report’)

Option 3 represents the most involved response to the Equity Performance Framework.

Coverage: Reporting in this context will necessitate an expansion in the number of Indicators being reported to include:

• Tier 1 Indicators; • Applications data by identifiable equity group by institution; • Graduate outcomes data (employment; further study; earnings) by institution; and • Input activity data.

In this proposal, Tier 1 reporting will take place, while input data collected by the Department will be reported in addition to other institutional-specific data.

Format: As with (1) and (2), framework data will be prepared by the Data Analysis Section of the Department of Education and Training. The reporting format will be separated from Selected Higher Education Statistics – Student Data, under Appendix 5: Equity Performance Data, and will include all Indicators covered in the Framework.

A number of formats exist for reporting under Option 3. These include:

A. Current reporting in Excel spreadsheets as per Appendix 2 and Appendix 5 of the Selected Higher Education Statistics – Student Data: Please see the following for details:

• Appendix 2: Equity Groups: https://docs.education.gov.au/documents/2014-

appendix-2-equity-groups • Appendix 5: Equity Performance Data:


Where these reports would be directly reported on a standalone page. Again, the drawback of Excel spreadsheet reports which rely on simple tables is that they are difficult to interrogate for data.

B. Data can be reported in a dashboard format, which combines simple Excel data collection (or database statistics) in a graphics-heavy reporting format. This can take the form of stand-alone data dashboards or, alternatively, sites with dashboards integrated into them. Examples include:

• Dedicated Enrolment Dashboards: These pages allow for the reporting of

higher education enrolment data using simple databases. A good example is the online dashboard maintained by Humboldt University in California:







http://www2.humboldt.edu/irp/Dashboards/Enrollment_University.html

• Headline Dashboards on dedicated topic sites: These pages present data dashboard in the context of more general discussions, for instance see: https://www.hesa.ac.uk/

http://www2.humboldt.edu/irp/Dashboards/Enrollment_University.html

https://www.hesa.ac.uk/



C. An online database format: This allows for a convention database query feature to be used to access information from an underlying data base. Examples include:

a. The Department of Education and Training’s QILT website

http://www.qilt.edu.au/study-areas:

http://www.qilt.edu.au/study-areas



b. The Department of Education and Training’s Higher Education Data Cube (ucube): https://education.gov.au/ucube-higher-education-data-cube

c. The NCSEHE’s equity data page which combines an online database and mapping system (beta version only): http://www.data.ncsehe.edu.au/

https://education.gov.au/ucube-higher-education-data-cube

http://www.data.ncsehe.edu.au/



In terms of complexity, the work required to design and implement any of these three solutions increases considerably from the shift from Excel spreadsheet to Data Dashboard, with both these options potentially being viewed as ‘in house projects’ within either the Equity or Data Analysis Sections. Beyond these solutions, a web-based database will require specialised expertise, either from within the Department (Datamart) or through external consultants.

Resourcing: The additional reporting requirements under the proposed Framework are minimal in the context of current reporting and primarily involve additional data requests:

• Social Research Centre – Graduate Outcomes Survey data and AEDC data; • ACARA – NAPLAN school results data; • ACER – PISA test results data; • Applications data – sourced internally from the Department; and • Input data – sourced internally from the Department.

It is envisaged that these data should be sourced from a similar report elsewhere in the Department of Education and Training for compulsory education (pre-school; K12 education).

Option 3 will require resourcing for additional reporting through a separate equity report. This includes a scoped project for the design, development and maintenance of a dedicated online database – the most involved solution in this context.



Stakeholders: Option 3 includes Tier 1 Indicators as well as the entire range of higher education Indicators of Option 2, with the addition of data on inputs (financial and activity) which allows users to generate comparative assessments of performance in view of overall resourcing.



The Requirements of Implementation

Among the three options, Option 1 provides the easiest extension to existing collections, with the lowest level of new resourcing requirements, while Option 3 will require additional collection responsibilities among data managers but offers the greatest potential for extended on higher education equity issues.

Figure F: Comparing the Three Options – (1) Most favourable to (3) Least favourable

Indicator

Format – Development Requirements

Coverage – Provision of Information

Resourcing - Requirements

Stakeholders – Access to Information

Option 1 (Continue Reporting in the Current Context)

1 3 1 3


2 2 2 2


3 1 3 1

After selection of an Option, implementation requires the following:

1. Selection of an option from (1) to (3) above, or the development of a new option; 2. An outline of existing data reporting functions which can be extended to include the

rollout of the Framework; 3. Operations guidance in relation to the Framework and existing data collections on:

a. Development of responsibilities; b. Assignment of responsibilities; c. Resourcing requirements; d. Infrastructure and data security issues; and e. Data transfer protocols for Framework Indicators.

4. Initial implementation of the Framework; and 5. Management plan for future operation of the Framework.

Contact Details

National Centre for Student Equity in Higher Education

Email: [email protected]

Web: www.ncsehe.edu.au

mailto:[email protected]

http://www.ncsehe.edu.au/

a framework for measuring equity performance in australian ... · a framework for measuring equity...

Documents