Background

GAML Brief 2:Good Practices in Learning Assessment

January 2018

GAML Brief 2: Good Practices in Learning Assessment

2

Background

Assessments are a key way to track and monitor learning outcomes for both national education objectives and international policy initiatives like the Sustainable Development Goal for education (SDG 4). However, correct implementation is crucial for reliable, high quality results that can feed into impactful policy decisions.

To this end, the UNESCO Institute for Statistics (UIS) and the Australian Centre for Educational Research (ACER), together with the Global Alliance to Monitor Learning (GAML), have developed a practical set of global guidelines called the Good Practices in Learning Assessments (GP-LA) to help countries implement assessments in a way that follows established scientific principles. These guidelines are laid out in a manual.

Further, two Quick Guides provide a snapshot of the process and outline some of the considerations that go into choosing a learning assessment.

Why are Learning Assessments important?

From a national perspective, assessments can inform policy decisions, and help decision makers understand the impact of the value of an investment in education. Further, learning assessments can keep stakeholders accountable, support teacher training, and help parents understand if their student is meeting expectations. As stated in Quick Guide 2: “investing in education without assessing learning is like paying for a product you cannot see.” (p. 8)

Moreover, assessments are a key component in helping countries meet their international obligations for SDG 4. In order to measure the full set of targets related to the SDG goal for education, each country needs to measure learning outcomes.

What are Good Practices in Learning Assessment?

The Good Practices in Learning Assessment (GP-LA), offers conceptual guidance on how to implement an assessment that results in high quality, consistent, reliable and comparable data.

Typically, each country that administers a large-scale assessment does so in a slightly different way. This variety is expected as assessments need to meet the diverse needs of national education systems around the world. Yet, while meeting particular national objectives based on the local context and learning traditions is key, it is just as important that assessments meet international expectations to ensure the quality, consistency and reliability of the resulting data. This is why the GP-LA framework is so important.

The GP-LA is based on six key quality concepts of learning assessments:

Fitness for purpose: Countries should choose a type of assessment that generates data that are appropriate for their stated purpose. In the case of SDG 4, the assessment must have the technical rigour to address the specific reporting needs. An assessment designed for other reasons will not produce the data that meets these objectives.

Clarity and consistency of purpose: An assessment may address more than one purpose, but they should be clearly stated so that consistency is ensured throughout.

GAML Brief 2: Good Practices in Learning Assessment

3

Objectivity and independence: For an assessment to be credible, the data collection and the results cannot be influenced by special interests of any individual or group.

Transparency and accountability: The assessment should be conducted in a transparent manner, and the results open to critique and feedback.

Technical rigour: The methodology, analysis and interpretation of the assessment should follow scientific principles.

Ethicality and fairness: The assessments should be inclusive and not give an unfair advantage to any group or individual, nor it should be harder or easier for certain groups. No harm should come from participating.

The GP-LA further identified 14 key principles of a high-quality assessment and offers suggestions on how countries can meet them. The 14 key areas describe what a good assessment looks like at each step of the process, from conceptualization to goal setting to reporting the results (see Figure 1).

How was the GP-LA developed?

The GP-LA was developed by ACER, in collaboration with the UIS and two task forces of the Global Alliance to Monitor Learning (GAML). These task forces, the Assessment Implementation Task Force and the Task Force for SDG 4.1, are made up of education and assessment experts from around the world.

As the GP-LA is meant to serve as an independent description of good practices that is not connected to any specific study or approach, it is aligned with the UN Fundamental Principles of Official Statistics, which sets professional and scientific standards for official data reporting. However, while the UN Principles are targeted toward national governments, the GP-LA is designed for any organization that implements learning assessments. Also, the GP-LA establishes data collection, analysis and guidelines specifically for learning assessments rather than more general purposes.

Are there other tools to evaluate the quality of learning assessments?

The UIS and its partners are also developing the Data Alignment Record Tools (DART) to evaluate the quality of national and cross-national assessments based on content and procedure. The content alignment includes a questionnaire (along with a ‘how to’ manual) to help countries ensure that the content of their assessments meets global standards. Similarly, from a procedural perspective, another questionnaire guides countries in the technical requirements necessary to obtain high quality data that can be used for global reporting.

More generally, revisions to the Data Quality Assurance Framework (DQAF), which offers conceptual direction on reporting statistics and is described in more detail in Brief 8, are also underway and are intended to address the particular needs of SDG 4 reporting.

How can governments and other stakeholders use these tools to implement learning assessments?

The DQAF, GP-LA, the Quick Guides and DART will all work together to set international expectations for the quality of large-scale assessments and the data produced by them. Governments and other actors who design and carry out assessments can use DQAF for a conceptual framework for ensuring high quality data, the GP-LA as a set of guidelines for implementation, the Quick Guides for a ‘snapshot’ of the process, and DART for

GAML Brief 2: Good Practices in Learning Assessment

4

content alignment and evaluation, to determine whether their assessments are meeting global standards.

These tools can also be used by international agencies to determine whether a country’s data are appropriate for SDG 4 reporting. If a country is not meeting expectations, it can use the GP-LA to improve its assessment system. In this case, the UIS can work directly with countries to help them build their technical and logistical capacity to create, implement and analyse learning assessments.

Figure 1. 14 key areas of a good assessment

5. The test questions are reliable (the results are consistent and accurate each time the test is administered), valid (the questions accurately measure what they claim to measure), and fair (all test-takers have an equal advantage when taking the test).

1. The goals and priorities of parents, students, teachers, ministry officials, and other education stakeholders are clearly defined and used to develop the assessment program.

2. There is a dedicated team with the skills to carry out the assessment program.

3. There is a document detailing the management of the assessment. This covers the expectations for the quality of the data collected, and how the results will be used to track whether expectations are being met.

4. A document that clearly describes the purpose of the assessment program and the learning topics (competencies, skills, content, etc.) it will cover. This helps guide those working on the assessment and also a wider audience including journalists, researchers, academics, etc. what the assessment will include.

6. The tests include a set of questions about the personal characteristics of the test-takers including their socio-economic status, home environment, classrooms, and schools. These questions are also reliable, valid and fair (see above).

7. The test questions are sensitive to cultural differences. For example, test-takers in urban areas should not find it easier than those in rural areas because the test contains words they are more familiar with. If the test is translated into multiple languages, the translation should be accurate and does not change the characteristics of the items.

8. The test is designed in such a way that it covers all the necessary topics, and asks the right amount of questions to get the information needed. Some of the same questions should be asked each year the test is administered, to note whether learning is improving or declining.

9. The subset (sample) of the population tested is chosen using scientific methods, and has the same characteristics as the whole population (same percentage of girls/boys, urban/rural, etc).

10. Before administering the main test, a pre-test (field trial) is conducted to identify and fix any issues with the test. When the main test is administered, the procedures are documented and the test is done in the same way in each place (school, home, etc.).

11. There is a secure database to keep the scores and test-takers’ information private. It is free from errors and organized for easy access and analysis.

12. The answers to the test questions are organized on a scale from the easiest questions to the most difficult. Scores for individuals and groups of test-takers can be placed on this scale in order to assess their performance and make comparisons across groups.

13. After the tests are complete, the results will be analysed to show emerging trends and patterns. The analysis is undertaken for the purpose of reporting and interpretation. This information will be publicly available.

14. The results of the test will be available in different formats for different stakeholder groups, and should be used to help stakeholders act on the results.