3 strengths and weaknesses of american students in ... - oecd · strengths and weaknesses of...

Strengths and Weaknesses of American Students in Mathematics

STRONG PERFORMERS AND SUCCESSFUL REFORMERS IN EDUCATION – LESSONS FROM PISA 2012 FOR THE UNITED STATES – © OECD 2013 55

3

This chapter looks in more detail at American students' strengths and weaknesses in the PISA 2012 mathematics assessment. It examines the success rates of students on individual items of the test, compared with the success of students in OECD countries on average and in comparison with five comparator or reference countries/economies. The analysis points to aspects of mathematics teaching that might be strengthened in order to improve the performance of the United States.

33STRENGTHS AND WEAKNESSES OF AMERICAN STUDENTS IN MATHEMATICS

56 © OECD 2013 – STRONG PERFORMERS AND SUCCESSFUL REFORMERS IN EDUCATION – LESSONS FROM PISA 2012 FOR THE UNITED STATES STRONG PERFORMERS AND SUCCESSFUL REFORMERS IN EDUCATION – LESSONS FROM PISA 2012 FOR THE UNITED STATES – © OECD 2013 57

INTRODUCTIONHaving presented the performance of 15-year-old students in the United States in the previous chapter, this chapter looks in more detail at their strengths and weaknesses in the PISA 2012 mathematics assessment. It examines the success rates of students on individual items of the test, compared with the success (measured by solution rates) of students in OECD countries on average and in comparison with five comparator or reference countries/economies. The five countries chosen for comparison with the United States are: two top performing Asian countries, Shanghai-China and Korea; two European countries performing significantly above the OECD average, the Netherlands and Germany; and one of the United States’ neighboring countries, Canada.

This analysis examines the performance of the United States students in terms of the percentage of students who correctly answered each of the 84 mathematics items that were administered in the United States as part of PISA 2012. It compares the performance of students in the United States with both the OECD average and with the performance of the five reference countries/economies to identify specific relative strengths and relative weaknesses of the country’s 15-year-olds. The analysis identifies the so-called “conspicuous items”, that is those items where the United States’ students performed unexpectedly well or unexpectedly badly compared with their overall distance from the OECD average or from the reference countries. Altogether the analysis reveals 33 such conspicuous items, in 16 of which the United States was notably strong and 17 in which it was notably weak. The analysis shows that the relatively strong items are mostly easy ones whereas the relatively weak items are much more demanding. This analysis identifies certain patterns, with clusters of conspicuous items that have similar cognitive requirements. Some of the items from these patterns are then analyzed more deeply in order to understand the clusters better. An analysis of student solutions helps to clarify further the strengths and weaknesses of the students.

What is PISA mathematics about? The conceptual core of PISA mathematics is mathematical literacy. According to the PISA 2012 mathematics framework (see OECD 2013, p. 25) this concept means “an individual’s capacity to formulate, employ, and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to recognize the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective citizens.” In short, mathematical literacy describes one’s capacity to use mathematics in a well-founded manner in order to solve real world problems, where “real world” means “the physical, social and mental world” (Freudenthal, 1983), including the mathematical world itself. Annex A1 gives an overview of the framework for assessing mathematics in PISA 2012.

OVERALL SOLUTION RATESAcross all of the 84 mathematics items administered in PISA 2012 in the United States, on average the United States has a significantly lower solution rate than the OECD average. The solution rate of the United States is 43.78% against an average of all OECD countries of 47.47%. In only 18 items did the United States reach a higher solution rate than the OECD average (see Table A1.1 for a full list of the item solution rates). Other countries with which the United States has to compete economically reached much higher averages in PISA 2012 mathematics. Table 3.1 shows the average percentage success rates of these reference countries.

• Table 3.1 •Average solution rates in PISA 2012 mathematics (%)

United States OECD average Canada Germany Netherlands Korea Shanghai-China

43.78 47.47 52.16 51.34 53.50 58.76 69.44

These lower overall solution rates of course reflect the relative overall performance in the PISA 2012 mathematics assessment of 15-year-olds in the United States as reported in Chapter 2. The remainder of this chapter will examine the solution rates item by item to reveal where the United States does relatively better or relatively worse.

SELECTION OF CONSPICUOUS ITEMSThe analysis seeks to identify so-called “conspicuous items”, which stand out as indicating particular strengths and weaknesses among United States’ students, compared with those of the average of all OECD countries and the five



reference systems: Canada, Germany, Korea, the Netherlands and Shanghai-China. Different approaches can be followed to identify such conspicuous items. The simplest approach would be simply to compare solution rates. However, such an approach would be distorted when comparing items with relatively low or high solution rates overall. For instance a difference of 10% between solution rates is less notable when the two solution rates are 95% and 85% than when they are 55% and 45% – in other words, the relationship between getting items correct and the items’ difficulty is not linear (see Table A1.1 for comparative solution rates for all 84 mathematics items).

Therefore it is necessary to transform the percentage solution rates into a linear metric. This can be achieved by transforming the percentages into “logits”. This transformation has the effect of “stretching out” very low and very high solution rates in comparison with solution rates close to 50%. A logit value of 0 means that the item has a solution rate of 50%, positive logits mean higher solution rates and negative logits mean lower solution rates.

How U.S. students compare with the OECD averageAs a first step the logit differences between the United States and the average of all OECD countries were compared in order to identify the items with notably different solution rates. Significant items are those with a difference of at least one standard deviation (both sides)1 between the United States and the OECD average. The analysis also refers to the categorization of the items within the PISA 2012 mathematics framework, an overview of which is given in Annex A1 of this report. This categorization, for instance, identifies the content area of mathematics to which the item belongs (Quantity, Uncertainty and data, Change and relationships, Space and shape) and also the mathematical process that is called on in order to answer the question (Formulating situations mathematically, Employing mathematical concepts, facts, procedures and reasoning and Interpreting, applying and evaluating mathematical outcomes).

Relative strengthsTable 3.2 shows the 12 conspicuously strong items identified by this method, all with significant positive differences in the logits of the United States compared with the average of all OECD countries. It is remarkable that there are no items from the content area “Space and shape” and only one item from “Quantity”. The others are either “Change and relationships” or “Uncertainty and data” items. It is equally remarkable that only one item belongs to the process category “Formulate”, whereas seven items stem from the category “Interpret” and six items from “Employ”.

• Table 3.2 •Relative strengths of the United States compared with the average of all OECD countries

(in logits)

Name Item Code Logit USALogit average for all OECD

countriesDifference Content Process

Charts PM918Q01 2.43 1.92 0.51 Uncertainty and data Interpret

Bike rental PM998Q02 1.43 0.92 0.51 Change and relationships Interpret

Speeding fines PM909Q01 2.57 2.13 0.45 Quantity Interpret

Transport PM420Q01T 0.32 0.00 0.31 Uncertainty and data Interpret

Thermometer cricket PM446Q01 1.08 0.78 0.30 Change and relationships Formulate

Medicine doses PM954Q01 0.86 0.64 0.23 Change and relationships Employ

Drip rate PM903Q03 -0.85 -1.06 0.21 Change and relationships Employ

Carbon tax PM915Q01 -0.26 -0.40 0.14 Uncertainty and data Employ

Diving PM411Q02 -0.04 -0.17 0.13 Uncertainty and data Interpret

Employment data PM982Q02 -0.70 -0.81 0.11 Uncertainty and data Employ

Carbon dioxide PM828Q02 0.35 0.24 0.11 Uncertainty and data Employ

Population pyramids PM155Q01 0.82 0.74 0.08 Change and relationships Interpret

Relative weaknessesA further 12 conspicuous items show significant negative differences, indicating the relative weaknesses of students in the United States (Table 3.3). In contrast to the relatively strong items, seven of the items where United States students are weak are in the domain of “Space and shape”. There is only one item (PM955Q02 i.e. item 2 of the unit “Migration”) which calls on students’ skills in interpreting a mathematical problem.



• Table 3.3 •Relative weaknesses of the United States compared with the average

of all OECD countries (in logits)


countriesDifference Content Process

Wheelchair basketball PM00KQ02 -2.59 -1.75 -0.84 Space and shape Formulate

Arches PM943Q02 -3.67 -2.89 -0.79 Space and shape Formulate

Migration PM955Q03 -2.76 -1.99 -0.76 Uncertainty and data Employ

Migration PM955Q02 -1.41 -0.65 -0.76 Uncertainty and data Interpret

Running tracks PM406Q02 -2.33 -1.59 -0.74 Space and shape Formulate

Computer game PM800Q01 1.35 2.03 -0.68 Quantity Employ

Carbon tax PM915Q02 0.12 0.77 -0.65 Change and relationships Employ

The fence PM464Q01T -1.81 -1.17 -0.64 Space and shape Formulate

Map PM305Q01 -0.14 0.42 -0.56 Space and shape Employ

Running tracks PM406Q01 -1.61 -1.07 -0.54 Space and shape Employ

An advertising column PM00GQ01 -2.87 -2.34 -0.53 Space and shape Formulate

Sauce PM924Q02 0.05 0.55 -0.50 Quantity Formulate

Conspicuous items by “Content” and “Process”In order to get a first impression of the similarities and differences of these conspicuous items, it is helpful to have a look at the “Content” and “Process” categories of these items. Figures 3.1 and 3.2 examine the correlations between the contents and the processes of the conspicuous items.

• Figure 3.1 •Bubble chart of process and content for the relative strengths

0

0

0

0 0

0

Proc

ess

Empl

oyFo

rmul

ate

Inte

rpre

t

Content

Change and relationships

Quantity Space and shape

Uncertainlyand data

2

2

1

1 3

3

It is apparent that the overlap between these bubble charts for the strengths and weaknesses is almost negligible. Obviously, these items fall into several different clusters. In other words, the strengths of the United States students lie primarily in items which require employing or interpreting mathematical concepts or facts and are either from the mathematical content area “Change and relationships” or from “Uncertainty and data”. In contrast, their weaknesses lie primarily in items which require formulating or employing mathematical concepts or facts from the mathematical content area “Space and shape”.



Comparison with the reference countriesTable 3.4 compares the United States with the 5 reference countries/economies for the 12 conspicuous items that showed relative strength when compared with the OECD average (from Table 3.2).

Relative strengths

Table 3.4 shows that in none of these items did the United States perform better than Shanghai-China. Additionally there were three items – “Diving 02” (i.e. item 2 of the unit “Diving”), “Carbon dioxide 02” and “Population pyramids 01” – where the United States did not perform better than any of the reference countries. Only in two items (“Charts 01” and “Transport 01”) did the United States score considerably better than three out of the five reference countries. Among the comparator countries, the United States performed best in comparison with Germany, performing better than Germany in nine out of the 12 items. It performed better than the Netherlands in six items, better than Korea in three and better than Canada in only two.

• Table 3.4 •Conspicuous items compared with the reference countries (in logits)


countries

Logit Canada

Logit Germany

Logit Netherlands

Logit Korea

Logit Shanghai-

China Charts PM918Q01 2.43 1.92 1.89 1.99 2.25 2.29 2.52

Bike rental PM998Q02 1.43 0.92 1.73 1.16 2.10 1.39 1.96

Speeding fines PM909Q01 2.57 2.13 2.82 2.50 3.11 2.67 2.83

Transport PM420Q01T 0.32 0.00 0.58 0.03 0.31 -0.94 0.43

Thermometer cricket PM446Q01 1.08 0.78 1.26 0.62 1.00 1.34 2.00

Medicine doses PM954Q01 0.86 0.64 0.94 0.77 0.36 1.46 2.51

Drip rate PM903Q03 -0.85 -1.06 -0.53 -1.11 -0.82 -0.15 0.88

Carbon tax PM915Q01 -0.26 -0.40 -0.38 -0.76 -0.02 0.69 0.98

Diving PM411Q02 -0.04 -0.17 0.08 -0,01 0.49 0.26 0.90

Employment data PM982Q02 -0.70 -0.81 -0.58 -0.91 -0.84 -0.22 -0.51

Carbon dioxide PM828Q02 0.35 0.24 0.51 0.40 0.82 0.68 1.16

Population pyramids PM155Q01 0.82 0.74 1.06 0.83 1.38 1.24 0.98

Note: Shading indicates the United States performs better than the comparator country.

• Figure 3.2 •Bubble chart of process and content for the relative weaknesses

0

0

00

Proc

ess

Empl

oyFo

rmul

ate

Inte

rpre

t

Content

Change and relationships

Quantity Space and shape

Uncertainlyand data

0

0 1

1 1

1

1 2

5



Relative weaknessesConsidering the 12 items where the United States is relatively weak compared with the OECD average, and comparing these with the comparator countries (Table 3.5), it is evident that in all of these items, the United States performs worse than each of the comparator countries. This is most notable in “Wheelchair basketball 02” and “Map 01”, the latter being an item in which all of the other comparator countries had solution rates of more than 50% (positive logits), whereas the solution rate for the United States was less than 50% (negative logit).

• Table 3.5 •Conspicuous items compared with the reference countries (in logits)


countries

Logit Canada

Logit Germany

Logit Netherlands

Logit Korea

Logit Shanghai-

China Wheelchair basketball PM00KQ02 -2.59 -1.75 -1.44 -1.31 -1.53 -1.77 -0.17

Arches PM943Q02 -3.67 -2.89 -3.14 -2.78 -3.39 -1.42 0.04

Migration PM955Q03 -2.76 -1.99 -1.74 -1.79 -1.25 -1.16 -0.29

Migration PM955Q02 -1.41 -0.65 -0.62 -0.56 -0.06 0.25 0.31

Running tracks PM406Q02 -2.33 -1.59 -1.14 -1.42 -1.21 -0.84 -0.20

Computer game PM800Q01 1.35 2.03 1.93 1.71 2.02 3.30 3.43

Carbon tax PM915Q02 0.12 0.77 1.16 1.42 1.12 1.05 2.21

The fence PM464Q01T -1.81 -1.17 -0.83 -0.73 -1.18 -0.32 0.40

Map PM305Q01 -0.14 0.42 0.55 0.68 0.71 0.51 0.94

Running tracks PM406Q01 -1.61 -1.07 -0.60 -0.80 -0.86 -0.40 0.32

An advertising column PM00GQ01 -2.87 -2.34 -2.28 -2.06 -1.70 -1.72 -1.52

Sauce PM924Q02 0.05 0.55 0.43 0.69 1.47 1.01 1.74

Note: See Table 3.3.

Comparison with the average of the reference countriesFurther conspicuous items can be found by generating an average logit value for each item, referring only to the five reference countries. This average is compared with the logits of the United States and again, “unexpectedly large” and “unexpectedly small” differences are examined. More precisely, all those items where the difference between those two logits (the United States minus the average of the reference countries) is at least by one standard deviation larger or smaller than the average of all differences2 are considered conspicuous.

Relative strengthsThis comparison reveals four new conspicuous items, highlighted in dark blue. The item “Employment data 01” shows, remarkably enough, a positive value compared with the two Asian countries, the other three items show a positive logit value in comparison with the Netherlands. The results for “Employment data 01” might also reveal one of the rare relative weaknesses of the Asian countries.

• Table 3.6 •Relative strengths of the United States compared with the reference countries (in logits)

Name Item Code Logit USALogit average for comparator

countries

Logit Canada

Logit Germany

Logit Netherlands

Logit Korea

Logit Shanghai-

China Charts PM918Q01 2.43 2.19 1.89 1.99 2.25 2.29 2.52

Transport PM420Q01T 0.32 0.08 0.58 0.03 0.31 -0.94 0.43

Employment data PM982Q01 1.92 1.88 1.97 2.10 2.06 1.76 1.49

Employment data PM982Q02 -0.70 -0.61 -0.58 -0.91 -0.84 -0.22 -0.51

Thermometer cricket PM446Q01 1.08 1.25 1.26 0.62 1.00 1.34 2.00

Speeding fines PM909Q01 2.57 2.79 2.82 2.50 3.11 2.67 2.83

Bike rental PM998Q02 1.43 1.67 1.73 1.16 2.10 1.39 1.96

Migration PM955Q01 1.02 1.26 1.33 1.33 0.87 1.11 1.68

Population pyramids PM155Q01 0.82 1.10 1.06 0.83 1.38 1.24 0.98

Population pyramids PM155Q04T 0.16 0.47 0.56 0.35 0.04 0.59 0.81

Arches PM943Q01 0.05 0.37 0.15 0.08 -0.20 0.52 1.30

Note: Shading indicates the United States performs better than the comparator country.



DETECTING PATTERNS

The analysis so far has revealed 33 conspicuous items: 16 items in which the United States is relatively strong (12 in Table 3.4 and four in Table 3.6) and 17 items in which the United States is relatively weak (12 in Table 3.5 and five in Table 3.7). The next question is whether, when considering these 33 items, different items have the same or similar cognitive requirements – are there certain patterns that can be detected? Of course, every single PISA item has its own characteristics. A pattern is thus composed of items that share some common features although other features remain item specific. Seven such patterns were detected, three for relative strengths and four for relative weaknesses of the United States. The vast majority (27) of the conspicuous items fit into one of these patterns.

For each pattern, the following analysis selects one “illustrating item”, an item that is particularly typical of this pattern. For some of the patterns, a second item is selected that is also fairly typical and which belongs to the set of publicly released PISA items.3

In some of the patterns, certain “demarcating items” are identified. These are items which seem, at first glance, to belong to that pattern, but which in practice require different competencies in order to solve them. A demarcating item for a strength pattern might therefore be relatively harder for U.S. students than the pattern suggests, while one for a weakness pattern might be relatively easier.

Three out of the 33 items do not fit the patterns and therefore are not considered further in the analysis. These are “Population pyramids 04”, “Diving 02”, and “Sauce 02”. Another three are used as demarcating items for some of the patterns: “Arches 01” for a weakness pattern, and “Computer game 01” and “Carbon tax 02” for strength patterns.

• Table 3.7 •Relative weaknesses of the United States compared with the reference countries (in logits)

Name Item Code Logit USALogit average for comparator

countries

Logit Canada

Logit Germany

Logit Netherlands

Logit Korea

Logit Shanghai-

China Arches PM943Q02 -3.67 -2.14 -3.14 -2.78 -3.39 -1.42 0.04

Migration PM955Q03 -2.76 -1.24 -1.74 -1.79 -1.25 -1.16 -0.29

Running tracks PM406Q02 -2.33 -0.96 -1.14 -1.42 -1.21 -0.84 -0.20

Wheelchair basketball PM00KQ02 -2.59 -1.24 -1.44 -1.31 -1.53 -1.77 -0.17

The fence PM464Q01T -1.81 -0.53 -0.83 -0.73 -1,18 -0.32 0.40

Migration PM955Q02 -1.41 -0.14 -0.62 -0.56 -0.06 0.25 0.31

Carbon tax PM915Q02 0.12 1.39 1.16 1.42 1.12 1.05 2.21

Roof truss design PM949Q01T 0.31 1.47 1.15 1.00 1.25 1.76 2.17

Running tracks PM406Q01 -1.61 -0.47 -0.60 -0.80 -0.86 -0.40 0.32

Spacers PM992Q03 -2.86 -1.73 -2.35 -2.23 -2.61 -0.98 -0.49

Computer game PM800Q01 1.35 2.48 1.93 1.71 2.02 3.30 3.43

Thermometer cricket PM446Q02 -2.87 -1.74 -2.33 -2.55 -1.84 -1.64 -0.36

Revolving door PM995Q01 -0.12 0.95 0.30 0.58 0.37 1.33 2.16

Tennis balls PM905Q02 -0.46 0.60 0.30 0.26 0.48 0.54 1.43

This method reveals five new conspicuous items, again highlighted in dark blue.

Relative strengths of the United StatesIt is remarkable that the following “strength patterns” include the majority of easy PISA items; there are only a few more challenging ones here. So the relative strengths of the United States lie mostly in the easy items. In particular, the easiest PISA items are part of the first pattern, A1. In fact it might be said that pattern A1 represents the strengths of the United States best. It includes almost exclusively easy items, in particular “Speeding fines 01”, “Charts 01”, “Employment data 01” and “Bike rental 02”, which have the highest solution rates for the United States of all the items (from 92.91 % for “Speeding fines 01” to 80.71 % for “Bike rental 02”).



Pattern A1: Read data directly from tables and diagrams (one step)The items in this pattern require students only to understand a short text and read single values directly from a representation provided such as a table or a bar diagram.

Items in this pattern “Transport 01”, “Speeding fines 01”, “Charts 01”, “Migration 01”, “Bike rental 02” and “Employment data 01”

Illustrating items “Migration 01” and (released) “Charts 01”

Demarcating items “Computer game 01” and “Running time 01”

The demarcating items, “Computer game 01” and “Running time 01”, are two items which at a first glance seem to fit this pattern as well but where the Unites States performed conspicuously poorly in comparison with the OECD average. “Computer game 01” not only represents a weakness of the United States, but in fact shows a significant negative difference when compared to the OECD average and the average of the comparator systems. In order to correctly answer this item, more than one step is needed and the first step is not reading data directly. Rather, students must first calculate total scores and then compare them in a second step. Similarly, in “Running time 01” more than one step is needed to solve this item. Students must first arrange the numbers according to their size and then identify the third-lowest number.

Pattern A2: Simple handling of data from tables and diagramsThe items in this pattern require students to understand a short text, read two values from a given representation and then perform some straightforward operation such as adding or comparing the values.

Items in this pattern “Population pyramids 01”, “Carbon dioxide 02”, “Carbon tax 01” and “Employment data 02”

Illustrating item “Carbon tax 01”

Demarcating items “Charts 02”, “Charts 05” and “Carbon dioxide 03”

The three demarcating items for this pattern, “Charts 02”, “Charts 05” and “Carbon Dioxide 03”, also seem to fit at first sight. However, the first two of these require more than a straightforward operation. In “Charts 02” a whole development process must be considered over time and “Charts 05” requires the continuation of a trend. “Carbon dioxide 03” is more challenging since students need to handle units (the data in the diagram are given in million metric tonnes) where it is easily possible to make mistakes.

Pattern A3: Handling directly manageable formulaeThe items in this pattern require students to use a formula provided, e.g. inserting numbers for variables, and do some easy calculation. The formula can be used directly, without any re-structuring.

Items in this pattern “Thermometer cricket 01”, “Drip rate 03” and “Medicine doses 01”

Illustrating items “Medicine doses 01” and (released) “Drip rate 03”

Demarcating items “Carbon tax 02”, “Medicine doses 04” and “Drip rate 01”

The demarcating items “Carbon tax 02”, “Medicine doses 04” and “Drip rate 01”might seem to fit into this pattern as well since there are formulae involved, but on closer inspection they do not. In “Carbon tax 02” the formula given cannot be used directly because it is necessary to apply some arithmetic laws first (“x before +”). Ignoring this will easily lead to mistakes. In “Medicine doses 04” there are two formulae that have to be compared, the values have to be switched from years to months for the second formula, and the values have to be rounded off appropriately. The item “Drip rate 01” contains a formula that has to be interpreted functionally (“what happens if…”) so it does not fit into this pattern either.

Relative weaknesses of the United StatesIt is remarkable that the patterns which display not only the weaknesses but the relative weaknesses of the United States include nearly all of the most difficult PISA items. The five items with the lowest OECD average solution rates are “Revolving door 02”, “Arches 02”, “Thermometer cricket 02”, “Spacers 03” and “An advertising column” (ranging from 3.5% for “Revolving door 02” up to 8.8% for “An advertising column”). All these items turn up in the following patterns. This suggests that the United States has a particular weakness in the most challenging items.

Pattern B1: Curricular requirement: π is necessaryThe items in this pattern require the explicit use of π in a calculation.

Items in this pattern “Wheelchair basketball 02”, “Running tracks 01”, ”Running tracks 02”, “An advertising column 01” and “Revolving door 02” (part of this pattern, but not so significant)

Illustrating item “Running tracks 01” and (released) “Revolving door 02”

Demarcating item “Arches 01”



To underline that this pattern is the result of a curricular requirement (being familiar with the number π) we chose “Arches 01” as a demarcating item. As in all other items in this pattern, students have to identify the concrete geometric form of a circle and operate with it in some way. In the item “Arches 01”, students have to find the relation of the height (radius) and the width (diameter) of a circular arch. Therefore π is not necessary here, although π is used in two of the four possible answers in this multiple-choice item. So it is not circles themselves that seem to be the problem for the U.S. students since they did particularly well in “Arches 01” (see Table 3.6).

Pattern B2: Substantial mathematization of a real world situationThe items in this pattern require students to establish a mathematical model of a given real world situation in the form of a term or an equation with variables for geometric or physical quantities, before further actions (especially calculations) can take place. They have to understand the situation and activate and apply the appropriate mathematical content.

Items in this pattern “Thermometer cricket 02”, “The fence 01”, “Arches 02”, “Spacers 03” and “Revolving door 02”

Illustrating tem “Arches 02” and (released) “Revolving door 02”

Pattern B3: Genuine interpretation of real world aspectsThe items in this pattern require students to take a given real world situation seriously and genuinely interpret aspects of it. The popular superficial classroom strategy “don’t care about the context, just extract the numbers from the text and do some obvious operations” fails here.

Items in this pattern4 “Tennis balls 02”, “Migration 02”, “Migration 03”, (“An advertising column 01”), (“Running Tracks 02”) and (“Drip rate 01”)

Illustrating item “Migration 02”

Pattern B4: Reasoning in a geometric contextThe items in this pattern require genuine reasoning in a planar or spatial geometric context by using geometric concepts and facts. The transformation from the given real world context into the corresponding geometric context is straightforward and not the hard part of the task.

Items in this pattern4 “Map 01”, “Roof truss design 01”, “Revolving door 01”, (“Arches 02”), (“Running tracks 01”) and (“Running tracks 02”)

Illustrating item “Map 01” and (released) “Revolving door 01”

ANALYSES OF ILLUSTRATING ITEMSThis section illustrates some of the patterns identified in the previous section using relevant publicly released PISA (not all of the patterns can be well illustrated using publicly released items). This should render the patterns more transparent and concrete. As already stated, each single item in the PISA test has its own structure and specific cognitive requirements. A certain pattern is therefore composed of items that share some common features whereas other features are specific to that item. The illustrating items contain these common features, alongside other features that are not typical for the pattern.

Strength patterns (A1 and A3)

Pattern A1: Read data directly from tables and diagrams (one step)Understanding a short text and reading single values directly from a given representation such as a table or a bar diagram.

“Charts 01”is chosen to illustrate this pattern (Figure 3.3). With its solution rate of 91.93% for the United States, and 87.27% on average for OECD countries it is a rather easy item, but the items “Employment data 01” and “Speeding fines 01” in this pattern have even higher solution rates. In this pattern, only “Transport 01”, with a solution rate of 57.82% for the United States and 50.03% for the OECD average, has a much lower solution rate (because of its complex-multiple-choice structure – each individual statement in “Transport 01” is easy). “Charts 01” requires students to interpret given data and belongs to the mathematical content area “Uncertainty and data”. This combination of content and process represents a specific strength of the United States students.



• Figure 3.3. •CHARTS 01 (PM918Q01)

In January, the new CDs of the bands 4U2Rock and The Kicking Kangaroos were released. In February, the CDs of the bands No One’s Darling and The Metalfolkies followed. The following graph shows the sales of the bands’ CDs from January to June.

Month

Sales of CDs per month

Num

ber

of C

Ds

sold

per

mon

th

4U2Rock

The Kicking Kangaroos

No One’s Darling

The Metalfolkies

2250

2000

1750

1500

1250

1000

750

500

250

0Jan Feb Mar Apr May Jun

QUESTION 1

How many CDs did the band The Metalfolkies sell in April?A. 250B. 500C. 1 000D. 1 270

Ideal-typical solution:

In order to answer the question one needs to have a look at the April bars and to identify the fourth bar as representing the number of CDs sold by The Metalfolkies. This bar reaches up to 500.

Therefore the correct answer is: B

The task “Charts 01” is embedded in a real world situation and starts with a short stimulus which explains the structure of the following diagram. The number of CDs sold per month from four different bands is given for the six months from January to June. For each month from February onwards there are four bars. Each bar represents the number of CDs sold in a specific month by a single band. The task for the students in item 01 is to find out how many CDs the band The Metalfolkies sold in April. Therefore one needs to have a look at the white bar for April which represents the band The Metalfolkies. It reaches 500 so answer B is correct.

Pattern A3: Handling directly manageable formulaeUsing a given formula directly to perform some easy calculation.

The item “Drip rate 03” can be analyzed to exemplify this pattern. This item has a relatively low solution rate for the United States, at 29.92%, but the OECD average is even lower at 25.72%, so the solution rate is relatively low overall. The items in this pattern are exclusively from the mathematical content area “Change and relationships”, and the process of “Drip rate 03” is “Employ”. This combination represents another specific strength of the United States students.



• Figure 3.4. •DRIP RATE 03 (PM903Q03)

Infusions (or intravenous drips) are used to deliver fluids and drugs to patients.

Nurses need to calculate the drip rate. D, in drops per minute for infusions.

They use the formula D = dv where —— 60n

d is the drop factor measured in drops per millilitre (ml) v is the volume in ml of the infusion n is the number of hours the infusion is required to run.

QUESTION 3

Nurses also need to calculate the volume of the infusion, v, from the drip rate, D. An infusion with a drip rate of 50 drops per minute has to be given to a patient for 3 hours. For this infusion the drop factor is 25 drops per millilitre.What is the volume in ml of the infusion?

Volume of the infusion ……………………. ml


First one has to identify the given values and to find out which value is wanted. Then the formula can be converted for v so that afterwards one has only to insert the right values in the new formula and calculate the result. For this task, D is 50 (drops per minute), d is 25 (drops per ml) and n is 3 (hours).

Thus D = , v = = = 360 (ml)60n d 25dv D x 60n 50 x 60 x 3

Therefore the correct answer is: 360.

The stimulus of the “Drip rate” task introduces the topic and explains a formula usedby nurses to calculate the drip rate (in drops per minute) for infusions. This formula contains four variables which are each separately explained in the stimulus. One challenge of the item “Drip rate 03” is to identify the relevant variables and their values. Another requirement is to insert these values correctly into the formula and then calculate the unknown volume of the infusion. A third challenge is to rearrange the formula with respect to the unknown volume of the infusion. This can either be achieved by inserting all given values and then converting the formula, or by converting the formula first and inserting the given values afterwards. The insertion is straightforward and the calculation is easy to perform. The formula is directly manageable since it is pre-structured by the fraction line. The relatively low solution rates in comparison with other items in this pattern are due to the number of algebraic steps required.

Weakness patterns (B1, B2 and B4)In order to illustrate the patterns for the relative weaknesses of the United States, items 01 and 02 of the unit “Revolving door” were analyzed. Item 02 has a wide range of cognitive requirements and therefore exemplifies several patterns. The following section explains which of the typical features of the patterns are part of the items “Revolving door 01” and “Revolving door 02”.

Analysis of the item “Revolving door 01”The item “Revolving door 01” (Figure 3.5) exemplifies pattern B4 (Reasoning in a geometric context). The United States has a solution rate of 46.96% for this item whereas the OECD average is 57.71%, so this is an item of intermediate difficulty. Pattern B4 has three items which are more difficult (“Arches 02”, “Running tracks 01” and “Running tracks 02”) but also an easier item (“Roof truss design 01”) and one of similar difficulty (“Map 01”). Like all the items in this pattern, the mathematical content area of “Revolving door 01” is “Space and shape”, and the process is “Employ”.



• Figure 3.5. •REVOLVING DOOR 01 (PM995Q01)


The door wings divide the circular-shaped space into three equal sectors; consequently the angle formed

by two door wings is = 120°3360°

Therefore the correct answer is: 120°

The stimulus of the item describes a revolving door and presents three diagrams which include information like the diameter and different positions of the door wings. This description already refers to a mathematical model, that is the geometric form of a circle, which is divided into three congruent parts by three linear segments. A difficulty is the rotation of the door wings, so there is no fixed shape. The students are faced with three slightly different diagrams instead of one. Even if the students do not have to mentally revolve the door wings, they need to recognize that the angle between the door wings stay the same when the door is rotated. Such a mental operation with the moving door wings is central for the second item of this task “Revolving door 02” (see below).

The essential cognitive requirement for “Revolving door 01” is to detect that the three sectors of the revolving door are congruent and therefore have the same angles. After that the students have to conclude that one of the angles will be 120°.

For this the students need knowledge about angles, namely that the full angle of a circle comprises 360°. A further difficulty could be that students might be unfamiliar with determining angles in the sectors of a circle. These requirements are mainly features of pattern B4 (Reasoning in a geometric context).



Some mistakes can be expected (as illustrated in the next section). For example, some students will not understand the geometric model and therefore simply pick out the only given value which is the diameter (expected wrong solution 200°). Some students will recognize that they have to divide the full angle by three but do not know how many degrees a full angle comprises or do not have a proper concept of angles, so they might use an avoidance strategy like simply using the value of the diameter for the full angle (expected wrong solution 66. 6°). Another incorrect strategy would be to guess or to measure the angle.

Analysis of the item “Revolving door 02”Analyzing the item “Revolving door 02” (Figure 3.6) exemplifies particularly patterns B1 (Curricular requirement: π is necessary) and B2 (Substantial mathematization of a real world situation). “Revolving door 02” has the lowest solution rate on average for the OECD in the whole test. The solution rate for the United States for this item is only 2.26% and the OECD average is 3.47%. The mathematical content area is “Space and shape” and the process is “Formulate”. As discussed earlier, this content area represents a particular weakness of the United States.

• Figure 3.6. •REVOLVING DOOR 02 (PM995Q02)


The minimal arc length on the left side of the door has to be one third of the circumference because the sector formed by two door wings is – according to the conditions about air flow – at least one-third of the whole circle.

For reasons of symmetry the minimal arc length on the right side of the door is also one-third of the circumference.

Thus at most one-third of the circumference is left for the two openings, which implies that the maximum opening for exit and entrance is one-sixth of the circumference each:

circumference = 200×π ≈ 628 cm Arc length ≈ 628.32 ÷ 6 ≈ 104 cm

Therefore the correct answer is (rounded off): 104 cm



The stimulus and the first item of this unit have already been analyzed to illustrate pattern B4.

Like “Revolving door 01”, the item “Revolving door 02” describes a real world situation through a text and a diagram. Neither the text nor the diagram in “Revolving door 02” contain the necessary value of the diameter, so the students have to transfer this information from the original stimulus to the new situation given in this item. The diagram leads to a mathematic model, but a substantial difficulty is that the item requires a dynamic conception, because the lengths of the openings are flexible and the rotation of the door wings has to be considered. In particular, in order to identify the arc length of the right side of the circle, it is necessary to modify the diagram and transfer it for a different position of the door wings. Therefore the mathematization of the real world situation is much more difficult than in the first part of the task, “Revolving door 01”. These cognitive requirements are typical for items of pattern B2 (Substantial mathematization of a real world situation). For that reason the item “Revolving door 02“can be used to exemplify this pattern.

It can be expected that a lot of students are not able to formulate an adequate mathematic model of the given situation. Some students might use an improper model, e.g. calculating the length of the arc between two door wings instead of the length of the arc of each opening. Furthermore, the task is unfamiliar and gives students no suggested approaches or instructions. Students have to invent their own strategies. For example, they could consider two extreme cases with the door at the entrance nearly opened and just closed; another possibility is to use a “dividing strategy”, that is to divide the circle into six equal parts and to argue that entrance and exit have to be one sixth of the circumference each.

No matter which strategy they use, students must calculate the circumference of the circle in order to solve the item. For this calculation the students need to know how to use the formula of the circumference (π times diameter or 2π times radius). This requirement is the core of pattern B1 (Curricular requirement: π is necessary) and therefore the item “Revolving door 02” can also be used to exemplify pattern B1.

Students who are not able to use the formula to calculate the circumference can be expected to use empirical strategies to measure the length. For example they may try to measure the length with a ruler or may use the given diameter as a scale.

ANALYSIS OF STUDENT SOLUTIONS

In this section a random sample of 500 student solutions to two of the illustrating items which include important features of the B-patterns are analysed, in order to clarify the results reported so far.

“Revolving door 01”A look at the student solutions of “Revolving door 01” reveals three common mistakes. For most of the wrong answers we cannot interpret the reasons behind the mistakes because most students have only written their answer without indicating a reason for it (they were not required in this item to show the complete calculation).

One of the most common (visible) mistakes is the answer “240”. The students divided the circle correctly into three parts of 120° each and then multiplied one segment by two. The question states “what is the size in degrees of the angle formed by two door wings”. It is possible that these students misunderstood the word “wings” and thus calculated the angle of two segments instead of the angle between two wings. It is also possible that they simply regarded the “two” in the question as a prompt to multiply their first result (120°) by 2 without regarding the geometric situation.

This mistake is shown in the two following solutions and exemplifies pattern B4 (Reasoning in a geometric context).



Another common wrong answer is “100”. This suggests that these students have flawed knowledge about circles and believe them to have a total angle of 300° or they were simply using the length of the radius as a value for the angle. Instead of reasoning in this geometric situation they simply took the given diameter (200) and divided it by 2 (because of the “two door wings”).

The answers “66.7”, “66.6” and “67” represent a third common mistake. For these answers we assume that the students simply divided 200 by 3, because 200 cm and (doors) are the numbers given in the text. These students did not reason in the geometric context but simply extracted the number 200 from the stimulus and did the obvious operation of dividing it by three. One of these mistakes is shown in the following solution.



“Revolving door 02”For this item, the sample of 500 solutions contains a great variety of answers and also lots of missing answers, which is certainly due to the high level of difficulty of the item. This item also did not require the students to show their working. The most common answer was 100 cm, which is the radius of the circle. Since the diameter is the only given value it can be expected that a lot of students used it to estimate the arc length of one of the openings. An example is the following solution.

In this solution the student took the diameter and measured the approximate arc length of the opening. This solution illustrates that the student reconstructed the situation but over-simplified it. He or she tried to figure out the arc length of the opening but ignored the curvature or replaced it by a linear segment. Apparently this student is not able to calculate the circumference.

The next solution also shows that the student was not able to deal with arc lengths. Indeed the solution displays an adequate mathematical model of the situation, and 60° is the correct angle, but the student simply used angles instead of arc lengths.



Both solutions therefore exemplify the requirements of pattern B1 (Curricular requirement: π is necessary). The fact that 100 cm and 60 cm were mentioned so often leads to the assumption that a lot of United States students were not able to calculate circumferences. Of course, since most of the students only wrote down the result it is difficult to interpret the underlying mistakes.

The second most common answer was 200 cm, which is the value of the diameter. The students offering this solution just indicated the value 200 cm without further explanations or considerations written down. Probably they did not take the situation seriously and picked out the only given value.

This might be caused by using the superficial classroom strategy “Don’t care about the context, just extract the numbers from the text and do some calculation”. This solution therefore also fits with pattern B3 (Genuine interpretation of real world aspects).

Nevertheless there are also solutions which display a correct calculation of the circumference. The following two solutions exemplify that some of the United States students have been able to calculate the circumference but had problems with understanding the real world situation or had not been able to formulate an adequate mathematical model. So these two solutions do not fit into pattern B1 (Curricular requirement: π is necessary) but into pattern B2 (Substantial mathematization of a real world problem).



The answer 209 cm was quite frequent. This student first calculated the circumference and then divided it by three. The result is the arc length of one sector. Maybe the student did not understand the situation and therefore used an incorrect mathematical model. But it is also possible that he or she recognized that the length of the opening of entrance and exit in total is one-third of the circumference.

The next student solution is slightly different:

Like the solution above, the student calculated one-third of the circumference which is about 209 cm. In contrast to the former solution it is clearly visible that the student calculated the arc length of a sector formed by two door wings and not the total length of the openings. It is noticeable that the student then wrote down the angle of the sector instead of the arc length, so this solution reveals two different student problems. The first problem is the use of the wrong mathematical model, namely considering the arc length of a sector instead of an opening. The second problem seems to be a confusion between angles and arcs.



FINAL OBSERVATIONS

The aim of this chapter was to identify the relative strengths and weaknesses of U.S. students in the PISA 2012 mathematics test. For that purpose, the U.S. solution rates (technically transferred to logits in order to get a linear scale) for all 84 PISA mathematics items that have been administered in the United States are compared with the solution rates of the OECD average and of five reference countries. The reference countries, which have all performed significantly above the OECD average and are, economically speaking, important partners and also competitors of the United States were: Canada, Germany, Korea, the Netherlands and Shanghai-China. Altogether 33 “conspicuous” items were identified, where the differences between the United States and the OECD average, or the five reference countries, were in some way statistically significant. These 33 items revealed certain patterns of relative strengths and weaknesses, where each pattern contains items with similar cognitive requirements. For 16 of these 33 items, the United States performed relatively well, resulting in three strength patterns. For 17 items the United States performed relatively badly, resulting in four weakness patterns.

It is interesting to see that the U.S. students do not perform uniformly compared with the OECD average or with the reference countries. Altogether, the U.S. average (43.8%) in PISA mathematics is below the OECD average (47.5%) and further below the averages of the other five countries. However, there are some items where the U.S. students performed nearly as well as Shanghai-China and better than the OECD average, as well as better than up to four of the other reference countries; these items constitute the relative strengths of the United States. On the other hand, there are items where the United States is much further below the OECD average and the reference countries than the average differences may suggest; these items constitute the relative weaknesses of the United States.

It is remarkable that the strength patterns consist nearly exclusively of easy PISA items and the weakness patterns mostly of demanding items. Both the absolutely easiest and hardest PISA items occur in the patterns, the easiest in the strength and the hardest in the weakness patterns. This is not as obvious as it may seem. It is clear that the United States, like all other countries, performs rather well in easy items and rather moderately in difficult items; this is the definition of “easy” and “difficult”. However, it is not at all obvious that the United States should perform relatively well or badly in these items. It seems that the U.S. students have particular strengths in cognitively less-demanding mathematical skills and abilities, such as extracting single values from diagrams or handling well-structured formulae. And they have particular weaknesses in demanding skills and abilities, such as taking real world situations seriously, transferring them into mathematical terms and interpreting mathematical aspects in real world problems. These are tasks where the well-known superficial classroom strategy “Don’t care about the context, just extract the numbers from the text and do some obvious operations” is bound to fail. This strategy is popular all over the world and frequently helps pupils and students to survive in school mathematics and to pass examinations. However, in a typical PISA mathematical literacy task, the students have to use the mathematics they have learned in a well-founded manner. The American students obviously have particular problems with such tasks. Of course, in more demanding tasks, some basic skills such as those mentioned (extracting values or handling formulae) are needed too, so the relative strengths of the U.S. students are a necessary prerequisite for solving higher-order tasks. Therefore, when it comes to the implications of these findings, one clear recommendation would be to focus much more on higher-order activities such as those involved in mathematical modeling (understanding real world situations, transferring them into mathematical models, and interpreting mathematical results), without neglecting the basic skills needed for these activities.



Notes

1. This standard deviation is 0.29, and the average of all differences is -0.21. The resulting threshold values are therefore 0.08 respectively -0.50.

2. This standard deviation is 0.36, and the average is -0.69. The resulting threshold values are therefore -0.33 respectively -1.05.

3. After each round of PISA a number of items are publicly released in order to illustrate what has been assessed in the assessment. These are typically used to illustrate the published framework and the analysis that appears in the international reports from PISA. Not all of the patterns identified in the analysis within Chapter 3 can be illustrated with the publicly released items.

4. Items in parentheses are second category items, which primarily fit in another pattern.

References

Freudenthal, H. (1983), Didactical Phenomenology of Mathematical Structures, Reidel, Dordrecht.

OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264190511-en

http://dx.doi.org/10.1787/9789264190511-en

3 strengths and weaknesses of american students in ... - oecd · strengths and weaknesses of...

Documents