assessing student learning in economics: the mexican experience adrián de león arias, ph.d....

Assessing Student Learning in Economics: The Mexican

ExperienceAdrián de León Arias, Ph.D.

Economics NPM, Mexico

Washington DCJuly 11, 2013

AHELO—What Can We Learn from OECD’s Feasibility Study?

Assessing Student Learning in Economics 2

Issues to be addressed

1. Why was Mexico Economics Strand included in the AHELO Feasibility Study?

2. What was anticipated in Mexico from the participation in the Feasibility Study?

3. What was obtained for Mexico from the participation in the Feasibility Study?

7/11/2013


1. Why was Economics included in the AHELO Feasibility Study?

Motivation (1)In the interest of the OECD, the Economics strand was included in the AHELO Feasibility Study. (Paris meeting, 5-6 July 2007, OECD).

“The experts recommended conducting the feasibility study in subject areas, which have a rather stable core of methodologies such as engineering or economics and possibly expand the range of covered subjects over time” (Paris meeting, 5-6 July 2007, OECD).

7/11/2013



Motivation (2)“It was not certain at the beginning of the feasibility study that academics from different countries would agree on what to measure in the disciplines as well as on an assessment instrument, especially in a social science like economics. Therefore an economics strand was included in the feasibility study” (OECD 2012, Feasibility Study Report, Volume 1).

Mexico had accepted to be part of the General Skills and Engineering strands, so getting involved in the Economics module was just a natural consequence.(About October 2010?)

7/11/2013



Disciplinary field tests in Mexico• Even before the AHELO Feasibility Study, disciplinary field tests already

existed in Mexico.

• Those tests are offered by CENEVAL (for its acronym in Spanish, National Center for Higher Education Assessment).

• They are called EGEL® (General Tests for Undergraduate Egress).

7/11/2013



EGEL® by CENEVAL (1)• Purpose of the test. EGEL identifies whether BA graduates have the knowledge and skills

necessary to begin effectively in practice.

• Target population. Graduates of BA in Economics who have covered 100% of the credits, and if necessary students enrolled in the final semester of the program, as long as the HEI so requests.

• Type of instrument: MCQ• Instrument length: 5 areas, 16 subareas, 53 topics, 108 contents.• Number of items that are scored: 179.

Source: http://www.ceneval.edu.mx/ceneval-web/content.do?page=2176

7/11/2013

http://www.ceneval.edu.mx/ceneval-web/content.do?page=2176




EGEL® by CENEVAL (2)

Current general structure of EGEL-ECONO by areas.Mexico, June 2013.

7/11/2013

* Additionally, a percentage of pilot items is included but not used to grade.Source: http://www.ceneval.edu.mx/ceneval-web/content.do?page=2176

Area % in the test Items distributionA Business Economics 23.5 42

B Financial Economics 16.2 29

C Public Economics 25.7 46

D Economic Growth and Development 12.8 23

E International Economics 21.8 39

Total* 100 179




EGEL® by CENEVAL (3) As a second generation test, EGEL assesses the application of knowledge, not knowledge itself (i.e.,

no conceptual questions are included).

• In the past, basic concepts from “traditional” areas -such as Microeconomics, Macroeconomics, Quantitative Methods, etc.- were evaluated.

• But as of 2010, there was a shift in focus: to evaluate the development of applied knowledge (though not mentioned the concept of competencies, assessment seems to go in that direction).

• In addition, there was a restatement of topics. For instance, Economic History and Mathematics questions were replaced by items in Applied Micro-and Macroeconomics.

In terms of grading, a cutoffs review is carried out annually by the test council, on the basis of results obtained by the examinees in the period before.

The criticism from some HEI to EGEL current structure is that not all Economics undergraduate degree programs cover such contents in their compulsory subjects block .

7/11/2013



7/11/2013

First admission Total enrollmentBA graduates

(previous year)

Year Men Women Total Men Women Total Men Women Total

Public 2,521 1,750 4,271 10,173 7,895 18,068 1,197 910 2,107

1999 Private 311 206 517 2,481 1,267 3,748 422 232 654

Total 2,832 1,956 4,788 12,654 9,162 21,816 1,619 1,142 2,761

Public 2,380 1,886 4,266 9,843 7,550 17,663 1,474 1,205 2,679

2004 Private 350 194 544 2,113 1,173 3,276 413 225 638

Total 2,730 2,080 4,810 11,956 8,723 20,939 1,887 1,430 3,317

Source: Ahumada & Butler (2009). La enseñanza de la Economía en México. Cuadro A.1, page 55.

Undergraduate programs in Economics – Mexico (1)

Economics students in Mexican HEI, 2010 and 2011.



Undergraduate programs in Economics – Mexico (2)• There are difficulties in knowing "what an economist should be" through educational

systems. In other words, there is a problem on this “disciplinary field” concept.

• For instance, in the USA, a student would choose a major in Economics and a minor in something else, while a student in Mexico would choose Economics as an entire undergraduate degree (four or five years).

• Furthermore, business schools teach Economics in a different fashion, and undergraduate in business include some Economics courses.

• Hence, if there is no common curriculum within our country –either for studying Economics in general or any sub-strands more specifically-, much less across countries.

7/11/2013



Undergraduate programs in Economics – Mexico (3)

Amount of HEI that offer Economics undergraduate programs.Mexico, 2010.

7/11/2013

B.A. Programs Amount of HEI

Economics 118Economics and Management 1Economics and Business Management 2Agricultural economics 2Financial economics 13Total 136

Source:http://www.observatoriolaboral.gob.mx/wb/ola/ola_anuies_profesional?idSubarea=32&idEntidad=-1&carrera=Econom%EDa



Participating HEI• Ten HEI responded positively to the call of the Economics

NPM in Mexico.

• In this group, there were eight public state universities, one polytechnic-type institution and the remaining one was private.

7/11/2013


2. What was anticipated from the participation in the Feasibility Study?

Participating HEI outlook• All HEI considered the learning

outcomes captured in the framework as important.

• Their main goal was to receive feedback on the performance of their students.

• Except for one HEI (the private one), they did not care about a ranking.

7/11/2013

Learning Outcomes in the Framework

i. Subject knowledge and understandingii. Subject knowledge and its application to

real world problemsiii. Effective use of relevant data and

quantitative methodsiv. Effective communication (to specialists

and non-specialists)v. Acquisition of independent learning skills

Competenciesa. Abstraction b. Analysis, deduction and induction c. Quantification and design d. Framing



Students participation (1)• Students were invited to participate without too many

information.

• Their participation was enthusiastic, and they keep asking for results which cannot be delivered. Arguably, the way to treat them was unfair.

7/11/2013



Effective participation of students per institution.AHELO Feasibility Study, Economics module. Mexico, 2012.

7/11/2013

Institution StudentsID Institution name Ex-post Ex-ante Attainment1 Universidad Autónoma de San Luis Potosí 46 59 78%2 Universidad de Guadalajara 40 48 83%3 Universidad Autónoma de Yucatán 18 39 46%

4Instituto Tecnológico y de Estudios Superiores de Monterrey, Campus Monterrey 35 38 92%

5 Universidad Veracruzana 19 23 83%6 Instituto Politécnico Nacional 131 200 66%7 Universidad Autónoma de Coahuila 15 15 100%8 Universidad Autónoma de Chihuahua 29 32 91%9 Universidad Autónoma de Zacatecas 45 61 74%10 Universidad de Colima 24 26 92%

Total 402 541 80%*

Source: Compiled from the databases of the feasibility study.* Average of means. Total mean was 74%.


3. What was obtained from the participation in the Feasibility Study?

Unfortunately, mainly nothing. Just a score which represents students’ general test

performance, but that is not broken down by learning outcomes or competencies.

A mapping (correspondence) was expected between learning outcomes (captured in the framework) and items, but this did not occur either in the test or in the scores.

7/11/2013


Cumulative student score distributions for all institutions and for this institution (ID 2). Mexico, 2012.

Source: OECD 2012, AHELO Feasibility Study Institution Report.


7/11/2013

0

10

20

30

40

50

60

70

80

90

100

50 150 250 350 450 550 650 750 850 950

Studen

ts (c

um

ula

tive

per

cen

t)

Economics Score

This institution Mean of all institutions All institutions


Expected mapping between learning outcomes/competencies and items (example)

7/11/2013 Assessing Student Learning in Economics 18

Learning Outcome

iMCQ#

Learning Outcome ii CRT1

-Item i

Learning Outcomes in the Framework

i. Subject knowledge and understandingii. Subject knowledge and its application to

real world problemsiii. Effective use of relevant data and

quantitative methodsiv. Effective communication (to specialists

and non-specialists)v. Acquisition of independent learning skills

Competenciesa. Abstraction b. Analysis, deduction and induction c. Quantification and design d. Framing



Students opinion (1)As results of focus group implementation, we found out that…

• On the assessed skills and knowledge, students considered them as relevant for an economist. Also, students thought the test helped themselves to know their academic level.

• As to the assessment duration, students considered the test seemed to evaluate “speed”; they felt more time was needed to respond properly. They suggested to allocate separate periods of time to each section (CRT and MCQ). They noticed that completing the test during the meal time was not appropriate.

7/11/2013



Students opinion (2)As results of focus group implementation, we found out that…

• Regarding the students' abilities to analyze and synthesize, they considered them as undeveloped, in order to respond the test correctly. They were afraid of the unknown, such as assessment content, types and amount of items. Furthermore, the pilot test made students feel uncertain, because it was challenging for them. Students thought that completing CRT was more challenging and innovative than answering MCQ.

• On other aspects, students suggested that an answer sheet should be designed, so they could validate their answers. Examinees related this kind of test to GRE and to EGEL (applied by CENEVAL).

7/11/2013



Conclusions

1. The AHELO Feasibility Study did serve to know the operability of an international AHELO in terms of costs and organization.

2. Nevertheless, this Feasibility Study did not serve in terms of instrument efficiency. The AHELO test was not even close to the instrument that should have been, one which truly assessed learning outcomes. While MCQ replicated the traditional knowledge assessment, CRT were described as “more challenging” by students.

3. Everything was done in "good faith" (voluntary participation), but it is not sustainable in the long term.

7/11/2013

assessing student learning in economics: the mexican experience adrián de león arias, ph.d....

Documents