Common language levels: mirage or panacea for the English industry?

Kieran O’Loughlin

Background #1ELICOS offered in Australia since mid-1980s

ELICOS delivered to student body 100% international

Offered at 240 institutions - private “standalone, VET, TAFE, secondary and university.

2005: 101,000 students whose visas were 49% student, 37% visitor, 14% working holiday and other.

65% of student visa holders in 2002 continued to further study.

Background #2Status and diversity of Australian ELICOS industry

Rationale for a national language levels framework

Nature of consultancy

Aims of the studyidentify factors the Australian ELT/ELICOS industry would need to consider in determining whether to make use of a language levels framework e.g. CEFR

assess benefits of adopting such a framework by the sector

consider the implications for providers and students of adopting such a system

MethodologyStep1 Background research2 Design of data collection instruments3 Review of methodology4 Distribution of draft interim report5 Collection of survey data6 Follow-up interviews7 Preparation of draft final report8 Circulation of draft final report9 Submission of Final (edited) report

Nature of language levels frameworks

Definition

Purposes

How frameworks differ from one another

What is a language levels framework?

A language levels framework is a series of language descriptions arranged sequentially to indicate an expected order of language progression or development over time.

A framework is NOT a test.

Purposes of language level frameworks

1. Establish a framework of reference which can describe achievement in a way that scores from test items cannot.

2. Provide coherent internal links within one system between pre-course or entry testing, syllabus planning, materials organisation, progress, exit assessment and certification using a common yardstick.

3. Enable comparison between systems or populations using a common set of standards.

How frameworks differ from one another

Target language/language variety

Learner groups

Purpose

Audience

Views of language proficiency/ language learning

Aids to implementation

Examples of currently used frameworks

ACTFL proficiency guidelinesALSPR/ ILSPRNLLIA ESL BandscalesCanadian Language BenchmarksCertificates of Spoken and Written English (CSWE)National Reporting System (NRS)Common European Framework of Reference (CEFR)

Common European Framework of Reference (CEFR) #1

Six sequential levels:

A B CBasic user Independent user Proficient user

A1 A2 B1 B2 C1 C2Breakthrough Waystage Threshold Vantage Effective Mastery

OperationalProficiency

Descriptors framed as “can do” statementsE.g. “Can summarise information from different spoken

and written sources”.

Common European Framework of Reference (CEFR) #2Accompanying tools:

Dialang

European language portfolio

User’s manual

Common European Framework of Reference (CEFR) #3CEFR has been used

to develop curricula e.g. English Profile Project -British Council

as a basis for test development e.g. Dutch CEFR construct project

to link tests to framework documents e.g. ETS mapping project

Range of expert viewsThose who support CEFR’s aim to create a

common language for describing learners progress and unifying curriculum and assessment

VERSUSThose who see the institutionalisation of

CEFR as politically motivated and as having unwarranted global influence.

Criteria for effective implementation of a frameworkNeed for the framework to:

be clear in purpose for its usershave a sound theoretical basespan the ability range of learners in the ELICOS sectormeaningfully describe learner development be taken up seriously by relevant stakeholders be highly practical be accompanied by appropriate materialsbe underpinned by standardised tests

Focus groupsHeld in Sydney and Melbourne, October 2006

Participants from a range of centres:Endorsed proposed methodology and timeline.Made suggestions for refinement of a) sampling methods and b) content and format of survey.

Survey of Directors of StudiesInitial sample of 46 ELICOS providers

proportionally representative in terms ofNEAS and non-NEAS accreditedthe various Australian statesprivate (stand alone), VET private,

TAFE, secondary and university.

Survey design

First part designed to elicit information from DoS about relevant features of their institution and their views about assessment needs and challenges.

Second part canvassed DoS’ knowledge, experience and views of different language frameworks as well as their attitudes regarding feasibility of implementing such a system within the sector.

Survey respondentsCompleted by 36 institutions across Australia:

34 NEAS 12 Private (stand alone) 6 university7 Private VET4 TAFE4 secondary schools1other

2 Non-NEAS

Survey results - Part 1Respondent profile (students, staff, type of operation)Type, level and duration of coursesSyllabi and course materialsAssessment practicesPathway arrangementsStaff with examiner credentialsAssessment problems and needs

Respondent profileStudents

Average enrolments for 2006:25.6%: < 10037.2%: 100-30037.2: > 300

StaffTotal number from 3.3 -120 with average of 28 across all providersAlmost 50% employed on contractMost had either CELTA or Grad Dip Ed

Type, level and duration of courses

Wide variety of courses and levels listed

General and Academic English as well as IELTS Preparation the most common.

Also large number of specific purpose courses e.g. linked to workplace needs, business travel and tourism.

Duration: range from 2 - 50 weeks. Most of limited duration, no longer than 12 weeks.

Syllabi and course materialsMajority of centres relied on syllabi produced in-house or mixture of in-house and textbook based.

13 centres based their syllabi exclusively on textbooks.

Only 7 centres relied solely on their own teaching materials.

Assessment practicesFor placement:Descriptors such as “Elementary”, “Intermediate” are most commonMajority of centres use placement testsFor promotion to higher levels:Most centres use in-house “off the shelf” assessment tools For certification:Most centres use continuous assessmentCertificate usually only awarded when student completes chosen programInformation about level of achievement on back of certificates varies considerably.

Pathway arrangementsMajority of providers had “direct entry”pathways with other institutions, mainly universities (67%).

Large majority prepared students for IELTS (83%). Minority of centres had other test preparation courses e.g. Cambridge exams, especially FCE, TOEIC, TOEFL.

Staff with examiner credentialsOn average, 20% of staff in ELICOS centreswere accredited IELTS examiners (although this varied widely across providers)

There were also accredited examiners of the Cambridge, TOEIC and Trinity College examinations amongst the providers surveyed.

Assessment problems and needsProblems (in order):Teachers apply standards inconsistentlyTeachers lack understanding about assessmentAssessments/tests are not sufficiently related to the

curriculumTeachers are reluctant to use explicit criteriaAssessments/tests are not appropriate for students’

proficiency level

Needs (in order):End-of-level testsDiagnostic testsSelf-assessment toolsPlacement proceduresExit tests

Survey results - Part 2English language frameworksViews on desirability of a frameworkType of framework favouredReasons for choice of frameworkEducation about frameworksNature of preferred frameworkNumber of levelsUse of a frameworkAcceptability to studentsDisadvantages of a framework

English language frameworksMost familiar frameworks (in order):ISLPR, CSWE, CEFR, NRS

Most commonly used frameworks (in order): ISLPR, CSWE, NRS and CEFR

Views on desirability of a framework

76% of respondents (30 of 36) were positive in principle about the idea of a common national framework

However, many respondents gave a mixture of positive and negative views when asked to explain their initial responses.

Positive views: transparency, portability, objectivity, professionalism, quality and credibility.

Negative views: inflexibility, homogenization, impracticality and redundancy.

Transparency and portability“It would of course help every institution

understand the English language skills of any student who wishes to enrol in either an ELICOS program or a university/TAFE program.”

“I believe that a national framework would make ELICOS results much more useful for students in future study & employment.”

Objectivity“Currently it is difficult for education providers to

make accurate judgements about a student's proficiency based on a certificate from an ELICOS centres; one's opinion of the centre or knowledge of its reputation is factored into the judgement of the value of the certificate.”

“It would be great to have a degree of standardisation between schools and set tests for teachers to use. It is very hard to design valid and reliable tests with little time available.”

Professionalism“A national framework would hopefully

educate staff in assessment.”

“[It would] aid in the development of curricula and syllabi.”

Quality and credibility“To ensure quality across the board, which will in

turn improve the industry”

“It is time to develop a true international test and reporting system, which has global integrity and recognises the modern purpose for language study: international information technology, global employment, travel and communication.”

Inflexibility and homogenisation“(A common framework) might lead to a lack of

flexibility in future programs as every ELICOS centre would need to teach to the same objectives or thereabouts, leading to a certain sameness in English progammes, that may harm future trade.”

“Niche markets developed by institutions may be harmed if the focus of the students narrows to finding the ELICOS program that other students have found easy to get the levels they need. “

“Should not be imposed. One-size-fits-all approach would not work. Australian context/market too small.”

Impracticality

“Too often the processes involved to meet criteria with training, ongoing PD, recording and reporting are too time consuming or inadequate.”

“There is a great deal of work involved for all centres in aligning current assessment andmarking criteria to a national standard. This would be at considerable cost and many centres would be unwilling to bear this.”

Redundancy“…just yet another reporting mechanism to

add to a full basket.”

“…there are already external working alternatives (IELTS and other tests, ESL Scales, etc) and what are these scales used for ? Where it counts (ie. in dealing with other institutions) most colleges have direct entry agreements already.”

Type of framework favouredThree most popular (in order of preference):

1. CEFR

2. ISLPR

3. NLLIA bandscales

Reasons for choice of frameworkCEFR:International currencyWell developed Specifies both proficiency levels and types of language study (e.g.social, academic, business)

ISLPR:Already in use

NLLIA bandscales:School-based and thus suitable for younger learners

Education about frameworksFour most popular options (in order):

In-house meeting or professional development sessions

Published information forwarded to each ELICOS centre

Externally run training sessions by industry

Government led training sessions

Nature of preferred framework and number of levelsNature of framework (in order):General EnglishAcademic EnglishWorkplace English

Number of levels:6 or more

Use of a frameworkMost important:

PlacementCertificationExit assessment

Also:End of level assessment Curriculum planning Self assessment

Acceptability to students“They will accept it only if it becomes the new

gatekeeper, they will welcome it if it can be shown to have status outside Australia (e.g. through its equivalence to European Reporting standards and standardised international tests).”

“I think they would believe it gives authority to their assessment. They would like it to provide a useful and recognised certificate for future work or study.”

Disadvantages of a framework

Two additional concerns:Two additional concerns:

Costs and resources e.g. Costs and resources e.g. need for an need for an independent moderating authorityindependent moderating authority

Lack of staff expertise in using standards Lack of staff expertise in using standards appropriately.appropriately.

Interviews7 of the questionnaire respondents and another DoS from a non-NEAS centre were interviewed.

Interviewees reinforced and extended the range of views expressed in the surveys.

Summary of key findingsMajor issues:

Variability of ELICOS providersAssessment a problem areaAttractiveness of a national frameworkProposed uses vary widelyMixed understandings about frameworksType of framework favouredProblems of implementationConcerns about homogenizing effect of a framework

ConclusionsMandating adoption of national framework

not feasible for ELICOS sector

However, could link existing frameworks, tests and curricula loosely to CEFR

RecommendationsIn-service and action research program allowing providers to familiarise themselves with CEFR

Further research into the creation of a national EAP certificate for “direct entry” into further education courses.

Development of an inventory of existing test materials already mapped onto CEFR and identifying gaps where new tests might be needed

ELICOS language levels ELICOS language levels feasibility study feasibility study

Final report Final report

Prepared for the Department of Prepared for the Department of Education, Science and TrainingEducation, Science and Training

byby Cathie Elder & Kieran OCathie Elder & Kieran O’’LoughlinLoughlin

April 2007April 2007