acer test of abstract reasoning manual

Australian Council for Educational Research

M A N U A LTest of Abstract ReasoningACER

TAR

TEST

OF

ABST

RACT

REA

SONI

NGTE

ST O

F AB

STRA

CT R

EASO

NING

TE

ST O

F AB

STRA

CT R

EASO

NING

TE

ST O

F AB

STRA

CT R

EASO

NING

TE

STOF

ABST

RACT

ACER TEST OF ABSTRACT REASONING

TAR can be used for various courses and occupations, including:Managers and executives

Business managers

Professionals and specialists

Para-professionals (e.g. technicians)

Arts professionals

Finance and data analysts

Medical professionals

Scientists

The ACER Test of Abstract Reasoning (TAR) is a quick and easily administered test ofabstract reasoning in adults. Abstract reasoning is linked to the ability to thinkstrategically, analyse information, learn new information quickly, engage in creativeproblem-solving and make logical decisions. TAR is suitable for use as a selection ordevelopment tool for graduate, managerial and other positions that require critical thinkingskills. The test is free from language, cultural and gender biases.

TAR is available online and in paper-based formats, and is easy to administer andinterpret. It provides immediate results using Australian norms and can be used inindividual or group testing situations.

TARISBN 10: 0-86431-733-6ISBN 13: 978-0-86431-733-9

9 7 8 0 8 6 4 3 1 7 3 3 9

TAR 1/11/06 3:21 PM

ACER Press

First published 2007by ACER PressAustralian Council for Educational Research Ltd19 Prospect Hill Road, Camberwell, Victoria, 3124

Copyright © 2007 Australian Council for Educational Research

All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publishers. Only material contained in the photocopy masters may be reproduced, and then only in quantities sufficient for the purchaser’s own proper use and not for resale to, or use by, any other person or organisation.

Edited by Elisa WebbCover design by R.T.J. KlinkhamerTypeset by Cannon Typesetting, MelbournePrinted by Print Impressions

National Library of Australia Cataloguing-in-Publication data:


ACER test of abstract reasoning manual.

Bibliography. For secondary students. ISBN 9780864317339.

ISBN 0 86431 733 6.

1. Reasoning (Psychology) – Testing – Handbooks, manuals, etc. I. Title.

153.43

Visit our website: www.acerpress.com.au

iii

ContentsList of tables .................................................................................................................................................vList of figures ................................................................................................................................................viList of appendices ............................................................................................................................................vi

About ACeR test of Abstract Reasoning (ACeR tAR) ............................................................................ 1Background .............................................................................................................................................1Description of the tests ............................................................................................................................1Typical uses .............................................................................................................................................2

Occupational uses ..............................................................................................................................2Educational and vocational guidance uses ..........................................................................................2

test options ............................................................................................................................................... 3Short or long form ...................................................................................................................................3

Short Form (TAR45) .........................................................................................................................3Long Form (TAR60) ..........................................................................................................................3

Paper or online administration ................................................................................................................3Paper .................................................................................................................................................3Online ...............................................................................................................................................3

scoring and Reporting ............................................................................................................................. 4Paper administration ...............................................................................................................................4

ACER Test Scoring Services ...............................................................................................................4Online Response Entry System ..........................................................................................................4

Online administration .............................................................................................................................5Automated email reporting ................................................................................................................5ACER Online Testing System ............................................................................................................5Direct request ....................................................................................................................................5

Interpretation of ACeR tAR Reports ....................................................................................................... 7Interpreting norm scores .........................................................................................................................7Percentile ranks .......................................................................................................................................7Stanine scores ..........................................................................................................................................7Rasch scores .............................................................................................................................................8Comparison with Raven’s Standard Progressive Matrices and Advanced Progressive Matrices ...............8

Case studies ............................................................................................................................................. 11Example 1: Recruitment for a public sector department .......................................................................11Example 2: Call centre selection ............................................................................................................11Example 3: Promoting floor staff to store manager ...............................................................................11

Directions for Administration ................................................................................................................ 13General information ..............................................................................................................................13General information for online testing ..................................................................................................13

Minimum system requirements ........................................................................................................14Specific instructions...............................................................................................................................14Instructions for paper administration ...................................................................................................15Instructions for online administration ..................................................................................................19

technical Information ............................................................................................................................. 22Summary of research studies .................................................................................................................22Study 1 – ACER TAR Short Form (TAR45) .........................................................................................23

Study design ....................................................................................................................................23Study sample ...............................................................................................................................23Testing procedure .........................................................................................................................24

iv

Item analysis ....................................................................................................................................25Reliability ........................................................................................................................................25TAR45 subsets .................................................................................................................................26Correlations .....................................................................................................................................26

Set correlations ............................................................................................................................26Set reliability ..............................................................................................................................27

Validity ............................................................................................................................................27Correlations with Raven’s Standard Progressive Matrices ................................................................27

Scale scores ......................................................................................................................................28Demographic variables .....................................................................................................................28

Gender .......................................................................................................................................28Education level ...........................................................................................................................29Occupational classification ...........................................................................................................29Language background ..................................................................................................................30Age groups ..................................................................................................................................30Test order ....................................................................................................................................31

Conclusion ......................................................................................................................................31Study 2 – ACER TAR Long Form (TAR60) ..........................................................................................32

Study design ....................................................................................................................................32Study sample ...............................................................................................................................32Testing procedure .........................................................................................................................33

Item analysis ....................................................................................................................................33Reliability ........................................................................................................................................34TAR60 subsets .................................................................................................................................34Correlations .....................................................................................................................................34

Set correlations ............................................................................................................................34Set reliability ..............................................................................................................................35

Validity ............................................................................................................................................36Correlations with Raven’s Advanced Progressive Matrices ...............................................................36Set correlations ............................................................................................................................36Calculating the scale scores ...........................................................................................................37Equating TAR60 to APM ............................................................................................................37

Background variables .......................................................................................................................38Gender .......................................................................................................................................38Education level ...........................................................................................................................38Language background ..................................................................................................................39Study type ...................................................................................................................................39Test order ....................................................................................................................................40

ACER TAR Short Form (TAR45) ....................................................................................................40Background variables .......................................................................................................................40

Gender .......................................................................................................................................40Education level ...........................................................................................................................41Language background ..................................................................................................................42Study type ...................................................................................................................................42Test order ....................................................................................................................................43

Conclusion ......................................................................................................................................43Study 3 – ACER TAR Short Form (TAR45) 2006 ................................................................................44

Study design ....................................................................................................................................44Reliability ...................................................................................................................................44Gender differences .......................................................................................................................45

Conclusion ......................................................................................................................................46

References ...............................................................................................................................................47Appendices ...............................................................................................................................................48

C o n t e n t s

v

LIst of tAbLes

Table 1 Source of items in ACER TAR ..........................................................................................1Table 2 Example occupation groups ..............................................................................................2Table 3 Examples of appropriate forms for selected jobs ................................................................3Table 4 Stanine groups and descriptors ..........................................................................................7Table 5 Simple conversion table between ACER TAR45 and Raven’s SPM ...................................9Table 6 Simple conversion table between ACER TAR60 and Raven’s APM .................................10Table 7 Key statistical findings in Study 1 ...................................................................................22Table 8 Key statistical findings in Study 2 ...................................................................................22Table 9 Key statistical findings in Study 3 ...................................................................................22Table 10 Demographics – Gender .................................................................................................23Table 11 Demographics – Highest level of education completed ...................................................23Table 12 Demographics – Occupational classification ...................................................................24Table 13 Demographics – Age group .............................................................................................24Table 14 Demographics – Language spoken at home ....................................................................24Table 15 Demographics – Order of tests .......................................................................................25Table 16 TAR45 set descriptions ...................................................................................................26Table 17 TAR45 internal set correlations .......................................................................................26Table 18 TAR45 set correlations with total score ...........................................................................26Table 19 TAR45 set reliabilities .....................................................................................................27Table 20 TAR45 mean and standard deviation by gender ..............................................................28Table 21 ANOVA TAR45 scores on gender ...................................................................................28Table 22 TAR45 mean and standard deviation by education level .................................................29Table 23 ANOVA TAR45 scores on educational level....................................................................29Table 24 TAR45 mean and standard deviation by occupational classification ................................29Table 25 ANOVA TAR45 scores on occupational classification .....................................................30Table 26 TAR45 mean and standard deviation by language ...........................................................30Table 27 ANOVA TAR45 scores on language ................................................................................30Table 28 TAR45 mean and standard deviation by age groups ........................................................30Table 29 ANOVA TAR45 scores on age groups .............................................................................31Table 30 TAR45 mean and standard deviation by test order ..........................................................31Table 31 ANOVA TAR45 scores on test order ...............................................................................31Table 32 Demographics – Study types ...........................................................................................32Table 33 Demographics – Highest level of education completed ...................................................32Table 34 Demographics – Gender .................................................................................................33Table 35 Demographics – Language spoken at home ....................................................................33Table 36 Demographics – Order of tests ......................................................................................33Table 37 TAR60 set descriptions ...................................................................................................34Table 38 TAR60 internal set correlations .......................................................................................35Table 39 TAR60 set correlations with total score ...........................................................................35Table 40 TAR60 set reliabilities .....................................................................................................35Table 41 TAR60 set correlations with APM total score ..................................................................36Table 42 Correlations of TAR60 items with APM total score ........................................................36Table 43 TAR60 mean and standard deviation by gender ..............................................................38Table 44 ANOVA TAR60 scores on gender ...................................................................................38Table 45 TAR60 mean and standard deviation by education level .................................................38Table 46 ANOVA TAR60 scores on educational level....................................................................39Table 47 TAR60 mean and standard deviation by language ...........................................................39Table 48 ANOVA TAR60 scores on language ................................................................................39Table 49 TAR60 mean and standard deviation by study types .......................................................39Table 50 ANOVA TAR60 scores on study types ............................................................................40

v

vi

Table 51 TAR60 mean and standard deviation by test order ..........................................................40Table 52 ANOVA TAR60 scores on test order ...............................................................................40Table 53 TAR45 mean and standard deviation by gender ..............................................................41Table 54 ANOVA TAR45 scores on gender ...................................................................................41Table 55 TAR45 mean and standard deviation by education level .................................................41Table 56 ANOVA TAR45 scores on education level ......................................................................41Table 57 TAR45 mean and standard deviation by language ...........................................................42Table 58 ANOVA TAR45 scores on language ................................................................................42Table 59 TAR45 mean and standard deviation by study types .......................................................42Table 60 ANOVA TAR45 scores on study types ............................................................................43Table 61 TAR45 mean and standard deviation by test order ..........................................................43Table 62 ANOVA TAR45 scores on test order ...............................................................................43Table 63 Number of males and females in the reference group ......................................................44Table 64 Descriptive statistics for TAR45 ......................................................................................44Table 65 Reliabilities of TAR45 subscales ......................................................................................45Table 66 Correlations between subsets in TAR45 ..........................................................................45Table 67 TAR45 scores for males and females ...............................................................................45Table 68 Statistics from independent sample t tests .......................................................................46

LIst of fIguRes

Figure 1 Example of a group report from ACER Test Scoring Services ............................................4Figure 2 Example of an automated email report ..............................................................................5Figure 3 Example of a group report from the Practitioner’s site .......................................................6Figure 4 Bell-shaped distribution ....................................................................................................8Figure 5 95% confidence belt for equating between TAR45 logits and SPM logits .......................27Figure 6 95% confidence belt for equating between TAR60 logits and APM logits .......................37

LIst of AppenDICes

Appendix A QUEST item analysis for all 60 TAR items (Study 1) .....................................................48Appendix B QUEST output for TAR45 (Study 1) .............................................................................50Appendix C Detailed conversion table between TAR45 and SPM (Study 1) .......................................52Appendix D QUEST item analysis for all 60 TAR items (Study 2) .....................................................53Appendix E QUEST output for TAR45 (Study 2) .............................................................................55Appendix F Detailed conversion table between TAR60 and APM (Study 2) ......................................57

L i s t o f Ta b l e s

�

About ACeR test of AbstRACt ReAsonIng (ACeR tAR)

backgroundACER Test of Abstract Reasoning (ACER TAR) has been developed in response to demand for an abstract reasoning assessment with Australian adult data.

The items from ACER TAR have been sourced from two existing, well-researched ACER tests of abstract reasoning: the Middle Years Ability Test (MYAT) and the Aptitude Profile Test Series (APTS). Items from these two existing tests have been combined to create the ACER TAR Long and Short forms:

Table 1 Source of items in ACER TAR

Test form Item source

ACER TAR Short Form (TAR45) 45 abstract reasoning items from MYAT

ACER TAR Long Form (TAR60) 45 abstract reasoning items from MYAT15 abstract reasoning items from APTS

Items were chosen to reflect an appropriate range of difficulty and task sub-type.The data samples detailed in this manual are based on:

(a) a study conducted on 151 adults in Australia who completed both ACER TAR and the Raven’s Standard Progressive Matrices (SPM);

(b) a study conducted on 295 adults at an Australian university who completed both ACER TAR and the Raven’s Advanced Progressive Matrices (APM); and

(c) a study conducted on 329 adults seeking work who had completed ACER TAR online.

Further research studies and analyses of ongoing data collection projects will be reported in future editions of this manual.

Description of the testsACER TAR is a test of abstract (or non-verbal) reasoning.

The MYAT Manual defines abstract (non-verbal) reasoning as ‘the extent to which [a person] can discover principles and rules, and apply them to solve problems using abstract visual patterns rather than numbers and words’ (p. 1).

The APTS Manual defines abstract reasoning as the ability to ‘discover the principle/s [that] determine… the rule or rules that govern the progression of a pattern from one diagram to another in a series, or to identify the part which is missing from a diagram’ (p. 3).

Tests of abstract reasoning such as the ACER TAR are nearly language-free and so may avoid problems asso-ciated with the use of verbal reasoning tests in some contexts where candidates’ written English skills are not well developed, such as where candidates are from non-English speaking backgrounds or have specific language learning difficulties. Statistical analyses detailed in this manual (see Technical information, p. 22) confirm that language background is not a significant source of variance in performance on the ACER TAR.

�

A C E R Te s t o f A b s t r a c t R e a s o n i n g

typical uses

Occupational uses

ACER TAR is typically used as a measure of general ability for selection into occupations which involve a moderate to high level of demand on reasoning ability and for other purposes where the ability to think clearly, analyse infor-mation, solve problems and make rational decisions are important.

Examples of selection criteria linked to measures of abstract reasoning are:

• Planning and coordinating activities• Creating innovative solutions to problemsCreating innovative solutions to problems• Making decisions in a rational mannerMaking decisions in a rational manner• Learning new material quicklyLearning new material quickly• Collecting and analysing informationCollecting and analysing information

(Power, 2004, p. 24)

Table 2 provides examples of occupational groups for which the ACER TAR can be used.

Table 2 Example occupation groups

Main occupation group Examples of occupations

Managers and administrators General and specialist managers, managing supervisors, business managers

Professionals Scientists, architects, engineers, medical practitioners, school teachers, social and business professionals

Para-professionals Medical and science technical officers, technicians, technical workers, registered nurses, police officers

Tradespersons Fitters, mechanics, electrical and electronic tradespersons

Artistic and creative Graphic artists, photographers, pianists, actors, dancers, desktop publishers

Figures and computational Accountants, clerks, data analysts, computer programmers, mathematicians, physicists

Literary Authors, journalists, editors, script writers

Medical Surgeons, general practitioners, nurses

Technical and engineering Electrical technicians, mechanical engineers, architects

Scientific Biologists, chemists, physicists, mathematicians, surveyors, teachers

(APTS, 2000, pp. 6–7)

Educational and vocational guidance uses

The ACER TAR tests may also be used in conjunction with a record of attainment and other data, in counselling a candidate in selecting an appropriate course of study, and in the selection of occupations where further study and training are involved. (See Nelson Bolles, 2002; Holland, 1997; Kummerow, 1991; Anastasi, 1976; Cronbach, 1970.)

�

test optIons

short or long formACER TAR has two forms: a Short Form (TAR45) and a Long Form (TAR60).

Short Form (TAR45)TAR45 is the easier of the two tests and is intended for use with candidates who have completed at least Year 11 and who are applying for a variety of technical, clerical and administrative positions where in-service training, or part-time or after work study is involved.

Long Form (TAR60)TAR60 includes the 45 items in the Short Form, plus an additional 15 items. TAR60 is the more challenging test and is appropriate for use with candidates who have completed at least Year 12 and are applying for positions where the work requires a high level of reasoning ability or where these qualities will become increasingly important as staff are promoted in a career system.

Table 3 provides examples of appropriate forms for selected jobs.

Table 3 Examples of appropriate forms for selected jobs

Short Form (TAR45) Long Form (TAR60)

Personal assistantMarket research interviewerAccounts clerkInsurance investigatorCustomer service operatorCall centre operatorWorkplace trainerLegal clerk

Journalist/copywriterComputer programmer/analystMarket research analystFinancial advisorsGraduatesManagersExecutives

paper or online administrationACER TAR is available for online administration or administration in paper and pencil format. In both cases ACER TAR should always be conducted under appropriate supervision. In no circumstances should ACER TAR be administered in an unsupervised environment. Candidates should not be provided with login codes to access the online test from home or other remote locations unless appropriate supervision has been put in place.

PaperTest administrators should provide one ACER TAR reusable test booklet and one ACER TAR answer sheet for each candidate. Completed ACER TAR OMR answer sheets should be sent to ACER Test Scoring Services for scoring and report generation. Alternatively, responses on non-OMR answer sheets can be keyed into the Online Response Entry System by the test administrator for immediate access to results.

OnlineTest administrators should provide each candidate with a unique login code, which allows candidates to access the tests online. On completion of the online test, results will be automatically generated and a report emailed to the test administrator. Further details regarding online testing are provided in Directions for Administration on p. 13. Contact ACER Press Customer Service on 1800 338 402 for more information about setting up an online testing account.

�

sCoRIng AnD RepoRtIng

There are no score keys or norm tables in this manual (apart from those relating to conversion between ACER TAR and Raven’s tests on pp. 9–10). ACER TAR can only be scored through one of the automated scoring systems described below.

paper administrationACER TAR can be administered in paper and pencil format using the ACER TAR test booklet and an ACER TAR answer sheet. There are two options for scoring and reporting.

ACER Test Scoring Services

Information on how to use the ACER Test Scoring Services can be found at www.acer.edu.au/tss. ACER TAR OMR answer sheets sent in for test scoring should be accompanied by a copy of the Test Scoring Order Form that comes with orders of the answer sheet.

Reports will be provided to test administrators in a format similar to the report shown in Figure 1 below. Candidates can be listed alphabetically or based on test scores.

Figure 1 Example of a group report from ACER Test Scoring Services

Online Response Entry System

Test administrators can manually enter data from non-OMR administrations into the online system. ACER TAR Online Response Entry login codes can be purchased from ACER Press Customer Service. Once the response data has been confirmed, the online system will automatically score the responses and generate a report. See the Online administration section (p. 5) for more details on getting access to candidate results.

�

online administrationTests completed online are automatically scored by the system. Reports can be obtained in one of three ways:

• automated email reporting;automated email reporting;• via the Practitioner’s site of the ACER Online Testing System; orvia the Practitioner’s site of the ACER Online Testing System; or• direct request to ACER Press Customer Service.direct request to ACER Press Customer Service.

Automated email reporting

To obtain automated email reports, an email address must be supplied at the time of purchasing ACER TAR login codes. The email address supplied should be that of the test administrator or HR manager, not the candidate.

A separate email report (see Figure 2) is generated for each ACER TAR candidate and automatically sent to the test administrator once the responses are submitted online.

Figure 2 Example of an automated email report

ACER Online Testing System

Test administrators can set up an Online Testing System account by contacting ACER Press Customer Service on 1800 338 402. Once this is done, individual or group results can be accessed at any time from the Practitioner’s site. An example of how results are displayed is shown in Figure 3 (p. 6).

Direct request

Test administrators who are conducting large-scale testing may prefer to wait until the last candidate has completed testing and then request ACER Press Customer Service to compile the results of all candidates in the group in a single report.

Direct request reports are also available for test administrators using the Online Response Entry System for scoring and reporting of paper tests.

S c o r i n g a n d R e p o r t i n g

�


Figure 3 Example of a group report from the Practitioner’s site.

�

InteRpRetAtIon of ACeR tAR RepoRtsThe raw score on a general ability test has little meaning in itself. Usually the first step in interpreting performance is to convert the number of correct answers (raw scores) into terms that will allow for comparison with a reference group who have taken the test (normative data).

Normative data for TAR45 have been obtained from a sample of 151 adults in Australia. See Study design, p. 23 for more information about the sample.

Normative data for TAR60 have been obtained from a sample of 295 adults at an Australian university. See Study design, p. 32 for more information about the sample.

Converted scores in the ACER TAR reports are presented in terms of percentile ranks, stanines and Rasch scale scores.

Interpreting norm scoresNorm scores such as percentile ranks and stanines (see below) for use with selection instruments such as ACER TAR enable recruitment officers to place candidates in order of merit, and to select an appropriate cut off point below which candidates will be excluded from further consideration. (See Case studies, p. 11 for examples of how this can be done.)

percentile ranksPercentile ranks show the percentage of the reference group with scores below the candidate’s score. For example, a percentile rank of 63 means that the candidate has performed better than 63 per cent of the candidates in the reference group; a percentile rank of 23 means that the candidate has performed better than only 23 per cent of the candidates in the reference group.

While percentile ranks are useful as a means of locating a candidate’s standing relative to the norm group, one drawback is that percentile ranks are not evenly spaced along a measurement scale. Percentile ranks for a large, representative sample tend to be ‘bunched up’ somewhere around the middle of a bell-shaped distribution, with smaller numbers of candidates towards the upper and lower ends of the distribution. Because small changes in raw scores around the middle of the distribution will have a considerable impact on the percentile rank, it is easy to over-emphasise differences near the middle set of scores and to under-emphasise differences near the extremes where very few candidates are located.

stanine scoresTo help overcome the problem of using a unit that is not evenly spaced along the measurement scale, stanine scores have been included. In order to derive stanine scores, nine categories are used. Stanines (except for stanines 1 and 9) are of equal length along the variable axis underlying the normal curve.

A stanine of 9 indicates a very high level of achievement relative to the reference group while a stanine of 1 indicates a very low relative achievement.

Table 4 shows the percentage of the reference group assigned to each stanine score and a descriptor.

Table 4 Stanine groups and descriptors

Stanine Percentage of group Descriptor

9 4 Very high

8 7Above average

7 12

6 17

Average 5 20

4 17

3 12Below average

2 7

1 4 Very low

�


Figure 4 Bell-shaped distribution

Rasch scoresRaw scores (number of correct scores) on the ACER TAR tests are not on an interval scale. In other words, equal numerical differences between raw scores do not represent equal differences in ability. To provide scores on an interval scale, raw scores have been transformed to scale scores using the Rasch measurement model. Measurement errors associated with these scale scores are also reported. The resulting scale is called the TAR Scale. The development of the TAR Scale is outlined in Scale scores, p. 28.

Comparison with Raven’s standard progressive Matrices (spM) and Advanced progressive Matrices (ApM)The provisional data from the various studies in Technical Information on p. 22 included a comparison of ACER TAR with the Raven’s SPM and APM. Rasch modelling of the data collected has allowed the construction of a table for approximate comparisons between TAR45 raw scores and SPM raw scores as well as TAR60 raw scores and APM raw scores. Raw scores on the SPM and APM can be used to generate percentiles and stanines on TAR45 and TAR60 using the norm data available, and vice versa.

Table 5 is a simplified version of the full table comparing TAR45 and SPM raw scores that can be found in Appendix C.

�

Table 5 Simple conversion table between ACER TAR45 and Raven’s SPM

TAR45 raw score TAR45 scale score SPM raw score

15 45 28

16 46 29

17 48 31

18 49 32

19 50 33

20 51 34

21 52 35

22 53 36

23 54 37

24 55 38

25 56 38

26 58 39

27 59 40

28 60 41

29 61 42

30 62 43

31 64 44

32 65 44

33 66 45

34 68 46

35 69 47

36 71 48

37 73 49

38 74 50

39 76 51

40 79 52

41 81 53

42 84 54

43 88 55

44 94 57

45 100 58

* Numbers in bold are obtained by linear interpolation

I n t e r p r e t a t i o n o f A C E R TA R R e p o r t s

�0


Table 6 is a simplified version of the full table comparing TAR60 and APM raw scores that can be found in Appendix F.

Table 6 Simple conversion table between ACER TAR60 and Raven’s APM

TAR60 raw score TAR60 scale score APM raw score

37 59 12

38 60 13

39 61 14

40 62 15

41 63 16

42 64 17

42 64 18

43 65 19

44 66 20

45 67 21

46 68 22

47 69 23

47 69 24

48 71 25

49 72 26

50 73 27

50 73 28

51 75 29

52 77 30

53 78 31

54 80 32

55 80 33

56 82 34

57 88 35

58 92 36

��

CAse stuDIes

example 1: Recruitment for a public sector departmentA recruiting officer for a public sector department has been put in charge of the graduate selection program. There are 100 initial applicants and the recruiting officer’s aim is to reduce that number to approximately 25 for the next stage. Knowing that successful applicants will need to demonstrate a high level of general ability, she decides to screen all applicants with TAR45.

The recruiting officer has access to a computer training lab with 25 computer terminals.She arranges for all applicants to attend one of four testing sessions over two days. After the last testing session

is completed, she contacts ACER Press Customer Service and requests a group report on the results of the 100 applicants. The spreadsheet is emailed to her showing the candidates’ achievement by percentile rank, stanine and scale score.

Using the spreadsheet, the recruiting officer orders the list by percentile rank. In scanning down the list from highest percentile rank to lowest, she notes that selecting a cut off score of the 80th percentile would retain 27 candidates. This number is in line with the number of candidates the recruiting officer was hoping to select for the next stage, and confirms that all the selected candidates demonstrate a high level of general ability.

example 2: Call centre selectionA recruitment agency has been asked to fill five vacancies for a special marketing project in a call centre. On advertising the position, 200 suitable applications are received. This number is reduced to 50 by the recruitment agency consultant on reviewing the applicants’ CVs.

The consultant speaks with the manager of the call centre about the job. In their discussion the consultant notes that the level of detail required for this project is relatively high, the training window is relatively short and that information will need to be relayed to customers in a clear, logical and informed manner. The consultant concludes that the successful applicant will need to have a moderately high level of general reasoning ability.

The manager indicates that she would like to have the applicants undergo some ability testing and attend a group interview in which various role-play activities will be conducted. From this exercise, 10 applicants will be selected for individual interviews.

The consultant invites the 50 applicants to attend the group interview. Before starting the role-play activities, he administers TAR45 to all the applicants in paper and pen format. While the group interviews are being con-ducted, the consultant directs his assistant to enter the candidates’ responses into the ACER Online Response Entry System. When all responses are entered, the assistant accesses the ACER Online Testing System and downloads a group report on the results of the 50 applicants. The assistant uses the report to rank the candidates by achievement, and gives the spreadsheet to the consultant.

At the completion of the group interview, the consultant makes a list of the 10 most likely candidates for the next stage and a further 10 ‘possibles’. He compares this list against the results of the test. He notes that of the list of most likely candidates, two have performed poorly or very poorly on the test, with percentile ranks of 28 or below. The other eight have performed well or very well, with percentile ranks above 70. He looks at the scores of the next list of ‘possibles’ and sees that two of them have performed very well on the test with percentile ranks above 70. He adds these to the list of most likely candidates for further interviewing.

example 3: promoting floor staff to store managerA trucking company is looking to promote one of the warehouse floor staff to the position of store manager. In addition to general organisational skills, the store manager must be able to work with basic mathematical concepts in areas such as stock control.

Five of the warehouse staff apply for the position, and the general manager decides to interview them all. Three of the five applicants are from a non-English speaking background. A high level of English language proficiency is not required, although an assessment of the candidates’ general ability would be useful. The HR manager decides

��


to administer two tests of general ability: ACER Numerical Select for assessing numerical reasoning skills, and the ‘language-free’ ACER TAR for assessing general ability.

The HR manager asks each of the candidates to come into the office in turn to complete the test session on the computer. Both tests are completed online. As each of the candidates completes the test, the results are auto-matically generated and emailed to the HR manager. When all five applicants have completed the testing, the HR manager transfers their results (percentile ranks and stanines) to a spreadsheet and orders them first by ACER Numerical Select results and then by ACER TAR results.

Of the five applicants, two score low or very low, at stanine 3 or below, on both tests. Two of the applicants score well on both tests, at stanines 7 or 8, showing both above average numerical reasoning and abstract reasoning ability. The last candidate scores an average stanine 5 on ACER Numerical Select, but stanine 9 (a very high result – the highest of the group) on ACER TAR. The HR manager points out to the general manager that while this last candidate scored significantly lower than the other two on numerical reasoning, the ACER TAR score indicates a very high level of abstract reasoning. This is indicative of the ability to learn new things quickly and engage in strategic thinking. Concluding that specific skills in analysing data can be taught, the general manager and HR manager decide to promote the person with the high ACER TAR score and provide him with training in data analysis using their inventory tracking software.

��

DIReCtIons foR ADMInIstRAtIon

general informationACER TAR Short Form (TAR45) has 45 items and should be completed in 20 minutes. ACER TAR Long Form (TAR60) has 60 items and should be completed in 25 minutes. Please note these details and ensure that the instructions given to candidates are accurate.

The data for the interpretation of the test were collected using standardised instructions. It is important that the examiner does not depart from these instructions when administering the test. To achieve this, the following points should be carefully observed:

• When administering these tests, do not vary in any way from the instructions given in the manual.• No candidate should see a test booklet before testing time, and there should be no opportunity for a

candidate to see any test booklet other than his/her own, during the testing session. All test booklets should be accounted for at the completion of testing.

• These are timed tests; it will be necessary to have either a stop watch or an ordinary watch with a second hand. If the latter is used, the starting and finishing time of the testing session should be recorded. The time limits given for the working of each section of the test should be observed exactly.

• If a test candidate asks a question during the practice items, it should be dealt with as follows:(a) If the question concerns details of procedure, e.g. ‘Where do I write the answers?’, it should be answered

directly.(b) If the question concerns the explanation of an answer, the relevant instructions should be repeated.

Paraphrasing should be used with great caution to ensure that the explanation does not differ in any substantial way from that given in the instructions.

(c) Once candidates have started work on the actual test paper, no questions other than procedural questions should be answered.

• To ensure that the candidates perform at their best, the anxiety level of the candidates should be minimal. Therefore while adhering strictly to the instructions, try to avoid any feeling of strain or strangeness. The physical comfort of the candidates should be given careful consideration; the testing environment should have adequate seating, work space, lighting and ventilation.

• All practical arrangements should be considered well before the testing session. The following points should be noted:(a) The tester should be familiar both with the directions for administration and with the relevant test

booklet well before the testing session.(b) A place for testing should be arranged and the seating and room layout checked.(c) Adequate quantities of test materials should be available.(d) If more than 30 candidates are to be tested, an assistant should be present.(e) Although candidates usually provide their own pens/pencils, the tester is advised to provide a few

spares.(f ) Measures should be taken to ensure that external noise is minimal and that the testing session will be

completed without interruption.

general information for online testingAll practical arrangements should be considered well before the testing session. The following points should be noted:

• A place for testing should be organised with a sufficient number of computers arranged for testing purposes (so that, for example, the test administrator can see all screens, candidates can only see their own screen, etc.)

• All computers should be checked (screen, keyboard, mouse, browser settings, internet connection, etc.)All computers should be checked (screen, keyboard, mouse, browser settings, internet connection, etc.) prior to the testing session.

��


• If required, an assistant should be present to monitor candidates. This is essential if the number ofIf required, an assistant should be present to monitor candidates. This is essential if the number of candidates is above 20.

• Paper and pencils for working should be provided for each candidate.Paper and pencils for working should be provided for each candidate.• Measures should be taken to ensure that external noise is minimal and that the testing session will beMeasures should be taken to ensure that external noise is minimal and that the testing session will be

completed without interruption.

During the test sessions, candidates should not use any other program or navigate to any other website other than the testing website. In addition, candidates should only use the buttons in the browser window (e.g. NEXT, PREVIOUS) and not use the web browser’s Back or Forward buttons.

Consideration should be given to removing software and/or access to programs and websites on the computers that will allow candidates to access information that would assist them in answering the test questions (e.g. calculators, dictionaries, general internet access). In any case, candidates should be closely monitored to ensure that they do not navigate away from the online testing window.

Minimum system requirements

Users will need access to computers with Internet Explorer version 6 or greater.AutoComplete for forms should be disabled.Sufficient internet connection bandwidth should be available – at least 10 kb per second per candidate. This

should be checked with the administrator of the internet/network. The requirements for ACER TAR are slightly higher than text-based tests due to the use of image files.

In general terms:

• a single candidate should have the equivalent bandwidth (at a minimum) of a dial-up connection• 20 candidates should have the equivalent of a broadband connection (512K).20 candidates should have the equivalent of a broadband connection (512K).

specific instructionsThe detailed instructions for paper and online administrations set out exactly what should be done and said. The spoken instructions are enclosed in boxes and should be given verbatim. Do not say anything other that what is indicated in the boxes.

Co p y r i g h t © 2 0 0 7 A u s t r a l i a n Co u n c i l f o r E d u c a t i o n a l R e s e a r c h ��

D i r e c t i o n s f o r A d m i n i s t r a t i o n

Instructions for paper administrationMake sure all candidates have a pencil, test booklet and answer sheet.

Say:

The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams. On your desk there should be a test booklet and an answer sheet.

Hold up the test booklet.

Say:

This is the test booklet. Please do not open the test booklet until you are told to.

Hold up the answer sheet.

Say:

This is the answer sheet. All your answers are to be marked on the answer sheet. Do not make any marks on the test booklet.

Say:

The answer sheet you are using will be scored by computer. It is important that you mark it properly. You will see that on the answer sheet there are many oval shapes or bubbles with letters or numbers in them. To give your answers to the test, you need to fill in these bubbles with a pencil. You should fill in the bubble completely, not just mark it with a tick, line, cross or circle. If you change your mind about an answer, you should completely erase the mark you have made and fill in another bubble.

To practice using the answer sheet, we will all now fill in our personal details in the top half of the sheet.

Hold up the answer sheet.

Say:

Where the answer sheet says ‘Name’, write in your name. Now look across the sheet and print your surname in the squares under SURNAME/FAMILY NAME. Now fill in the bubbles underneath. If your surname is Eade, you would fill in the bubbles E, then A, then D, then E. Now print your initials in the squares under INITS. Now fill in the bubbles underneath.

Check that the candidates are correctly filling in the answer sheets.

Say:

Where the answer sheet says TEST FORM, fill in the bubble for (Short Form/Long Form).

Say:

Now fill in your sex and date of birth. Please note that this information will be used solely for identification and research purposes. They will not be reported in your results.

Wait for a few seconds. Hold up the test booklet.


�� Co p y r i g h t © 2 0 0 7 A u s t r a l i a n Co u n c i l f o r E d u c a t i o n a l R e s e a r c h

Say:

We are now ready to look at the test booklet and do some practice questions. Please open your test booklet to the first page. Do not turn the page until I tell you to.

Hold up the test booklet open to the first page.

Say:

Look at the first practice question P1. The instruction says:

The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them?

You will see that the five shapes in the bottom row each have a letter above them. On your answer sheet, look for the shaded rectangle that says ‘Practice Examples’. Next to P1, fill in the bubble with the letter of the shape you think is most like the top three shapes.

Wait for a few seconds.

Say:

The correct answer is shape A, so you should have filled in the bubble A next to P1 on the answer sheet. Remember not to mark the test booklet, only the answer sheet.

Now have a look at the second practice question P2. The instruction says:

On the top row there are five squares with shapes arranged in order. One shape is missing. Which shape from the bottom row best completes the pattern?

On the answer sheet next to P2 fill in the bubble with the letter of the shape you think best completes the pattern.


Say:

The correct answer is shape E, so you should have filled in the bubble E next to P2 on the answer sheet.

Now have a look at the third practice question P3. The instruction says:

There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern?

On the answer sheet next to P3 fill in the bubble with the letter of the shape you think best completes the pattern.


Say:

The correct answer is shape A, so you should have filled in the bubble A next to P3 on the answer sheet.

That completes the practice questions.

A few things to remember:

– Fill in your answer on the answer sheet, not the test booklet

– When you choose an answer, fill in the whole bubble on the answer sheet – don’t tick, cross or circle the answer



– If you change your mind about an answer, erase the mark you have made and fill in the bubble of your new choice.

Are there any questions?

Answer any questions about filling in the answer sheet.

If you are administering TAR45, say:

In the test booklet there are 60 questions. Today we are only doing the first 45 questions. You should finish at question 45 where it says END OF SECTION 1.

Hold up the answer sheet and point to the bottom half of the sheet where the candidates fill in their answers.

Say:

You can see on the answer sheet that there is space for answers to 60 questions. We will only fill in the first 45 down to where it says END OF SECTION 1.

Are there any questions?


In the test booklet there are 60 questions. After 45 questions, there is a message END OF SECTION 1. Please ignore that message and work through the test booklet all the way to the last page.

Answer any questions about the number of items to be completed. Hold up the test booklet.

Say:

In Section 1 of the test booklet there are five sets of questions. The instruction for the questions is given once at the beginning of the set. So for each instruction there are several questions. Each new set of questions has a new instruction for all the questions in that set.

Pause.


You will have 20 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you.

Don’t spend too much time on any one question – try to answer as many questions as you can in the time. After 20 minutes I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet.

Before we begin, are there any questions?


You will have 25 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you.


�� Co p y r i g h t © 2 0 0 7 A u s t r a l i a n Co u n c i l f o r E d u c a t i o n a l R e s e a r c h

Don’t spend too much time on any one question – try to answer as many questions as you can in the time. After 25 minutes, I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet.


Answer any questions.

Say:

We are now ready to begin the test. You have (20/25) minutes to complete the test, please turn the page now and begin.

If administering TAR45, wait 20 minutes.

If administering TAR60, wait 25 minutes.

When the time limit is up, say:

Please stop writing and put your pencils down.

Collect the test booklets and answer sheets. Make sure that each candidate has correctly filled in their details on the answer sheet.



Instructions for online administrationMake sure all candidates have a login code and a pencil and some paper for working. All the candidates’ computers should already display the ACER Press Online Testing System login page.

Say:

Please enter your login code and click the ENTER button.

or you may enter the login codes into the screens and click the ENTER button prior to the candidates’ arrival.

Say:

Please fill in the personal details requested. Do not click submit until you are told to do so. Please note that the personal details requested are used for identification and research purposes only and will not be provided along with test results.

Allow time for filling in the personal details. When all candidates have filled them in, say:

Now click the SUBMIT button (Pause). The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams.


Look at the screen and follow while I read. This test is a timed test and will take 20 minutes. There are 45 questions in this test. There are some practice questions to complete before the test begins.


Look at the screen and follow while I read. This test is a timed test and will take 25 minutes. There are 60 questions in this test. There are some practice questions to complete before the test begins.

Say:

Now click on BEGIN PRACTICE QUESTIONS.

On the next few screens you will see a number of practice questions to help you in the test that follows. All the questions require choosing one answer from the list of possible answers. To answer each question, click the small circle below the answer.

Look at the first practice question. The instruction says:

The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them?

You will see that the five shapes in the bottom row each have a small circle below them. Click on the circle under the shape you think is most like the top three shapes. The circle should now have a dark spot inside it.


Say:

The correct answer is the first shape, so you should have clicked the small circle under the first shape.

Now click NEXT and have a look at the second practice question. The instruction says:


�0 Co p y r i g h t © 2 0 0 7 A u s t r a l i a n Co u n c i l f o r E d u c a t i o n a l R e s e a r c h

On the top row there are five squares with shapes arranged in order. One shape is missing.

Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern.


Say:

The correct answer is the fifth shape, so you should have clicked the small circle under the fifth shape.

Now click NEXT and have a look at the third practice question. The instruction says:

There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern.


Say:

The correct answer is the first shape, so you should have clicked the small circle under the first shape. That completes the practice questions.

Click NEXT and wait. Do not click ahead until I tell you to.

Say:

In the test there is an instruction for each of the questions. Several questions in a row will have the same instruction, then there will be a change of instruction. Be careful to note when the instruction changes.


You will have 20 minutes to complete all the questions.


You will have 25 minutes to complete all the questions.

Say:

Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you.

Don’t spend too much time on any one question – try to answer as many questions as you can in the time.


Answer any questions.

Say:

We are now ready to begin the test.



During the test, use only the NEXT and PREVIOUS buttons under each question or the numbered navigation panel on the left hand side of the screen to move through the test. Under no circumstances use the Back or Forward buttons on your browser or the backspace button on your keyboard.

If you finish ahead of time, you may go back over your answers if you wish or click the FINISH button.

During the test you may not use any other software or programs on the computer or navigate away from the testing system website.


You will have 20 minutes to complete the test. Please click on BEGIN TEST.


You will have 25 minutes to complete the test. Please click on BEGIN TEST.

If administering TAR45, wait approximately 20 minutes.

If administering TAR60, wait approximately 25 minutes.

The system is timing each individual but you should also keep an independent watch on the time.

NOTE: If a candidate goes over time but does not click NEXT, PREVIOUS or FINISH on the item they are currently working on, it may appear that the system has not timed them out. When they eventually click NEXT, PREVIOUS or FINISH, the system will time the session out and close the test. In such a case when you are sure the time has gone past the time limit, ask the candidate to click one of the buttons.

When all the candidates have been timed out, say:

The time limit is up. Please click on the FINISH button to submit your responses.

��

teChnICAL InfoRMAtIon

Users of ACER TAR should note that the study samples reported in this manual are not stratified samples of the Australian population. However, percentiles, stanines and scale scores have been derived from these reference groups to provide a point of comparison with an individual candidate’s performance on the test. Demographic details of the samples used in each study are provided.

summary of research studiesStudy 1 was conducted on 151 adult volunteers from university, TAFE campuses, Centrelink and Job Network Providers using mainly TAR45 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below.

Table 7 Key statistical findings in Study 1

Statistic Value

Mean raw score 35.37 out of 45 items

Std. deviation 6.58

Internal reliability 0.87 (Good)

Correlation with SPM 0.80 (High)

Demographic variable(s) that showed a significant difference in test scores

Age groups

Study 2 was conducted on 295 adults using mainly TAR60 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below.


Statistic Value


Std. deviation 6.71

Internal reliability 0.84 (Good)

Correlation with APM 0.76 (High)

Demographic variable(s) that showed a significant difference in test scores

Highest level of education completed

Data for Study 3 was obtained from a total of 329 adult jobseekers who had completed TAR45 online. Key descriptive and reliability findings are given in the table below.


Statistic Value


Std. deviation 7.29

Internal reliability 0.904

��

Te c h n i c a l I n f o r m a t i o n

study 1 – ACeR tAR short form (tAR45)

Study design

Study sampleThe data reported in this study were collected from a sample of 151 adults in Australia. The study participants were adult volunteers, including 30 applicants from a graduate recruitment program. The candidates were recruited from university and TAFE campuses, Centrelink and Job Network Provider premises, community organisations and other sources.

The participants were asked to provide information on age, gender, occupation and highest level of education.Tables 10 to 15 describe the sample by the demographic data collected.

Table 10 Demographics – Gender

Gender Number of participants

Female 87

Male 51

No data 13

Total 151

Table 11 Demographics – Highest level of education completed

Highest level of education completed Number of participants

Postgraduate degree 22

Undergraduate degree 35

TAFE diploma 13

Year 12 68

Year 11 8

Year 10 1

No data 4

Total 151

��


Table 12 Demographics – Occupational classification

Occupational classification Number of participants

Student 77

Unskilled 2

Skilled/Trade 2

Office 22

Retail 5

Artist 3

Professional 26

No data/Unemployed 14

Total 151

Table 13 Demographics – Age group

Age group Number of participants

Under 20 years 6

20–29 years 79

30–39 years 26

40–49 years 11

50 years or over 16

No data 13

Total 151

Table 14 Demographics – Language spoken at home

Language spoken at home Number of participants

English 78

Other 40

No data 33

Total 151

Testing procedureThe tests were administered using standardised instructions provided in the test manual. Each study participant completed both ACER TAR45 and Raven’s Standard Progressive Matrices in a single session. The participants were given 20 minutes to complete each test. Approximately half the participants completed ACER TAR45 before the SPM, and the other half in reverse order. Some participants managed to complete some of the extra 15 items that comprise TAR60.

��


Table 15 Demographics – Order of tests

Test order Number of participants

ACER TAR45 – SPM 70

SPM – ACER TAR45 78

No data 3

Total 151

Item analysis

QUEST analysis provides measures of general acceptability, content validity, contribution to internal consistency and sensitivity of individual items in a test. This information can be used to determine the usefulness and/or suitability of items for inclusion in a test. QUEST analysis was performed on all 60 TAR items and the results were tabulated (See Appendix A). This provided the:

(a) number of correct responses obtained on each item and the number of persons who attempted the question, from which the difficulty estimates (measured in logits) for each item could be calculated;

(b) the margin of error associated with the difficulty level of each item;(c) the infit mean squares, which are an indication of how well the item discriminates between poor and high

performers. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.6 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.4 have poor discriminating power.

Because not all participants carried on after item 45, the item analysis was done first on all 60 TAR items and then on the first 45 TAR items. The item analysis done on all the 60 TAR items shows a high level of missing responses after item 45. As seen in the difficulty estimates, the difficulty levels of items 46 to 60 are generally much higher than the first 45 items.

As seen from the results, the items cover a range of difficulties with 9 of the 60 items (items 1, 3, 8, 12, 15, 37, 38, 40, 42) showing some deviation from acceptable discrimination levels, as measured by the infit mean squares. Further research and analysis would need to be conducted to determine if these items should be excluded from ACER TAR.

A further QUEST analysis was done on TAR45 that provided the following data:

(a) facility, which is the percentage of individuals who got the item correct;(b) point biserial, which is the correlation between scores on items and overall test scores. This is an

indication of how related the item is to the overall test. A positive number is considered appropriate;(c) difficulty estimates, measured in logits, which provide an indication of how difficult each item is; and(d) infit mean squares, which are an indication of how well the item discriminates between poor and high

performers. Again, a value of between 0.6 and 1.4 is considered acceptable.

As seen in Appendix B, the items cover an appropriate range of difficulty levels and all items are positively correlated with the overall test scores. All items are also within acceptable levels of discrimination. The difference in results of item analyses on TAR45 versus all 60 items may be due to the high number of missing scores from items 46 to 60.

Reliability

QUEST analysis on the data in this study indicated that the internal consistency of the TAR45 items was 0.87. This compares to 0.90 for the full 60 items of the SPM for the study sample. These are considered strong measures of internal consistency, indicating that the items are highly inter-correlated and measure the same general construct. The slightly smaller value of the consistency of TAR45 items is likely to be related to the inclusion of only 45 items in the analysis compared to 60 items for the SPM.

��


TAR45 subsets

TAR45 has five sets of items each with its own instruction.

Table 16 TAR45 set descriptions

Set Instruction text

Set 1 (items 1–7) The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them?

Set 2 (items 8–16) In the top row there are five squares with shapes arranged in order. One shape is missing. Which shape in the bottom row best completes the pattern?

Set 3 (items 17–26) If the five shapes in the row are placed in logical order, which shape comes in the middle?

Set 4 (items 27–36) Four of the five shapes are alike in some way. Which one is different?

Set 5 (items 37–45) There is one missing shape in the pattern on the top. Which shape in the bottom row best completes the pattern?

Correlations

Set correlationsAn analysis of the internal correlations between the five sets in TAR45 was carried out and the results are shown in Table 17.

Table 17 TAR45 internal set correlations

Set 1 Set 2 Set 3 Set 4

Set 1 (items 1–7) 1

Set 2 (items 8–16) 0.27 1

Set 3 (items 17–26) 0.31 0.55 1

Set 4 (items 27–36) 0.36 0.39 0.32 1

Set 5 (items 37–45) 0.27 0.32 0.22 0.53

The low correlations between the sets indicate that each set of items may be measuring slightly different skills.

Table 18 TAR45 set correlations with total score

Correlations with total score on first 45 items

Set 1 (items 1–7) 0.53

Set 2 (items 8–16) 0.69

Set 3 (items 17–26) 0.66

Set 4 (items 27–36) 0.79

Set 5 (items 37–45) 0.76

Each set, however, has correlations of considerable size with the total TAR45 score. This shows that the various items are measuring different skills that contribute to a measure of general ability.

��


Set reliabilityInter-item correlations were performed on each set of items, which produced the results shown in Table 19.

Table 19 TAR45 set reliabilities

Number of items Reliabilities

Set 1 (items 1–7) 7 0.34

Set 2 (items 8–16) 9 0.61

Set 3 (items 17–26) 10 0.72

Set 4 (items 27–36) 10 0.68

Set 5 (items 37–45) 9 0.88

As seen in the table, set 1 has poor inter-item reliability. This lower correlation in set 1 could be due to the fact that it has fewer items than the other sets. However, inter-item correlations show that items 2 and 3 do not correlate well with other items in the same set. The deleted item statistics show that if these two items are removed, the reliability will be higher. Removal of item 2 raises the reliability from 0.34 to 0.39, and removal of item 3 raises the reliability from 0.34 to 0.36. It was decided not to remove these items from the final test form until further data had been collected to substantiate these conclusions.

Validity

Correlations with Raven’s Standard Progressive Matrices (SPM)The correlation between the SPM raw scores and the TAR45 raw scores is 0.80. Correlation between the logits is 0.63. This value shows that there is little difference between the two tests and they basically measure the same abilities.

Removing 18 students brings most of the points within and near the 95% confidence belts. The shift constant, which is a measure of the difference between TAR45 and SPM, is 0.64.

Figure 5 95% confidence belt for equating between TAR45 logits and SPM logits

A conversion table between SPM raw scores and TAR45 raw scores was calculated in the following way:

(a) Corresponding to each SPM raw score, there is an SPM logit measure of the ability, which can be derived from the QUEST analysis.

��


(b) Corresponding to each TAR45 raw score, there is also a TAR45 logit measure of the ability, which can also be obtained from the QUEST analysis.

(c) Through equating, the TAR45 logit is put on the same scale as the SPM logit scale. This is done by adding the shift constant of 0.64 to each of the TAR45 logits.

(d) The adjusted TAR45 logit is equated to the nearest SPM logit.

So, a TAR45 raw score can provide a TAR45 logit measure of ability, which after adjusting for the shift constant is equated to the nearest SPM logit, from which an SPM raw score can be determined. A detailed conversion table for TAR45 raw scores to SPM raw scores is provided in Appendix C.

Scale scores

TAR45 has been calibrated using the Rasch measurement model (Fisher and Molenaar, 1995). The average diffi-culty was then set at 65 with a standard deviation of 15.

The Rasch model gives an ability estimate for each person in terms of logit. The logits are then transformed into scaled scores with a range from 0 to 100 and a standard deviation of 15. The mean of the scaled scores is chosen so that a student answering all TAR45 items correctly will be given a scaled score of 100. The mean scale score obtained from this sample of 151 students is 65.

Demographic variables

Means and standard deviations were calculated for the study participants by five background variables: gender, level of education, occupational classification, language background, age and order of test. An analysis of variance (ANOVA) was also performed for each background variable to find out if they had significantly different scores. In each case candidates providing no data were counted as one group. A significance value in the ANOVA tables of less than or equal to 0.05 is indicative of a statistically significant difference in the performance of the different groups.

GenderResults of males and females were compared and analysed. The results in Tables 20 and 21 show that there was no significant difference in the performance of males and females on TAR45.

Table 20 TAR45 mean and standard deviation by gender

Missing scores Female scores Male scores Group total

Count 13 87 51 151

Mean 32.46 35.54 35.82 35.37

Maximum 41 44 45 45

Minimum 19 22 15 15

Std. deviation 7.38 6.09 7.10 6.58

Table 21 ANOVA TAR45 scores on gender

Sum of squares df Mean square F Significance value

Between groups 11.17 2 5.58 0.128 0.880

Within groups 6474.07 148 43.74

Total 6485.23 150

��


Education levelIndividual results were grouped by education level and analysed. The results in Tables 22 and 23 show that there was no significant difference in the scores of individuals in different educational levels even though there was some difference in the mean scores of Year 10 students and TAFE diploma students.

Table 22 TAR45 mean and standard deviation by education level

Postgrad degree

Undergrad degree

TAFE diploma

Year 12

Year 11

Year 10

No data

Group total

Count 22 35 13 68 8 1 4 151

Mean 35.73 35.26 37.15 35.38 32.25 31.00 35.75 35.37

Maximum 43 45 44 44 44 31 41 45

Minimum 23 15 23 19 23 31 30 15

Std. deviation 5.19 7.74 5.61 6.64 6.94 – 4.79 6.58

Table 23 ANOVA TAR45 scores on educational level


Between groups 142.18 6 23.70 0.538 0.779

Within groups 6343.05 144 44.05

Total 6485.23 150

Occupational classificationIndividual results were grouped by occupational classification and analysed. The results in Tables 24 and 25 show that there was no significant difference in the scores of individuals in different occupational groups. However, the mean score of those who are unemployed or have provided no data was considerably lower than that of the other occupational groups.

Table 24 TAR45 mean and standard deviation by occupational classification

Student

Unskilled

Trade

Office

Retail

Artist

Professional

No data/Unemployed

Group total

Count 77 2 2 22 5 3 26 14 151

Mean 35.71 38.50 38.50 34.77 34.80 37.33 36.19 31.79 35.37

Maximum 45 43 44 44 42 42 43 43 45

Minimum 15 34 33 19 22 31 23 22 15

Std. deviation 6.75 6.36 7.78 6.63 8.35 5.69 5.74 6.57 6.58

�0


Table 25 ANOVA TAR45 scores on occupational classification


Between groups 266.79 7 38.11 0.876 0.527

Within groups 6218.44 143 43.49

Total 6485.23 150

Language backgroundIndividual results were grouped by whether they were from an English-speaking background and analysed. The results in Tables 26 and 27 show that there was no significant difference in the scores of individuals from different language backgrounds.

Table 26 TAR45 mean and standard deviation by language

English Other no data Group total

Count 78 40 33 151

Mean 35.68 34.40 35.82 35.37

Maximum 44 45 43 45

Minimum 19 15 23 15

Std. deviation 6.22 8.32 4.84 6.58

Table 27 ANOVA TAR45 scores on language

Sum of squares df Mean squares F Signigicance value

Between groups 51.74 2 25.87 0.595 0.553

Within groups 6433.50 148 43.47

Total 6485.23 150

Age groupsIndividual results were grouped by age and analysed. The results in Tables 28 and 29 show that there was a significant difference in the scores of individuals of different ages. Specifically, mean scores on TAR45 increased until age 29 and then started to decline in those aged 30 and above.

Table 28 TAR45 mean and standard deviation by age groups

<20 20 to 29 30 to 39 40 to 49 50 or over Missing Group total

Count 6 79 26 11 16 13 151

Mean 33.50 37.00 34.42 31.45 31.75 36.00 35.37

Minimum 22 15 22 19 20 25 15

Maximum 44 44 45 42 42 41 45

Std. deviation 7.79 6.09 6.86 7.34 6.52 4.97 6.58

��


Table 29 ANOVA TAR45 scores on age groups


Between groups 637.66 5 127.53 3.162 0.010

Within groups 5847.57 145 40.33

Total 6485.23 150

Test orderTwo groups of individuals were given the SPM and TAR45 in different order. One group was given the SPM first and then TAR45. The other group was given the two tests in the reverse order. This was to negate any possible order or practise effect. The results in Tables 30 and 31 show that there was no significant difference in the scores of the two groups.

Table 30 TAR45 mean and standard deviation by test order

SPM/TAR45 TAR45/SPM No data Group total

Count 78 70 3 151

Mean 35.81 34.70 39.67 35.37

Maximum 43 45 40 45

Minimum 15 19 39 15

Std. deviation 5.88 7.35 0.58 6.58

Table 31 ANOVA TAR45 scores on test order


Between groups 101.75 2 50.88 1.180 0.310

Within groups 6383.48 148 43.13

Total 6485.23 150

Conclusion

In these analyses of variance, the only variable to show a degree of significance was age groups. No significant differences were found for gender, education level, occupation, language background or test order.

��


study 2 – ACeR tAR Long form (tAR60)

Study design

Study sampleThe data reported in this study were collected from a sample of 295 adults in Australia. The participants were asked to provide information on gender, education level, study type and language background.

Participants in the study consisted of 212 undergraduates (72%), 28 postgraduates (9%) and 55 participants (19%) who were not students or who did not provide information. 225 participants (76%) had completed Year 12. 48 participants (16%) possessed undergraduate degrees. 8 (about 3%) held postgraduate degrees. Only 5 (2%) in the sample had not completed Year 12.

The sample is fairly evenly represented in gender. There were 139 (47%) females and 154 (52%) males. Only two participants (1%) did not provide their gender information.

Tables 32–36 describe the sample by the demographic data collected.

Table 32 Demographics – Study types

Study type Number of participants

Undergraduate/humanities, law, education 55

Undergraduate/science, IT, engineering 88

Undergraduate/commerce, finance, accounting 22

Undergraduate/medicine 15

Undergraduate/double degree 26

Undergraduate/not stated 6

Postgraduate/commerce 6

Postgraduate/humanities 6

Postgraduate/science 16

Not stated/not a student 55

Total 295

Table 33 Demographics – Highest level of education completed

Highest level of education completed Number of participants

Postgraduate degree 8

Undergraduate degree 48

TAFE diploma 8

Year 12 225

Year 11 3

Year 10 2

No data 1

Total 295

��


Table 34 Demographics – Gender


Female 139

Male 154

No data 2

Total 295

Table 35 Demographics – Language spoken at home

Language spoken at home Number of participants

English 165

Other 128

No data 2

Total 295

Testing procedureThe tests were administered using standardised instructions provided in the test manuals. Each study participant completed both TAR60 and the Raven’s Advanced Progressive Matrices (APM) in a single session. The participants were given 25 minutes to complete each test. Approximately half the participants completed the TAR60 before the APM, and the other half in reverse order. This was to check for possible order or practise effects.

Table 36 Demographics – Order of tests

Test order Number of participants

ACER TAR60 – APM 152

APM – ACER TAR60 143

Total 295

Item analysis

Because of the high number of missing responses after item 45 on the TAR60, the item analysis was done first on all 60 TAR items and then on the first 45 items. The QUEST Analysis provided the following information (see Appendix D):

(a) number of correct responses obtained on each item and the number of persons who attempted the question, from which the difficulty estimates (measured in logits) for each item could be calculated;

(b) the margin of error associated with the difficulty level of each item;(c) the infit mean squares, which are an indication of how well the item discriminates between poor and high

performers. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.7 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.3 have poor discriminating power.

As seen from the results in Appendix D, the items cover a range of difficulties with 3 of the 60 items (items 8, 16 and 37) showing any deviation from acceptable discrimination levels, as measured by the infit mean squares. The difficulty levels of items 46 to 60 are generally much higher than the first 45 items.

��


A further QUEST analysis was done on the first 45 TAR items, which provided the following information (See Appendix E):

(a) facility, which is the percentage of individuals who got the item correct;(b) point biserial, which is the correlation between scores on items and overall test scores. This is an

indication of how related the item is to the overall test and a positive number is considered appropriate;(c) difficulty estimates, measured in logits, which provide an indication of how difficult each item is; and(d) infit mean squares, which are an indication of how well the item discriminates between poor and high

performers. Again, a value of between 0.6 and 1.4 is considered acceptable.

Reliability

The internal reliability of TAR60 items was 0.84. For TAR45, the internal reliability is 0.74. The internal reliability of the APM, which has 36 items and which is also administered to this sample of students, has a reliability of 0.78. These are all acceptable levels of internal consistency, which means that the items in each test inter-correlate and generally measure the same underlying dimension.

TAR60 subsets

TAR60 has seven sets of items, each with its own instruction.

Table 37 TAR60 set descriptions

Set Instruction text

Set 1 (items 1–7) The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them?

Set 2 (items 8–16) In the top row there are five squares with shapes arranged in order. One shape is missing. Which shape in the bottom row best completes the pattern?

Set 3 (items 17–26) If the five shapes in the row are placed in logical order, which shape comes in the middle?

Set 4 (items 27–36) Four of the five shapes are alike in some way. Which one is different?

Set 5 (items 37–45) There is one missing shape in the pattern on the top. Which shape in the bottom row best completes the pattern?

Set 6 (items 46–55) The four shapes in the top row are in order. Which shape in the bottom row logically follows the fourth shape in the top row?

Set 7 (items 56–60) There is one missing shape in the pattern at the top. Which shape in the bottom row best completes the pattern?

Correlations

Set correlationsAn analysis of the internal correlations between the five sets in TAR45 was carried out and the results are shown in the tables below.

��


Table 38 TAR60 internal set correlations

Set 1 Set 2 Set 3 Set 4 Set 5 Set 6


Set 2 (items 8–16) 0.24 1

Set 3 (items 17–26) 0.25 0.17 1

Set 4 (items 27–36) 0.17 0.18 0.30 1

Set 5 (items 37–45) 0.28 0.21 0.34 0.30 1

Set 6 (items 46–55) 0.31 0.15 0.24 0.28 0.39 1

Set 7 (items 56–60) 0.15 –0.01 0.14 0.24 0.12 0.51

Table 39 TAR60 set correlations with total score

Correlations with total score on first 45 items

Correlations with total score on all 60 items

Set 1 (items 1–7) 0.55 0.49

Set 2 (items 8–16) 0.49 0.35

Set 3 (items 17–26) 0.67 0.54

Set 4 (items 27–36) 0.72 0.61

Set 5 (items 37–45) 0.67 0.59

Set 6 (items 46–55) – 0.81

Set 7 (items 56–60) – 0.62

The low correlations between the sets seen in Table 38 indicate that each set of items may be measuring separate components. Each set, however, has correlations of considerable size with the total TAR60 score.

Set reliability

Table 40 TAR60 set reliabilities

Number of items Reliabilities

Set 1 (items 1–7) 7 0.51

Set 2 (items 8–16) 9 0.23

Set 3 (items 17–26) 10 0.58

Set 4 (items 27–36) 10 0.55

Set 5 (items 37–45) 9 0.63

Set 6 (items 46–55) 10 0.80

Set 7 (items 56–60) 5 0.81

��


Reliability depends on the length of the test and longer tests usually have higher reliability. This is reflected to a certain degree in Table 40. Set 6, with 10 items, has a reliability of 0.80 and set 1, with 7 items, has a reliability of only 0.51. Set 2 has a low reliability based on data from this sample. Additional item analysis showed that item 15 has a low point biserial coefficient and item 16 has a low facility. Removing these two items from the set, though shortening its length, raises the reliability to 0.38. Apart from set 2, the internal reliability for each set is acceptable. If each set is allowed to have 20 items, the reliability in each set, with the exception of set 2, will reach 0.7 using the Spearman-Brown formula.

Validity

Correlations with Raven’s Advanced Progressive Matrices (APM)Correlations between TAR60 and APM were conducted to demonstrate construct validity, which shows that the two tests measure the same underlying ability. The correlation between APM raw scores and TAR60 raw scores is 0.76.

Table 41 TAR60 set correlations with APM total score

Correlations with APM total score

Set 1 (items 1–7) 0.41

Set 2 (items 8–16) 0.23

Set 3 (items 17–26) 0.32

Set 4 (items 27–36) 0.27

Set 5 (items 37–45) 0.35

Set 6 (items 46–55) 0.53

Set 7 (items 56–60) 0.40

Set correlationsSet 6 of TAR60, containing items 46–55, is the set with the highest correlation with the APM total scores. This explains why the correlation between TAR60 scores and APM scores is higher if the last 15 items are included. Set 2 has the lowest correlation with APM total.

Table 42 Correlations of TAR60 items with APM total score

Correlations with APM total scores (Set 1)

Correlations with APM total scores (Set 2)

1st item 0.09 0.07

2nd item 0.35 –0.04

3rd item 0.02 0.09

4th item 0.19 0.16

5th item 0.19 0.22

6th item 0.30 0.11

7th item 0.20 0.19

8th item 0.08

9th item –0.01

��


As an illustration, correlations between item scores in set 1 and set 2 in ACER TAR and the APM total are given in Table 42. These in fact are the point biserial of the items but use APM total scores in the correlation instead of the ACER TAR total. If ACER TAR and APM are measuring the same construct, the point biserial should be similar whether using ACER TAR total or APM total. It is seen that items in set 1 are satisfactory but there are some items in set 2 with negative correlations or correlations very close to zero.

Calculating the scale scoresTAR60 has been calibrated using the Rasch measurement model (Fisher and Molenaar, 1995).

The Rasch model gives an ability estimate for each person in terms of logit. The logits are then transformed into scaled scores with a range from 0 to 100 and a standard deviation of 15. The mean of the scaled scores is chosen so that a student answering all TAR60 items correctly will be given a scaled score of 100. The mean scale score obtained from this sample of 295 students is 65.

How well the data fit the Rasch model is indicated by the infit mean squares. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.7 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.3 have poor discriminating power.

Equating TAR60 to APMThe APM logits and the TAR60 logits of the same students are plotted in a graph after making adjustments for the scale. Not all of the 295 students lie within the 95% confidence belts.

This means that after adjustment of the scale, the logit measures from TAR60 and those from APM are not likely to come from the same person. In the process of test equating, these persons will be removed in order to find an appropriate constant for adjustment of the scale.

For TAR60, the shift constant found is –0.508.A conversion table between APM raw scores and TAR60 raw scores was calculated in the following way:

(a) Corresponding to each APM raw score, there is an APM logit measure of the ability, which can be derived from the QUEST analysis.

(b) Corresponding to each TAR60 raw score, there is a TAR60 logit measure of the ability, which can also be obtained from the QUEST analysis.

(c) Through equating, the TAR60 logit is put on the same scale as the APM logit scale. This is done by adding the shift constant of –0.508 to each of the TAR60 logits.

(d) The adjusted TAR60 logit is equated to the nearest APM logit.

Figure 6 95% confidence belt for equating between TAR60 logits and APM logits

��


So, a TAR60 raw score yields a TAR60 logit measure of ability, which is equated to the nearest APM logit. After adjusting for the shift constant this is then used to determine an APM raw score (See Appendix F).

Background variables

Means and standard deviations were calculated for the study participants by five background variables: gender, level of education, language background, study type and order of test. An analysis of variance (ANOVA) was also performed for each background variable.

GenderResults of males and females were compared and analysed. The results in Tables 43 and 44 show that there was no significant difference in the performance of males and females on TAR60.


Male scores Female scores Missing scores Group total

Valid N 154 139 2 295

Mean 48.36 47.76 47.00 48.07

Minimum 25 22 44 22

Maximum 59 59 50 59

Std. deviation 6.47 7.01 4.24 6.71



Between groups 28.12 2 14.06 0.311 0.733

Within groups 13214.52 292 45.26

Total 13242.64 294

Education levelIndividual results were grouped by education level and analysed. The results in Tables 45 and 46 show that there was no significant difference in the scores of individuals with different educational levels, although Year 12, undergraduate and postgraduate students scored higher than Years 10 and 11 students.


Postgrad degree

Undergrad degree

TAFE diploma

Year 12

Year 11

Year 10

Missing

Group total

Valid N 8 48 8 225 3 2 1 295

Mean 48.00 45.54 42.88 48.98 40.67 35.00 54.00 48.07

Minimum 41 27 22 28 30 25 54 22

Maximum 55 58 53 59 49 45 54 59

Std. deviation

5.76 6.50 10.20 9.71 9.71 14.14 – 6.71

��




Between groups 1251.26 6 208.54 5.009 0.000

Within groups 11991.39 288 41.64

Total 13242.64 294

Language backgroundIndividuals were grouped according to whether or not they were from an English-speaking background and their results were analysed. The results in Tables 47 and 48 show that there was no significant difference in the scores of individuals from different language backgrounds, indicating that TAR60 is a relatively language-free measure of general ability.


English Other Missing Group total

Valid N 165 128 2 295

Mean 48.65 47.47 38.50 48.07

Minimum 25 22 33 22

Maximum 59 59 44 59

Std. deviation 6.67 6.64 7.78 6.71



Between groups 100.32 1 100.32 2.264 0.134

Within groups 12897.49 291 44.32

Total 12997.81 292

Study typeIndividual results were grouped by study type and analysed. The results in Tables 49 and 50 show that there was no significant difference in the scores of individuals in different types of study.

Table 49 TAR60 mean and standard deviation by study types

1 2 3 4 5 6 7 8 9 10 11

Valid N 55 88 22 15 26 6 6 6 16 55 295

Mean 46.02 49.64 49.45 48.67 48.77 46.17 47.00 46.67 47.06 47.33 48.07

Minimum 28 36 27 41 33 30 40 37 34 22 22

Maximum 57 58 58 56 59 54 55 55 58 59 59

Std. deviation 7.49 5.86 6.88 5.29 6.69 9.30 6.20 7.34 6.88 6.83 6.71

Key: 1. Undergraduate humanities, law or education degree; 2. Undergraduate science, IT or engineering degree; 3. Undergraduate commerce, finance or accounting degree; 4. Undergraduate medicine degree; 5. Undergraduate double degree; 6. Undergraduate degree – discipline not stated; 7. Postgraduate commerce degree; 8. Postgraduate humanities degree; 9. Postgraduate science degree; 10. Not stated / not a student; 11. Group total.

�0


Table 50 ANOVA TAR60 scores on study types


Between groups 594.68 9 66.08 1.489 0.151

Within groups 12647.96 285 44.38

Total 13242.64 294

Test orderTwo groups of individuals were given the APM and TAR60 in different order. One group was given the APM before TAR60. The other group was given the two tests in the reverse order. This was to negate the possible order or practise effect. The results in Tables 51 and 52 show that there was no significant difference in the scores of the two groups.


TAR60/APM APM/TAR60 Group total

Valid N 152 143 295

Mean 48.63 47.48 48.07

Minimum 25 22 22

Maximum 59 59 59

Standard deviation 6.65 6.74 6.71



Between groups 97.36 1 97.36 2.170 0.142

Within groups 13145.29 293 44.86

Total 13242.64 294

ACER TAR Short Form (TAR45)

The following is analysis on data from the first 45 items of TAR60 completed by the sample in Study 2. Because there were fewer missing responses on the first 45 items, this provides more accurate data.

Background variables

GenderResults of males and females were compared and analysed. The results in Tables 53 and 54 show that there was no significant difference in the performance of males and females.

��



Male scores Female scores Missing Group total

Valid N 154 139 2 295

Mean 39.11 38.82 37.50 38.96

Minimum 19 20 36 19

Maximum 45 45 39 45

Std. deviation 3.79 4.20 2.12 3.98



Between groups 10.46 2 5.23 0.329 0.720

Within groups 4638.13 292 15.88

Total 4648.59 294

Education levelIndividual results were grouped by education level and analysed. The results in Tables 55 and 56 show that there is no significant difference in the scores of individuals in different educational levels, however, there is some difference in the mean scores of Year 10 students compared to undergraduate and postgraduate students.


Postgrad degree

Undergrad degree

TAFE diploma

Year 12

Year 11

Year 10

Missing

Group total

Valid N 8 48 8 225 3 2 1 295

Mean 39.00 38.19 36.25 39.33 36.33 29.00 42.00 38.96

Minimum 31 20 22 27 28 19 42 19

Maximum 42 43 42 45 42 39 42 45

Std. deviation

3.78 4.21 6.82 3.49 7.37 14.14 – 3.98



Between groups 347.11 6 57.85 3.873 0.001

Within groups 4301.48 288 14.94

Total 4648.59 294

��


Language backgroundIndividual results were grouped by language background and analysed. The results in Tables 57 and 58 show that there is no significant difference in the scores of individuals from different language backgrounds.


English Other Missing Group total

Valid N 165 128 2 295

Mean 39.26 38.67 33.00 38.96

Minimum 19 20 30 19

Maximum 45 45 36 45

Std. deviation 3.91 4.00 4.24 3.98



Between groups 24.98 1 24.98 1.604 0.206

Within groups 4534.01 291 15.58

Total 4559.00 292

Study typeIndividual results were grouped by study type and analysed. The results in Tables 59 and 60 show that there is no significant difference in the scores of individuals in different types of study.

Table 59 TAR45 mean and standard deviation by study types

1 2 3 4 5 6 7 8 9 10 11

Valid N 55 88 22 15 26 6 6 6 16 55 295

Mean 37.65 39.76 39.41 38.80 39.04 37.83 39.50 39.17 38.75 38.93 38.96

Minimum 27 31 20 33 30 30 36 28 34 19 19

Maximum 43 44 44 44 45 42 42 43 43 45 45

Std. deviation

4.38 2.87 5.01 3.45 4.03 4.83 2.35 5.56 2.72 4.75 3.98

Key: 1. Undergraduate humanities, law or education degree; 2. Undergraduate science, IT or engineering degree; 3. Undergraduate commerce, finance or accounting degree; 4. Undergraduate medicine degree; 5. Undergraduate double degree; 6. Undergraduate degree – discipline not stated; 7. Postgraduate commerce degree; 8. Postgraduate humanities degree; 9. Postgraduate science degree; 10. Not stated / not a student; 11. Group total.

��


Table 60 ANOVA TAR45 scores on study types


Between groups 165.61 9 18.40 1.170 0.314

Within groups 4482.98 285 15.73

Total 4648.59 294

Test orderTwo groups of individuals were given the APM and TAR60 in different order. One group was given the APM before TAR60. The other group was given the tests in the reverse order. This was to negate possible order or practise effects. The results in Tables 61 and 62 show that there was no significant difference in the scores of the two groups.


TAR45/APM APM/TAR45 Group total

Valid N 152 143 295

Mean 39.12 38.80 38.96

Minimum 19 22 19

Maximum 45 45 45

Std. deviation 4.13 3.81 3.98



Between groups 7.60 1 7.60 0.480 0.489

Within groups 4640.99 293 15.84

Total 4648.59 294

Conclusion

In these analyses of variance no significant differences were found for gender, language, study type and order of test. However, a significant difference was found between groups with different levels of completed education on the TAR60 and TAR45.

��


study 3 – ACeR tAR short form (tAR45) 2006

Study design

This study was conducted using data from the completion of TAR45 online from January 2005 to April 2006. The individuals included in this study were recruitment candidates or applicants for various job positions. There were a total of 329 individuals in this reference group. Percentiles and stanines derived from this reference group will be available in all automated email reports.

The number of males and females in the reference group is provided in the table below.

Table 63 Number of males and females in the reference group


Males 203

Females 126

Total 329

The descriptive statistics for TAR45 are provided in Table 64. The mean raw score on TAR45 is 37.06 out of 45 questions for this group. The raw scores tend to be skewed towards the higher end for this group.

Table 64 Descriptive statistics for TAR45

Value

Valid N 326

Missing 3

Mean 37.06

Std. deviation 6.36

Variance 40.40

Range 32

Minimum 13

Maximum 45

Test score at 25th percentile 34



ReliabilityThe reliabilities of the subscales comprising TAR45 are given below. The internal consistency of the items contained in the subsets is relatively high, ranging from 0.52 to 0.91. This means that all the items in the subsets are measuring the same construct. The internal reliability of TAR45 was 0.904, which is very high.

��


Table 65 Reliabilities of TAR45 subscales

TAR45 Number of items Reliabilities

Set 1 (items 1–7) 7 0.517

Set 2 (items 8–16) 9 0.623

Set 3 (items 17–26) 10 0.751

Set 4 (items 27–36) 10 0.732

Set 5 (items 37–45) 9 0.911

Table 66 gives the correlations between the various item subsets in TAR45. There are low to moderate correlations between the various subsets, which show that the constructs measured by each subset are related to some extent.

Table 66 Correlations between subsets in TAR45

TAR45 Set 1 Set 2 Set 3 Set 4 Set 5


Set 2 (items 8–16) 0.520 1

Set 3 (items 17–26) 0.501 0.534 1

Set 4 (items 27–36) 0.368 0.410 0.517 1

Set 5 (items 37–45) 0.300 0.262 0.438 0.595 1

Gender differencesData from this study was split into male and female scores and compared.

Table 67 TAR45 scores for males and females

Female raw score Male raw score

Mean 37.33 36.89

Maximum 45 45

Minimum 18 13

Range 27 32

Std. deviation 5.71 6.74

Count 126 203

No significant difference between males and females was seen, based on the results of an independent samples t test, which checks for statistically significant differences between two or more groups. This shows that there is no difference in the performance of males and females on TAR45.

��


Table 68 Statistics from independent sample t tests

Test t statistics df Significance

Test of Abstract Reasoning (TAR45) 0.609 324 0.543

Conclusion

TAR45 is a reliable, valid instrument that correlates well with Raven’s SPM as a measure of general ability.

��

RefeRenCes

Anastasi, A. 1976, Psychological Testing, 4th edn, Macmillan, New York.Australian Council for Educational Research 2000, Aptitude Profile Test Series (APTS) Occupational, ACER,

Camberwell.Australian Council for Educational Research 2005, Middle Years Ability Tests (MYAT), ACER, Camberwell.Cronbach, J. L. 1970, Essentials of Psychological Testing, 3rd edn, Harper & Row, New York.Fisher, G. H. and Molenaar, I. W. 1995, Rasch Models: Foundations, Recent Developments and Applications,

Springer-Verlag, New York.Holland, J. L. 1997, Making Vocational Choices: A Theory of Vocational Personalities and Work Environments,

3rd edn, Psychological Assessment Resources, Florida.Kummerow, J. M. (ed.) 1991, New Directions in Career Planning and the Workplace: Practical Strategies for

Counselors, Davies-Black, California.Nelson Bolles, R. 2006, What Color Is Your Parachute?, Ten Speed Press, California.Power, M. 2004, A Quick Guide to HR Testing, ACER Press, Camberwell.Raven, J. C. et. al. 1983, Raven’s Advanced Progressive Matrices, OPP Limited, Oxford. Raven, J. C. et. al. 2000, Raven’s Standard Progressive Matrices 2000 Edition, OPP Limited, Oxford.

��

AppenDICes

AppenDIX A: Quest item analysis for all 60 tAR items (study 1)

Item number

Correct responses

Number of people

attempting item

Difficulty estimates (in logits)

Error

Infit mean

squares

1 146 151 –2.12 0.47 1.66

2 127 150 –0.31 0.24 0.68

3 150 151 –3.78 1.01 0.46

4 119 151 0.14 0.22 1.13

5 138 150 –1.14 0.32 0.96

6 131 151 –0.51 0.26 1.14

7 136 151 –0.87 0.29 0.76

8 149 151 –3.08 0.72 0.38

9 140 151 –1.24 0.33 0.78

10 123 150 –0.10 0.23 0.92

11 136 151 –0.87 0.29 0.54

12 136 151 –0.87 0.29 0.48

13 143 151 –1.61 0.38 0.77

14 102 151 0.82 0.19 0.74

15 136 151 –0.87 0.29 0.47

16 52 151 2.48 0.19 1.32

17 142 149 –1.73 0.40 1.10

18 139 149 –1.33 0.34 0.82

19 127 149 –0.36 0.25 0.63

20 93 149 1.10 0.19 1.23

21 140 148 –1.58 0.38 1.05

22 132 145 –0.99 0.31 1.46

23 129 148 –0.54 0.26 0.85

24 127 149 –0.36 0.25 0.70

25 128 149 –0.42 0.25 0.67

26 83 139 1.22 0.19 0.78

27 135 149 –0.95 0.30 1.47

28 85 148 1.33 0.18 1.12

29 117 146 0.03 0.23 0.98

��

A p p e n d i c e s

Item number

Correct responses

Number of people

attempting item


Error

Infit mean

squares

30 123 147 –0.24 0.24 0.59

31 127 143 –0.72 0.28 1.12

32 97 142 0.79 0.20 0.95

33 88 144 1.15 0.19 0.83

34 97 142 0.77 0.20 0.71

35 60 142 2.09 0.19 1.18

36 119 141 –0.29 0.25 0.87

37 131 139 –1.52 0.38 0.46

38 133 139 –1.84 0.43 0.35

39 128 136 –1.50 0.38 0.91

40 129 134 –1.97 0.47 0.43

41 120 129 –1.19 0.36 0.55

42 77 122 1.20 0.21 1.63

43 111 120 –0.97 0.36 0.56

44 82 109 0.60 0.24 0.99

45 78 106 0.72 0.24 0.91

46 59 95 1.39 0.23 0.78

47 73 95 0.59 0.26 1.28

48 55 89 1.42 0.24 0.90

49 57 78 0.95 0.28 0.93

50 53 71 0.90 0.29 0.74

51 44 65 1.39 0.29 1.33

52 36 62 1.87 0.28 0.98

53 34 50 1.51 0.33 0.88

54 23 35 1.68 0.39 1.02

55 21 31 1.48 0.42 1.40

56 14 25 2.05 0.44 1.18

57 16 22 1.20 0.52 0.90

58 6 15 2.77 0.57 0.98

59 7 14 2.24 0.59 0.86

60 11 13 0.06 0.86 0.92

Appendix A (cont.)

�0

A p p e n d i c e s

AppenDIX b: Quest output for tAR45 (study 1)

Item number

Facility

Point biserial


Infit mean squares

1 96.7 0.12 –1.64 1.01

2 84.7 0.48 0.17 0.86

3 99.3 0.11 –3.29 0.98

4 78.8 0.31 0.61 1.05

5 92.0 0.16 –0.66 1.12

6 86.8 0.18 –0.04 1.12

7 90.1 0.31 –0.40 0.98

8 98.7 0.26 –2.59 0.93

9 92.7 0.28 –0.77 1.00

10 82.0 0.41 0.38 0.94

11 90.1 0.47 –0.40 0.87

12 90.1 0.51 –0.40 0.83

13 94.7 0.17 –1.13 1.06

14 67.5 0.53 1.30 0.86

15 90.1 0.54 –0.40 0.80

16 34.4 0.26 2.96 1.08

17 95.3 0.13 –1.25 1.06

18 93.3 0.32 –0.85 0.94

19 85.2 0.51 0.12 0.84

20 62.4 0.28 1.58 1.13

21 94.6 0.12 –1.10 1.08

22 91.0 0.19 –0.52 1.07

23 87.2 0.29 –0.07 1.04

24 85.2 0.52 0.12 0.82

25 85.9 0.57 0.06 0.78

26 59.7 0.56 1.70 0.83

27 90.6 0.05 –0.47 1.17

28 57.4 0.28 1.81 1.08

29 80.1 0.18 0.50 1.18

30 83.7 0.58 0.23 0.79

31 88.8 0.25 –0.25 1.05

��

A p p e n d i c e s

Item number

Facility

Point biserial


Infit mean squares

32 68.3 0.52 1.26 0.89

33 61.1 0.52 1.62 0.88

34 68.3 0.59 1.25 0.81

35 42.3 0.29 2.57 1.17

36 84.4 0.29 0.19 1.07

37 94.2 0.49 –1.05 0.80

38 95.7 0.46 –1.37 0.82

39 94.1 0.35 –1.02 0.90

40 96.3 0.38 –1.50 0.86

41 93.0 0.39 –0.71 0.90

42 63.1 0.05 1.67 1.39

43 92.5 0.51 –0.50 0.80

44 75.2 0.33 1.07 1.00

45 73.6 0.31 1.19 1.03

Appendix B (cont.)

��

A p p e n d i c e s

AppenDIX C: Detailed conversion table between tAR45 and spM (study 1)

TAR rawTAR45 scale

scoreTAR logit

(pre-equating)

TAR logit (post equating: including

shift constant) SPM logit SPM raw

15 45 –0.89 –0.25 –0.51 28

16 46 –0.77 –0.13 –0.35 29

17 48 –0.66 –0.02 –0.19 31

18 49 –0.54 0.10 –0.02 32

19 50 –0.42 0.22 0.14 33

20 51 –0.30 0.34 0.27 34

21 52 –0.19 0.45 0.40 35

22 53 –0.07 0.57 0.52 36

23 54 0.04 0.68 0.66 37

24 55 0.16 0.80 0.79 38

25 56 0.27 0.91 0.79 38

26 58 0.39 1.03 0.93 39

27 59 0.51 1.15 1.06 40

28 60 0.63 1.27 1.20 41

29 61 0.75 1.39 1.34 42

30 62 0.88 1.52 1.49 43

31 64 1.01 1.65 1.64 44

32 65 1.14 1.78 1.64 44

33 66 1.29 1.93 1.80 45

34 68 1.43 2.07 1.96 46

35 69 1.59 2.23 2.13 47

36 71 1.75 2.39 2.30 48

37 73 1.93 2.57 2.49 49

38 74 2.12 2.76 2.69 50

39 76 2.33 2.97 2.90 51

40 79 2.56 3.20 3.13 52

41 81 2.83 3.47 3.38 53

42 84 3.15 3.79 3.66 54

43 88 3.56 4.20 3.97 55

44 94 4.15 4.79 4.75 57

45 100 5.33 5.97 5.27 58

* Numbers in bold are obtained by linear interpolation

��

A p p e n d i c e s

AppenDIX D: Quest item analysis for all 60 tAR items (study 2)

Item number

Correct responses

Number of people

attempting them


Error

Infit mean

squares

1 286 295 –1.47 0.34 0.93

2 273 295 –0.50 0.23 0.58

3 290 295 –2.07 0.46 1.22

4 251 294 0.30 0.17 0.97

5 275 293 –0.72 0.25 1.16

6 242 295 0.57 0.16 0.92

7 266 295 –0.18 0.20 0.78

8 293 295 –2.99 0.71 0.34

9 278 295 –0.79 0.26 1.38

10 256 295 0.18 0.18 1.15

11 276 294 –0.72 0.25 0.90

12 280 295 –0.92 0.27 0.78

13 290 295 –2.07 0.46 0.57

14 261 294 –0.02 0.19 0.74

15 273 295 –0.50 0.23 0.98

16 78 294 3.45 0.14 1.53

17 283 293 –1.34 0.33 1.33

18 285 295 –1.36 0.33 0.73

19 279 294 –0.92 0.27 0.82

20 192 295 1.58 0.13 1.15

21 285 295 –1.36 0.33 0.75

22 263 294 –0.09 0.20 1.12

23 273 295 –0.50 0.23 0.58

24 279 291 –1.12 0.30 0.91

25 272 294 –0.49 0.23 0.69

26 232 287 0.67 0.16 0.90

27 265 290 –0.34 0.22 0.96

28 195 294 1.52 0.13 1.21

29 237 293 0.67 0.16 1.06

30 273 293 –0.59 0.24 1.01

31 258 292 0.03 0.19 0.98

��

A p p e n d i c e s

Item number

Correct responses

Number of people

attempting them


Error

Infit mean

squares

32 218 294 1.10 0.14 1.01

33 188 290 1.59 0.13 1.02

34 242 290 0.47 0.17 0.97

35 127 285 2.55 0.13 1.04

36 255 293 0.16 0.18 1.18

37 291 294 –2.58 0.58 1.67

38 284 294 –1.35 0.33 1.11

39 288 293 –2.05 0.46 0.81

40 289 292 –2.56 0.58 0.66

41 283 289 –1.84 0.42 0.40

42 243 289 0.43 0.17 0.91

43 271 287 –0.79 0.26 0.59

44 223 286 0.87 0.15 1.05

45 253 279 –0.21 0.21 1.01

46 172 276 1.75 0.13 1.07

47 269 283 –0.93 0.28 0.73

48 212 278 0.98 0.15 0.86

49 215 273 0.83 0.16 0.83

50 209 271 0.95 0.15 0.75

51 197 265 1.11 0.15 0.89

52 208 263 0.81 0.16 1.06

53 186 254 1.18 0.15 0.97

54 167 239 1.36 0.15 0.81

55 160 228 1.36 0.16 0.82

56 148 224 1.57 0.15 1.26

57 139 205 1.48 0.16 1.01

58 117 201 1.94 0.15 0.98

59 123 194 1.71 0.16 1.12

60 164 189 0.22 0.23 0.95

Appendix D (cont.)

��

A p p e n d i c e s

AppenDIX e: Quest output for tAR45 (study 2)

Item number

Facility

Point biserial


Infit mean squares

1 96.9 0.27 –1.11 0.95

2 92.5 0.45 –0.14 0.88

3 98.3 0.16 –1.72 0.97

4 85.4 0.29 0.67 1.02

5 93.9 0.32 –0.36 0.95

6 82.0 0.31 0.94 1.03

7 90.2 0.31 0.18 0.99

8 99.3 0.25 –2.64 0.94

9 94.2 0.12 –0.43 1.07

10 86.8 0.21 0.54 1.07

11 93.9 0.20 –0.36 1.02

12 94.9 0.35 –0.57 0.92

13 98.3 0.26 –1.72 0.94

14 88.8 0.33 0.34 0.98

15 92.5 0.07 –0.14 1.11

16 26.5 0.13 3.86 1.18

17 96.6 0.17 –0.98 1.01

18 96.6 0.26 –1.00 0.97

19 94.9 0.16 –0.56 1.04

20 65.1 0.34 1.95 1.05

21 96.6 0.23 –1.00 0.99

22 89.5 0.28 0.27 1.00

23 92.5 0.44 –0.14 0.89

24 95.9 0.17 –0.76 1.02

25 92.5 0.37 –0.12 0.94

26 80.8 0.33 1.04 1.00

27 91.4 0.27 0.02 1.01

28 66.3 0.24 1.89 1.10

29 80.9 0.30 1.03 1.02

30 93.2 0.27 –0.23 0.99

31 88.4 0.28 0.39 1.01

32 74.1 0.37 1.47 0.99

��

A p p e n d i c e s

Item number

Facility

Point biserial


Infit mean squares

33 64.8 0.39 1.96 0.99

34 83.4 0.48 0.84 0.89

35 44.6 0.31 2.93 1.02

36 87.0 0.26 0.53 1.03

37 99.0 0.01 –2.23 1.03

38 96.6 0.11 –1.00 1.05

39 98.3 0.27 –1.69 0.93

40 99.0 0.14 –2.21 1.00

41 97.9 0.36 –1.48 0.91

42 84.1 0.36 0.80 0.98

43 94.4 0.35 –0.43 0.93

44 78.0 0.35 1.24 1.00

45 90.7 0.29 0.15 0.99

Appendix E (cont.)

��

A p p e n d i c e s

AppenDIX f: Detailed conversion table between tAR60 and ApM (study 2)

TAR60 raw score

TAR60 scale scoreBest predicted

TAR60 logit

Best predicted TAR60 logit in

TAR60 scale APM logit APM raw score

37 59 0.14 0.65 –1.02 12

38 60 0.23 0.74 –0.85 13

39 61 0.32 0.83 –0.69 14

40 62 0.41 0.92 –0.52 15

41 63 0.50 1.01 –0.35 16

42 64 0.59 1.10 –0.19 17

42 64 0.68 1.18 –0.03 18

43 65 0.77 1.27 0.14 19

44 66 0.86 1.37 0.31 20

45 67 0.94 1.45 0.47 21

46 68 1.04 1.54 0.64 22

47 69 1.13 1.63 0.81 23

47 69 1.22 1.73 0.99 24

48 71 1.32 1.82 1.16 25

49 72 1.42 1.93 1.35 26

50 73 1.52 2.03 1.54 27

50 73 1.63 2.13 1.74 28

51 75 1.75 2.25 1.96 29

52 77 1.87 2.38 2.19 30

53 78 2.00 2.51 2.44 31

54 80 2.16 2.67 2.73 32

55 80 2.34 2.85 3.07 33

56 82 2.57 3.08 3.50 34

57 88 2.91 3.41 4.12 35

58 92 3.55 4.06 5.32 36


M A N U A LTest of Abstract ReasoningACER

TARTE

ST O

F AB

STRA

CT R

EASO

NING

TEST

OF

ABST

RACT

REA

SONI

NG

TEST

OF

ABST

RACT

REA

SONI

NG

TEST

OF

ABST

RACT

REA

SONI

NG

TEST

OFAB

STRA

CT

ACER TEST OF ABSTRACT REASONING

TAR can be used for various courses and occupations, including:Managers and executives

Business managers

Professionals and specialists

Para-professionals (e.g. technicians)

Arts professionals

Finance and data analysts

Medical professionals

Scientists

The ACER Test of Abstract Reasoning (TAR) is a quick and easily administered test ofabstract reasoning in adults. Abstract reasoning is linked to the ability to thinkstrategically, analyse information, learn new information quickly, engage in creativeproblem-solving and make logical decisions. TAR is suitable for use as a selection ordevelopment tool for graduate, managerial and other positions that require critical thinkingskills. The test is free from language, cultural and gender biases.

TAR is available online and in paper-based formats, and is easy to administer andinterpret. It provides immediate results using Australian norms and can be used inindividual or group testing situations.

TARISBN 10: 0-86431-733-6ISBN 13: 978-0-86431-733-9

9 7 8 0 8 6 4 3 1 7 3 3 9

TAR 1/11/06 3:21 PM Page 1

acer test of abstract reasoning manual

Documents