ORIGINAL ARTICLE

Introducing contextual laddering to evaluate the likeabilityof games with children

Bieke Zaman

Received: 3 July 2006 / Accepted: 4 March 2007

� Springer-Verlag London Limited 2007

Abstract It becomes more and more recognized that

children should be involved in a product’s design and

evaluation process. Many findings report on the method-

ology for usability research with children. However, there

has been relatively little analysis of likeability research

with children. In this paper, we propose the laddering

method—traditionally a marketing method among

adults—for likeability research in the domain of child–

computer interaction. Three exploratory cases will be de-

scribed. The cases report on the use of the laddering

method with children aged between 7 and 16 to evaluate

the likeability of two games. The lessons learnt about the

use of the laddering method will be discussed in great

detail. In order to adapt the laddering method to work with

children, we recommend a variation of this method and call

it the ‘contextual laddering method’.

Keywords Likeability � Contextual laddering � Children �Evaluation technique � Computer game

1 Introduction

The development and evaluation of user friendly and

pleasant products for children needs an adapted or new

research method to involve children since adults cannot

correctly judge their expectations, skills or media context

(Hanna et al. 1997; Druin et al. 1999; Bruckmann and

Bandlow 2003). The instructive input children can give

regarding the design and evaluation of products can no

longer be neglected (Druin et al. 1999). Although sub-

stantial research effort has been devoted to usability re-

search with children, relatively little has been written about

the evaluation of the likeability of a product. In the domain

of child–computer interaction, several studies report on the

effectiveness, appropriateness or adaptation of existing

methods for usability research with children (Hanna et al.

1999; Als et al. 2005a, b; Markopoulos and Bekker 2003).

However, more research is needed to successfully deduce

information about the reasons why children like or dislike a

product. To this end, we introduce the laddering technique.

This paper begins with a discussion of usability and

likeability. Then, we review some techniques used to elicit

opinions from children. One of these elicitation techniques

is the laddering method. This method will be framed by

referring to previous research. The next section of the pa-

per describes three case studies were laddering was used

with children to evaluate the likeability of two games. The

results of these studies are presented and discussed. The

discussion in the paper focuses on lessons learnt about the

use of the laddering method in the three cases. In particular,

we give suggestions for the design of likeability tests with

children as well as some recommendations to adapt the

laddering interview to working with children. Finally, ideas

for further work are presented.

2 Usability, likeability

Both usability and likeability must be investigated for the

design and evaluation of products; and this applies to

games in particular. When evaluating productivity appli-

cations like for example spreadsheet software, the focus is

primarily on performance measurement. One then assesses

B. Zaman (&)

Centre for Usability Research, Katholieke Universiteit

Leuven, Parkstraat 45, BUS 3605, 3000 Leuven, Belgium

e-mail: Bieke.Zaman@soc.kuleuven.be

URL: http://soc.kuleuven.be/com/mediac/cuo/index.php

123

Cogn Tech Work

DOI 10.1007/s10111-007-0067-y

how quickly or efficiently tasks can be accomplished. For

the evaluation of games however, one should distinguish

the overall game elements such as missions, story, graph-

ics, mood and sound from the in-game interface elements

like a map, navigation keys or a scoreboard. As for the in-

game interface, a usability approach predominates. On the

contrary, when looking past a game’s interface, it becomes

clear that playability and likeability are of great impor-

tance. In this context Pagulayan et al. (2003) state that ‘‘the

goal of iterative usability testing on games is to reduce the

obstacles to fun, rather than the obstacles to accomplish-

ment’’. Games are developed to be entertaining and fun.

Experiences that are fun give an intrinsic reward to the

users and operate as an important motivator (Carroll and

Thomas 1988; Barendregt et al. 2003).

The International Organization for Standardization

(ISO 1998) defines usability as: ‘‘The extent to which a

product can be used by specified users to achieve speci-

fied goals with effectiveness, efficiency and satisfaction in

a specified context of use’’. Nielsen (2003) defines the

term usability in the broad sense by five components:

learnability, efficiency, memorability, errors and satisfac-

tion. Shackel (1991) only discerns four dimensions of

usability: effectiveness, learnability, flexibility and atti-

tude. These definitions of usability (in the broad sense)

show the importance of the likeability of a product, re-

ferred to as ‘satisfaction’ or ‘attitude’. However, like-

ability covers more than satisfaction and attitude alone.

Likeability is a stronger concept that differs from usabil-

ity. The likeability of a product can influence the per-

ceived usability and vice versa. For instance, if a website

scores badly at usability, users might find it also less

pleasant to use. In contrast, when a website is evaluated as

attractive and pleasant, the perceived usability may be

biased in a more positive way.

The users’ satisfaction (Nielsen 2003; ISO 1998), fun

(Read et al. 2002; Pagulayan et al. 2003) or attitude

(Shackel 1991) are manifestations of the same and broader

concept that is the likeability of a product. In this context,

Trevino and Webster (1992) also mention Csikszentmih-

alyi’s concept of flow. The ‘flow state’ refers to a playful,

pleasurable experience in which a person is fully immersed

in what he or she is doing (Trevino et al. 1992; Hoffman

and Novak 1996). It becomes clear that the likeability of a

product encompasses different aspects like a positive atti-

tude towards the product, satisfaction, flow experiences,

fun, play and so on. According to Federoff (2002) the

satisfaction of a game not only involves fun but also im-

mersive environments and compelling experiences. In this

paper, we will use terminology according to the literature

we are referring to (e.g., ‘fun’ for Read et al. 2002).

Nevertheless, it should be clear that all terms refer to the

likeability of the product in the broad sense.

3 Elicitation techniques with children

Our research purpose was to find an appropriate method-

ology to gain information about the reasons why children

like or dislike games. Current research has already pro-

posed different methods for the measurement of fun.

Baauw et al. (2005) defined an expert evaluation method

(SEEM) based on Malone’s concepts of fun to predict

usability and fun problems (Malone 1980). The SEEM fun

heuristics were not useful for our research purpose because

we wanted to involve children in the evaluation process.

Read et al. (2002) developed three tools to measure ‘how

much fun’ different computer interfaces for children are.

Hanna et al. (1997) evaluated fun by observing children’s

non-verbal behaviour (e.g., laugh, frowns). Although Read

et al. (2002) and Hanna et al. (1997) involve children in the

evaluation process, nothing is said about the reasons why

the product is fun or not. To this purpose, several

researchers (Fortini-Campbell 1990; Gutman 1982; Su-

bramony 2002; Jordan 2000; Grunert and Bech-Larsen

2005) propose the laddering method that accord to our two

conditions: (i) the involvement of children and (ii) an

analysis of the reasons why a product is fun or not. To

investigate the reasons why people choose or prefer a

product, Fortini-Campell (1990) argues in favour of the

laddering method, rather than other quantitative or quali-

tative social scientific research methods.

4 Laddering method

The laddering method is mainly used with adults in mar-

keting and consumer research (Grunert et al. 2005; Rey-

nolds and Gutman 2001; Peter and Olson 1994; Olsen and

Reynolds 2001). Subramony (2002) proved the applica-

bility of the laddering method in the context of human–

computer interaction research with adults. To our knowl-

edge, the combination of the laddering technique and

child–computer interaction research has never been made.

That is why our research purpose aimed at testing this

method with children for the likeability evaluation of

computer games.

The laddering method is based on the means-end theory

(Gutman 1982; Olsen and Reynolds 2001). This theory

helps to understand and describe how consumers perceive

products by revealing the core underlying values that

motivate consumers to desire certain product conse-

quences. According to this theory, people unconsciously

categorize incoming stimuli into a hierarchically chain of

beliefs, referred to as the means-end chain which consists

of attributes, consequences and values (Subramony 2002;

Gutman 1982). The beliefs are linked to each other

and vary in level of abstractness (Gutman 1982). The

Cogn Tech Work

123

means-end theory was introduced in consumer research

because it explains that not the product but the conse-

quences of specific product attributes determine consum-

ers’ preferences (Olsen and Reynolds 2001; Jordan 2000;

Grunert et al. 2005; Kjeldal 2002; Gutman 1982). One

woman might for example prefer crisps of brand X over

brand Y. When asking why brand X is better, the respon-

dent answers ‘because X is a light product’. ‘Light’ is then

an attribute of product X. If the interviewer asks why she

prefers light crisps, she replies that she will not get too fat

if she eats this product. ‘Not getting too fat’ is a relevant

consequence of the attribute ‘light’ of product X. Again,

the interviewer probes into the reasons why it is important

not to get fat. During the laddering interview, the answers

of the respondent will become more and more abstract.

This process continues until the personal values of the

respondent are known (Jordan 2000). In our example, the

answers of the respondent reveal the importance of staying

slim. It seems that good looks and beauty are important

personal values to her. Figure 1 shows how the informa-

tion gained during the laddering interview can be presented

in a ‘hierarchical value map (HVM)’ (Subramony 2002). A

HVM is a ‘ladder model’ of connected answers: the most

concrete answers form the bottom of the ladder the most

abstract answers the top (Reynolds and Gutman 2001;

Subramony 2002).

In this paper, we figure out how the laddering method

can be used for likeability research with children. To this

end, we propose a variation of the laddering method in

which we stress the importance of context: the contextual

laddering method. Reynold et al. (2001) already propose to

evaluate products in the particular context of the setting in

which they are naturally used. We conceptualized context

by adding two other elements. We further see context as

the relevant information gained in the run-up to the inter-

view (e.g., hobbies, language, attitudes). Finally, context

also refers to the application that stays within the child’s

reach during the interview.

5 Case studies

Three exploratory cases were conducted to test the lad-

dering method with children. More particularly, the like-

ability of two PC games was tested with children either in

their natural environment or in a controlled setting. In our

research, we were somewhat limited by the company

context that demanded results quickly. We therefore opted

for an exploratory working method rather than an experi-

mental one. In this paragraph, we will first discuss the

methodology of three cases. We focus on the test location,

test participants, software and test procedure. Second, we

go more deeply into the details of the data analysis. Finally,

we describe some of the results.

5.1 Method

The method of three exploratory cases will be discussed.

The cases vary with regard to the location (usability lab

versus natural environment), age of the children, software

(Music Community and KetnetKick) and the test procedure

(usability and/or likeability test).

5.1.1 Case 1

Case 1 aimed at investigating the usability and likeability

of a music game. The tests were conducted in a controlled

setting with children aged 7–16.

Location The first case took place in a usability lab. This

lab was divided into two rooms. One room was arranged as

a living room so that test-users would experience new

applications in a situation that is close to real time expe-

rience. The other room concerned a control room where an

observer took notes and captured video.

Why this location Depending on the initial research pur-

pose, we observed and interviewed children in a specific

test environment. As for case 1, the main aim was to

evaluate the usability of a game before its commercial

launch. Quick results were demanded so that the usability

suggestions could be implemented on time. This is why the

first case took place in a controlled setting, a usability lab.

Effect of the location The test set-up of case 1 began with

a usability session that took place in the living room withFig. 1 Example of a hierarchical value map (HVM)

Cogn Tech Work

123

one researcher called ‘the facilitator’. Then, the interview

happened in the control room with another researcher: ‘the

observer’. The change of test set-up was problematic for

three reasons. First, children had become used to the

presence of the facilitator during the observation session.

During the interview, children were suddenly introduced to

another researcher, the observer. Second, children became

familiarized to the living room that represented a cozy,

homelike place. After 1 h and a half of play, they were

invited in the control room for the interview. The control

room is a workplace, organized with a lot of technical

infrastructure. Third, the role of the test children changed

from being observed to an active participation in the

interview. Consequently, children had to get familiar with

the new setting and researcher as well as with the new

interaction method. During the interview, each test child

sat in front of the observer’s computer that showed his/her

video images captured during the test session. Using cap-

tured video as a reminder had some limitations because of

the exclusion of immediate interaction (e.g., to show

examples when answering the laddering questions). How-

ever, showing video suppressed the child’s temptation to

start playing the game and proved to be a useful aid to

memory.

Participants Ten children of which seven boys and three

girls participated in case 1. Their age varied between 7 and

16 years. Table 1 shows the number of test participants

according to the age category.

Software The computer game tested in case 1 concerned

a music game for children between 8 and 15 years old,

called the Music Community. By the time of the test ses-

sions, the game was still in its development phase. Con-

sequently, we could only test the music creation studio (see

Fig. 2) and the website (see Fig. 3). In the studio, children

created samples and uploaded these to their personal

website, a mobile phone or sent it to a public broadcaster.

The personal website assembled some favourite photo-

graphs, created songs, the latest blog updates and a top five

of most popular songs/artists.

Procedure Preceding the laddering interview, we had

been observing the test child while he/she was playing with

the computer game. More particularly, a usability test of a

computer game was conducted before the interview session

took place (see Fig. 4). The usability test lasted 1 h and a

half. We incited the test child to play by giving a few

general tasks like for example ‘Can you please go and play

with the music creation studio?’ or ‘Please, visit your

personal website’. During the usability test, we opted for

the think aloud method, an effective usability evaluation

technique (Wright and Monk 1990; Rubin 1994). By ask-

ing children to talk/think out loud while performing tasks,

the observer not only gained information about what was

happening but also why certain things occurred. After the

usability test, we conducted an interview session (see

Fig. 4). In total, ten questions were asked. Eight ques-

tions related to the usability of the game; two questions

Table 1 Case 1: number of test

participants according to ageAge N

7 1

8 0

9 2

10 1

11 1

12 1

13 2

14 0

15 1

16 1

Fig. 2 Music creation studio (copyright Larian)

Fig. 3 Personal website (copyright Larian)

Cogn Tech Work

123

(1 positive and 1 negative) to the likeability. We found that

children looked somewhat tired after the usability session.

That is why we decided to shorten the length of the

usability session in case 2.

5.1.2 Case 2

During case 2, the usability and likeability of a creative and

adventurous game was tested with children aged 7–9.

Whereas case 1 took place in a controlled setting, case 2

occurred in children’s natural environment (at home).

Location During case 2, children were observed and

interviewed in their natural environment. We asked parents

in which room their children normally play with the game

to choose the appropriate test setting. Half of the children

usually play in the living room, the other half in their

bedroom. For one family, we had to switch rooms to ob-

serve and interview all siblings as close to real life expe-

rience as possible.

Why this location The initial focus of case 2 related to the

usability in the broadest sense. Not only the user-friendli-

ness but also the social implications and the likeability of

the product were investigated. Consequently, a real-life

setting was the most appropriate choice.

Effect of the location Observing and interviewing chil-

dren at home had some pros and cons. On the one hand,

children did not need time to get familiar with the setting.

Parents and siblings could help children feel at ease.

Researchers received a lot of information about the child

by interpreting contextual cues like for example the inter-

actions between child and other family members or the

decoration in the child’s bedroom. On the other hand,

children were more easily distracted in their own envi-

ronment than in a usability lab: siblings often asked the test

child’s attention, parents sometimes influenced the answers

during the interview, the sound of the television distorted

the test child’s concentration etc. Contrary to case 1, the

observation and interview took place in the same room,

which resulted into a smooth change between the two

sessions. Children could explain their answers during the

interview by showing real time examples on the computer.

The computer game functioned as an aid to memory, which

made it easier to formulate opinions and mention relevant

information.

Participants Nine children participated in case 2 (4 boys,

5 girls). Their age varied between 7 and 13 years. Table 2

shows the number of test participants according to the age

category.

Software Figures 5 and 6 illustrate KetnetKick, a multi-

media game for children between 6 and 12 years. Ketnet-

Kick is part of a ‘PCtoTV’-project of the public

CONTEXT

INTERVIEW:

2 LADDERING QUESTIONS

INTERVIEW :

8 USABILITY QUESTIONS

USABILITY SESSION

Fig. 4 Timeline case 1 and

case 2

Fig. 5 Screenshot of the square

which is the centre of the

KetnetKick world and an

entrance to the different sub

games (copyright VRT)

Table 2 Case 2: number of test

participants according to ageAge N

7 1

8 1

9 2

10 3

11 1

12 0

13 1

Cogn Tech Work

123

broadcaster of the Flemish community in Belgium, VRT

and is developed by Larian. The computer game encom-

passes a mix of 2D creative studios and a 3D play world

that is updated regularly. At the time of the test sessions,

the 3D world offered single player games. The studios al-

lowed children to make movies, drawings, comics, etc.

Their creations could be sent to friends or family but also to

VRT. The latter daily broadcasted a selection of the cre-

ations.

Procedure The procedure was the same as case 1 with the

exception of the location and the duration of the usability

test. Whereas the usability test in case 1 lasted about 1 h

and a half, the length of the test during case 2 was 1 h. Still,

we found that 1 h was too long because an interview fol-

lowed. The learning process of a new game demanded a lot

of effort so that little energy was left for the interview.

Consequently, we decided to shorten the length of the

observation session of case 3 to half an hour. Beside this,

we also found that we overloaded the children by asking

ten questions, especially since an observation session pre-

ceded. As a result, we left out the eight more general

questions in case 3 and focused exclusively on the lad-

dering questions.

5.1.3 Case 3

Case 3 concerned a full likeability test. The observation

session that preceded the interview was no longer set up as

a usability test and the interview session only encompassed

laddering questions.

Location The location was the same as in case 2. Chil-

dren were observed and interviewed at home.

Why this location Since case 3 was conceived as a full

likeability test, we preferred a natural test setting. Children

behave more naturally in a familiar setting. Consequently,

the researcher will receive more contextual clues that help

to correctly interpret the answers of the children.

Effect of the location Although conducting the likeability

test at home was quite time consuming (e.g., because of

unexpected technical problems and interference of sib-

lings), it was very useful to gain contextual information

like for example the language of the children, how they use

the product, which disturbing factors occur, etc.

Software As in case 2, the likeability of the game Ket-

netKick was tested. However, in the mean time KetnetKick

Fig. 6 Screenshot of a 2D

creative game (film studio) and

a 3D adventurous play game

(copyright VRT)

Fig. 7 Karrewiet, a news

broadcast for children, imported

into KetnetKick via video

streaming

Cogn Tech Work

123

was updated with video-on-demand. Consequently, chil-

dren could view clips of ‘Eurosong for Kids’ and ‘Kar-

rewiet’, a news broadcast for children on television (see

Figs. 7, 8).

Participants Three girls and five boys participated in case

3. The number of test participants according to the age

category, is illustrated in Table 3.

Procedure Case 3 exclusively focused on the likeability

of the game KetnetKick. No usability issues (in the narrow

sense) were investigated. The main test set-up consisted of

half an hour free play on the one hand (‘observation ses-

sion’) and the laddering interview on the other hand (see

Fig. 8). Half an hour of observation seemed sufficient to

obtain the most important contextual cues and keep the

child’s attention. One should take care not to skip the

usability/observation session. Giving the child some time

to play with the computer game proved to have several

advantages. The observation session broke the ice. The

child became familiar to the test situation, the researcher

and her equipment (see also Christensen and Allison 2000;

Pellegrini 1996). The researcher received relevant infor-

mation (‘contextual clues’) to formulate adequate ques-

tions, interpret the answers correctly and prompt the child

without evoking social desirable answers. Recent game

experiences reactivated the child’s memory, which made it

easier to formulate opinions and mention relevant infor-

mation. Beside this, the child’s memory was also reacti-

vated when the application stayed within the child’s reach

during the interview. The application then functioned as a

visual stimulus. Visual stimuli make the interview issues

far more concrete than verbal representations alone

(Christensen and Allison 2000).

5.2 Analysis

According to Elliot and Adelman (1973), Lincoln and

Guba (1985), and McNiff (2002), triangulation is an

appropriate technique to enhance the internal validation of

a case study. The technique gathers evidence from different

sources bearing on the same findings. In this study, the

triangulation was centered on (i) the positive laddering

interview, (ii) the negative laddering interview and (iii) the

observation session. As a result, children’s responses were

analyzed from different angles in order to improve the

validity of the laddering data.

For the analysis of the laddering data, one should define

the attributes (dis)liked and the explanatory consequences.

If the most important consequences are revealed, one can

determine the underlying values. We will explain this

process by two examples of case 2 in which the game

KetnetKick was tested. One child mentioned that he likes

the Snowboard race (see example 1 below). Another child

said he does not like too many obstacles in KetnetKick (see

example 2 below). When analyzing these answers, the re-

searcher distinguished two attributes: the ‘Snowboard race’

and ‘(too many) obstacles’. This information was worthless

if we could not understand why these attributes were

respectively, liked or disliked. For instance, the interviewer

found that whether the second child likes the obstacles or

not depends on the situation. It is thus necessary to gain

insight into the consequences of the attributes. As for the

Snowboard race, the first child mentioned that he likes the

glorious feeling of jumping. When asking for more

explanation, the child answered he prefers the tunnel where

he can catch a lot of marbles. It is also the most difficult

tunnel. As for the second child, he made clear that he does

not like the consequences of the obstacles (that is probably

losing time and energy) if he has a particular goal/game in

mind. However, if he is not eager to play a predetermined

game, he does not mind an eventful trip full of obstacles.

Example 1 (positive laddering)

Interviewer: ‘Please tell me something that you liked

about the game?’

Test child: ‘The Snowboard race is the greatest

game’.

Interviewer: ‘What is so great about the Snowboard

game?’

Test child: ‘It’s just fun ...’

Interviewer: ‘mmm’

Test child: ‘And jumping gives me a glorious feel-

ing’.

Interviewer: ‘What makes you like to jump?’

Test child: ‘Well, for me I prefer to race through the

tunnel with the most marbles.’

Interviewer: ‘What is so special about that tunnel?’

Test child: ‘It’s the most difficult tunnel where you

can win most points’.

Example 2 (negative laddering)

Interviewer: ‘Please tell me, what didn’t you like

about the game?’

Test child: ‘I don’t like it when there are too many

obstacles in the game world’.

Table 3 Case 3: number of test

participants according to ageAge N

8 1

9 0

10 2

11 3

12 1

13 0

14 1

Cogn Tech Work

123

Interviewer: ‘So the obstacles aren’t nice ...’

Test child: ‘Well sometimes they’re fun, but some-

times they’re not.’

Interviewer: ‘Can you explain when you don’t like

the obstacles?’

Test child: ‘Sometimes I just want to be somewhere

very quickly. For example when I want to go from

one game to another. At that moment, I do not want

to loose time with the obstacles.’

Interviewer: ‘And when do you like the obstacles?’

Test child: ‘Sometimes, I don’t know what to play. At

that moment, I like walking around in the game world

and taking obstacles.’

The labels for the attributes, consequences and values

should correspond to the actual responses of the intervie-

wees (Subramony 2002). However, it rarely happened that

test children explicitly mentioned their personal values in

the interview. Consequently, the researcher had to decide

for herself, which values applied. The responses of exam-

ple 1 ‘It’s the most difficult tunnel where you can win most

points’ and example 2 ‘[...] I like walking around in the

game world and taking obstacles’ were coded under the

label ‘challenge’ whereas the answer ‘Sometimes I just

want to be somewhere very quickly. [...] At that moment, I

do not want to loose time with the obstacles’ was coded

under the label ‘control’.

The combination of the observation session and the

interview was a great help to label the values together with

the interpretation of contextual clues. The laddering inter-

view gave a good opportunity to probe the child to give

more elaborate answers than during the observation ses-

sion. However, as the child could think more consciously

about an answer, the chance of a social desirable answer to

please the adult increased. The contextual clues gained

during the test made it easier to correctly analyze the an-

swers. We illustrate this with an example of case 3. During

the observation session, a boy turned up his nose twice and

said ‘oh no, Karrewiet’. He mentioned casually that he is

not interested in the current events. Karrewiet, a news

broadcast for children on television, was imported into the

game via video streaming. When the researcher asked

about Karrewiet during the interview, the child answered:

‘Karrewiet is a very important programme to watch, for

myself, to keep up to date’. We can conclude that the boy

knows it is important to keep up to date, but that he prefers

watching other programmes than the news broadcast be-

cause it does not really interest him.

6 Results

The results of a laddering interview are typically presented

in a HVM (Reynolds and Gutman 2001). A HVM provides

a concise answer on the question why a product is (dis)-

liked since it explains which specific attributes cause rel-

evant consequences corresponding with the user’s personal

values.

As stated earlier, the likeability of a product cannot be

explained without investigating the positive and negative

consequences the user perceives. Reynolds and Gutman

(2001) propose negative laddering when a respondent

cannot answer the positive laddering question. The

respondent then thinks about the reasons why he/she does

not like certain attributes/consequences. By definition, we

conducted both positive as negative laddering in order to

get a complete overview of the most important personal

values in relation to the tested computer game. Conse-

quently, we had to code all the laddering results first so that

we could merge them into one HVM. To this end, we

reformulated the answers of the negative laddering in a

positive way. For instance, the value ‘lack of control’ be-

came ‘control’ or the consequence ‘lose time’ became

‘save time’. Table 4 shows the codes for ‘example 1’ and

‘example 2’ that we discussed earlier in the analysis sec-

tion.

As shown in Fig. 9, the codes can be mapped out in a

HVM. Each value map consists of different layers or

‘rungs’. The bottom layers present the attributes children

said to like or dislike. The middle layers give the reasons

(or consequences of the product) why children (dis)liked

CONTEXT

OBSERVATION SESSION

FREE PLAY

INTERVIEW:

2 LADDERING QUESTIONS

Fig. 8 Timeline case 3

Table 4 Laddering codes (po-

sitive and negative laddering)Attributes

+ Jumping (Snowboard game)

+ Difficulties (tunnel with most

marbles)

± Too many obstacles

Consequences

+ Glorious feeling

+ Winning points

+ Save time (and energy)

+ Exploration and taking

obstacles

Values

+ Challenge (action)

+ Control

Cogn Tech Work

123

the attributes. The top of the ladder shows the most

important values of the test users.

7 Discussion

In this paragraph, we describe the lessons we learnt about

the use of the laddering method for likeability research

with children. First, we focus on the design of likeability

tests. As regards the laddering interview, special interview

techniques and the correct use of contextual clues are

discussed.

7.1 Design of likeability tests

In general, a likeability test consists of two successive

sessions: an observation session (this can be a usability

test) and the laddering interview. As for the observation

session, children approximately should have about half an

hour free play. The researcher takes notes and interprets

contextual clues. Then, the laddering interview follows.

The same researcher starts two probing sessions. The first

session begins with a positive question to reveal the rea-

sons why the child likes the product. The second probing

session is initiated by a negative question that reveals the

reasons why the child does not like the product.

Our cases show that both usability lab as well as chil-

dren’s natural environment are possible locations to con-

duct a likeability test. Each location has its pros and cons.

A usability lab is a controlled setting where the researcher

can prevent disturbing elements such as background noise

or interference of parents and siblings. Researchers can

easily prepare this test setting in advance in order to pre-

vent unexpected technical or infrastructural problems.

Since the test setting can easily be controlled, quick results

are obtained. With regard to the natural environment,

researchers receive a lot of additional information about the

participants. For instance, children behave more naturally

in their daily environment. The disadvantage of conducting

likeability tests in children’s natural environment is the

lack of control. Children are easily distracted by the

environment like for example the sound of the television,

siblings asking for attention or parents who meddle. Fur-

ther, researchers cannot prepare the test setting in advance

so that unexpected technical or infrastructural problems

might occur. Consequently, testing in natural environment

is much more time consuming.

Although both locations are possible to conduct a like-

ability test, we prefer children’s natural environment be-

cause it meets our third condition of context. The first

condition is that researchers interpret children’s verbal and

non-verbal behaviour, the decoration of their bedroom, the

toys they are playing with, the shortcuts and screensaver of

their computer, etc. These contextual clues, gained during

the observation session, help the researcher adapting his/

her language to the children and correctly interpreting their

answers. Further, we recommend that the interview occurs

at the same location as the observation session and that the

application stays within the child’s reach during the inter-

view in order to reactivate his/her memory and help

expressing his/herself by giving real time examples. Fi-

nally, the likeability test should happen in children’s nat-

ural environment to fulfill our third condition. Since

children behave more naturally in a familiar setting (e.g.,

when they are at home), researchers get a representative

impression of how they actually use the product. One also

receives information about the computer’s capacity, the

disturbing elements and so on. The findings of our cases

showed that the laddering method is appropriate to evaluate

the likeability with children on condition that it meets our

conceptualization of context. We call this variation of the

laddering method the ‘contextual laddering method’.

7.2 Laddering interview

The laddering method refers to a specific one-to-one elic-

itation interview technique. During the laddering interview,

a researcher probes into the reasons why children like or

dislike a product. The results of our case studies show that

when questions are only asked positively, children tend to

forget to mention relevant information. We therefore ad-

vise to have two probing sessions: one that starts with a

positive question and another starting with a negative

question.

In general, we tried to reduce cognitive efforts to a

minimum by asking relatively short and clear laddering

questions. Whereas longer questions provide helpful

memory cues for adults, they do load children’s verbal

comprehension and memory. The shorter the questions, the

less cognitively demanding they are for children and the

more reliable their responses tend to be (Borger et al.

2004). During our cases, we also found that children may

not be overloaded with questions, especially because an

observation session preceded. To this end, one should not

ask other questions than the positive and negative laddering

questions. Further, we opted for clear (non-hypothetical)

and open-ended questions as they elicit more information

per question and lead to more objective answers (Walker

et al. 1992; Christensen and Allison 2000). Another tech-

nique to receive more appropriate and definite answers is

the use of silence or a ‘communication check’ (Reynolds

and Gutman 2001). In our cases, we alternately used si-

lence and a ‘mmm’ when a more elaborate answer was

expected. Both solutions proved to be efficient to give

children more time to think about their answers and to

reformulate or specify what they had said before. As for the

Cogn Tech Work

123

‘communication check’, it refers to a paraphrase of the

respondent’s answer (Reynolds and Gutman 2001). The

communication check helps to organize the respondent’s

thoughts and feelings and results in increased verbal

interchange (Walker et al. 1992). In an example of our

cases, the interviewer replied ‘So the obstacles aren’t nice

...’ when the child previously said ‘I don’t like it when

there are too many obstacles in the game world’. In gen-

eral, children did not realise this communication check.

Most answers after the ‘communication check’ were in-

deed more refined. Further, we explicitly allowed ‘don’t

know’-answers in order to avoid children mentioning

information to please the adult (Christensen et al. 2000;

Buck et al. 1999). In this context, the interviewer also

stressed that every answer is a correct answer and that there

are no wrong answers (Reynolds and Gutman 2001).

Although the aim is to reveal why products are liked or

disliked, one might better not literally ask why-questions.

In our cases, we received more and better replies from

children when a ‘why’-question was specified into a what-

question. More particularly, we rephrased questions such as

‘Why did you (dis)like the game?’ into ‘What did you

(dis)like about the game?’. Why-questions are complex

because of two reasons (Berry 2002). A why-question is

abstract so that it results in an infinite number of possible

answers. Whereas a great number of response options im-

proves the response quality of interviewed adults, the

reliability of children’s responses decreases when too many

response options are offered (Borgers et al. 2004). Sec-

ondly, ‘why’ as such has two meanings. One meaning

implies a purpose (‘what was the reason for ...?’), the other

causality (‘what caused ...?’) (Berry 2002). On the con-

trary, what-questions are more specific and concrete, which

makes it easier to be answered (Berry 2002).

It is important that the laddering interview does not

replace the observation session and vice versa. Both ses-

sions are complementary. In our cases, the laddering

interview gave a good opportunity to probe the child to

give more elaborate answers than during the observation

session. However, since the child could think more con-

sciously about an answer, the chance of a social desirable

answer to please the adult increased. The contextual clues

gained during the observation made it easier to correctly

analyze the answers. We also found that the observation

session helped in judging the child’s cognitive capacity so

that the interviewer could adapt his spoken language (e.g.,

appropriate vocabulary, sentence structure, points of

interests). Finally, interviewers were able to ask concrete

questions by giving situational examples. This would not

have been possible if they could not refer to contextual

clues. We thus stress the importance of the interpretation of

contextual clues, which is one of the conditions of the

contextual laddering method.

8 Further work

The three cases discussed in this paper were set up to test

the laddering method with children and find ways to im-

prove the methodology. The set-up was rather exploratory

than experimental. However, further work should conduct

controlled and comparable experiments to test the ladder-

ing method in more detail. For instance, it would be

interesting to investigate the effect of age on the feasibility

of the laddering method. Another interesting topic is a

comparison of laddering data when the interview occurred

in a lab versus in the child’s natural environment.

In our cases, we found that children almost never

mentioned their personal values. This probably can be

explained by the fact that children younger than eleven

have difficulties to think abstractly (Piaget 1970). Conse-

quently, a researcher has to decide for him/herself which

values correspond to the (dis)liked attributes and conse-

quences. Further research should think about a methodo-

logical solution so that values can be determined in a

strictly repeatable way.

The laddering method was found appropriate to inves-

tigate the likeability of (creative computer) games. The

question then arises whether this technique is also suitable

for other applications than (creative computer) games. An

experimental set-up of a likeability evaluation session of

other applications would help to clarify this issue.

CHALLENGE / ACTION CONTROL …………………………………………………………………………………………………………

glorious feeling difficulties win points the option walking

to be somewhere around& very quickly taking

obstaclesjumping around………………………………………………………………………………………………………. Snowboard Race Tunnel with Too many obstacles most marbles

Fig. 9 HVM for ‘example 1’

and ‘example 2’

Cogn Tech Work

123

Acknowledgments The current research was part of a project

regarding creative communities for children in cooperation with

Larian (Flemish computer game development company) and funded

by VRT (public broadcaster of the Flemish community in Belgium). I

am grateful to all the children who participated. Further, I would like

to thank Licia Calvi, Janet Read and David Geerts for the constructive

feedback on my paper.

References

Als BS, Jensen JJ, Skov MB (2005a) Comparison of think-aloud and

constructive interaction in usability testing with children. In:

Proceedings of the fourth international conference on interaction

design and children IDC2005. ACM Press, New York

Als BS, Jensen JJ, Skov MB (2005b) Exploring verbalization and

collaboration of constructive interaction with children. In:

Proceedings of Interact 2005. Springer, Germany

Baauw E, Bekker M, Barendregt W (2005) A structured expert

evaluation method for the evaluation of children’s computer

games. In: Proceedings of Interact 2005. Springer, Germany

Barendregt W, Bekker M, Speerstra M (2003) Empirical evaluation of

usability and fun in computer games for children. In: Proceed-

ings of Interact 2003. IOS Press, Switzerland

Berry M (2002) Why? url: http://www.bbc.co.uk/dna/h2g2/A747047

(visited 14/02/2006)

Borgers NJ, Hox J, Sikkel D (2004) Response effects in surveys on

children and adolescents: the effect of number of response

options, negative wording, and neutral mid-point. Qual Quant

38(1):17–33

Bruck M, Ceci SJ (1999) The suggestibility of children’s memory.

Annu Rev Pscychol 50:419–439

Bruckman A, Bandlow A (2003) Human–computer interaction for

kids. In: Jacko J et al (eds) Handbook of human computer

interaction. Lawrence Erlbaum Associates Inc., New Jersey, pp

428–440

Carroll JM, Thomas JC (1988) Fun. Sigchi Bull 19(3):21–24

Christensen P, Allison J (2000) Research with children. Perspectives

and practices. Routledge, Taylor & Francis Group, UK

Druin A, Bederson B, Boltman A, Miura A, Knotts-Callaghan D, Platt

M (1999) Children as our technology design partners. In: Druin

A (ed) The design of children’s technology. Morgan Kaufmann,

San Francisco, pp 51–72

Elliot J, Adelman C (1973) Reflecting where the action is: the design

of the Ford teaching project. Educ Teach 92:8–20

Federoff MA (2002) Heuristics and usability guidelines for the

creation and evaluation of fun in video games. Faculty of the

University Graduate School, Master of Science in the Depart-

ment of Telecommunications of Indiana University, Indiana

Fortini-Campbell L (1990) The customer insight book. The Copy

Workshop, Chicago

Grunert GK, Bech-Larsen T (2005) Explaining choice option

attractiveness by beliefs elicited by the laddering method. J

Econ Psychol 26(2):223–241

Gutman J (1982) A means-end chain model based on consumer

categorization processes. J Market 46:60–72

Hanna L, Risden K, Alexander K (1997) Guidelines for usability

testing with children. Interactions 4(5):9–14

Hanna E, Risden K, Czerwinski M, Alexander KJ (1999) The role of

usability research in designing children’s computer products. In:

Druin A (ed) The design of children’s technology. Morgan

Kaufmann, San Francisco, pp 3–26

Hoffman DL, Novak TP (1996) Marketing in hypermedia computer-

mediated environments: conceptual foundations. J Market

60:50–68

ISO (1998) Ergonomic requirements for office work with visual

display terminals (VDTs) Part 11. Guidance on usability (Rep.

No. ISO No. 9241–11). International Organization for Standard-

ization, Switzerland

Jordan PW (2000) Designing pleasurable products. An introduction to

the new human factors. Taylor & Francis, London

Kjeldal S (2002) Back to basics: the sequencing of inductive and

deductive. Research methodologies in fresh fruit and vegetable

research. Forum Qual Soc Res 3(3). Online available at: http://

www.qualitative-research.net/fqs-texte/3-02/3-02kjeldal-e.pdf

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage, Beverly Hills

Malone TW (1980) What makes things fun to learn? A study of

intrinsically motivating computer games. Technical Report CIS-

7, Xerox parc, Palo Alto

Markopoulos P, Bekker M (2003) On the assessment of usability

testing methods for children. Interact Comput 15(2):227–243

McNiff J, Whithead J (2002) Action research: principles and practice.

Routledge, UK

Nielsen J (2003) Usability 101: introduction to usability. In: Alertbox

2003, http://www.useit.com/alertbox/20030825.html (visited 08/

02/2006)

Olson JC, Reynolds TJ (2001) The means-end approach to under-

standing consumer decision making. In: Reynolds TJ et al (eds)

Understanding consumer decision making. The means-end

approach to marketing and advertising strategy. Lawrence

Erlbaum associates, New Jersey, pp 3–24

Pagulayan JR, Keeker K, Wixon D, Romero RL (2003) User-centered

design in games. In: Jacko J et al (eds) Handbook of human

computer interaction. Lawrence Erlbaum Associates Inc., Lon-

don, pp 883–906

Pellegrini AD (1996) Observing children in their natural worlds: a

methodological primer. Lawrence Erlbaum associates, New

Jersey

Peter JP, Olson JC (1994) Understanding consumer behavior. Irwin,

Burr Ridge

Piaget J (1970) Science of education and the psychology of the child.

Orion Press, New York

Read J, MacFarlane S, Casey C (2002) Endurability, engagement and

expectations: measuring children’s fun. In: Proceedings of Inter-

action Design and Children (IDC’02), Eindhoven, pp 189–198

Reynolds TJ, Gutman J (2001) Laddering theory, method, analysis,

and interpretation. In: Reynolds TJ et al (eds) Understanding

consumer decision making. The means-end approach to market-

ing and advertising strategy. Lawrence Erlbaum associates, New

Jersey, pp 25–62

Rubin J (1994) Handbook of usability testing. Wiley, Canada

Shackel B (1991) Usability-context, framework, definition, design

and evaluation. In: Shackel B et al (eds) Human factors for

informatics usability. Cambridge University Press, Cambridge,

pp 21–37

Subramony DP (2002) Why users choose particular web sites over

others: introducing a ‘‘means-end’’ approach to human–com-

puter interaction. J Electron Commer Res 3(3):144–161

Trevino LK, Webster JF (1992) Flow in computer-mediated com-

munication. Electronic mail and voice mail evaluation and

impacts. Comm Res 19(5):539–573

Walker CE, Roberts MC (1992) Handbook of clinical child psychol-

ogy. Wiley, New York

Wright PC, Monk AF (1990) The use of think-aloud evaluation

methods in design. SIGCHI Bull 23(1):55–57

Cogn Tech Work

123