insights into knowledge representation: the influence of amodal and perceptual variables on event...

Insights Into Knowledge Representation: The Influence ofAmodal and Perceptual Variables on Event Knowledge

Retrieval From Memory

Susanne Raisig, Tinka Welke, Herbert Hagendorf, Elke van der Meer

Department of Psychology, Humboldt University, Berlin

Received 29 September 2008; received in revised form 17 February 2009; accepted 23 February 2009

Abstract

Event sequences or scripts are the conceptual representations of activities in memory. Traditional

views hold that events are represented in amodal networks and are retrieved by associative strategies.

The embodied cognition approach holds that knowledge is grounded in perception and retrieved by

mental simulation. We used a script generation task where event sequences of activities had to be

produced. Activities varied in their degree of familiarity. In a regressional design we investigated

whether amodal or perceptual variables best predicted knowledge retrieval to gain insight into the

underlying representation. Retrieval depended on the familiarity of the activity. While novel activi-

ties mainly relied on perception-based simulation and to a lesser extent on associative strategies,

moderately familiar activities showed the opposite pattern, and events of familiar activities were

retrieved by association alone. We conclude that amodal structures exist in all representations that

become stronger with increasing frequency and finally prevail over perceptual structures.

Keywords: Scripts; Knowledge representation; Retrieval; Frequency effects

1. Introduction

Everyday activities like shopping for groceries, doing the laundry, or going to the restau-

rant consist of a goal-oriented sequence of component events. The conceptual knowledge

structures of these event sequences in memory have been called scripts (Schank & Abelson,

1977) or schemata (Rumelhart, 1977), and they encode not only which agents, objects, and

actions one can expect to encounter in a given situation but also the order in which events

Correspondence should be sent to Susanne Raisig, Department of Psychology, Humboldt University Berlin,

Unter den Linden 6, 10099 Berlin, Germany. E-mail: [email protected]

Cognitive Science 33 (2009) 1252–1266Copyright � 2009 Cognitive Science Society, Inc. All rights reserved.ISSN: 0364-0213 print / 1551-6709 onlineDOI: 10.1111/j.1551-6709.2009.01044.x

unfold over time. Thereby, these knowledge structures enable us to ‘‘foresee’’ and plan our

own as well as others’ actions and anticipate their consequences, something that has been

termed ‘‘situated anticipation’’ (Barsalou, Breazeal, & Smith, 2007). Scripts further form

the building blocks for mental models (Johnson-Laird, 1983) or situation models (van Dijk

& Kintsch, 1983) that are constructed during the reading and comprehension of text. Since

the early work by Schank and Abelson many studies have explored how event sequences

are represented in memory.

Originally, Schank and Abelson (1977) proposed that event sequences are represented in

an amodal script structure whereby knowledge is transduced into propositions. These are

organized in a linear fashion along a time dimension and hold the temporal relations

between events. Event propositions are connected via learned associations that are meaning-

ful rather than arbitrary since events mostly occur in a typical temporal order (i.e., to order

food in a restaurant one must have read the menu). Knowledge retrieval from the network is

achieved by an associative strategy whereby either one event primes the following events in

the sequence or the activity primes its event sequence, respectively. The linear structure is a

key characteristic of scripts, and studies have shown that the processing of chronological

temporal relations benefits over inverse temporal relations (e.g., Barsalou & Sewell, 1985;

Nuthmann & van der Meer, 2005; Raisig, Welke, Hagendorf, & van der Meer, 2007).

According to Grafman (1995), the strong linear structure evolved from the repeated (active

or passive) experience with the activity. Consequently, scripts can be considered frequency-

based knowledge structures. Indeed, in a set of normative studies Rosen, Caplan, Sheesley,

Rodriguez, and Grafman (2003) found that the number of events that were generated for an

activity depended on the frequency with which an activity was performed. Frequency of per-

formance can also be described as the activity’s familiarity. That is, the more familiar an

activity was, the more events were retrieved from memory in a script generation task. Sirigu

et al. (1995) also found that the retrieval of scripted knowledge was frequency based.

Across different participant groups (normal control participants and brain-damaged

patients), retrieval speed of event knowledge was shortest for highly familiar event

sequences and longest for unfamiliar or novel sequences. Moreover, especially patients with

lesions to frontal areas made more errors upon the retrieval of unfamiliar than familiar event

sequences. The results suggested that the strength of the knowledge representation was

graded and that highly familiar events and often used information were not affected by

structural damage to the same extent as information that was only retrieved rarely. Due to

these results it was suggested that knowledge representation was frequency based. Neurosci-

entific evidence came from an fMRI study by Krueger, Moll, Zahn, Heinecke, and Grafman

(2007), who found that the medial prefrontal cortex (MPFC) mediates event sequence

knowledge and different subregions within the MPFC are activated depending on the famili-

arity of an event sequence. While unfamiliar sequences activated anterior medial areas,

familiar event sequences were processed in posterior areas. The authors suggested that

the anterior MPFC encodes the representation of complex information. As an event

sequence becomes increasingly familiar and is used more frequently, the representation

becomes simpler, coding ‘‘sparser cognitive information’’ (p. 2352) in posterior regions of

the MPFC.

S. Raisig et al. ⁄ Cognitive Science 33 (2009) 1253

From the perspective of the original script theory (Schank & Abelson, 1977), scripts do

not exclusively consist of linearly structured propositions. Propositions can also be struc-

tured hierarchically into scenes in the semantic network. Retrieval from the hierarchy is

achieved by an associative strategy, too. The most central (i.e., important) events are repre-

sented on the scene level, which is the highest level of the hierarchy. Retrieval of these

events is facilitated when the activity name is given as a prime because the associative

strength between the activity and central events is greatest (see Galambos & Rips, 1982).

Less central events that are represented on lower hierarchical levels are not as strongly asso-

ciated with the activity, and retrieval of these events is slower. In a hierarchical, amodal rep-

resentation, the distinctiveness of events can also affect retrieval. Distinct events that have a

unique associative connection with a specific activity are retrieved more readily than events

low in distinctiveness that have connections with many different activities (‘‘fan effect’’;

Anderson, 1974). Here, knowledge retrieval also relies on an associative strategy.

A different approach regarding the representation of event sequences was described by

embodied theories of cognition. According to these approaches, knowledge is grounded in

perception and action. For example, Barsalou (1999) suggested that aspects of the percep-

tual and motor experience are stored as perceptual symbols in modality-specific areas of the

brain and are retrieved by a mental simulation (or mental reenactment) of the experience.

Retrieval, therefore, is achieved by reactivating the visual, auditory, tactile, and motor infor-

mation of the initial experience. While some tasks like autobiographical tasks (Conway,

2002) or complex reasoning tasks (Hegarty, 2004) require ‘‘explicit’’ simulations, others

depend on ‘‘implicit’’ simulation processes. For example, solving a property verification

task (e.g., is mane a property of a pony?) would involve constructing the sensory-motor sim-

ulation of a property (e.g., mane) and attempting to find it in the previously constructed sen-

sory-motor simulation of the object (e.g., pony). In this case, the simulation is mainly visual

in the sense that the property is mentally visualized and then looked for in the mental image

of the object. Consequently, the imageability of the property and the object strongly deter-

mines the simulation. Further, the complexity (e.g., the size of the object or how many

details it has which makes the property more difficult to find) influences the simulation per-

formance. This is precisely what Solomon and Barsalou (2004) found: Knowledge retrieval

was indeed mainly affected by perceptual variables but only in cases where distracters con-

sisted of highly associated object-property pairs (e.g., owl—tree). In these cases the verifica-

tion task could not be solved by the degree of associativeness between object and property,

but rather a different decision strategy, namely mental simulation, had to be applied that

allowed ‘‘deeper’’ processing. When object-property pairs were not associated, a more

superficial associative strategy could be applied. Despite a flood of evidence that has been

interpreted in favor of the embodied cognition approach (e.g., Glenberg & Kaschak, 2002;

Zwaan, Madden, Yaxley, & Aveyard, 2004), many questions remain in the representation

debate (see Mahon & Caramazza, 2008; Markman & Dietrich, 2000).

In this study, we wanted to address some of the open questions. Since the retrieval strat-

egy serves as a window on the underlying representation (c.f., Barsalou & Sewell, 1985),

we specifically focused on the question of how amodal and perceptual variables contributed

to the retrieval of event knowledge from memory and how they interacted with the

1254 S. Raisig et al. ⁄ Cognitive Science 33 (2009)

familiarity of an activity (i.e., the frequency with which an activity is performed). Therefore,

we applied a script generation task where participants had to retrieve all events that

belonged to an activity. Retrieval performance was operationalized by the mean number of

events that were generated for an activity. By splitting activities into one of three frequency

classes (low, moderate, and high) we wanted to replicate the frequency effect that more

events would be retrieved for highly and moderately familiar event sequences than for unfa-

miliar or novel event sequences.

Subsequently, we collected ratings of amodal and perceptual variables for each event of

the sequence and performed a regression analysis (for a more detailed description of the

variables, please refer to the methods section). The collected ratings were regressed onto

retrieval performance (outcome variable) under consideration of the familiarity of the event

sequence in order to draw inferences about the nature of the representation of event knowl-

edge in memory. On the one hand, if the representation is in a script-like structure, retrieval

should rely on associative strategies and amodal variables should be the best predictors of

the outcome variable regardless of the familiarity of the activity. On the other hand, a

greater influence of the perceptual predictor variables is expected if the underlying represen-

tation is tied to perceptual pathways and knowledge is retrieved by a sensory-motor simula-

tion of the experience. Moreover, if we assume that the representation of unfamiliar and

moderately familiar activities is more complex (see Krueger et al., 2007), which therefore

demands ‘‘deeper’’ processing, we expect that a simulation strategy is applied and accord-

ingly perceptual variables best predict retrieval. On the other hand, amodal variables should

determine retrieval of events from highly familiar activities since this knowledge is repre-

sented in a simpler code and superficial processing suffices.

2. Method

2.1. Participants

Ninety German native speakers from Humboldt University Berlin participated in the

study for course credits (72 female). Mean age was 23.4 years.

2.2. Materials

In preparation of the script generation task, we collected 60 activities that generally pro-

ceed in a typical sequence. The activities were then randomly assigned to one of six book-

lets, each containing 10 activities. Hence, 15 participants generated a script for each

activity. The first page of the booklet contained the instructions as well as an example. The

example consisted of the name of an activity that was always written in bold letters (here

riding in a taxi) as well as a sample event sequence. At the top of each of the following

pages of the booklet only the names of the activities were written with lines underneath on

which to write the events. On the last page of the booklet participants were asked to rate

how often they engaged in each of the 10 activities on a 5-point scale (0 = less then once a


year or never, 1 = once or twice a year, 2 = once or twice a month, 3 = once or twice a

week, or 4 = every day).

2.3. Procedure

The script generation task was carried out as a group experiment. The experimenters

handed out the booklets in random order but made sure that neighboring participants did not

receive the same booklet. Participants read instructions describing the experiment that asked

them to imagine that they wanted to carry out the activity and write down all events from

the beginning of the activity until its completion. If they forgot an event they could insert it

into the appropriate time slot. Participants were not given any time constraints. It took them

about 45 min to complete the task.

2.4. Scoring

All events that had been generated for an activity were entered into a spreadsheet

according to their sequential position within the script. While doing so, the frequency totals

of the events (i.e., how often each event was named in total) were noted. It occurred that

events with the same meaning were expressed differently across individuals. For example,

one participant would write, ‘‘switch on the oven,’’ whereas another participant would write,

‘‘preheat oven’’ in order to bake a cake. A preheat oven event was created to represent these

analogous events.

2.4.1. Classification into frequency classes, script selection, and event selectionWe calculated the mean performance frequencies for each activity. Sequences with a

mean below 1 were defined as being of low frequency because they were performed less

than once a year or had never been performed by an individual. Moderate-frequency

activities had a mean engagement frequency between 1 and 2 because they were performed

at least once a year or even once a month. High-frequency activities were activities that

were engaged in every week or daily and had a mean frequency score above 2. Mean

frequency ratings were used to classify activities into frequency classes (low, moderate,

or high).

Out of the 60 event sequences we chose 30 that fit Schank and Abelson’s (1977)

definition of a ‘‘strong’’ script according to the independent opinions of the experimenters.

People may agree upon which events typically occur when preparing a salad but not on

their temporal order (this could then be described as a ‘‘weak’’ script). In other cases very

diverse scripts were produced for the activities. Since many people have different

preferences concerning their breakfast, very different events for the activity making

breakfast were produced. These diverse event sequences were also excluded. In the end we

had 30 scripts, 10 of low, 10 of moderate, and 10 of high frequency (see Table 1).

Finally, all events of a sequence were sorted by descending number of frequency totals,

which were then transformed into percentage values. Only events that had been named by at

least 30% of participants were included in the final event sequence. In the end each event


sequence had a different number of events, some being longer than others (see Table 1,

column 3). Table 2 gives three examples of generated event sequences, one for each fre-

quency class. The other event sequences can be requested from the authors or can be

obtained from http://www.cogsci.rpi.edu/CSJarchive/Supplemental/index.html

2.5. Rating of predictor variables

To perform a regression analysis, we obtained ratings for different variables for each

event within the sequence. Similar to Solomon and Barsalou’s (2004) procedure we had 50

Table 1

Activities, mean frequency with which an activity is performed, and mean number of events generated for an

activity

Event Sequence Mean Frequency Mean Number of Events Generated

Low-frequency class

Going parachuting 0.00 13

Going on an airplane 0.94 18

Going bowling 0.89 15

Changing batteries in an alarm clock 0.87 11

Changing a flat tire on a bicycle 0.33 13

Playing tennis 0.31 13

Order a pizza 0.86 15

Going to a theater audition 0.12 13

Being in a car accident 0.06 14

Going scuba diving 0.60 14

Moderate-frequency class

Getting a haircut 1.11 17

Making a bonfire 1.62 13

Going to the dentist 1.00 17

Having a barbeque 1.44 15

Writing a letter 1.47 14

Going on a train 1.78 17

Baking a cake 1.42 17

Fueling the car 1.44 19

Shopping for clothes 1.79 18

Going to a restaurant 1.89 18

High-frequency class

Riding on a bus 2.68 14

Making coffee 3.07 14

Getting ready for work 3.68 17

Going to the supermarket 3.13 17

Cooking pasta 2.56 17

Washing one’s hair 3.44 17

Borrowing a book from the library 2.50 14

Doing the dishes 3.25 14

Doing the laundry 2.79 17

Taking the subway 3.57 15


new participants rate the selected events. A short overview of each variable will be provided

along with a description of how ratings were obtained.

2.5.1. CentralityCentrality assesses the relative importance of the event and indicates the associative

strength between an activity and a component event. Galambos and Rips (1982) found

that highly central events were retrieved faster in a membership-judgment task than

events low in centrality. For example, since order food was more central (i.e., impor-

tant) during a restaurant visit than prebook a table, the former received greater asso-

ciative priming and was retrieved faster. Accordingly, order food should receive a

higher centrality rating in our rating task than prebook a table. Centrality ratings are

made on grounds of comparisons of the relative importance of each event of the

activity. We assume that if the underlying representation was an amodal script-like

structure, retrieval is achieved by an associative strategy making centrality a signifi-

cant predictor of the outcome variable (i.e., how often a certain event was generated).

In other words, centrality should make a significant contribution to retrieval perfor-

mance in our task.

To rate centrality, participants rated each event of a sequence on a 5-point scale indicat-

ing how central it was for the activity. The average rating across participants was calculated

for each event. A high rating represented high centrality.

Table 2

Example scripts for the low-, moderate-, and high-frequency classes

Low Frequency Moderate Frequency High Frequency

Changing batteries in an alarm clock Fueling the car Going to the supermarket

Get new batteries

Take batteries out of packet

Get alarm clock

Open battery case

Take out empty batteries

Check position of the battery poles

Insert new batteries

Close battery case

Set right time on alarm clock

Dispose of old batteries

Put alarm clock on bedside table

Drive to gas station

Stop next to gas pump

Stop the engine

Get out of the car

Open fuel tank cap

Take nozzle off pump

Put nozzle into tank

Pull lever on the nozzle

Fill up the tank

Shake off nozzle

Put nozzle back on pump

Close fuel tank cap

Go into gas station

Pay for gas

Return to car

Get into car

Drive off

Consider what is needed

Write shopping list

Get money ⁄ wallet

Drive to supermarket

Get trolley ⁄ shopping cart

Go down aisles

Get everything that is on the list

Compare prices

Put groceries in trolley ⁄ cart

Queue up at cash desk

Put groceries on belt

Hand in receipt of returned empties

Pack groceries that have been registered

Pay for groceries

Put groceries in car

Take back trolley ⁄ shopping cart

Drive home

Unpack bags


2.5.2. DistinctivenessDistinctiveness assesses the polysemy of events and indicates the associative strength

between a component event and an activity. It defines how uniquely an event is associated

with an activity. For example, the event buying a ticket is part of many activities like going

on a train, riding on a bus, or going to the cinema and therefore is low in distinctiveness. On

the other hand, the event wait for boarding call is a highly distinct event of the activity going

on a plane. Ratings of distinctiveness are derived by judging in how many different activi-

ties a certain event occurs. Assuming an amodal associative network, we hypothesize that

distinct events should be retrieved better as they are uniquely associated with an event

sequence (Anderson, 1974). Distinctiveness should therefore make a significant contribution

to retrieval if the underlying representation was amodal and propositional.

The relationship between centrality and distinctiveness is unclear. While Galambos

(1983) found the variables to be relatively independent, Rosen et al. (2003) found that they

were closely related. If distinctiveness and centrality are indeed closely related as suggested

by Rosen et al. then both variables should contribute to knowledge retrieval to the same

degree.

To rate distinctiveness, 50 new participants rated the distinctiveness of each event of a

sequence on a 5-point scale. The average rating across participants constituted the value for

each event. A high mean indicated that the event was very distinct.

2.5.3. ImageabilityAccording to embodied theories of knowledge representation, the representation contains

sensory-motor information from the initial experiential input. Since vision is our dominant

sensory experience, visual information (agents, objects, colors, and other details) should be

preserved in the representation of activities (cf., Barsalou, 1999). Imageability ratings reflect

the ease with which a stimulus elicits a mental image. It has been shown that imageability

affects retrieval speed (Bird, Franklin, & Howard, 2001), and we therefore assume that

the ease with which an event can be visualized will affect retrieval performance. In other

words, when participants can easily visualize an event, it will be retrieved more often from

memory.

To rate imageability, participants rated the imageability of each event of a sequence on a

5-point scale. The average rating across participants constituted the value for each event,

whereby a high mean indicated better mental imageability.

2.5.4. ComplexityThere exists evidence in the literature on imagery that mental images are more difficult to

construct when the stimulus was complex and more parts had to be inserted into the image.

For example, more time was required to imagine objects of increasing complexity (e.g.,

Kosslyn, Reiser, Farah, & Fliegel, 1983). Similarly, more time was required when a scene

that had to be visualized contained a greater number of objects (e.g., Beech & Allport,

1978). We defined the complexity of an event as the number of subordinate actions consti-

tuting it, that is, the more actions an event consists of, the more complex it is (e.g., the event

administer first aid in the script being in a car accident was considered very complex).


Therefore, upon constructing a mental image of an event, participants derive the complexity

of an event on grounds of whether the mental image contains many additional actions. In

the property verification task by Solomon and Barsalou (2004) the equivalent perceptual

variable turned out to be a significant predictor of task performance. In line with previous

findings, we expect that the complexity of an event will decrease retrieval performance.

To rate complexity, 50 new participants rated the complexity of each event of a sequence

on a 5-point scale. The average rating across participants constituted the value for each

event. A high rating indicated that the event was very complex.

2.6. Data analysis

For analysis of a frequency effect, we submitted the mean number of events that had been

generated across participants for each activity to an analysis of variance (anova) where the

factor frequency had three levels (low, moderate, high). The significance level was set at

p < .05. To identify which conditions differed significantly we calculated planned contrasts.

Effect sizes are indicated as r values for each contrast.

Subsequently, we ran a stepwise multiple regression analysis in which only the best pre-

dictors for the outcome variable were included into the regression model. First, the best pre-

dictor variable was determined by selecting the variable that had the highest simple

correlation with the outcome variable. If this predictor significantly improved the model it

was retained in the model and, by algorithm, the second best predictor was determined that

had the largest semipartial correlation with the outcome. This procedure continued until all

redundant predictors that did not contribute to the outcome were removed. We ran the anal-

yses separately for each frequency class to check whether different predictors contributed to

the outcome depending on the familiarity of the activity. Here, too, the significance level

was set at p < .05.

3. Results

3.1. Frequency effects

A main effect of frequency was found, F(2, 27) = 6.21, p < .05. Planned contrasts

between the low-frequency class and the moderate- and high-frequency class together

showed that significantly fewer events were generated for scripts in the low-frequency class

(M = 14) than for scripts in the moderate- (M = 17) and high-frequency classes (M = 16),

t(27) = 3.37, p < .05, r = .54. Moderate- and high-frequency scripts did not differ in the

mean number of retrieved events, t(27) = )1.04, p = .37, r > .05 (see also Table 1).

3.2. Regression analysis

Table 3 presents the estimates of beta, standard errors for beta, and standardized beta

values, b. When running the regression analysis including only ratings of events from


high-frequency activities, only centrality made a significant improvement to the ability of

the model to predict the outcome. All other predictors were removed. Centrality accounted

for 15% of variance within the outcome variable (R2 = .147, p < .001).

In the analysis of moderate-frequency activities, the best predictor was centrality,

accounting for 19% of the variance in the outcome variable (R2 = .19, p < .001). However,

imageability had the largest semipartial correlation with the outcome and significantly

improved the prediction strength of the model. Including imageability increased the total

amount of explained variance by 4% so that now 23% of the variance was explained

(R2 = .229, p < .001). The other two variables were redundant and were therefore not

included into the model.

In the low-frequency regression analysis the same predictors contributed to the outcome

variable, however, in a different order. The best predictor of the outcome was imageability,

which explained 11% of the variance in the outcome (R2 = .107, p < .001). However,

a significant influence of centrality was also found: including this variable in the model

increased the explained variance by 9%. In total, 9% of variance was explained by both

predictors. The other two variables were redundant and were therefore not included in the

model.

4. Discussion

One of the most interesting research questions remains how knowledge is represented in

memory. We were particularly interested in a certain class of knowledge, namely event

sequences that are known by the name of scripts. These knowledge structures are vital in the

understanding of text (c.f., mental or situation models; Johnson-Laird, 1983; van Dijk &

Kintsch, 1983) as well as in the planning and the anticipation of events. In the introduction

we have outlined two representation formats, namely the amodal view that knowledge is

stored in semantic networks where propositions are structured hierarchically and connected

associatively; and the embodied cognition view that holds that knowledge is grounded in

Table 3

Coefficients of the regression analysis

Frequency B SE (B) b**

High

Centrality 1.76 0.34 .38

Moderate


Imageability 2.4 0.84 .22

Low

Imageability 1.86 0.42 .34


Note: **p < .01.


perception whereby perceptual and motor information is stored in modality-specific areas of

the brain. The aim of our study was to investigate how amodal and perceptual variables con-

tribute to the retrieval of event knowledge of different familiarity in order to gain insights

into the underlying representation.

In a script generation task, we obtained event sequences of everyday activities that were

free of individual differences in the form of idiosyncratic events. We could replicate a fre-

quency effect that more events were retrieved for familiar than unfamiliar activities. This is

in perfect accord with Grafman (1995) who assumes a frequency-based representation, a

suggestion that has found neuropsychological and neuroscientific support (Krueger et al.,

2007; Sirigu et al., 1995). However, in our opinion the question remains why such frequency

effects occur. What constitutes frequency-based representations?

In a second step, we regressed retrieval performance onto ratings of amodal and percep-

tual variables, respectively. The centrality and the distinctiveness of the component events

served as amodal predictor variables since they indicate the associative strength between an

activity and its component events. The imageability and complexity served as perceptual

variables. The results revealed that the retrieval of events from low-frequency activities was

best predicted by the imageability of events and to a lesser extent by the centrality of events.

This suggests that the representation of activities from the low-frequency class is embodied

and perception based because retrieval relied heavily on a mental simulation strategy. That

is, events that could easily be visualized or rather mentally ‘‘acted out’’ were generated

more often. Centrality contributed as a second but not as important predictor to retrieval per-

formance in the low-frequency class, suggesting that some amodal structures exist that also

guide retrieval. These do not seem to be very strong yet (i.e., do not consist of strong associ-

ations) so that a perception-based simulation is necessary for appropriate retrieval.

The retrieval of events from moderate-frequency activities was best predicted by central-

ity and to a lesser extent by imageability, which was only the second best predictor here.

The retrieval strategy of ‘‘mentally acting out’’ an activity aided retrieval but the associative

strategy was superior, which implies that the underlying representation entails partly percep-

tual but dominantly amodal structures. A mixture of simulation and associative strategies

in both low- and moderate-frequency classes is in line with the finding by Solomon and

Barsalou (2004) who found that associative as well as perceptual variables accounted for

performance in a property verification task.

When retrieving events from highly familiar activities like cooking pasta or riding on a

bus, there was no mixture of strategies. In fact, simulation did not aid the retrieval at all and

instead retrieval was guided by the centrality of events alone. In other words, an associative

strategy prevailed where events on higher levels of the hierarchical representation were

accessed more readily. An associative strategy implies that the relations between an activity

and its component events are associations that elicit priming (i.e., the activity elicits priming

of an event) and since the associations vary in strength as a function of centrality, more cen-

tral events receive stronger priming (c.f., Rumelhart, 1977). This suggests that activities in

the high-frequency class are represented in amodal, hierarchical networks.

Taken together these results point to interesting questions: Do representations change as

a function of the familiarity of activities and, if they do, how do they change? Is it likely that


two representation formats coexist? Or is it the case that one representation is replaced by

the other when an activity becomes more and more familiar with practice and experience?

We agree with Markman and Dietrich (2000) or more recently Mahon and Caramazza

(2008) that the idea of an amodal representation should be modified to include aspects of

the embodied cognition approach, thus bringing together the best of both worlds. The fact

that centrality contributes to retrieval performance in all frequency classes does imply that

amodal structures exist and remain while perceptual information becomes less important

with increasing familiarity. We suggest that the initially perception-based representation

develops into an amodal representation where amodal structures (i.e., associations between

amodal symbols) have evolved. These structures become stronger and stronger with famil-

iarity (i.e., the frequency with which information is activated) so that this knowledge can be

accessed without a simulation. In the language comprehension literature Louwerse (2007)

has proposed the symbol interdependency hypothesis where amodal symbols can be

grounded in experience of the physical world but do not necessarily have to be. Grounding

only takes place when the situation affords it. Otherwise, readers can take a ‘‘short cut’’ and

can capture the meaning of amodal symbols through their relations with other symbols.

Symbol interdependency argues ‘‘that symbols are built onto embodied representations’’

(p. 15). Although Louwerse’s (2007) hypothesis applies to language comprehension, our

results are compatible with this approach. We showed that unfamiliar activities must be

grounded in the sensory, (at least) visual, experience. In familiar activities amodal symbols

have developed that can be retrieved via the short cut of association from ‘‘activity-symbol’’

to ‘‘event-symbol,’’ enabling faster and more effective retrieval. We believe that grounding

is necessary in knowledge retrieval when amodal symbols do not yet have sufficiently strong

associations with other symbols so that the short cut is not available.

In effect, many perceptual theorists acknowledge that there may be systems other

than the perceptual system that contribute to conceptual processing. For example, Barsalou,

Santos, Simmons, and Wilson (2008) have proposed a framework that integrates linguistic

and perceptual systems (LASS theory—language and situated simulation). The linguistic

system is activated quickly upon processing a word to produce other linguistic forms like

word associates. Barsalou et al. emphasize that the linguistic system does not consist of

amodal symbols (an assumption that corresponds to Barsalou’s perceptual symbol system

theory; Barsalou, 1999). Only the slower responding simulation system activates a simula-

tion that integrates words into a spatial and temporal context and represents meaning via

access to conceptual information. Crucially, LASS theory postulates that the two systems

both contribute to knowledge retrieval to a different degree depending on stimuli and task

conditions. We agree to a mixture of strategies on grounds of our results: event concepts can

be retrieved from a perception-based representation via (visual) simulation and from an

amodal representation by association between amodal symbols. In our opinion, amodal sym-

bols develop through the repeated experience with an activity and organize themselves in a

form that is relatively detached from the initial perceptual experience. This does not mean

that perceptual structures become completely irrelevant (see also Gabora, 1999), but they

are—according to our results—only necessary under the condition that knowledge is

unfamiliar and only accessed infrequently. Our assumption would have to be supported by


neuroscientific measures: Novel activities should activate the situational context and there-

fore more widespread brain areas than familiar activities.

Our results support the interpretation that the representation becomes simpler as a func-

tion of familiarity and frequency of use (Krueger et al., 2007). Beyond this interpretation,

we suggest that the ‘‘sparser’’ coding in effect marks a different representation, namely a

representation based on associative connections between amodal symbols. Nevertheless, we

believe that perceptual information remains (according to Krueger et al., 2007 in anterior

regions of the MPFC) and can be accessed when the situation demands it. Therefore, knowl-

edge from the same activity can be accessed either rapidly (see also Louwerse, 2007) or

slowly and with deeper processing. However, a rapid access is only possible when the activ-

ity is sufficiently familiar. Otherwise a more elaborate processing is necessary. Another

advantage of an altered representation (we do not want to call it a simpler representation) is

that in becoming more amodal in nature the representation becomes more stable and is less

affected by brain lesions. Neuropsychological findings suggest that frequently used informa-

tion (that is, information that can be accessed via the ‘‘short cut’’) is not affected by struc-

tural damage to the same extent as information that was retrieved infrequently (e.g., Sirigu

et al., 1995). Since the amodal structures are not yet as strong in infrequently used knowl-

edge, this information is likely to take damage easily after brain lesions.

Before concluding, let us point out why we did not find a significant contribution of dis-

tinctiveness or complexity on knowledge retrieval. According to Anderson (1974), the

strength of the association between an activity and an event should vary as a function of the

uniqueness, that is, the distinctiveness of an event (‘‘fan effect’’). However, distinctiveness

did not turn out to be an important predictor of the outcome variable. An explanation is that

in a script generation task, retrieval can be considered a ‘‘forward’’ association task where

items are retrieved from activity to event. Hence, the fan effect does not come into play in

our task since it rather applies to cases of ‘‘backward’’ associations where items would be

retrieved from event to activity. In such a backward case, events that were associated with

many different activities would indeed be more difficult to retrieve. More specifically, if

activities had to be associated for the indistinct event buying a ticket (which is part of many

different activities) retrieval would be more difficult than associating activities for a very

distinct event (like boarding the plane in the going on an airplane activity).1 The different

contributions of centrality and distinctiveness also give hints about the nature of the rela-

tionship between these two variables. Although a positive relationship exists, they are still

relatively independent, which was confirmed by a medium correlation between them

(r = .4).

Further, the complexity of an event had no effect on retrieval performance in any of the

frequency classes. If we assume a ‘‘mental acting out’’ strategy for low- and partially for

moderate-frequency activities, a complexity effect should certainly manifest itself since

complex stimuli have been shown to impair mental imagery (Kosslyn et al., 1983). An

explanation for the missing effect comes from studies of object recognition where a basic

level is accessed first before retrieving more detailed information at a more subordinate

level (cf., Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976). Such a basic level has also

been determined in the script literature. When asked to generate events of an activity,


Humphreys and Forde (1998) found that individuals typically do not name events on the

‘‘microstructure’’ (e.g., motor actions) but events from a higher, basic level. Accordingly,

participants did not construct a detailed mental image of objects in a visual imagery task

and no effects of complexity were found (Farah & Kosslyn, 1981). Hence, our participants

may have retrieved basic events by simulation but without forming highly elaborate visual-

izations of the activity. Although this explanation is somewhat speculative, it can explain

the missing complexity effect.

In conclusion, our findings have offered some interesting insights, but the representation

debate is far from being resolved. Many highly interesting questions remain unanswered,

for example, how does the amodal network evolve and could it possibly deteriorate? Which

other conditions determine access to the amodal or perception-based representation, respec-

tively? However, the findings we present here give some meaningful impulses for the direc-

tion of future research.

Note

1. We would especially like to thank Fergus Craik for pointing out this interpretation.

Acknowledgments

We would like to thank Fergus Craik and two anonymous reviewers for helpful com-

ments and interesting suggestions concerning alternative interpretations of our results. This

research was supported by the German Science Foundation (DFG) grant ME1362-10.

References

Anderson, J. R. (1974). Retrieval of propositional information from long-term memory. Cognitive Psychology,

6(4), 451–474.

Barsalou, L. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.

Barsalou, L. W., Breazeal, C., & Smith, L. B. (2007). Cognition as coordinated non-cognition. CognitiveProcessing, 8(2), 79–91.

Barsalou, L. W., Santos, A., Simmons, W. K., & Wilson, C. D. (2008). Language and simulation in conceptual

processing. In M. de Vega, A. M. Glenberg, & A. C. Graesser (Eds.), Symbols, embodiment, and meaning(pp. 245–284). Oxford, England: Oxford University Press.

Barsalou, L. W., & Sewell, D. R. (1985). Contrasting the representation of scripts and categories. Journal ofMemory and Language, 24(6), 646–665.

Beech, J. R., & Allport, D. A. (1978). Visualization of compound scenes. Perception, 7(2), 129–138.

Bird, H., Franklin, S., & Howard, D. (2001). Age of acquisition and imageability ratings for a large set of words,

including verbs and function words. Behavior Research Methods, Instruments, & Computers, 33(1), 73–79.

Conway, M. A. (2002). Sensory-perceptual episodic memory and its context: Autobiographical memory. In

A. D. Baddeley (Ed.), Episodic memory. New directions in research (pp. 53–70). Oxford, England: Oxford

University Press.


van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.

Farah, M. J., & Kosslyn, S. M. (1981). Structure and strategy in image generation. Cognitive Science, 5(4), 371–

383.

Gabora, L. (1999). Grounded in perception yet transformed into amodal symbols. Behavioral and Brain Sci-ences, 22, 617.

Galambos, J. (1983). Normative studies of six characteristics of our knowledge of common activities. BehaviorResearch Methods & Instrumentation, 15, 327–340.

Galambos, J., & Rips, L. J. (1982). Memory for routines. Journal of Verbal Learning and Verbal Behavior,

21(3), 260–281.

Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9,

558–565.

Grafman, J. (1995). Similarities and distinctions among current models of prefrontal cortical functions. Annalsof the New York Academy of Sciences, 769, 337–368.

Hegarty, M. (2004). Mechanical reasoning by mental simulation. Trends in Cognitive Sciences, 8(6), 280–285.

Humphreys, G. W., & Forde, E. M. E. (1998). Disordered action schema and action disorganization syndrome.

Cognitive Neuropsychology, 15, 771–811.

Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language inference and conscious-ness (1st ed.). Cambridge, England: Cambridge University Press.

Kosslyn, S. M., Reiser, B. J., Farah, M. J., & Fliegel, S. L. (1983). Generating visual images: Units and relations.

Journal of Experimental Psychology: General, 112(2), 278–303.

Krueger, F., Moll, J., Zahn, R., Heinecke, A., & Grafman, J. (2007). Event frequency modulates the processing

of daily life activities in human medial prefrontal cortex. Cerebral Cortex, 17(10), 2346–2353.

Louwerse, M. M. (2007). Symbolic or embodied representations: A case for symbol interdependency. In T. K.

Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 107–

120). Mahwah, NJ: Erlbaum.

Mahon, B. Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal

for grounding conceptual content: Links and interactions between language and motor systems in the brain.

Journal of Physiology-Paris, 102(1–3), 59–70.

Markman, A. B., & Dietrich, E. (2000). Extending the classical view of representation. Trends in CognitiveSciences, 4(12), 470–475.

Nuthmann, A., & van der Meer, E. (2005). Time’s arrow and pupillary response. Psychophysiology, 42(3), 306–

317.

Raisig, S., Welke, T., Hagendorf, H., & van der Meer, E. (2007). Investigating dimensional organization in

scripts using the pupillary response. Psychophysiology, 44(6), 864–873.

Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural cate-

gories. Cognitive Psychology, 8(3), 382–439.

Rosen, V. M., Caplan, L., Sheesley, L., Rodriguez, R., & Grafman, J. (2003). An examination of daily activities

and their scripts across the adult lifespan. Behavior Research Methods, Instruments, & Computers, 35(1),

32–48.

Rumelhart, D. E. (1977). Understanding and summarizing brief stories. In D. LaBerge & S. J. Samuels (Eds.),

Basic processes in reading. Perception and comprehension (pp. 265–303). Hillsdale, NJ: Erlbaum.

Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals and understanding: Inquiry into human knowledgestructures. Hillsdale, NJ: Erlbaum.

Sirigu, A., Zalla, T., Pillon, B., Grafman, J., Agid, Y., & Dubois, B. (1995). Selective impairments in managerial

knowledge following pre-frontal cortex damage. Cortex, 31(2), 301–316.

Solomon, K., & Barsalou, L. W. (2004). Perceptual simulation in property verification. Memory & Cognition,

32(2), 244–259.

Zwaan, R. A., Madden, C. J., Yaxley, R. H., & Aveyard, M. E. (2004). Moving words: Dynamic representations

in language comprehension. Cognitive Science, 28(4), 611–619.


insights into knowledge representation: the influence of amodal and perceptual variables on event...

Documents