an exploration of a semiotic model of interaction through interactive media

8/2/2019 An Exploration of a Semiotic Model of Interaction Through Interactive Media

1/4

An exploration of a semiotic model of interaction through

interactive media

Shaleph O'NeillThe HCI Group, School of Computing,

Napier University,

Edinburgh, EH10 5DT,+44 0131 455 2699,

[email protected]

David BenyonThe HCI Group, School of Computing,

Napier University,

Edinburgh, EH10 5DT,+44 0131 455 2699,

[email protected]

1 Introduction

Semiotics has been called the mathematics of the arts

because it deals with concepts that are relevant to so

many different disciplines; cultural studies, literature,art, anthropology and so on. It has also been applied to

HCI where Benyon (Benyon 2001) has argued that it

could form the basis of a 'new HCI. Semiotics centralcritical focus on the signs, messages and texts that are

used to create meanings within different domains make

it a prime candidate for inclusion in HCI theory. Over

the past ten years or so a number of experts have

conducted research into Semiotics in relation to HCI.

Most notable among them are Peter Bogh Andersen

(Andersen 1993, Andersen 2001, Andersen 1990) and

the members of the semiotics-engineering group

(SERG) (Barbosa, et al. 1999, deSouza, et al. 2001a,

deSouza, et al. 2001b, deSouza, et al. 2000, Prates, etal. 2000a, Prates, et al. 2000b). However as technology

continues to advance and the diversity of interaction

continues to develop computer semiotics, as it is, has

not kept pace. Like most HCI research, computer

semiotics has focused on traditional ideas of

interaction. A re-evaluation of this approach needs to

be conducted in new technological domains, taking

into account the already existing spectrum of semiotic

theory. Semiotics as a whole has a rich history of

critiquing a wide variety of domains and as such it

offers HCI a unique perspective and insight into areas

such as virtual environments, CSCW and ubiquitous

computing. The aim of research from this perspectiveis to offer accounts of these new domains, which may

extend our concepts of analysis and design moving

towards a semiotics of interaction.

2 Position

Semiotics concern with the nature and use of signs is a

good place to approach HCI from because some of the

central concerns of HCI parallel those already present

in semiotics. The notion of the sender and reader in

semiotics is not dissimilar to the notion of designer and

user in HCI. From this perspective and other research

around semiotic ideas in HCI a number of questionshave emerged that we are interested in looking at.

1. Using a qualitative methodology, is it possible

to find evidence to support a theoretical

semiotic model of interaction?

2. What other elements can be found in the data

to develop this model so that it is more

effective at describing interaction from this

semiotic point of view?

3. Can we begin to move away from purely

denotational meanings in a semiotic

evaluation and start to consider the richer

world of connotations during interaction?

4. What, if anything, can this semiotic model ofinteraction say about the notion of presence in

virtual environments, given that interaction ina virtual world is a mediated experience?

3 Towards A Semiotic Model

The focus of our early research into semiotics has

taken a broad view of many strands of semiotic theory.

In our first paper (O'Neill, et al. 2002) we focused on

the work of Umberto Eco (Eco 1976) in relation to a

semiotic analysis of mobile phone interfaces. Starting

from the SERG perspective that interfaces can be

considered to bea one shot message from designer touser (Prates, et al. 2000b), We applied Ecos revised

KF model as a tool by which to analyse interactions

looking at the meanings associated with the signs

within the interface.

The revised KF model is built around the notion that

meanings can be extrapolated from signs as either

denotations or connotations that are dependent on the

context and circumstances in which the signs are

encountered. The revised KF model then is a dynamic

tool that looks at the way the meanings of signs change

depending on where they are encountered. Applying

these ideas to mobile phone interfaces uncovered how

the meanings of individual signs were dependent on

the context provided by the concurrent and sequential

signs in the interface (Andersen, 1990).

This early work showed that the KF model is useful as

an analytical tool for studying the intricacies of


2/4

interaction at the interface and that semiotics in general

can be applied to interface concepts. However,

although its level of detail is very useful for looking at

interface problems it is a particularly cumbersome tool

to use for looking at complex interfaces, capturing

nothing of the human aspects of interaction in general.

Clearly a more general model of interaction with muchmore scope across different domains could be a useful

contribution from semiotics to HCI.

3.1 Sequential and concurrent syntagms Peter Bogh Andersens notions of concurrent and

sequential syntagms (Andersen 1990) provide an

insight into HCI by abstracting a point of view direct

from structural semiotics. By drawing on semiotic

concepts from Theatre and Dance, Andersen focuses

on the notions of the sequence of events in relation to

the actors and props present on the stage. For Andersen

computer based signs exist as two-dimensional objects

that occupy both sequential and concurrent planes.During interaction computer based signs occupy a

place in the interface, which is relative to other signs

on the screen. As they are interacted with they are

brought into relation temporally to other signs in the

interface that occur as a result of system response.

Andersen proposes a model here that looks at the

process of interaction based on the notion that it takes

place through the manipulation of the signs within an

interface over a period of time. What is unique in this

description of interaction is that it can be viewed as a

kind of pseudo discourse that takes place between the

computer and the user in terms of the meanings each

one can attribute to the signs as they are activatedduring the interaction. These ideas provide the

backbone for our interaction models as they are evident

in various domains such as theatre, dance, cinema and

wayfinding (Andersen 1990) (Passini 1992) to name

but a few.

3.2 The UmweltJacob Von Uexkulls conception of the Umwelt (Allot

1994, Allot 1992, Deely 2001, Kull 1998, Sebeok

1979) is built upon the unique notion that all

significations take place within the bounds of firstly,

our genetic codes in terms of hereditary aspects of

species, and secondly, the social codes within which

we live as aspects of our environment into which we

become indoctrinated as we develop and grow as

people. There can be no signification outside these

constraints because they are what give us a) the need to

communicate and b) the means by which to do it. The

Umwelt then is effectively the mass of knowledge that

we carry around with us into every interaction, which

has been formed and continues to form as a result of

those interactions. This is an important idea in our

model for two reasons. Firstly it takes the place of

existing cognitive models in HCI in representing

knowledge and memory. Secondly, It is clearly linked

to ideas about semiosis and connotation because this

knowledge of codes is the well that the semiotic

process draws from to allow signification to take place.

Ecos revised KF model then, is really a dynamic

model of the semiotic process in relation to the

Umwelt (Eco 1976).

3.3 The Perception/Action LoopContained within Uexkulls conception of the Umwelt

is a model of the relationship between organism and

environment, which is a perception/action model.

Again it is very similar to those proposed in HCI by

cognitive psychology. This is very useful because it

frames these ideas in semiotic terms that allow us to

build an integrated model of interaction from a

semiotic perspective. The fundamental difference

between Uexkulls perception/action model and the

cognitive perspective is that Uexkull characterises its

operation in terms of signs rather than in terms of

processing raw sensory data. This is an important shift

in perspective that considers these signs as

phenomena which are encountered during interaction

which provide possibilities for interaction rather than

the goal, task, action, evaluation approach originated

by the ecological psychologist Gibson and

subsequently extended and applied to HCI in a major

strand of Normans work (Norman 1998).

3.4 Information artefacts According to Andersen information artefacts are the

different types of signs that make up an interface.

These are the buttons, graphics, words and such like

that he categorised in his book Computer Semiotics(Andersen 1990). Since then however, as pointed out

earlier, many new forms have come to be included in

an interface to the point where we now have the new

media Metalanguage (Manovich 2001). So the

information artefacts in our model are considered to be

all of the elements that now go into an interface which

constitute the beginnings of this new metalanguage.

3.5 The ModelThe model (Figure 1) works like this: Uexkulls

perception/action loop encounters and manipulates

phenomena or information artefacts (Benyon 2000)

that exist in an interface. This activity, which producessequential chains from the concurrent chains of the

system throughout an interaction, is a sense making

process that occurs in relation to aspects of denotation,

connotation and meta-language. In other words our

semiotic model of interaction describes an

interface/information space as a number of signs

brought together to form a text. This text is

experienced as a pseudo-discourse, mediated through

the concurrent and sequential chains of the system.

This results in the production of meaning

(interpretation) by the user who acts through the

messages of the text as if it were a medium.


3/4

Fig 1. A semiotic model of interaction through a computer Medium

5 Future work

In relation to our future work we are focusing on the

development of this model as a tool for exploring

interaction in Virtual Environments. As part of the

Benogo project (Arnspang 2002) work is scheduled to

take place that will look at the concept of presence in

interactive mediated environments. It is hoped that

this model, and the methods associated with it will

bring some insight into this type of mediated

interaction from a semiotic point of view.

There is an interesting phenomenological perspective

to presence that might link the notion of the Umwelt

in this model to the notion of the embodied mind as

proposed by Lakoff and Johnson (Lakoff, Johnson

1999). In their work the utterances of individual

subjects uncover the metaphors by which they

understand meanings. This is similar to the semiotic

processes in the Umwelt and might help us to

understand more about the part connotations play in

interaction. Lakoff and Johnson provide a number of

categories of metaphor that they consider as being

active in the mind. Further development of a method

to explore the notion of the Umwelt in this model

might do well to include some of this research,

particularly in relation to a sense of presence in

mediated environments.

ReferencesAllot R (1994) Language and the origin of semiosis.

In: Noth W (ed.) Origins of Semiosis: sign Evolutionin Nature and Culture. Morton de Gruyter, Berlin

Allot R (1992) The Motor Theory of Language:

Origin and function. In: Winderval J (ed.) Language

Origin: A Multidisciplinary Approach. Kluner

Academic Publishers, Dordrecht

Andersen PB (1993) A semiiotic approach to

programming. In: Peter bogh Andersen BH, Jens F.

Jensen (ed.) The Computer as Medium. Cambridge

University Press, Aarhus (pp 16-67)

Andersen Pb (2001) What Semiotics can and cannot

do for HCI. Knowledge-Based Systems Volume 14:

419-424.

Andersen PB (1990) A Theory of Computer

Semiotics. Cambridge University Press, Cambridge

Arnspang J, Benyon, D. R., Fahle, M. W., Granum,E., Madsen, C. W., Pajdla, T., Peleg, S., Smyth, M.,

Turner, P., Turner, S. and Weinshall, D. (2002) An

Investigation Into Virtual Representations Of Real

Places. In Proceedings of PRESENCE 2002:.

Barbosa DJ, Prates R, deSouza CS (1999) Direct and

Indirect user-to-developer messages through

communicability evaluation. Representational

Support for User Developer Communication

workshop, INTERACT'99.

Benyon D (2000) Beyond the Metaphor of Navigation

in Information Space. CHI2000


4/4

Benyon D (2001) The new HCI? Navigation of

information space. Knowledge-Based Systems

Volume 14: 425-430.

Deely J (2001) Umwelts Semiootika osakonna

kodulehekulg. Semiotika 134 special volume about

Jakob von Uexkull: 125-135

deSouza TS, Barbosa SDJ, Prates RO (2001a) A

semiotic engineering approach to user interface

design. Knowledge-Based Systems Volume 14: 461-

465

deSouza S, Barbosa SDJ, Prates RO (2001b) A

Semiotic Engineering Approach to User Interface

Design. Knowledge-Based Systems 14: 461-465

deSouza S, Prates R, Carey T (2000) Missing and

Declining Affordances: Are these Appropriate

Concepts? Journal of the Brazilian Computer Societyvol.6

Eco U (1976) A theory of Semiotics. Indiana

University Press, Indiana

Kull K (1998) On semiosis, Umwelt, and

semiosphere. Semiotica vol. 120: 299-310

Lakoff G, Johnson M (1999) Philosophy of the Flesh

Manovich L (2001) The Language of New Media.

MIT Press

Norman D (1998) The Psychology of Everyday

Things. MIT Press, London

O'Neill S, Benyon DR, Turner SR (2002) Semiotics

and Interaction Analysis. ECCE 11.

Passini R (1992) Wayfinding in Architechture.

Reinhold, New York

Prates O, deSouza CS, Barbosa S (2000a) A case

Study for Evaluating Interface Design through

Communicability. ACM Designing Interactive

Systems, DIS'2000.

Prates O, deSouza CS, Barbosa S (2000b) A methodfor evaluating the communicability of User Interfaces.

Interactions: 31-38

Sebeok TA (1979) The sign and its Masters .

University of Texas Press, Austin

an exploration of a semiotic model of interaction through interactive media

Documents