an e-learning module on negotiation analysissal.aalto.fi/publications/pdf-files/meht.pdf · 1 an...

25
An E-learning Module on Negotiation Analysis Harri Ehtamo Raimo P. Hämäläinen Ville Koskinen Helsinki University of Technology Systems Analysis Laboratory P.O. Box 1100, FIN-02015 HUT, Finland Tel. 358-9-451 3054, fax 358-9-451 3096 E-mails: [email protected], [email protected], [email protected]

Upload: nguyenkiet

Post on 11-Mar-2018

236 views

Category:

Documents


4 download

TRANSCRIPT

An E-learning Module on Negotiation Analysis

Harri Ehtamo Raimo P. Hämäläinen

Ville Koskinen Helsinki University of Technology

Systems Analysis Laboratory P.O. Box 1100, FIN-02015 HUT, Finland Tel. 358-9-451 3054, fax 358-9-451 3096

E-mails: [email protected], [email protected], [email protected]

1

An E-learning Module on Negotiation Analysis

We describe a web-site containing material and tools for learning mathematical

models of negotiation analysis and discuss students’ experiences of its use. It is part

of e-learning decision making site, www.dm.hut.fi, which is developed at Systems

Analysis Laboratory, Helsinki University of Technology.

The negotiation analysis site provides an introduction to theory and gives online

practice in negotiation support. The learning material is modular and it can be used

both as an independent introduction to the topic and in parts to create supplementary

modules for other courses. The material consists of theory sections, case studies and

assignments. It also includes quizzes for self-evaluation and video clips illustrating

the use of the Joint Gains software.

Models of negotiation analysis are especially suitable for e-learning because of their

interactive nature. The Joint Gains is an interactive negotiation support system for

multi player multi issue negotiation problems and it is intended for real-life

negotiations; for instance, it has been applied to create new regulation policies for

lake-river systems. Here, we apply the Joint Gains as an interactive tool for active

learning through role playing experiments in educational context.

“Introduction to game theory and negotiation” learning module was evaluated in a

web-course on mathematical modelling. The experiences were encouraging and

mainly positive; the students considered the material very useful and they succeeded

well in the assignments without need for any personal guidance.

2

Key words: Negotiation Analysis, Negotiation Support, e-Learning, e-Democracy,

Interaction

1 Introduction

It is an educational challenge to teach decision making and negotiation skills for the

students by applying possibilities offered by electronic media and especially e-

learning. Basically, e-learning means education that takes advantage of electronic

media, which can offer new dimensions and possibilities for learning. Particularly

personal computers and the Internet have established a sound basis for e-learning

because their costs are nowadays relatively low and hence they have become a

standard which allows their widespread applications. Electronic media offer

possibilities for learning “any time, anywhere”, more extensive use of colorful

graphics, animations, voice etc., and new ways of communication, such as e-mail, on-

line chat, newsgroups and video conferencing. Especially the new ways of

communication can play a crucial role in practice of negotiation.

By negotiation we mean an interactive process by which different parties try to reach

compromises and make an agreement. According to Raiffa, Richardson and Metcalfe

(2002) negotiation analysis is a field of study primarily focusing on prescriptive

models of negotiation processes for generating acceptable compromise outcomes and

reaching agreements having its roots in decision analysis and game theory. These

prescriptive models can be implemented as negotiation support systems (NSSs) for

solving real-life negotiation problems by applying electronic media.

3

Besides their intended real-life use, NSSs can be used for teaching. In this case, the

students apply them to solve imaginary negotiation problems by having role-playing

experiments as suggested, e.g., by Winham (2002). This lets the students to learn by

doing through simulating the negotiations and seeing how their actions affect the

interactive negotiation process and its final results. This cannot normally be

accommodated in a traditional course that takes no advantage of the electronic media.

Therefore, the negotiation analysis seems a fruitful area for e-learning, simply because

of its interactive nature and practical essence.

We aim to create e-learning material for teaching mathematical models of negotiation

analysis. The material consists of theory sections complemented with relevant

references to the literature, case studies, quizzes and assignments. These are

complemented with multimedia presentations such as video clips, animations and

colorful graphics where necessary. We provide facilities for interactive and active

learning by the Joint Gains NSS1. The Joint Gains is a web-based implementation of

an interactive negotiation method that can be used for real-life negotiation problems

online. See Hämäläinen et al. (2001) for an example on its use in Lake-River

regulation policy problems.

There is a variety of potential applications of negotiation analysis, e.g., in political and

environmental decision making, in e-business, and in e-commerce (Raiffa, Richardson

and Metcalfe, 2002; Susskind, Levy and Thomas-Larmer, 2000; Hämäläinen et al.,

2001; Kersten and Lo, 2001). Our e-learning material is modular to make it possible

to tailor it to support different needs of different students. The students can work with

the material either with a pair, in a small group or perhaps alone. The latter is not

4

recommended, however, because in such a case learning may lose its social

dimensions. These are very important when studying negotiation skills. Teachers can

use the material as such or possibly by including their own material in it and by

having face to face meetings between teacher and students.

In this paper, we present one e-learning module, “introduction to game theory and

negotiation”2, which corresponds to traditional few hours engineering lecture plus the

exercises, and discuss experiences of its use. So far we have taken the students of

OR/MS and engineering as primary users of the module. Nevertheless, the material

can also be applied to construct other modules for decision makers and mediators that

are preparing for web-based participation in real-life negotiations, and especially for

the use of the Joint Gains NSS. These possibilities are discussed in Chapter 5.2.

2 Related Material in the Web

There is also some other e-learning material on negotiation science available in the

Internet. For instance, Köszegi and Kersten (2003) have published “Negotiations and

e-negotiations: management and support” course3. It consists of an electronic textbook

presenting theory, which focuses on basic concepts of economics, game theory and

social psychology. It also contains some case studies and related role-playing

assignments. Our notions on e-learning of mathematical models of negotiation

analysis are somewhat similar to those presented by Köszegi and Kersten even if our

work has been carried out independently. For example, as Köszegi and Kersten we

also consider learning by doing very central concept and take advantage of

possibilities offered by the electronic media to produce complete learning modules

5

that can be used in combination with face to face sessions. Nevertheless, there are

some differences. First, the students can apply our module independently of any

instructor, which provides possibilities for completely self-paced learning and even

distance learning. Second, when browsing their learning material one easily gets the

impression of studying a traditional textbook on a computer screen, which has been

criticized, e.g., by Hobbs (2002). We have paid attention on developing a more on-

screen readable material, looking more like a “web-site”. Third, our approach includes

the use of the Joint Gains software that is completely available online any time from

anywhere for real-life negotiation cases defined by the negotiators or a third

intervening party. Thus it is not intended only for educational use as the INSPIRE

system (Kersten and Noronha, 1999) applied by Köszegi and Kersten in their e-

learning course.

Harvard university press e-learning division has published an e-learning program

“Yes! The On-Line Negotiator” based on the book by Fisher and Ury (1981). It is

commercially available and it contains slideshows summarizing the theory, presenting

some case problems and related quizzes4. That material is essentially a program and

hence it does not constitute a complete whole but interactive tools and related

slideshows for training negotiation skills.

On game theory, there is more e-learning material available in the Internet. For

instance, a link collection provided by the Vanderbilt University5 contains a number

of links to material in the web. That material includes lecture notes, textbooks,

quizzes and tools illustrating the interactive nature of different games.

6

There is much e-learning material and experiences of their use already available in

OR/MS field. For instance, our e-learning module has been developed as a part of

OR-World project (Suhl and Kassanke, 2000), which has focused on developing and

assessing new techniques for teaching. Similar work has been performed at University

of Strathclyde by the MENTOR project that aims to “improve the effectiveness and

the efficiency of teaching and learning OR/MS through the use of multimedia

computer based learning materials” (Thornbury et al., 1996). Those projects have

reported that the electronic media has been successfully applied to offer multimedia

presentations and tools for interaction in education of OR/MS. Also, International

Federation of Operations Research Societies (IFORS) has created tutOR learning

modules6 and Institute for Operations Research and the Management Sciences

(INFORMS) has a learning center7, which constitutes a set of links to existing

educational resources in Internet. That material, however, does not either form an

independent whole but rather a material collection for existing courses.

3 Contents of the Module

In “introduction to game theory and negotiations” e-learning module, we teach

mathematical models of negotiation analysis through practical examples. First, we

describe basic concepts of game theory and negotiation problems through two

classical game theory examples: prisoners’ dilemma game and problem of the

commons. For different variants and history of these games, we refer to textbooks by

Luce and Raiffa (1957), Myerson (1997) and Gibbons (1992). These games are

relatively simple but they are also rich enough to illustrate the central concepts and

7

even describe phenomena encountered in real life.

This is followed by an introduction to methods on negotiation analysis that can be

used to give prescriptive aid for negotiating parties to make joint decisions. Here we

refer to Raiffa, Richardson and Metcalfe (2002) and Sebenius (1992). In particular,

we discuss a classification given by Teich, Wallenius and Wallenius (1994) and

Ehtamo and Hämäläinen (2001). They divide various methods in negotiation analysis

according to: whether the utility functions of the negotiating parties can be elicited or

not, and whether negotiating parties take joint problem solving attitude or make

concessions. Thus, we can discuss roughly about four modeling categories:

1. utility function and concession based methods

2. utility function based joint gains searching methods

3. interactive methods based on concession making

4. interactive methods searching joint gains

In the module, we also present ideas of third party intervention, i.e., mediation and

arbitration to help the negotiating parties.

As a detailed example on methods in negotiation analysis we describe the jointly

improving direction method (Ehtamo, Verkama and Hämäläinen, 1999) which is a

mathematical formalization of the single negotiation text (SNT) procedure presented

by Raiffa (1982). In the jointly improving direction method, joint gains are searched

interactively and iteratively starting from a tentative agreement under assistance of a

mediator. Gradually, the method guides the parties to a Pareto optimal agreement. We

present the method by using verbal and geometrical reasoning through an example

8

where it is applied to reach Pareto points for problem of the commons.

9

4 Module Description

We divide the e-learning module conceptually into five main elements by following

an e-learning material structure originally developed for multiple criteria decision

analysis module8 at Systems Analysis Laboratory. The module consists of

1. theory

2. online quizzes

3. cases

4. assignments

5. video clips

Each of the elements supports different learning objectives and presents the topic

from different viewpoints. Besides those main elements, the module has a front page,

see Figure 1. The front page contains motivating introduction to the module, states

explicitly what the students are expected to learn and provides links to the material

included in the module. In the following, we describe each main element and give

some screenshots presenting the material from the students’ viewpoint.

Theory can be compared to an ordinary introductory textbook but it is designed to be

on-screen readable, which is taken into account by considering two major challenges

presented by Weitl et al. (2002). The first challenge is that students’ perception tires

out when studying new information on a computer screen. The other one is the loss of

overview, which is due to low amount of information that can be shown on a

computer screen at a time. Therefore, we have written the theory in a format of a story

telling about two friends, Harold and William. This story is complemented with

10

frequent headings and introductory overview sections that point out the thread of the

story and the core concepts of the theory.

Figure 1 Learning module front page

Weitl et al. (2002) have concluded that technical topics, such as pure mathematics, are

inconvenient to be studied through the web. Therefore, we include only basic, high

school level, mathematical definitions and analysis in the theory. Most of the

presentation is based on colorful graphs and their interpretations. Further

mathematical details can be found from articles and textbooks given in further reading

sections.

The theory is divided into subsections each of which physically consists of one

HTML-page. The table of contents of the theory is presented for the students in a

11

navigation frame on the left side of the body text to help the navigation and sharpen

the overview. As an example, Figure 2 illustrates a part of the theory section

presenting prisoners’ dilemma game.

Figure 2 An example on a theory section presenting prisoners’ dilemma game

Online quizzes are web-forms stored in Quiz Star9 server. They contain multiple

choice questions regarding the theory, see Figure 3. Their main purpose is to act as a

motivating self-evaluation tool and provide a variety of refreshing cognitive activities

for the students. The quizzes summarize the core concepts of the theory and thus they

also sharpen students’ overview on the topic through presenting simple games and

negotiation situations. After filling in and submitting a quiz a student gets

immediately a response pointing out the correct and incorrect answers. In case of

possible incorrect answers, the student should refer to the theory and try refilling and

12

submitting the quiz.

Figure 3 A part of the game theory quiz

Cases are simple problems encountered in real life. In this learning module, the

students study a description of a case, namely, buyer seller problem. That problem

represents a typical case in e-commerce, where a buyer and a seller negotiate about

price and delivery time. The case acts as an example connecting the theory to the real

life applications and to students’ personal context thus easing the adoption of new

knowledge.

The module contains two types of assignments: analytical and software assignments.

The analytical ones ask the students to apply mathematical tools they have learned

13

and thus familiarize themselves with somewhat theoretical concerns. In the software

assignment, the students learn by doing and solve the buyer seller problem presented

in the case description through role-plays as suggested by Winham (2002). The

assignments are manually graded by an instructor, unlike the quizzes.

The students apply the Joint Gains NSS, which is an implementation of the jointly

improving direction method, in the software assignment. They first create a

negotiation case by defining the negotiating parties, a buyer and a seller, and the

issues they negotiate about, i.e., price and time, see Figure 4.

Figure 4 Creating a case in the Joint Gains

After creating the case, the students take their roles and start the negotiation. In the

negotiation, the Joint Gains gathers local preference information from the parties with

simple pairwise comparison tasks and makes new jointly preferred proposals for the

parties, who may either accept or reject them; for details see Ehtamo and Hämäläinen

(2001). Figure 5 shows how the students answer the questions and apply an online

chat for verbal communication in the Joint Gains NSS.

14

NegotiatingNegotiating

On-line chatOn-line chat

Figure 5 Negotiating with the Joint Gains

The assignments are delivered both as HTML and MS Word documents for the

students. Those who are able to use MS Word can apply the assignment document as

a report template and fill their answers directly in it. Besides the questions, the

assignment document contains detailed instructions so that the students need not to

open a separate window for writing a report, reading the assignment and the

instructions. Figure 6 shows how a student reads the instructions and the questions

and works with the Joint Gains software filling her answers to the report template

directly.

15

Figure 6 A student fills her answers to the report template and works with the Joint Gains

software simultaneously

There are also some video clips that are recordings demonstrating the use of the Joint

Gains at a “click the OK-button”-level. The clips are available in three formats: AVI

with audio, AVI with no audio and GIF animations. AVI-format was chosen because

it allows including audio, pausing, stopping, rewinding or fast forwarding the clips.

Nevertheless, viewing AVI-files needs a media player to be installed. Therefore, GIF

animations are delivered, as well, because they can be viewed by an ordinary web

browser. Their usability is, however, poorer because they can only be played and

restarted and they may not contain audio. AVI files with no audio are included in the

material to serve the students who have slow internet connection and find the versions

with audio too large.

16

The main purpose of the clips is to reduce need for personal instruction and

communication between the teacher and students by e-mail or face to face directly.

The clips are mainly intended to help the students who consider themselves unfamiliar

with computers and encourage them to start working with the software assignment.

After completing the module, the students evaluate it by filling an online

questionnaire that is intended to measure their subjective experiences. The evaluations

are accomplished with the Opinions Online10 software, which is an online-platform

for voting, surveys and group decision making developed at the Systems Analysis

Laboratory.

5 Discussion

5.1 Experiences of Use

The learning module was used in November 2002 in an advanced web course on

mathematical modeling organized by the Virtual University of Finland. The course

consisted of eleven learning sessions and our module was one of them. The overall

structure of the other learning sessions was quite different from that of ours. For

example, they consisted of video lectures, lecture slides and assignments.

There were nine one or two student groups from different Finish universities

completing our learning module. The students were mainly graduate students in

industrial engineering, systems engineering, physics or other engineering. They

considered themselves quite experienced users of web and they had some familiarity

on e-learning from the earlier learning sessions of the course but not on e-

17

learning format similar to ours. The course was not compulsory for the students and

hence our experiences comprise of observing students who had chosen an e-learning

course voluntarily.

The results of the student evaluation were strikingly positive. The students found our

format of e-learning convenient despite the fact that the format was new for them.

Most of the students stated that they are willing to work in similar e-learning

environments in the future and that they are also willing to recommend their

experiences for their peers. This kind of positive attitude has been reported in

literature also earlier, see, e.g., e-learning experiences presented by Benbunan-Fich

and Hilz (2002). As in our case, the students had voluntarily chosen e-learning course

in their experiments.

The lack of face to face interaction between students and teacher, and among the

students, has been a realized problem in e-learning (Wesley, 2002). The same problem

was also reported by our students. Nevertheless, the students were able to work with

our learning module completely independently; they did not communicate with a

human instructor either personally, through e-mail or the newsgroup system.

Therefore, the lack of social interaction did not appear to be a severe problem in our

case. This is probably due to two reasons. First, the students had an option of working

together with a pair and hence they had possibilities for social contacts during their

learning process. Second, the instructions were very detailed, e.g., through the use of

the video clips.

Most of the students reported in student evaluation that they are relatively indifferent

18

between the format of the other learning sessions and that of our module. Hence, they

did not miss a talking head of the video lectures when working with our module. We

see this indicating that each format of learning has its own advantages and

disadvantages. According to opinions of our students, the greatest advantage in e-

learning is the freedom of learning almost everywhere and any time. This freedom has

the other side of the coin, too. Namely, e-learning requires strong responsibility and

self-motivation from the students. Not all our students felt themselves motivated

enough to complete the module.

5.2 Future Work

We aim to gain further experiences on the use of the e-learning material. These

experiences are obtained at least through the use of “introduction to game theory and

negotiation” learning module in the same course on mathematical modeling also in

next autumn.

So far, the users of our e-learning material have been university students. Other users

of the material could be experts already in the field of e-commerce, and persons

involved in environmental and political decision making. This is taken into account

when choosing the examples and cases presented in the material. For instance, the

problem of commons provides perspectives from the environmental decision

problems, and our buyer-seller case is a problem emerging frequently in supply chains

between suppliers and their customers in one form or another. For recent studies and

possible future cases in e-business, see special issues of the Group Decision and

Negotiation dealing with e-negotiations (2003) and of the Decision Sciences dealing

19

with e-business and supply chain management (2002).

According to Meerts (2002), if a student already has some real life negotiation

experiences it is beneficial to let her learn by doing by role-playing experiments and

analyzing them afterwards. Hence, our e-learning module could be even more

successful when training negotiators for web based negotiations than merely training

ordinary students. Nevertheless, to teach them efficiently we should introduce new

cases and new modules. In particular, the negotiators should be provided a more

compact learning module on applying the Joint Gains software in practice through a

case study close to their own case.

Currently, the e-learning material concentrates mainly on one particular negotiation

method, namely, the joint gains searching method. Therefore, the material should be

complemented by presenting other types of negotiation methods, such as concession

based methods, as well. Concession based methods can be illustrated, e.g., through a

case representing buyer-seller negotiations where the negotiating parties try to reach a

jointly accepted agreement through consecutive offers and counteroffers.

6 References

1. Ehtamo, H., M. Verkama and R.P. Hämäläinen (1999). “How to Select Fair

Improving Directions in a Negotiation Model over Continuous Issues”. IEEE

Transactions on Systems Man and Cybernetics – Part C: Applications and

Reviews, Vol. 29, 26-33.

20

2. Benbunan-Fich, R. and S. Hilz (2002). “Correlates of Effectiveness of

Learning Networks: The Effects of Course Level, Course Type and Gender on

Outcomes”. Proceedings of the 35th Hawaii International Conference on

Systems Sciences 2002. IEEE Computer Society Press, Hawaii.

3. Bichler M., G. Kersten, C. Weinhardt (2003). Group Decision and

Negotiation: Special Issue on Electronic Negotiations, Vol. 12, No 2.

4. Ehtamo, H. and R.P. Hämäläinen (2001) “Interactive Multiple-Criteria

Methods for Reaching Pareto Optimal Agreements in Negotiations”. Group

Decision and Negotiation, Vol. 10, 475-491.

5. Fisher, R. and W. Ury (1981). Getting to Yes: Negotiating Agreement Without

Giving In. Boston: Houghton Mifflin.

6. Gibbons, R. (1992). A Primer in Game Theory. Prentice Hall.

7. Hobbs, D.L. (2002). “A Constructivist Approach to Web Course Design: A

Review of the Literature”. International Journal on E-Learning, Vol. 1, 60-65.

8. Hämäläinen, R.P., E. Kettunen, M. Marttunen and H. Ehtamo (2001).

“Evaluating a Framework for Multi-Stakeholder Decision Support in Water

Resources Management”. Group Decision and Negotiation, Vol. 10, 331-353.

9. Kersten, G. and G. Lo (2001). “Negotiation Support Systems and Software

Agents in E-business Negotiations”. The First International Conference on

21

Electronic Business, Hong Kong.

10. Kersten, G. and S. Noronha (1999). “WWW-based Negotiation Support:

Desing, Implementation and Use”. Decision Support Systems, Vol 25, 135-

154.

11. Köszegi, S. and G. Kersten (2003). “On-line/Off-line: Joint Negotiation

Teaching in Montreal and Vienna”. InterNeg Research Papers,

http://interneg.org/interneg/research/papers/2003/03.pdf, referred 09.05.2003.

12. Luce, R. and H. Raiffa (1957). Games and Decisions. John Wiley & Sons.

13. Meerts, P. (2002). “Training of Negotiators”. In V. Kremenyuk (ed.),

International Negotiation. John Wiley & Sons, 455-464.

14. Myerson, R. (1997). Game theory: analysis of conflict. Harvard University

Press.

15. Narasimhan R. (2002). Decision Sciences: Special Issue on e-Business and

Supply Chain Management, Vol. 33, No 4.

16. Raiffa, H. (1982). Art and Science of Negotiation. Harvard University Press.

17. Raiffa, H., J. Richardson and D. Metcalfe (2002). Negotiation Analysis: The

Science and Art of Collaborative Decision Making. Belknap Press of Harvard

University.

22

18. Sebenius, J. (1992). “Negotiation Analysis: A Characterization and Review”.

Management Science, Vol. 38, 18-38.

19. Suhl, L. and S. Kassanke (2000). “OR-World-Using Learning Objects in a

Hypermedia Learning Environment”. Proceedings of EdMedia2000. AACE,

Montreal.

20. Teich, J., H. Wallenius and J. Wallenius (1994). “Advances in Negotiation

Science”. Transactions on Operational Research, Vol. 6, 55-94.

21. Thornbury, H., M. Elder, D. Crowe, P. Bennett and V. Belton (1996).

“Suggestions for successful integration”. CTI Active Learning, Vol. 2, 18-23.

22. Weitl, F., C. Süss, R. Kammerl and B. Freitag (2002). “Presenting Complex e-

Learning Content on the Web: A Didactical Reference Model”. Proceedings of

E-Learn 2002 World Conference on E-Learning in Corporate Government,

Healthcare & Higher Education, AACE, Montreal.

23. Wesley, D. (2002). “A Critical Analysis on the Evolution of E-Learning”.

International Journal on E-Learning, Vol. 1, 4148.

24. Winham, G. (2002). “Simulation for Teaching and Analysis”. In V.

Kremenyuk (ed.), International Negotiation. John Wiley & Sons, 465-480.

23

7 Web References

1 Systems Analysis Laboratory. “Joint Gains Negotiation Support in the Internet”.

http://www.jointgains.hut.fi/, referred 04.06.2003.

2 Systems Analysis Laboratory, Helsinki University of Technology. “Introduction to

Game Theory and Negotiation: Learning Module”.

http://www.negotiation.hut.fi/learning-modules/IntroToGTAndNego/, referred

04.06.2003.

3 Concordia Univeristy. “Negotiations and e-negotiations: management and support”.

http://mis.concordia.ca/projects/negocourse/nego_course/index.html, referred

04.06.2003.

4 Harvard Business School Publishing. “Yes! The On-Line Negotiator”.

http://www.elearning.hbsp.org/, referred 04.06.2003.

5 Vanderbilt University. “GameTheory.net: A resource for educators and students of

game theory”. http://www.gametheory.net/, referred 04.06.2003.

6 University of Melbourne, Department of Mathematics and Statistics, OR Group.

“Welcome to TutOR”. http://www.tutor.ms.unimelb.edu.au/, referred 08.06.2003.

7 Institute for Operations Research and the Management Science. “INFORMS

Learning Center”. http://www.anderson.ucla.edu/informs/SU/learning/, referred

08.06.2003.

24

8 Systems Analysis Laboratory, Helsinki University of Technology. “Multiple Criteria

Decision Analysis e-Learning Site”. http://www.mcda.hut.fi/, referred 04.02.2003.

9 University of Kansas. “QuizStar”. http://quiz.4teachers.org/index.php3, referred

04.06.2003.

10 Systems Analysis Laboratory, Helsinki University of Technology. “Opinions

Online – Platform for Global Participation, Voting, Surveys and Group Decisions”.

http://www.opinions.hut.fi/, referred 04.06.2003.