Self-observation Principle for Estimating the Other’s Internal State∗†

– New Computational Theory on Communication –

Makino, Takaki and Aihara, KazuyukiDepartment of Complexity Science and Engineering,

Graduate School of Frontier Science, Tokyo Universitymak@sat.t.u-tokyo.ac.jp aihara@sat.t.u-tokyo.ac.jp

October 24, 2003

Abstract

We propose a computational theory on estimatingthe internal states of others, which is the basis ofinformation processing in human communication.To estimate internal states of peers, we have todeal with two considerable difficulties, restricteddimension of estimator parameters and conver-sion of objective information into subjective. Tosolve these problems, we propose a new computa-tional theory based on the self-observation princi-ple. Learning one’s own dynamics provides priorknowledge on the dynamics of others, which re-duces restriction of the parameter dimension. Onthe other hand, learning the association betweenone’s own subjective state and the objective self-observation provides a mechanism for convert-

∗This article is a minor revision (based on commentsfrom an English native) of the technical report [MA03]:Makino, T. and Aihara, K. Self-observation Principle forEstimating the Other’s Internal State. Mathematical Engi-neering Technical Reports METR 2003-36, the Universityof Tokyo, October 2003.

†This research was partially supported by a Grant-in-Aid(No. 15016023) for Scientific Research on Priority Areas (2)Advanced Brain Science Project, and the Advanced and In-novational Research Program in Life Science, from the Min-istry of Education, Culture, Sports, Science, and Technologyof the Japanese Government. This research was also par-tially supported by Research Fellowships of the Japan Soci-ety for the Promotion of Science for Young Scientists.

ing objective information obtained from observ-ing others and their subjective information. Inthis paper, we formalize communication within aframework of dynamics-estimation problems, andexplain the two difficulties and our framework.We also discuss the relations our proposal haswith evolutional psychology and neuroscience.Keywords: communication, computational the-ory, model estimation, mirror neuron, self-observation

1 Introduction

The mechanism behind human communication isstill not understood sufficiently well in informa-tion science. Not only human beings, but mon-keys and dogs communicate with one another. Inadvanced communication, new strategies emerge,such as tactics, cooperation and cultural elements.However, modern computers cannot communi-cate. Only in a programmed manner, they com-municate with other computers, let alone with hu-mans.

Recent developments in technology are over-coming technical issues, such as the lack of“humanity” in computers, which have preventedcomputers from communication. There have beenrapid improvements in the synthesis and analysisof facial expressions, generation and recognition

1

of vocalized dialogue, and other areas. Some ofthe latest studies have been oriented toward hu-man communication through the bodies of hu-manoid robots [IOIK03, AHP+00]. Nevertheless,we still have found no way of achieving advancedcommunication that will cause emergence of newstrategies.

We expect improvements in brain science willreveal the information-processing mechanism re-sponsive for communication in the brain. How-ever, without a breakthrough theory, even theachievements of brain science will be limited.Even if we investigate all the synaptic connectionswithin the brain, we will still not know how thebrain engages in communication. Experiments incognitive science and neuroscience can only lookfor evidence in existing theories; they cannot elu-cidate a wholly-unknown mechanism.

We need a computational theory that explainswhat underlies communication. A computationaltheory is a basic framework for research on braininformation-processing, which states the purposeand requirements of communication and the in-put/output of information processing for commu-nication [Mar82, Kaw96]. Based on computa-tional theory, we can propose several algorithmsthat satisfy this purpose and requirements. Wecan then implement the proposed algorithms oncomputers, as well as look for evidence of the al-gorithms in the brain.

As for communication, one existing computa-tional theory states that communication consistsof interactions that occur through estimation ofother’s internal states [KDH01]. However, to thebest of our knowledge, no one has yet found analgorithm that conforms to this theory. This sug-gests some incompleteness in the theory, in otherwords, the theory may lack some key factors thatare essential in solving difficulties within actualalgorithms. If this is the case, we need a newcomputational theory, which contains key factorsin addition to the original theory.

In this paper, we investigate the existing com-

putational theory on communication within theframework of dynamics-estimation problem, andpoint out that it cannot be applied to practice be-cause of two difficulties, limits in the parameterdimensions of the estimator and conversion of ob-jective information to subjective. To solve theseproblems, we propose a new computational the-ory based on the self-observation principle, wherethe dynamics model, which is learned through ob-jective observation of the self, is applied to othersto estimate their internal states. The first difficultyis reduced by prior knowledge on the dynamicsof others provided by learning one’s own dynam-ics, while the second is resolved through learn-ing the association between one’s own subjectivestate and the objective self-observation. We alsodiscuss the proposed theory within the context ofrelated scientific domains, and explain why theself-learning dynamics model is related to humanself-consciousness and mirror neurons.

2 Communication in a frame-work of dynamics estimation

In this section, we formalize the computationaltheory of communication within the framework ofdynamics-estimation problems.

2.1 Existing computational theory ofcommunication

We first define what we mean by communication,to avoid ambiguity. We also introduce an existingcomputational theory on communication.

In this paper, we focus on communication,which is “information-based interaction betweenpeers, which has been developed for the purposeof surviving under pressure of natural selection”.We have excluded actions which are of no directbenefit to survival or reproduction, such as sim-ple observation and imitation. This is becausewe have no way of verifying algorithms that canundertake these actions. We have also excluded

2

reflex actions based on instincts because our fo-cus is on the mechanism responsible for informa-tion processing in the brain. In addition, we re-strict the partners of communication into peers,i.e., colleagues with equivalent constructions andcomplexity.

Existing studies have shown that estimatingpeers’ internal state is the main form of process-ing the information involved in communication[KDH01]. Here, we define the internal state asa set of elements, which is not observable fromone’s external world but affects actions, e.g. emo-tions, desire, intent, and knowledge.

If a colleague can help, one can survive betterthan if alone. To get this assistance, one needsto appeal to peers to act beneficially for oneself,and this is the purpose of communication. To dothis, however, one must know what form appealmust take to be effective. In other words, prior toappealing, one needs to predict others’ actions un-der various conditions. This is required if the in-formation for communication is to be processedeffectively. Furthermore, since a peer’s actionsare hardly predictable from observable informa-tion, one needs to estimate others’ internal states,prior to predicting actinos.

For example, it may be difficult to predict one’sfuture actions based only on external observationof his state (a thin man with unsteady gait). If,however, we are able to estimate his internal states(hungry and looking for food), this helps us topredict his future actions (when he finds food, hewill eat it). Such predictions help our actions forsurvival (we can hide our food) as well as the in-teractions that changes his internal states (givingthe thin man food will alleviate his hunger andmake him happy).

In advanced communications, internal statesare estimated recursively, i.e. “by estimating ‘howhe is estimating my internal state’ as part of hisinternal state”. For example, chimpanzees exhibitdeceiptful actions [PW78]. One chimpanzee pre-tended that some danger was approaching from a

distance, expecting this action would distract an-other chimpanzee and prevent him from scoldinghim. Recursive estimates, which underlie suchactions, also play a crucial role in human com-munication.

Despite its significance, a largely unknown ishow the human brain estimates peers’ internalstates. One reason for this is the lack of compu-tational studies. For any information processingdone by humans, it is difficult to find the corre-sponding region in the brain, if we do not knowwhat kind of representation is used or what algo-rithms can provide the process. Computationalstudies have tried to construct these representa-tions and algorithms, so that we can use them ashypotheses of the mechanisms in the brain.

To study this further, we need a computationaltheory, which would form a common basis for thealgorithms, such as the purpose of calculation andthe structure of input/output. In this study, we de-fine the computational theory for communicationas follows.

1. Improve one’s chances of survival and repro-duction through selection of actions. (Thisincludes the one selecting actions that causesaction by peers that is beneficial to him).

2. Estimate the peer’s actions to achieve 1.

3. Estimate the peer’s internal states and the re-lation between internal states and the envi-ronments/actions to achieve 2.

The remainder of this paper is devoted to prin-ciples of algorithms that conform to this compu-tational theory.

2.2 Dynamics-estimation framework

Estimating unobservable parameters involves theestimation of several components. We presenta general framework of a dynamics-estimationproblem to identify the components in estimatinghuman dynamics.

3

Let x(t) ∈ S be the state of the estimation tar-get at time t. The dynamics of the target, that is,the structure of the state change according to theenvironment and time, can be denoted as follows,using input a(t) of time t and state-transition mapf .

x(t + 1) = f (x(t), a(t)) (1)

However, we have no way of observing the cur-rent state x(t) directly. Instead, we can observeoutput y(t), which is determined from x(t) andoutput map g.

y(t) = g(x(t)) (2)

Let us assume we want to predict the future out-put of the target, y(t+1), y(t+2), · · · using only in-puts a(1), a(2), · · · and past outputs y(1), y(2), · · · ,y(t). For better results, the predictor needs to cal-culate an estimate for the current state x(t) ∈ S aswell as estimate of the target dynamics, f and g.This is a dynamics-estimation problem (Fig. 1).

If the target dynamics ( f and g) are known tothe estimator, the problem can be reduced to esti-mating the current state, which is generally easy.In the case only f or g is known, the predictor hasto estimate the other map, which makes the prob-lem more difficult.

However, the problem becomes extremely dif-ficult if f and g are both unknown. Since we haveno clues on original state space S , the estimatorhas to provide reconstructed state space S , wherestate estimation x is placed. Generally, in a deter-ministic system with constant input, reconstruc-tion is not guaranteed unless the dimensions ofS is larger than twice the dimensions of originalstate space S (embedding theorem [AIYK00]).The increase in the dimensions of S multiplies thenumber of parameters of estimated maps f andg, and as a result, estimation becomes less feasi-ble. Conversely, partial information on f or g mayhelp estimation through reducing the dimensionsof S .

2.3 Communication in a framework ofdynamics estimation

We can regard the process of estimating a peer’sinternal states as the dynamics-estimation frame-work extended to a non-stationary system. Al-though the two communicating peers are equal,we will only focus on one of them as the dynam-ics estimator, who we will call the self. The targetof the self’s estimates is called the other.

Figure 2 illustrates this framework, where theself is estimating the other’s internal dynamics aswell as the dynamical system being estimated bythe other. To represent this relation, the other’sdynamics model, f2 and g2, should be part ofthe self’s internal state, x1, and be constructed bymodel learner M in state-transition map f1. Sincethe other is also constructing a dynamics model ofthe self, the whole framework becomes symmet-rical. We can also see a fractal structure if the selfrecursively estimates a dynamics model of the selfin the other’s internal state.

Note that the self cannot obtain the other’s pre-cise input a2 or output y2. The self can only ob-tain them indirectly, through input a1 to the self.This problem will be discussed later in Section 4.Until then, let us temporarily use symbols withprimes a′2, y′2 to denote indirect and partial infor-mation.

3 Difficulties with parameter di-mensions

In this section, we describe one of the two diffi-culties involved in estimating the other’s internalstates, i.e., restricted parameter dimensions of theestimator.

3.1 Limits of parameter dimensions

In the previous section, we stated that estimat-ing the other’s internal states can be regarded as

4

Internal state

OutputVerify

Estimated internal state

InputState-transition map

Output map

x x

a

y y

f

g

f

g

Estimation target Estimator

Dynamics model

Model learner

M

Estimated state-transition map

Estimated output map

Prediction of the output

Figure 1: Framework of dynamics-estimation problem

Environment

Inputs

Outputs

Internal state

Peer 1 (the self)Peer 2 (the other)

a1x1a2

x2

y2 y1

g1g2g2

^ffff222f2f2f2

ggg222g2g2g2

^yyyyy222

a'2

yy'2

xx2

f2f2 f1f1

Dynamics model of the other

Estimate of the other's internal state

y

a

M

f^ffff111f1f1f1

gggg11

^yyyyyy111

a'1

yy'1

^xx1

'

1

M

1

2x

Figure 2: Communication in a framework of dynamics estimation

5

a problem within the dynamics-estimation frame-work.

However, the brains of animals have been per-formed dynamics estimation since ancient days,irrelevant of communication. Feedback-basedcontrol of muscle movement and environmentmap construction are good examples of howchanges in the external world are predicted. Thebrain estimates non-observable states and transi-tion/output maps to resolve the complex relationsbetween input and output. Researchers are stilltrying to reproduce this form of estimation in acomputational model [WK98].

Can estimating the other’s internal states,which is the basis of communication, be accom-plished through an extended, complicated versionof dynamics estimation? It seems that, in the areaof computational communication studies, there isan implicit consensus that the answer to this is‘yes’.

We, however, claim the reverse. This is be-cause the other, who is equivalent to the self, istoo complex to be estimated. An individual, suchas a human being, has a large dimension of inter-nal states and complex dynamics. Moreover, theestimator requires extra dimensions to reconstructthe state space. It is very difficult to estimate all ofthe target’s dynamics — state-transition map f2,output map g2, and the current state x2 — solelyby observing the target from the outside.

Theoretically, it is still possible to estimate thedynamics through a massive amount of obser-vation data. However, the estimator’s parame-ter dimensions need to be far larger than thoseof the target’s internal state. In other words, ifthe estimator’s number of parameter dimensionsis fixed, s/he can only estimate a very simpletarget, whose number of dimensions is substan-tially smaller than the estimator’s. A more com-plex target beyond that can only be approximatedvaguely.

For more advanced estimates, such as the recur-sive ones, it is necessary to estimate the internal

state of the target, who is just as complex as theestimator. It is obvious that the target’s param-eter dimensions exceed any preconceived limitsand cannot be estimated solely through externalobservation.

Even a human being has a very limited capac-ity to predict problems involving the estimation ofdynamics with hidden parameters. It is very un-likely that a capacity beyond this is necessary forcommunication.

3.2 Clues for estimating dynamics

The problem with limited parameter dimensionsis caused by the presumption that all dynamics areto be estimated only from observation. As dis-cussed in Section 2.2, we may be able to solvethis if we have some partial information on thedynamics, such as the state space, transition map,and output map.

One naive solution involves partial informationthat resides innately in the brain. That is, the hu-man brain has a priori mechanism of basic in-formation processing that extracts one’s internalstates from external observation. We know thatthe brain holds innate knowledge for some primi-tive elements, including that for facial expressionsand tones of voice.

However, this solution is insufficient in describ-ing human communication. We learn most ofwhat we know, including self knowledge, afterbirth. It is unlikely that a human being innatelyknows all about the other, and undergoes complexprocessing such as recursive estimation. In fact,psychologists have evidence of age-dependent de-velopment in their ‘theory of mind’ in humanchildren [Kuj97].

We need some description of how dynamics arelearned. In other words, the brain needs somesource for learning data, in addition to observingothers.

6

Dynamics model

Observation

Environment

Action

Internal state (of the self)

Theother

Dynamics model

Observation

Environment

Action

(of the self)Internal state

Estimate of the other's action

Estimate of the other's internal state

(a) Acquire a dynamics model from theself’s own dynamics

(b) Use the acquired dynamics modelto estimate the other’s internal statesand actions

Figure 3: Solution using self

3.3 Solution using self

We propose that the estimator’s brain learns theother’s dynamics from his/her own dynamics(Fig. 3). That is, the self assumes that the otherobeys similar dynamics to the itself. If this istrue, the brain can roughly gauge the other’s inter-nal states. Slight difference can be acquired afterthis.

Three advantages can be gained from this as-sumption.

• It complies with the computational theory ofcommunication, i.e., it enables the internalstate of a peer to be estimated, as s/he isequivalent to the self and so are his/her dy-namics. This is the most natural way of sat-isfying the constraint of equivalency.

• The self can obtain much more informationfrom its own dynamics than the other’s. Infact, a human being can observe his/her owninternal states, such as emotions and desires,as well as his/her externally non-observableinput, such as pain and touch. A dynamicsmodel constructed from this wealth of infor-mation makes it much easier to estimate theinternal states of others.

• We can also reduce state space dimensions.

The brain does not need to reconstruct inter-nal state space from scratch if it uses infor-mation on internal states and transition mapsfrom the self. In addition, knowledge on thestructure of internal state space simplifies thelearning of the output map.

The following example illustrates this withinthe context of a human communication scenario.A person (the self) sees another person (the other)accidentally bumping his hand on a table andbleeding. The self has already learned, from hisown experience, associations between input andthe state change (bumping hand causes pain), andthe associations between this observation and theinternal state (bleeding hand is painful). Using thelearned knowledge, the self estimates the inter-nal state of the other from observing him/her andhis/her environment (S/he bumped his hand andis bleeding, so this is painful). Combined with adecision on what action to take, the self can as-sist the other (apply ointment to the wound), andchange his/her internal state (relieve the pain andmake him/her happy).

Note that one’s dynamics model does not nec-essarily reflect himself/herself perfectly. Weknow that the framework for observing the selfinvolves substantial difficulties with representingits complexity within itself [Ros98]. In this study,

7

however, self-learning is only used to provide the‘seed’ for the dynamics, enabling the other’s in-ternal state to be estimated. Thus, an approximatedynamics model to assess the self’s dynamics issufficient for this purpose.

4 Difficulties with converting ob-jectivity to subjectivity

This section describes another difficulty, convert-ing objective information to subjective informa-tion. The other’s internal state cannot be esti-mated merely through the dynamics model onthe self, because the input for the model, theother’s subjective information, needs to be recon-structed from observed information. However, re-constructing subjective input is in the same classof difficulty as estimating internal states. In thissection, we point out that a naive self-applicationprinciple cannot escape from the necessity of re-constructing subjective input, and propose a solu-tion, a self-observation principle.

4.1 Self-application principle

One method, which naively achieves the pro-posal in the previous section, involves applyingthe self’s action rule to predicting the other’s ac-tions. We name this as self-application princi-ple (Fig. 4). Here, one’s brain projects (copies)his/her own state-transition map f1 and outputmap g1 to the corresponding maps in the dynam-ics model, f and g. After this, the brain uses thedynamics model to estimate the other’s internalstate x and predict the other’s output y.

The problem with this principle lies in the dif-ferent stances between the self and the other. Onecan only observe objective information from theother. (objective information is denoted by an at-tached asterisk, e.g. a∗2). This information is in-compatible with the input of map f , which hasbeen copied from the state-transition map of theself. The compatible and required input a′2 is the

input that the self would have received if s/he hadhad the other’s stance, i.e., information on subjec-tive input. To bridge this gap, the brain requiresa conversion process Va, which converts a∗2 intoa′2.

The same applies for output. Although pre-dicted output for the dynamics model y′2 is sub-jective, we need to know how this matters to theself, i.e. objective information of the output y∗2.Thus a conversion process Vy, which converts y′2to y∗2, is required.

We claim that these processes, which convertobjective information to subjective and vice versa,is the second difficulty with the computationaltheory of communication. Contrary to intuition,these processes are not trivial, because the twotypes of information are in totally different modesof representation. In terms of vision, the other’sviewpoint needs to be obtained from the self’sthrough a complex calculation that involves view-point conversion. Almost all senses of touch andpain needs to be reconstructed from other senses,such as visual and auditory sources.

For example, suppose that the self saw the otheraccidentally bumping his hand against a table.If the self puts himself/herself in the other’s po-sition, s/he could imagine the bump would cre-ate surprise in his emotion. This imagination,‘putting oneself in the other’s position’ is an es-timation by the self-application principle. How-ever, to do that, the self has to be able to con-vert his/her viewpoint into the other’s subjectiveseeing of the table and the hand, and reconstructthe sense of pain from that; otherwise, s/he can-not put himself/herself in the position of the other.Such complex conversion and reconstruction isunlikely to be acquired from only observing oth-ers, let alone be innate knowledge in human be-ings.

In other words, estimating the other’s subjec-tive input is as difficult as estimating the other’sinternal state. The other’s internal state is pri-vate, as is the other’s subjective input; knowledge

8

a1x1

y1

g1

^ff

gg

x

f1f1

Input of the dynamics model is a subjective

information of the self

Projection of the self'smaps provides the dynamics model

Internal state

Dynamics model

^yyy

fa'

(a) Self-projection stage

a1x1

y1

g1

^ff

gg

xx2

f1f1

Only objective inputs and outputscan be observed

Need to convert between observed objective infor-

mation and model-required subjective information

Estimate of the other's

internal state

VaVaa*2 a'2

VyyVyyy*yy*

y*2y

a

a2

y2

(b) Self-application stage

Figure 4: Self-application principle

of private processes, f and g, does not create away of extracting private from public information.We claim that the self-application principle doesnothing more than displace the problem of esti-mating internal state into that of converting objec-tivity to subjectivity, and does not solve the actualproblem.

4.2 Self-observation principle

We propose here a self-observation principle,which resolves the actual problem. This is doneby constructing a dynamics model from the ob-jective observation of the self, and then applyingit to estimate the other’s actions.

Figure 5 illustrates this. The state of the selfx contains a dynamics model f ∗ and g∗. The dy-namics model is different from that of the self-application principle in a point that the dynamicsmodel directly handles information that has beenobjectively observed.

In the model-learning stage (Fig. 5 (a)), modellearner M learns the dynamics of a∗1 and y∗1, whichare objective observations of the self’s input andoutput. Here, objective observation of the selfmeans observing the self through external envi-

ronment, where a similar observation is applica-ble to peers, e.g. sight of hand moving, bumpingsound, and allo-centric (objective space) arrange-ments of the table and hands. In this stage, thelearning is easy because actual internal state x1 ofthe self can be used as teacher data for internalstate x of the model.

Once the self has learned the dynamics model( f ∗, g∗), s/he can apply the model to observingthe other (Fig. 5 (b)). The dynamics model pro-cesses a∗2 (objective observation of the other’s in-put) and produces x (estimate of the other’s inter-nal state) and y∗2 (predicted objective observationof the other’s output).

For example, the self first learns that “bumpingthe hand causes pain” from the his/her own ex-perience. The self’s dynamics model learns fromobjective observation of himself/herself (such asthe sight of the hand moving, a bumping sound,and allocentric arrangements of the table andhand) and their relation to the pain in the self’sinternal state. After observing the other bumpinghis/her hand on the table, the self estimates thats/he feels pain from noticing such things as theother’s hand moving, a bumping sound, and the

9

a1 x1

y1

g1

fff*

gg*^yyyy**

a**1

yy**1xx1

f1f1

The information of thecurrent internal state helps model learning

y

a

M

ff

The input is an objective observation of the self

(a) Model-learning stage

a1x1

y1

g1

xx2

f1f1

An objective obser-vation of the other can

be used as the input

Estimate of theother's internal state

M

a2

y2

^fffff*

^gg*^yyyy***

aa**2

yyy****2

(b) Model-application stage

Figure 5: Self-observation principle

allocentric locations of the table and the hand.Note that these observations have no direct

benefit in determining the self’s actions. To avoidbumping his/her hand again, the self only needsto learn the relations between ego-centric (sub-jective space) arrangements of the table, mus-cle movements of the arm, and the resultingpain. However, such subjective information is notenough to estimate the other’s internal states. Todo this, the self needs to obtain objective obser-vations and relate them to his/her own subjectiveinformation and internal state.

4.3 Discussion

The proposed self-observation principle, which isapparently complicated, is actually simpler thanthe self-application principle. One reason is thereduced elements to be calculated due to eliminat-ing the conversion from objectivity to subjectivityand vice versa, Va and Vy.

Moreover, the idea underlying the self-application principle, “applying the self’s dynam-ics to the other” is implicitly involved in theself-observation principle. The mechanism thattranscripts the self’s dynamics into the dynamics

model is provided as model learner M in the self-observation principle, unlike simple projection inthe self-application principle. From this pointof view, the self-observation principle can be re-garded as a variant of the self-application princi-ple, which solves the problem of reconstructingsubjective information.

Another advantage of the self-observation prin-ciple is its ability to learn the other’s dynamics,using the same model learner M as in the self’sdynamics. it is not straightforward to design thelearner in the self-application principle; when theprediction is wrong, the learner needs to choosewhether map Va or f is to be corrected. That isnot the case with the self-observation principle,where the two maps are joined into f ∗.

4.4 New computational theory

The self-observation principle, which we pro-pose, solves the two difficulties with estimatingthe other’s internal states, i.e., the limited di-mensions of estimator parameters and the conver-sion from objectivity to subjectivity. However,the principle is so general that a number of al-gorithms can satisfy the principle’s requirements.

10

Of course, they have to solve many other practicaldifficulties.

To spur studies on these algorithms, we can in-tegrate the proposed principle into a new compu-tational theory. Although the existing computa-tional theory (Section 2.1) was so incomplete thatwe could hardly find any algorithm that could dealwith the difficulties, our new computational the-ory enables us to study new algorithms by clari-fying difficulties and approaches. These new al-gorithms can lead to the development of a newartificial intelligence as well as hypotheses on thefunction of the brain.

Consequently, we propose a new computa-tional theory here, which integrates the self-observation principle. The theory involves thefollowing, in addition to the original theory inSection 2.1:

4. Construct a dynamics model, which is ap-plicable to the other, through objective ob-servation of the estimator himself/herself, toachieve (3) (estimation of the other’s internalstates).

Of course, we do not deny the possibility ofan alternate computational theory of communica-tion. There may be another way to solve the lim-its of estimator dimensions, and the original com-putational theory ‘to estimate the other’s internalstate’, which we have used as a basis, may needsome modifications. We hope that our proposedprinciple will lead us to a deeper understandingof the computational studies of communication,including alternate theories.

5 Related studies

The self-observation principle is very simple butrelated to various research domains. This sectiondescribes the relations.

Sensory field

Internal state

Motor field

Action decision

Sense Action

Sense

Observation of the self

Observation of the other

Action

Sensory field Motor field

Dynamics model

Action decision

(a) Rizzolatti’s hypothesis (b) Hypothesis derived from[RA98] the self-observation principle

Figure 6: Comparison of hypotheses on role ofmirror neurons. Filled circles denote mirror neu-rons.

5.1 Psychology

The idea of “self-observation for a tool to esti-mate the other’s internal states” was proposed in1970s by an evolutional psychologist, NicholasHumphrey [Hum78, Hum84]. However, he madea purely theoretical hypothesis, which describedthe origin of self-consciousness through a Dar-winian process of evolution. His specificationsfor which part of the self should be observed weresomewhat vague.

Our study, on the other hand, can be regardedas an improvement to Humphrey’s theory for thefollowing reasons. Through formalizing the com-munication process, we stated the effectiveness ofthe self-observing principle in solving the diffi-culties with communication. We also found thatobservation needed to create the associations be-tween the self’s subjective state and objective ob-servation. Moreover, by defining the principle asa new computational theory, our study can be saidto migrate Humphrey’s theory into new scientificdomains, such as neuroscience and artificial intel-ligence.

Baron-Cohen, in his book on autism and thetheory of mind [BC95], claimed that Humphrey’shypothesis corresponds to the self-application

11

principle, which contradicts the dissociation ofsymptoms in autism patients. He also brieflysuggested that a different principle based on in-trospection (which resembles the self-observationprinciple) will match this dissociation. Al-though Humphrey’s hypothesis covers both self-application and self-observation principles, it isinteresting to find a study of autism that sup-ports the advantage of the self-observation prin-ciple over the self-application principle.

5.2 Computational neuroscience

Most communication studies on computationalneuroscience have not matched with the self-observation principle. Kawato et al. discussedthe communication model as an extension oftheir model of dynamics interaction [KDH01],but their model implicitly assumed identical rep-resentation for both subjective and objective in-formation.

To the best of our knowledge, no study hasinvolved the construction of a model based onHumphrey’s hypothesis or a similar framework.This would be because the requirements and theprinciples to estimate peers’ internal states havenot been declared as a computational theory. Wehope that our proposal promotes future studies ofcommunication in computational neuroscience.

Within a different context from communica-tion, Tani proposed a constructivist approach forthe study of the self and consciousness [Tan98].However, in this study, an individual was placedalone in an environment, and it did not considerinteractions between individuals, let alone esti-mates of the other’s internal state.

5.3 Experimental neuroscience

Based on the self-observation principle, we cansuggest a new role for mirror neurons [DFF+92,GFFR96]. A mirror neuron is a neuron that be-comes active during a certain action (e.g. grasp-ing an object) as well as when observing some-

one else performing the same action. The mirrorneurons are in the premotor area of the monkeybrain, and are supposed to be in Broca’s area ofthe human brain [RFGF96]. They are claimed tobe related to our ability to communicate throughlanguage, but details are unknown.

Rizzolatti et al. proposed a hypothesis wherea mirror neuron serves as a commander for anaction as well as a recognizer of that action(Fig. 6(a)). However, the mechanism of associ-ation is unknown. In addition, the response of amirror neuron to action being observed is claimedto be usually suppressed, without any reasonablediscussions on suppression mechanisms.

In contrast, the self-observation principle sug-gests the role of mirror neuron outlined in Fig. 6(b). The process in Section 4.2 described the de-velopment of mirror neurons very well. A neuronin the dynamics model learns the self’s actions,and after this, applies them to the other’s actions;as a result, the neuron begins to act as a mirrorneuron. Moreover, such a neuron does not requireany suppression mechanism because it does notdirectly trigger any action.

Oztop and Arbib [OA02] offered a hypothe-sis that the basic functionality of a grasping mir-ror system is to elaborate the appropriate feed-back for opposition-space-based control when anobject is manually grasped. They claim that,given this functionality, understanding of actionin the mirror system may be seen as an exaptationgained by generalizing from one’s own hand toanother’s hand. Since this hypothesis matches theself-observation principle, we suggest that such afeedback mechanism is also the evolutionary ori-gin of the self-observation mechanism in humancommunication.

5.4 Artificial intelligence

Existing strategic algorithms, e.g. chess-playingprograms, are based on being able to the predictthe opponent’s actions. Most existing algorithms,

12

such as the alpha-beta algorithm [Ish89], assumethat the opponent will evaluate and selects an ac-tion in the same way as the algorithm does. Thiscan be regarded as a sort of the self-applicationprinciple. The self-application principle is ap-propriate to this sort of strategic algorithm be-cause the rule symmetry makes it easy to designthe viewpoint translator. However, it is possibleto design a new strategic algorithm based on theself-observation principle. A program based onsuch an algorithm would make it possible to learnthe opponent’s characteristics through successivegames and adapt strategy.

6 Conclusion

We investigated existing computational theory oncommunication, i.e. the estimates of peers’ in-ternal states. We found that estimates are posedwith two difficulties, the limits of the estimator’sparameter dimension and conversion from objec-tivity to subjectivity. Our proposal for solvingthese difficulties was the self-observation princi-ple: one observes oneself objectively to establisha dynamics model, which is then applied to oth-ers. Since the dynamics model learns the asso-ciation between one’s internal state and objectiveself observation, one can use the model to esti-mate the internal states of others by observingthem objectively. Through clarifying the targetand purpose of the learning process and integrat-ing the self-observation principle into computa-tional theory, this study has opened the way inenabling communication to be studied construc-tively. We also discussed the relation our proposalhas with other research domains, including self-consciousness and a mirror neuron system.

References

[AHP+00] C. G. Atkeson, J. Hale, F. Pollick,M. Riley, S. Kotosaka, S. Schaal,

T. Shibata, G. Tevatia, S. Vijayaku-mar, A. Ude, and M. Kawato. Us-ing humanoid robots to study humanbehavior. IEEE Intelligent Systems:Special Issue on Humanoid Robotics,15:46–56, 2000.

[AIYK00] Kazuyuki Aihara, Tooru Ikeguchi,Taishi Yamada, and Motomasa Ko-muro. Fundamentals and Applica-tions of Chaos Time-Series Analysis.Sangyo Tosho, 2000. In Japanese.

[BC95] S. Baron-Cohen. Mindblindness: AnEssay on Autism and Theory of Mind.MIT Press, 1995.

[DFF+92] G. Dipellegrino, L. Fadiga, L. Fo-gassi, V. Gallese, and G. Rizzolatti.Understanding motor events - a neu-rophysiological study. ExperimentalBrain Research, 91:176–180, 1992.

[GFFR96] V. Gallese, L. Fadiga, L. Fogassi, andG. Rizzolatti. Action recognition inthe premotor cortex. Brain, 119:593–609, 1996.

[Hum78] Nicholas Humphrey. Nature’s psy-chologists. New Scientist, pages 900–904, June 1978.

[Hum84] Nicholas Humphrey. The inner eye:Social intelligence in evolution. Faberand Faber, 1984.

[IOIK03] H. Ishiguro, T. Ono, M. Imai, andT. Kanda. Development of an in-teractive humanoid robot “Robovie”– an interdisciplinary approach. InR. A. Jarvis and A. Zelinsky, editors,Robotics Research, pages 179–191.Springer, 2003.

[Ish89] Kiyoshi Ishihata. Algorithm and DataStructure: Iwanami Lecture on Soft-

13

ware Science 3. Iwanami Shoten,1989. In Japanese.

[Kaw96] Mitsuo Kawato. Computational The-ory of the Brain. Sangyo Tosho, 1996.In Japanese.

[KDH01] Mitsuo Kawato, Kenji Doya, andMasahiko Haruno. Extension of MO-SAIC and communication: Computa-tional neuroscience of human intelli-gence #5. Kagaku, 71:197–204,839–843, 2001. In Japanese.

[Kuj97] Takashi Kujiraoka. Aspects in Primi-tive Communication. Minerva Shobo,1997. In Japanese.

[MA03] Takaki Makino and Kazuyuki Ai-hara. Self-observation principle forestimating the other’s internal state.Mathematical Engineering TechnicalReports METR 2003–36, Departmentof Mathematical Informatics, Gradu-ate School of Information Science andTechnology, the University of Tokyo,October 2003.

[Mar82] David Marr. Vision: A ComputationalInvestigation into the Human Repre-sentation and Processing of Visual In-formation. W. H. Freeman, San Fran-cisco, 1982.

[OA02] E. Oztop and M. A. Arbib. Schemadesign and implementation of thegrasp-related mirror neuron system.Biological Cybernetics, 87:116–140,2002.

[PW78] D. G. Premack and G. Woodruff.Does the chimpanzee have a theoryof mind? Behavioral and Brain Sci-ences, 1:515–526, 1978.

[RA98] Giacomo Rizzolatti and Michael A.Arbib. Language within our grasp.

Trends in Neuroscience, 21:188–194,1998.

[RFGF96] G. Rizzolatti, L. Fadiga, V. Gallese,and L. Fogassi. Premotor cortex andthe recognition of motor actions. Cog-nitive Brain Research, 3:131–141,1996.

[Ros98] Otto E. Rossler. Endophysics: theworld as an interface. World Scien-tific Publishing, 1998.

[Tan98] Jun Tani. An interpretation ofthe ‘self’ from the dynamical sys-tems perspective: a constructivist ap-proach. Journal of ConsciousnessStudies, 5(5-6):516–542, 1998.

[WK98] D. M. Wolpert and M. Kawato. Multi-ple paired forward and inverse modelsfor motor control. Neural Networks,11:1317–1329, 1998.

14