cognitive infocommunications

8/15/2019 Cognitive Infocommunications

1/229

Péter Baranyi · Adam CsapoGyula Sallai

CognitiveInfocommunications(CogInfoCom)


2/229

Cognitive Infocommunications (CogInfoCom)


3/229


4/229

CognitiveInfocommunications(CogInfoCom)

1 3

Péter Baranyi • Adam Csapo • Gyula Sallai


5/229

Széchenyi István UniversityGy ˝ or, Hungary

Institute for Computer Science

and Control of the HungarianAcademy of Sciences

Budapest, Hungary

Gyula SallaiBudapest University of Technology

and EconomicsHungary

Future Internet ResearchCoordination Centre

University of DebrecenHungary

Institute for Computer Scienceand Control of the HungarianAcademy of Sciences

Budapest, HungarySzéchenyi István UniversityGy ˝ or, Hungary

ISBN 978-3-319-19607-7 ISBN 978-3-319-19608-4 (eBook)DOI 10.1007/978-3-319-19608-4

Library of Congress Control Number: 2015953263

Mathematics Subject Classification (2000): 01–01, 04–01, 11Axx, 26–01

Springer Cham Heidelberg New York Dordrecht London© Springer International Publishing Switzerland 2015This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,broadcasting, reproduction on microfilms or in any other physical way, and transmission or informationstorage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodologynow known or hereafter developed.The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoes not imply, even in the absence of a specific statement, that such names are exempt from the relevantprotective laws and regulations and therefore free for general use.The publisher, the authors and the editors are safe to assume that the advice and information in this bookare believed to be true and accurate at the date of publication. Neither the publisher nor the authors orthe editors give a warranty, express or implied, with respect to the material contained herein or for any

errors or omissions that may have been made.

Printed on acid-free paper

Springer International Publishing AG Switzerland is part of Springer Science+Business Media (www.springer.com)

Péter Baranyi Adam Csapo

http://www.springer.com/http://www.springer.com/http://www.springer.com/http://www.springer.com/http://www.springer.com/http://www.springer.com/


6/229

Preface

In the past decades, the role and value of information has significantly increased innearly all aspects of daily life. Information besides matter and energy has becomea primary conceptual component of human thinking, as evidenced by the growingappearance of the word “information” in a widening range of contexts. Alongsidethe factories, power plants and relevant industries which provide, store, processand transport matter and energy, a system of infocommunication services providinganalogous functionalities with respect to information now pervades our everydayexperience. As a result, access to infocommunications has become a basic need, in

much the same way as the ability to access electric power through outlets in walls istaken for granted. In fact, most services pertaining to matter, energy and informationare co-dependent at various levels, with each service relying on others for its ownsustained efficiency.

At the same time, it is important to realize that while services relevant to matterand energy primarily stem from and serve physical needs, infocommunicationsis motivated by and serves high-level mental activities, which are supervenienton human knowledge and intelligence. Given that human mental capabilities aregenerally more flexible to adaptation than material-energetic capabilities governed

purely by physical-biological reality, new patterns, modes and structures areconstantly appearing in the ways we interact with information. Coupled with recentadvances in technology, this inventivenessin means of interaction is culminating in aprocess of human entanglement with information and communication technologies(ICT). This process has been identified and described by many authors, and canbe observed from a number of different perspectives relevant at different scalesof time, including the key measures used to characterize the performance of newtechnologies, the multiple (physical) levels at which interaction can occur , andthe increasing relevance of what we refer to as “human-oriented fields” in the

development of new technologies.With respect to the ways in which the performance of new technology is char-acterized, it can be observed—at least in the case of personal informatics devicesand from the perspective of end users—that measures of hardware performanceare losing importance in comparison with high-level (soft) functional capabilities.

v


7/229

vi Preface

While 20 years ago the specifications of personal computers would have includeddetails on power consumption, processor clock frequency, cache size and otherlow-level hardware details, later the capacity of random access memory, as wellas networking and graphical capabilities gained relative importance. Today as endusers of personal informatics devices, we rarely take note of such specifications,despite their still being accessible. Instead, we take for granted that a new devicewill include capabilities for the most common communication protocols, and willinclude a camera, for instance, that produces images of decent quality for mostpractical purposes. Even our focus on battery life is much more oriented towardscapability than towards raw measures of hardware performance: if the batterylife of a device can last for a full working day, most practical requirements forfunctionality will be satisfied. As this tendency continues, the specifications of personal informatics devices most important to us will soon be—and already are

to some extent—those that focus on high-level capabilities built into and around theoperating systems running on them, such as the richness of associated applicationstores, the various capabilities of built-in personal assistants such as Siri, GoogleNow and Cortana (including capabilities for generating and understanding speech,text, gestures and other emotional behaviours), or capabilities for handling multipleor multi-user profiles for family use. In the end, our personal informatics devicescan be expected to evolve into tools that merely provide an “intelligent window”into a ubiquitously and collaboratively accessible augmented world of personalizedinformation services—a fact that yields significant motivations, and important

support for key notions behind research areas such as speechability, ethology-based CogInfoCom and mathability, as discussed extensively in the book. The capabilitiesof future devices will be judged based on the extent to which they are motivated byand fulfill the goals of such branches of CogInfoCom, and once this is the case, it canbe said that these are the research areas which supply the most relevant parametersrequiring specification.

With respect to the multiple levels at which interaction can occur (i.e. in terms of the second perspective behind entanglement listed earlier), it can be observed thatthe phenomenon of entanglement is gaining strength at levels of communication

ranging from the low level of cell-electronics connections (as in, e.g., brain-computer interfaces and some areas of bionics) through connections expressedat the representational, structural or command level (involving, e.g., multimodalinteraction), all the way to high-level collective relationships among populations of humans and ICT (involving e.g. high-level services based on data aggregation andanalysis). These three levels and their impacts are discussed extensively in the book,in relation to research areas such as CogInfoCom channels and socio-cognitive ICT .The point is also made that as the three levels of interaction gain strength, they canbe expected to influence each other in as yet unforeseen ways.

Finally, with respect to the increasing relevance of “human-oriented fields” innew technologies (i.e. in terms of the third perspective behind entanglement listedearlier), it can be observed that as our interaction with information evolves, boththe theory and practice of infocommunications are increasingly relying on resultsfrom fields of scientific inquiry motivated by the goal of better understanding how


8/229

Preface vii

humans think, feel and act. In particular, all fields dealing with high-level aspectsof human cognition, including even the humanities, can ultimately be expected tomake important contributions. It is true that this is a slower process, characterizedby a larger time constant than the processes of change behind functional capabilitiesand levels of interaction; nevertheless, it is important to note that the effects of such“human-oriented fields” can already be observed in the plethora of new researchdirections emerging within ICT. For example, the Future Internet concept that isactively supported by the EU has several branches that are strongly relevant to thisphenomenon, such as the Internet of Things (IoT) and 3D Internet . In IoT, the goalis to deploy as many sensors and actuators as possible into the surrounding physicalnetwork, enabling the measurement and actuation of previously inaccessible parts of physical-biological-mental reality. Irrespective of how the data collected in this wayis represented, it will serve as a kind of informational, or virtual image of everything

in the world that is important to us. Through the concept of 3D Internet, researchersare seeking ways to attribute 3D geometry to this growing collection of data—including both data that is naturally and directly amenable to 3D representationand abstract data sets requiring “virtual” visualization—in order to support ourcapabilities for interpretation and manipulation. The increasingly popular domain of Big Data also represents an initiative towards processing IoT data, albeit primarily inmore automated, less human-interpretable ways. A common effect of these researchareas is that a new infrastructure is emerging, which can be seen as a collectionof artificial “sensory organs” that extend across the globe and are capable of

extracting different kinds of information in a way that integrates humans into thepicture. It can be expected that this human integration will eventually bring aboutnew forms of entanglement, which extend through periods of time that are muchlonger than the relatively episodic interactions imagined today. New generationsare already growing up in a kind of “co-evolution” with ICT, as evidenced bythe growing number of psychological effects that have been documented withrespect to changing cognitive-social-technological relationships. In the book, theterm “cognitive entity” is introduced to characterize such entangled relationshipsin terms of cognitive capabilities. Further, the term “generation CE” (generation

of cognitive entities) is introduced by analogy with the generations X, Y and Z,to reflect the fact that members of today’s generation are growing up in this newworld starting from the first phases of their personal and social development, andthat through their entangled co-evolution with ICT, they can be seen to embodyan entirely new set of cognitive capabilities. The general conclusion is that if researchers would like to record, explain and understand the diverse phenomenasurrounding human-ICT entanglement and newly emerging cognitive entities, aswell as build new technologies based on those phenomena, then having recourse toresults from relevant “human-oriented” research fields will be essential, in much the

same way as the development of personal informatics devices today involves jointefforts from information and electrical engineers as well as experts of psychologyand ergonomics.

The goal of the emerging field of cognitive infocommunications (CogIn-

foCom) is to support the investigation of human-ICT entanglement related


9/229

viii Preface

phenomena as well as how such phenomena lead to new cognitive entities and

the appearance of the generation CE. The field places special emphasis on thequalification and quantification of cognitive capabilities, and aims to adopt a multi-and interdisciplinary approach in doing so. The distinction between multi- and inter-disciplinarity is an important one: while the formulation of the scope and goals of the field encourages the involvement of a large variety of pertaining disciplines, italso aims towards an integration of the perspectives and methodologies which theyrepresent. Thus, besides aiming to create a common language, it is also the goalof CogInfoCom to establish a common way of thinking about relevant problems,and a common approach towards designing new technologies. This is not withoutchallenges. The engineering sciences—and by extension, infocommunications—generally emphasize the synthesis of novel systems so as to provide functionalitiesthat are guaranteed to be stable and reliable in at least some respect. Crucially, even

the modeling of component structures and phenomena—serving as a prerequisite toengineering design—proceeds with these ultimate goals in mind. Considerationsin engineering often do not extend to the requirement of holistic completeness;instead, specific interpretations are focused on that are relevant to specific designgoals. In contrast, the primary focus of most branches of the cognitive sciencesis analytical in nature, with the goal of recording, understanding and explainingcognitive phenomena in a way that is phenomenologically complete. However, suchanalyses do not necessarily lead to functional reproducibility. As a result, it is oftenthe case that a set of experiments and models separately deemed precise from the

perspective of an engineering field are seen as inadequate from the perspective of a cognitive science-related field, and vice versa. If a personal computer from ourcentury were to have suddenly appeared functionally complete 100 years ago, surelyits operation would have been described from a different set of perspectives thanthose descriptions that are accepted as blueprints for their production today. Theformer set of descriptions would also likely have been inadequate for the design of a fully functional computer. From the perspective of CogInfoCom, the key point isthat while investigations in the cognitive sciences focus on a biological structure thathas been highly differentiated through the complex processes of evolution, and that

as a result cannot easily be “copied” or “reproduced” using today’s engineeringconcepts, it is nevertheless possible to obtain new knowledge by attempting toextend human cognitive capabilities through the synthesized design of technologiesin infocommunications. This latter goal in turn necessitates a broadening in scopeof the human-oriented methodologies used in the cognitive sciences to address(artificial) cognitive phenomena relevant to artificial systems.

It can often be observed that well-established research fields become morespecialized, or perhaps show increasing signs of segmentation, while specificsegments meet with previously distinct research fields to form completely new

synergies. This process can be likened to the roots of a tree, which meet at variouspoints to create the trunk, which in turn gives distinction to different branches, whichthen meet with the foliages of neighbouring trees. In this process, some researcherschoose to maintain focus and achieve completeness in their own specializedresearch, while others instead are in constant search of new possibilities for synergy


10/229

Preface ix

with other fields. Although seemingly there may be a contradiction between thetwo approaches, in fact both of them together are necessary for the emergenceof valuable new research directions. So it is the case with CogInfoCom: one canfind in it the influence of well-established research fields (e.g. augmented cognition,human-computer interaction, virtual reality and others), while at the same time newdirections and specializations are also appearing due to the unique viewpoint of thefield, as discussed in several chapters of this book. It is important to emphasizethat these new specializations born under CogInfoCom are often difficult, if notimpossible, to categorize into already existing fields. Interesting and novel changesin viewpoint are often possible by considering various phenomena from tighter, orbroader perspectives. All of these tendencies can be observed in parallel at the IEEECogInfoCom conferences series. In some cases, sessions with well-specified topicshave included presentations of researchers coming from widely different areas, such

as information, medical or cognitive sciences, which has led to valuable exchangeswith respect to a focused topic. In other cases, sessions named after completelynew research directions appeared, and have included presentations which aimedto establish novel perspectives. This heterogeneity can also be observed in thefields represented by researchers who have co-authored papers at conferences andspecial issues on CogInfoCom; for example, joint papers were published togetherby researchers with backgrounds in computer science, systems and control theory,mathematics, ethology and cognitive psychology.

As in the case of all newly established conference series, the CogInfoCom con-

ferences have been attended by both supporters and sceptics of the motivations andideas behind the field. At the 3rd IEEE International Conference on CogInfoCom,it happened that a professor dealing with human-computer interaction voiced hiscriticism, saying that “There is nothing new in CogInfoCom: : : it is just another name for what I am working on”. Later at the same event, another scientist fromthe field of computational linguistics made almost the same remark, expressinghis view that CogInfoCom was just another label given to the already existingfield he was dealing with. As the same criticism was later raised with respect tocognitive informatics and neurocomputing as well, it is worth considering what

these criticisms would entail, if accepted as true:

CogInfoCom D HCI

CogInfoCom D computational linguistics

CogInfoCom D cognitive informatics

CogInfoCom D neurocomputing

Were these equalities to be accepted as true, clearly they would also suggest

relationshipsof equivalence among their right-hand sides—something that has neverbeen claimed by any serious researcher acquainted with the scope and goals of thesefields:


11/229

x Preface

HCI D computational linguistics

computational linguistics D cognitive informatics

cognitive informatics D neurocomputing

HCI D cognitive informatics

: : :

No researcher has ever suggested that cognitive informatics is just another name forHCI, or that computational linguistics is actually the same as any of these fields.These facts go well to demonstrate the multi-disciplinary nature of CogInfoCom:the criticisms of novelty were raised by researchers who were all able to discoverthe motivations for their own work behind it, and yet, they all came to the same

conference representing different fields, never before seen to have much in common!However, in addition to having this strongly multi-disciplinary background, theCogInfoCom conferences have also led to the formulation of a wide range of newresearch perspectives and initiatives, which, as outlined in this book, cannot betrivially broken down into the contributions of previously existing individual fields.Thus, the emergence of CogInfoCom has shown that the time is ripe for all of thecognitive + social + technological phenomena discussed above to be investigatedfrom a unified perspective. For these reasons, we respectfully ask readers to lookfurther than the ways in which their own research perspectives are represented in the

book, and to support the development of CogInfoCom by actively looking for waysto approach this new and common perspective. In parallel to the process throughwhich humans and ICT are merging together and becoming increasingly entangled,we encourage readers to help develop both our analytical and synthesis-orientedunderstanding of newly emerging cognitive entities.

Writing a book that does justice to the influence provided by the large numberof fields behind CogInfoCom and also adequately represents the new perspectivesit has brought about is a significant challenge. No attempt at meeting this challengewould have been possible without the invaluable suggestions and contributions

of many people who have influenced our thinking about CogInfoCom over theyears. Although providing a complete list of mentors, colleagues and studentswho have made such contributions would be impossible to compile, we wouldlike to express our gratitude to Nick Campbell, Tom Gedeon, Hideki Hashimoto,Kristiina Jokinen, Toshikazu Kato, Tetsuo Kotoku, Joo-Hoo Lee, Gabor Magyar,Helen Meng, Adam Miklosi, Geza Nemeth, Mihoko Niitsuma and Csaba Plehfor their active involvement in initiating a common conceptual framework forfurther discussions at the very beginning of the CogInfoCom conference series.We would also like to thank Hassan Charaf, Valeria Csepe, Anna Esposito, Peter

Foldesi, Hamido Fujita, Attila Gilanyi, Andras Hajdu, Karoly Hercegfi, ThomasHermann, Ferenc Honbolygo, Laszlo Hunyadi, Lajos Izso, Joni Jamsa, Zsolt Janko,Anita Komlodi, Maria Koutsombogera, Laszlo Kovacs, Szilveszter Kovacs, MikaLuimula, Jean-Claude Martin, Frederic Noel, Harris Papageorgiou, Sakari Pieska,Claudiu Pozna, Radu-Emil Precup, Laura Slaughter, Bjorn Solvang, Wei Deng


12/229

Preface xi

Solvang, Peter Varlaki, Gabor Vattay, Klara Vicsi, Alessandro Vinciarelli, CarlVogel and Yeung Yam, who have significantly contributed to the success of theconference series by helping to organize sessions and tracks that were oftenpioneering in their subject matter, and also by supporting through their work theappearance of several special issues on CogInfoCom in international scientific

journals. We thank the entire CogInfoCom community for helping to create afriendly and scientifically stimulating atmosphere, as well as for bringing theirvaluable perspectives into the fruitful discussions held during and often—through e-mail and Skype—between the CogInfoCom conferences. Last but not least, specialthanks is due to Anna Szemereki for her Herculean efforts in the organization of every last detail of the CogInfoCom conferences year after year.

April 2015 Adam CsapoGyula Sallai

Budapest, Hungary Péter Baranyi


13/229


14/229

Contents

Part I Foundations of CogInfoCom

1 Emergence of CogInfoCom in Science and Technology . .. .. .. .. .. .. .. . 31.1 What Is CogInfoCom? . . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. . 41.2 Scientific Priming Effects . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . . 5

1.2.1 Convergence from a General Perspective . .. .. .. .. .. .. .. .. 51.2.2 Convergence from an Infocommunications Perspective .. 7

1.3 Technological Priming Effects. . .. . .. . .. . .. . .. . .. .. . .. . .. . .. . .. . .. . .. 8

2 Definitions, Concepts and Assumptions . . . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 132.1 Defining CogInfoCom . . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. . 132.2 Concepts Emerging from CogInfoCom . . . . . .. . . .. . . . . .. . . . . .. . . . . .. 14

2.2.1 Mode of Communication. .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 142.2.2 Type of Communication. . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 15

2.3 Implicit and Explicit Assumptions . . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . .. 162.3.1 Levels of Merging and Entanglement . . . . .. . . . . .. . . . . .. . . . 162.3.2 Levels of Cognitive Capability . . . . . .. . . .. . . . . .. . . . . .. . . . .. . 182.3.3 An Emergent Concept of Information . . . . .. . . . . .. . . . . .. . . . 21

2.3.4 Transitions from Operation to Functionality . . .. . .. . .. . .. . 213 ICT-Based Foundations of CogInfoCom . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. 23

3.1 The Digital Convergence.. . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 233.2 From Telecommunications to Cognitive Infocommunications .. .. . 25

3.2.1 Digitization of Separate Communication Sectors . .. . . .. . . 263.2.2 Unification of Telecommunications:

Electronic Communications . . . . . . . . . .. . . .. . . . . .. . . . . .. . . . .. 273.2.3 Expansion of Telecommunications: Infocommunications 283.2.4 Expanding Content Space: Cognitive

Infocommunications . . . . . . . . . . . . . . . . . . .. . . .. . . . . .. . . . . .. . . . . 313.3 ICT, TIM, DES and CogInfoCom . . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. 32

xiii


15/229

xiv Contents

Part II Research Directions Under CogInfoCom

4 Synergies Between CogInfoCom and Other Fields . . . . .. . . . . .. . . . . .. . . . 394.1 Affective Computing . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . . 40

4.2 Augmented Cognition. . . .. . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 414.3 Body Area Networks . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 434.4 Brain-Computer Interfaces. . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . 444.5 Cognitive Informatics . . . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. . 454.6 Cognitive Networks for Future Internet . . . .. . . . .. . . . . .. . . . . .. . . . . .. . 454.7 Human-Computer Interaction and Multimodal Interaction . .. .. .. . 474.8 Sensory Substitution . . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 504.9 Social Signal Processing . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. 514.10 Virtual and Augmented Avatars . . . . . .. . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . 52

4.11 Virtual and Augmented Reality. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 545 CogInfoCom-Driven Research Areas. . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 57

5.1 CogInfoCom Channels . . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. . 575.2 Speechability . . . . . .. . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . . 595.3 Socio-Cognitive ICT . . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 605.4 Common Features and Techniques . . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . .. 62

5.4.1 Icon-Based Approaches and Applications. .. . .. . .. . .. . .. . . 635.4.2 Context-Based Approaches and Applications . .. . .. . .. . .. . 665.4.3 Pattern-Based Approaches and Applications . .. . .. . .. . .. . . 69

5.5 A Summary of Common Concepts . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. 716 Novel Research Initiatives . . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 73

6.1 The Virtual Collaboration Arena . . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. . 746.1.1 Affordances for Multi-user Collaboration .. . .. . .. . .. . .. . .. 756.1.2 Synergies Supported by the VirCA Platform . .. . .. . .. . .. . . 76

6.2 Augmented Virtual Reality Mediated Neuroscience Research.. .. . 776.3 Ergonomics of Augmented Virtual Collaboration . . .. . .. . .. . .. . .. . . 78

6.3.1 Adaptation of Rasmussen and Reason’sFramework of Human Errors to VirCA. . . . .. . .. . .. . .. . .. . . 79

6.3.2 Types of Rule-Based Human Errors Identified . .. . .. . .. . .. 796.4 Ethologically Informed CogInfocom (EtoCom) . . .. . .. . .. . .. . .. . .. . 80

6.4.1 CogInfoCom Tanglefaces and Engines . . .. . . . . .. . . . . .. . . . . 816.4.2 Examples: The EtoMotor and Its Applications. .. . .. . .. . .. 82

6.5 CogInfoCom-Aided Industrial Engineering. . .. . . .. . . . . .. . . . . .. . . . .. 916.5.1 Design Issues Relevant to Cognitive Networks . . . . . . . .. . . 926.5.2 Industrial Robotics and Production Management . .. . . .. . . 926.5.3 Industrial Capabilities of VirCA . . . . . .. . . . .. . . . . .. . . . . .. . . . 93

6.6 Mathability . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . .. . . . . .. . . . . .. . . . . .. 95

6.6.1 Granular and Analytical Representations. .. . .. . .. . .. . .. . .. 976.6.2 Example Application: Human-Aided

Resolution of Linear Functional Equations . . .. . .. . .. . .. . . 98


16/229

Contents xv

Part III CogInfoCom Channels

7 Structure of Sensory Signals: Icons and Messages. . . . . . . . . . . . . . . . . . . . . . 1057.1 Motivations Behind CogInfoCom Channels . . . . . . . . . . . . . . . . . . . . . . . . 105

7.2 The Need for a Unified View of Communication:Overlaps, Ambiguities, and Contradictions . . . . . . . . . . . . . . . . . . . . . . . . . 1077.3 Unified Structure of Sensory Signals: Streams,

Icons and Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1097.3.1 CogInfoCom Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1107.3.2 CogInfoCom Icons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1117.3.3 CogInfoCom Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

7.4 Icons and Messages: A Summary of Key Points . . . . . . . . . . . . . . . . . . . 116

8 CogInfoCom Channels: Perceptual Unification of

Representation and Meaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1198.1 CogInfoCom Channels and Relevant Concepts. . . . . . . . . . . . . . . . . . . . . 120

8.1.1 CogInfoCom Message Generated Perceptual Concepts . . 1208.1.2 Definition of CogInfoCom Channels . . . . . . . . . . . . . . . . . . . . . . 1218.1.3 A Novel Interpretation of the Concept of Modality . . . . . . . 122

8.2 Formal Representation of CogInfoCom Channel Parameters . . . . . . 1238.2.1 Nomenclature and Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1238.2.2 Mathematical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1258.2.3 Generative Model of CogInfoCom Channels . . . . . . . . . . . . . 126

8.3 Design Challenges Behind CogInfoCom Channels:A Summary of Key Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

9 Mapping Meaning onto CogInfoCom Channels . . . . . . . . . . . . . . . . . . . . . . . . 1299.1 Concept Algebra Based Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

9.1.1 Key Definitions in the OAR Model . . . . . . . . . . . . . . . . . . . . . . . . 1309.1.2 Message Generated Perceptual Concepts

in Terms of OAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1319.1.3 Tools for Analogy: Interactive Concepts . . . . . . . . . . . . . . . . . . 134

9.2 Concept Mapping Techniques as Inference in OAR . . . . . . . . . . . . . . . . 1349.2.1 Direct Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1359.2.2 Analogy-Based Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1369.2.3 A Note on the Joint Use of Mapping Techniques . . . . . . . . . 138

9.3 Future Perspectives for Automated Mapping . . . . . . . . . . . . . . . . . . . . . . . 139

10 Tunability of CogInfoCom Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14110.1 Preliminary Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14110.2 Mathematical Background . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 144

10.2.1 Tensor Product Form of Multivariate Functions . . . . . . . . . . 144

10.2.2 HOSVD-Based Canonical Form of Discretized Multivariate Functions . . . . . . . . . . . . . . . . . . . . . . . . 14510.3 Generic Tuning Model for CogInfoCom Channels . . . . . . . . . . . . . . . . . 14610.4 The Spiral Discovery Method (SDM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147


17/229

xvi Contents

10.5 Implementation of SDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15110.6 Usability of SDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

11 Temporal Aspects of CogInfoCom Channel Design . . . . . . . . . . . . . . . . . . . . 157

11.1 Challenges Behind Temporal Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15711.2 Aspects of Low-Level Biological Communication . . . . . . . . . . . . . . . . . 15911.2.1 CogInfoCom Cues and Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16011.2.2 Cues, Signals and Messages in Past Research . . . . . . . . . . . . 162

11.3 Aspects of High-Level Human Communication . . . . . . . . . . . . . . . . . . . . 16211.3.1 Volition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16411.3.2 Directness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16511.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

11.4 Putting It All Together: Progressive CogInfoCom Channels . . . . . . . 166

Part IV Future Perspectives of CogInfoCom

12 Cognitive Capabilities in the Future Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . 17312.1 Challenges of Today’s Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17312.2 Future Internet Visions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17512.3 Recent Future Internet Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17812.4 Clustering Future Internet Research Themes . . . . . . . . . . . . . . . . . . . . . . . 179

12.4.1 Cluster 1: Internet Basic Research (Internet Science) . . . . 18112.4.2 Cluster 2: Future Internet Modeling, Analysis and Design 181

12.4.3 Cluster 3: Future Internet Network Architectures. . . . . . . . . 18112.4.4 Cluster 4: Data and Content Technologies . . . . . . . . . . . . . . . . 18212.4.5 Cluster 5: 3D Internet and Cognitive Infocommunications 18312.4.6 Cluster 6: Internet of Things (IoT). . . . . . . . . . . . . . . . . . . . . . . . . 18312.4.7 Cluster 7: Cyber-Physical Systems and Applications. . . . . 18412.4.8 Cluster 8: Future Internet Based Community

Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 18412.4.9 Cluster 9: Experimentation, Standardization, Regulation 184

13 Towards a Cognitive Internet Ecosystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18713.1 Reshaping Digital Ecosystems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18713.2 A Long-Term Vision: Cognitive Internet Ecosystem . . . . . . . . . . . . . . . 190

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 193

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213


18/229

About the Authors

Széchenyi István University, and head of the 3D Internet based Control andCommunications Laboratory at the Institute for Computer Science and Control of the Hungarian Academy of Sciences.

Gyula Sallai is a professor at professor at the Department of Telecommunicationsand Media Informatics of the Budapest University of Technology and Economics,and Scientific Director of the Future Internet Research Coordination Centre of theUniversity of Debrecen.

xvii

Péter Baranyi is a professor at the Multidisciplinary Doctoral School of the

Adam Csapo is a postdoctoral researcher at the Institute for Computer Science and

Control of the Hungarian Academy of Sciences, and an assistant professor at the

Department of Informatics of the Széchenyi István University.


19/229


20/229

About the Book

This book describes the theoretical foundations of cognitive infocommunications(CogInfoCom), and provides a survey on state-of-the-art solutions and applicationswithin the field. The book covers aspects of cognitive infocommunications inresearch fields such as affective computing, BCI, Future Internet, HCI, HRI,sensory substitution, and virtual/augmented interactions, and also introduces newlyproposed paradigms and initiatives under the field, including CogInfoCom channels,speechability and socio-cognitive ICT. The book focuses on describing the mergingbetween humans and information and communications technology (ICT) at the level

of cognitive capabilities with an approach towards understanding the perspectivesof generation CE (after generations X, Y and Z), and developing future cognitiveICT.

xix


21/229

Part I

Foundations of CogInfoCom

This part presents the scope and goal of cognitive infocommunications(CogInfoCom), and discusses the primary motivations behind the field from variousperspectives.

In Chap. 1, a research historical overview is given on the phases and synergiesleading to the emergence of cognitive infocommunications, both from a scientificand technological point of view. Based on the chapter, it can be concluded that theemergence of the field is a natural development in view of the broader evolutionof relevant research domains, including infocommunications and the cognitive

sciences.In Chap. 2, a deeper discussion on the scope and goals of CogInfoCom is

provided. This is followed by an overview of novel concepts which have emergedbased on the definition, as well as the various implicit and explicit assumptionsthat serve as a basis for CogInfoCom research, including the assumptions of levelof cognitive capability and entanglement between humans and information andcommunications technology (ICT).

Finally, in Chap. 3, the foundations of CogInfoCom are discussed from theperspective of the digital convergence process leading to the diffusion of ICT.

Although the discussion is motivated by the fact that CogInfoCom is stronglyinfluenced by engineering fields, focus is also placed on changes in associated valuechains and social-technological phases (in particular, the cognitive phase). Based onthe chapter, the position of CogInfoCom within the Digital Ecosystem is clarified.


22/229

Chapter 1

Emergence of CogInfoCom in Science

and Technology

In this chapter, an overview is given of the scientific and technological triggersleading to the emergence of cognitive infocommunications (CogInfoCom).Section 1.1 of the chapter begins with a brief introductory description of themotivations behind the field. This is followed by a discussion of how the fieldhas emerged, first from the point of view of the scientific influences (Sect. 1.2),and later from the point of view of technologies (Sect. 1.3) contributing to it. Thediscussions provided by the chapter reveal that the evolution of all scientific fieldsis characterized by a process of maturation and ultimate convergence into newer

fields of inquiry. In many cases such processes are supported by both conceptualand technological advances which bring to light possibilities for new theoreticalapproaches as well as previously unimagined application areas. It is demonstratedthat the emergence of CogInfoCom can be attributed to a similar process.

Some explanation is in order here regarding the term “ priming effect ”, whichis used in Sects. 1.2 and 1.3 to describe the nature of the influence of scientificresearch fields and technology on CogInfoCom. The term was taken from thestudy of cognitive biases, where a concept manifesting itself in any of a varietyof forms (i.e. as a subconscious impression, a conscious idea, or alternatively as

a behavior or emotion experienced) is capable of selectively priming other relatedconcepts, effectively rendering them easier to recall and/or experience (Kahneman2011; Gilovich et al. 2002). The loose analogy with this technical notion is that thescientific currents and technological advances that define our contemporary worldinevitably influence the way in which new problems, challenges and research goalsare defined. It is with respect to these influences that the chapter aims to position thefield of CogInfoCom.

© Springer International Publishing Switzerland 2015P. Baranyi et al., Cognitive Infocommunications (CogInfoCom),DOI 10.1007/978-3-319-19608-4_1

3


23/229

4 1 Emergence of CogInfoCom in Science and Technology

1.1 What Is CogInfoCom?

Cognitive infocommunications (CogInfoCom) is an interdisciplinary research field

that has emerged as a synergy between infocommunications and the cognitivesciences.1 One of the key observations behind CogInfoCom is that humans andICT are becoming entangled at various levels through a convergence processbetween these disciplines, as a result of which new forms of cognitive capabilityare appearing. Crucially, these capabilities are neither purely natural (i.e., human),nor purely artificial; therefore, it is suggested that they should be treated in away that unifies both engineering and human-oriented perspectives. Thus, whileCogInfoCom aims to provide qualitative and quantitative analyses of emergentcommunicative phenomena through the analogy of natural cognitive capabilities, it

also focuses on the development of approaches and methodologies for the synthesisof new human-ICT capabilities based on engineering principles.A more specific description of the scope and goals of CogInfoCom is provided

in Chap. 2. A deeper discussion of assumptions underlying the field and specificterminologies to which it has given rise is also provided in that chapter. At thispoint, it is nevertheless worth highlighting that some of the most important featuresof the field are as follows:

• Focus on cognitive capability: rather than merely focusing on ways in whichhumans, devices and ICT infrastructures interact, the field acknowledges the

notion of “cognitive capability”—an abstraction which allows for the introduc-tion of temporal and contextual considerations into the analysis and design of relevant phenomena;

• . . . from a generic perspective: rather than restricting the notion of cognitivecapability to humans alone, the field adopts the point of view that such capabili-ties are emergent properties of any continued interaction and communication thatpotentially involves both natural and artificial components and systems;

• . . . at various (particularly at large) temporal scales: rather than merelyfocusing on individual goal-oriented interactions at clearly specified points in

time, the field adopts the point of view that the entanglement between humansand ICT is resulting in the necessity to consider their co-evolution at larger scalesof time;

• . . . in the context of emergent functionality: rather than focusing exclusively onfunction-oriented interactions, the field also considers cases where functionalitiesdeveloped at one time to fulfill a specific goal acquire new roles and uses—aphenomenon that is caused by constant change and growth in complexity of theelements (and relationships thereof) defining human-ICT networks.

1The term “cognitive sciences” is used here as a comprehensive term for those fields that dealwith the capabilities of biological systems, including—among others—the fields of psychology,neuroscience, cognitive modeling, cognitive ergonomics and human factors, linguistics, biology,anthropology, some branches of artificial intelligence, etc.


24/229

1.2 Scientific Priming Effects 5

1.2 Scientific Priming Effects

In this section, it is argued that the appearance of CogInfoCom is a natural

development from a research historical point of view, and that this is well reflected inthe terminologies emerging from the field. In order to show this, the past evolution of informatics, media, communications and the cognitive sciences is briefly described.Chapter 3 adds further perspectives to the discussion provided here by consideringthe value chains and social-technological phases relevant to the convergence of thesefields.

1.2.1 Convergence from a General Perspective

It is a common phenomenon for newly established fields to go through a processof maturation and ultimate convergence. The evolution of informatics, media andcommunications is no different: although these fields initially had different goalsand applied different methodologies, their maturation and growing pervasivenesshas led to the emergence of new synergies. Thus the fields of infocommunications,media informatics and media communications appeared in the latter half of thetwentieth century (Sallai 2012b; Fransman 2002; Chaffee and Metzger 2001). Thesubsequent evolution of these disciplines, in turn, has resulted in still newer patternsof convergence. As modern network services aim to provide an increasingly holisticuser experience, the infocommunications sector now encompasses “all information processing and content management functions [. . . ] of information technology and

electronic media” (Sallai 2012a, 2007; Preissl and Muller 2006). Thus, the modernICT/TIM sector was born.

Parallel to these developments, with the enormous growth in scope and tech-nological relevance of the cognitive sciences, the new fields of cognitive media(Nannicelli and Taberham 2014; Hokanson and Hooper 2000; Recker et al. 1995;Kozma 1991), cognitive informatics (Wang 2002; Wang and Kinsner 2006; Vernonet al. 2007) and cognitive communication(s) (Roschelle 1996; Hewes 1995); andalso2 (WUN CogCom 2008; Mitola and Maguire 1999) are gradually emerging. Bytoday, these fields have either fully made their way, or are steadily on their way intostandard university curricula and will eventually become a natural part of collectiveawareness.

For example, a quick search reveals that several research groups and companiesaround the world have “ cognitive media” (sometimes together with the suffix“technologies”) in their name. While originally the field was strongly motivated bynew prospects of virtually augmented, interactive education, today it is driven by a

more general focus on how various forms of media can be analyzed in terms of their

2Cognitive communication and cognitive communications have different meaning in differentresearch communities, as described later.


25/229


effects on human cognition, and how they can be applied to sharing informationin ways that appeals to human cognitive capabilities. As a result, any researchfocusing on interactive media, or interaction technologies in general will have strongrelevance to the field. However, another important factor behind cognitive media isthe growing prevalence of artificial sensory capabilities implemented in media: ina way analogous to the human nervous system, the Internet as an interconnectionof globally distributed devices and nodes, together with the media applicationsbased on it can be seen as an artificial nervous system and an artificial set of sensory modalities. Data available on social network platforms such as Facebookand Twitter are increasingly used to predict and understand physical, mental andsocial processes taking place in the world. According to this view, cognitive mediatargets not only the cognitive effects that media has on humans, but also thecognitive capabilities of media itself. As this viewpoint gains increasing viability, it

can be expected that at a certain point the term “cognitive” will no longer be reservedexclusively for the description of human-oriented phenomena: although this was theoriginal meaning of the term (from a time when modern media, not to mention ICTwas still a dream of the future), today the ICT infrastructure surrounding us hasgrowing capabilities for sensing, representing and interpreting a growing scope of phenomena.

Similarly, the terms “ cognitive informatics” a n d “ cognitive communication”/“ cognitive communications” are omnipresent in delineations of researchfocus. A definition of cognitive informatics can be found on the official web page of

a number of research organizations (e.g., the Pacific Northwest Laboratory fundedby the U.S. Department of Energy). Today there are several research institutesdedicated to cognitive informatics, such as the Institute for Cognitive Informatics(ICON) in Sheffield, UK, which has an annual budget of over one million GBP.An IEEE International Conference on Cognitive Informatics has been held everyyear since 2002, and several symposia with leading researchers invited as keynotelecturers have been held in the past few years. Cognitive communication(s), whenused without the trailing “s”, can refer to the study of ways in which humansanticipate context as a contributor to the choice of communication style, and

perceive the consequences of communication choices (in short, it is the study of human mental models of communication). On the other hand, with the trailing“s”, cognitive infocommunications is used to refer to devices and networks whichcan assign resources and functionalities in dynamic and intelligent ways (as ine.g., cognitive radio or cognitive networks). It is possible that in the future, thesetwo directions will reach common formulations for certain problems—see e.g.Mitola and Maguire (1999), which specifically mentions user needs and applicationscenarios as possible contributors to the adaptivity of cognitive radio.

In a way analogous to the evolution of media informatics, media communications

and infocommunications, examples of results which can be categorized as belongingto cognitive media informatics, cognitive media communications and cognitiveinfocommunications are gradually emerging, even if—as of yet—these fields are notalways clearly defined. The relations between the fields discussed here are shown inFig. 1.1 (further significant details on this figure are provided later in Chap. 3).


26/229

1.2 Scientific Priming Effects 7

Fig. 1.1 Relationships between fields relevant to the convergence process behind CogInfoCom.

The figure highlights the fact that the emergence of CogInfoCom is a result of the way in whichthe fields of informatics, media and communications have evolved. From a historical perspective,CogInfoCom can also be regarded as an extension of cognitive communications (an in-depthdiscussion on this aspect is provided in Chap. 3 of this book)

1.2.2 Convergence from an Infocommunications Perspective

As a result of the convergence process described above, infocommunications

today adopts a broader focus than before. An up-to-date and comprehensiveoutline of the past evolution of infocommunications, as well as prospects of its future development, is provided in Sallai (2012a). The convergence leadingto the infocommunications of today was thoroughly analyzed in the mid-1990s,and was soon recognized by both by the Commission of European Communities(European Commission 1997) and by the International Telecommunication Union(ITU 1999).

The consensus is that this convergence took place at three levels, manifestingitself in the unification of technologies, the integration of markets and the har-monization of regulation (Sallai 2012a). Thus, the same devices that are used tocommunicate with others can also be used to access and process digital mediacontent (hence, both the technology and the market is available for unification).Further, it is due to the harmonization of regulation that the cost of using moderninfocommunications devices is transparent with respect to the kind of informationthat they are used to process and transmit. The convergence process and its futureprospects can be summarized in the following four steps (for further details, seeChap. 3):

1. Traditional separation, internal digitization. The technology, market and

regulation behind various content types (e.g., voice, text, audio-visual) aremanaged separately.2. Unified telecommunications. A partial unification was possible from a techno-

logical point of view with the rapid development of digital technology. Hence,it became possible to handle different kinds of content with the same devices.


27/229


On the other hand, the electronic media production industry had yet to becomeinvolved in the convergence process.

3. Infocommunications. The electronic media content producing industry, as wellas the appropriate institutions for regulation joined the convergence processto produce the technological and social-economic revolution that is today’sinfocommunications industry.

4. Cognitive infocommunications. A natural fourth step is the integration of cognitive sciences into the convergence process behind infocommunications. Onthe one hand, this involves an expanding content space, in which new informationtypes are gathered, inferred and reasoned upon in novel ways. On the other hand,it is essential that this content be used in effective ways to extend the cognitivecapabilities of both users and ICT, allowing for the formation of new cognitiveentities.3

As a result of the synergy between the cognitive sciences and infocommunica-tions, the content space that is accessed and manipulated by both users and ICT isexpected to grow in size and richness of interpretation. If the potential behind thisexpanding content space is to be harnessed, it can be expected that the respectiveunique capabilities of humans and ICT will be applied through new, long-terminteraction patterns, leading to the emergence of new cognitive entities. A significantchallenge in CogInfoCom is how to “power” these cognitive entities with the kindsof information and functionality that are suitable to their roles and requirements.

1.3 Technological Priming Effects

An important tool using which emerging technologies are often characterized is the technological hype cycle, a graphical representation developed by ICT research andadvisory firm Gartner Inc. (Fenn and Raskino 2008). The main premise behind thehype cycle is that new technologies most often evolve through a process that can bebroken down into the following five phases (Fig. 1.2):

1. Technology trigger. A potential breakthrough becomes apparent, and significantpublic interest is raised through early proof-of-concept success stories.

2. Inflated expectations. The number of success stories associated with the tech-nology increases, but is also accompanied by failures; although some enterprises

join efforts to develop and improve the technology,many are deterred by growingperceptions of risk.

3. Trough of disillusionment. Interest wanes as the technology fails to deliver onits (both real and perceived) promises. Most investments are tied to the condition

that existing implementations are improved to the minimum level of satisfactionamong early adopters.

3For a fully detailed explanation of what is meant by the term “cognitive entity”, the reader isreferred to Sect. 2.3.2.


28/229

1.3 Technological Priming Effects 9

Fig. 1.2 Five phases of Gartner’s “hype cycle” in terms of technological expectations with respectto time

4. Slope of enlightenment. Potential benefits of the technology become increas-ingly widely understood as an increasing number of successful use cases becomeapparent. Second- and third-generation implementations of the product appear,new pilots are increasingly funded, while more conservative actors remaincautious.

5. Plateau of productivity. Mainstream adoption begins to take off. The broadmarket applicability of the technology becomes clear, and its adoption yieldsclear advantages.

One of the messages of the hype cycle model is that almost all new technologiesare initially surrounded by a kind of hype that is, in a way true to the definition of the word, founded more on a combination of illusions, expectations and a desireto provide something “new” than on a true realization of benefits, drawbacks andassociated tradeoffs. This is what leads to the trough of disillusionment, whichin turn often (or in the case of successful technologies, always) brings about areassessment with respect to potential manufacturing approaches and use cases.The end result of this reassessment combined with a few instances of successfulapplications is a better informed proliferation of the technology.

Besides making possible these general conclusions, however, the model alsoallows for the tracing of technological advances through time, both in nominal andrelative terms. On the one hand the location of an individual technology on thehype cycle curve allows for assessments to be made with respect to the timelinessand the expected success of that technology in the near future. On the other hand,


29/229


technologies can also be assessed relative to one another, or alternatively in groupsif considered together based on some criterion. The latter type of group assessmentis especially informative if it is performed through a period of 5–10 years instead of at a specific point in time: such a perspective allows for the maturation process of technologies behind specific fields or industries to be evaluated.

In this section, we consider a group of those technologies and R&D initiativeswhich can be interpreted as ready-to-use components of cognitive entities; oralternatively as either catalyzing or at the very least influencing CogInfoComresearch (these technologies and R&D areas are further discussed in Chaps. 4 and 5).Figure 1.3 provides two hype curves with several examples as of 2010 and 2014.Based on a broad comparison of these two snapshots of technological progress,several points can be made in terms of relevance to CogInfoCom:

• Trends in 2010 were characterized by a comparatively higher proportion of technologies directed at what may be referred to as “elementary cognitive capa-bilities”—including speech recognition, gesture recognition, and location aware-ness. In contrast, the period between 2010 and 2014 has seen the appearance andcomparative proliferation of more refined, higher level cognitive capabilities—including emotional capabilities represented by affective computing, capabilitiesfor human augmentation, speech-to-speech translation and health monitoring.Capabilities belonging to this latter category are often directed at capturing the“global” state of an entire cognitive entity rather than merely providing details

on lower-level aspects of interaction.• As the former set of elementary capabilities have matured over the pastyears, they have evolved into “building blocks” for the latter, higher levelcapabilities. For example, earlier technologies directed at recognizing discretegestures, speech utterances or facial information have become a crucial partof applications directed at e.g. input control through fine joint movements andemotion recognition; while smart network technologies such as mesh sensornetworks have begun to form the basis of health monitoring infrastructures.

• Many of those technologies which focus on higher-level cognitive capabilitiesstill haven’t reached the slope of enlightenment and are often prognosticatedto require 8–10 more years before maturation—in much the same way as theywere 4 years ago (in case they were already present on the hype curve at

I

Fig. 1.3 Evolution of expectations between 2010 (top) and 2014 (bottom) in terms of CogIn-foCom. Technologies less relevant to the field, such as cryptocurrencies, machine-to-machinecommunication services and others were left out for improved clarity. Care was also taken to ensurethat technologies shown on the curve for 2014 were also included on the curve for 2010 if also

present in the originally published figure. Darker shades represent technologies expected to haverelatively longer maturation times. For example, the maturation process of human augmentationtechnologies has been, and still is expected to take over 10 years, while the maturation of speechrecognition can be expected to take less than 2 years as of 2014. The original hype cycle curves canbe found at e.g. http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/ technology-for-innovation-in-marketing/

http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/technology-for-innovation-in-marketing/http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/technology-for-innovation-in-marketing/http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/technology-for-innovation-in-marketing/http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/technology-for-innovation-in-marketing/


30/229

1.3 Technological Priming Effects 11

Fig. 1.3 (continued)


31/229


that time). This suggests that the maturation processes of such capabilities aregoverned by larger time constants and are expected to remain in the main focusof CogInfoCom-related research areas for years to come.

In summary, the evolution of hype curves from the past 4–5 year period stronglysuggests that the primary technological components underlying CogInfoCom havereached the phase of maturation, while the integration of these components asbuilding blocks into more complex capabilities has begun both in subtle ways aswell as at a very explicit conceptual level. This integration process can be expectedto gain further traction in the coming years as the technologies based on which it isgrounded remain at the stable end of maturation, and as new technological triggerscontinue to appear based on those technologies (as demonstrated by the hype curveof 2014, many new concepts are already appearing). Such salient processes of

technological integration are a strong motivating factor behind CogInfoCom.


32/229

Chapter 2

Definitions, Concepts and Assumptions

In this chapter, a description of the scope and goals of CogInfoCom is provided. Thisis followed by an overview of novel concepts—such as those of mode and type of communication, as well as the more general notion of cognitive capability—whichhave emerged through the field. Further, a set of assumptions, primarily founded onthe existence and consequences of the merging process between humans and ICT,are described in terms of their relevance to CogInfoCom research.

2.1 Defining CogInfoCom

Humans and the infocommunications network (ICT in a broader sense) surroundingthem are merging together and becoming entangled at various levels, ranging fromlow-level connectivity at the cellular and electrotechnical level, all the way to thehighest level of sensing collective behaviors such as mass movements, mass habitsetc. As a result, humans (more generally, living beings) and infocommunicationswill soon coexist as an entangled web, resulting in an augmentation of both

natural and artificial cognitive capabilities.1 This process is occurring today, andis expected to gain further impact in the near future. By analogy, it also necessitatesa merging process between the scientific fields related to natural cognitive systemsand the scientific and technological fields related to infocommunications. The abovemotivations have contributed significantly to the definition of CogInfoCom. A draftproposal of the definition was provided in Baranyi and Csapo (2010), and was laterrefined by Professors Nick Campbell, Tom Gedeon, Hideki Hashimoto, ToshikazuKato, Tetsuo Kotoku, Kristiina Jokinen, Joo-Hoo Lee, Gábor Magyar, Helen Meng,

1As we will see, in many cases this separation between natural and artificial is no longermeaningful.

© Springer International Publishing Switzerland 2015P. Baranyi et al., Cognitive Infocommunications (CogInfoCom),DOI 10.1007/978-3-319-19608-4_2

13


33/229

14 2 Definitions, Concepts and Assumptions

Géza Németh, Mihoko Niitsuma and Gyula Sallai at the 1st International Workshopon CogInfoCom, held in Tokyo, Japan in 2010. The finalized definition is as follows:

Definition 2.1. Cognitive infocommunications (CogInfoCom) investigates the

link between the research areas of infocommunications and the cognitive sciences,as well as the various engineering applications which have emerged as a synergiccombination of these sciences. The primary goal of CogInfoCom is to provide asystematic view of how cognitive processes can co-evolve with infocommunicationsdevices so that the capabilities of the human brain may not only be extended throughthese devices, irrespective of geographical distance, but may also interact with thecapabilities of any artificially cognitive system. This merging and extension of cognitive capabilities is targeted towards engineering applications in which artificialand/or natural cognitive systems are enabled to work together more effectively.

2.2 Concepts Emerging from CogInfoCom

As will be discussed later in this chapter in further detail, the implicit and explicitassumptions underlying CogInfoCom together form a unique viewpoint. As aresult, new notions and concepts capable of leading to new research directions arecontinuously emerging. In this section, two early concepts central to multi-sensory

communication between various levels of cognitive capability are introduced: themode of communication, and the type of communication. In the future, theseconcepts may be extended to provide a more detailed qualification (and in thelong run: quantification) of cognitive capabilities independent of the exchange of communicational messages.

2.2.1 Mode of Communication

The mode of communication refers to the way in which the relationship betweenactors at the two endpoints of can be characterized:

• Intra-cognitive communication: information transfer occurs between two cog-nitive entities with equivalent cognitive capabilities (e.g., between two humans,or between two humans in the same social-technological environment—asdetermined by what is relevant to the application).

• Inter-cognitive communication: information transfer occurs between two cog-nitive entities with different cognitive capabilities (e.g., between a human and an

artificially cognitive system, or between two humans in different social or tech-nological environments—as determined by what is relevant to the application).


34/229

2.2 Concepts Emerging from CogInfoCom 15

In accordance with the introductory remarks to this section, future developmentsare expected to lead to a deeper, perhaps quantified understanding of what is meantby level of cognitive capability. Further discussions on this notion can be found inSect. 2.3.2.

2.2.2 Type of Communication

The type of communication refers to the way meaning is conveyed between the twocommunication entities:

• Sensor-sharing communication: cognitive entities on both ends use the same

sensory modality to receive information.• Sensor-bridging communication: sensory information is not only transmitted,but also transformed to a different, more appropriate sensory modality of thereceiving cognitive entity.

• Representation-sharing communication: the same information representationis used on both ends of communication.

• Representation-bridging communication: sensory information is filteredand/or adapted so that a different information representation is used on thetwo ends of communication.

A sensor-sharing application brings novelty to traditional infocommunications inthe sense that it can convey any kind of normally perceptible signal (i.e., a signal thatcould be perceived if there were no distance to communicate across) to the other endof the communication line. The key determinant of sensor-sharing communication isthat the same sensory modality is used to perceive the information on the receivingend of communication as would be if there were no distance between the sendingand receiving ends.

Sensor bridging can reflect both the way in which information is conveyed (i.e.,by changing sensory modality) as well as the novelty of the information type that

is conveyed. Whenever the transmitted information type is imperceptible to thereceiving entity due to a lack of appropriate sensory modality, communication willnecessarily occur through sensor bridging.

A CogInfoCom application can be regarded as an instance of representationsharing even if it bridges between different sensory modalities. By considering thegeneral characteristics of a representation (e.g., its character-based, icon-based etc.nature) rather than the specific details of its physical or biological manifestation, itbecomes possible to describe representations of different modalities in unified ways.

As mentioned in the introductory remarks to this section, aspects encompassed

by the concept of type of communication are relevant only when explicit communi-cation occurs in a way that is directed at information sharing. As this is only a smallpart of what it means to communicate, further extensions can be expected to emergein future work (see also Sect. 2.3.3).


35/229


2.3 Implicit and Explicit Assumptions

The definition of CogInfoCom as well as the discussions on the scope and goals

of the field lay emphasis on the merging process between humans and ICT, aswell as on different levels of cognitive capabilities and long-term co-evolution of biological and artificial systems (Baranyi and Csapo 2010, 2012). In this section,a brief discussion is provided on these aspects to further highlight the focus of thefield. The section is concluded by observations on the new information concept thatis implicitly assumed by the CogInfoCom, as well as the transition from operationto emergent functionality that implicitly motivates research within the field.

2.3.1 Levels of Merging and Entanglement

As mentioned earlier in Sect. 1, one of the key observations behind CogInfoComis that there is a merging process between humans and ICT that is resulting inincreasingly complex forms of human-ICT entanglement, and is at the same timecreating the necessity for an analogous convergence between technology and thehuman-oriented cognitive sciences. The phenomena of merging and entanglementin the context of ICT are clear not only from everyday experience, but have also beenremarked and analyzed to various degrees and in various contexts by many authors,as in e.g. Romportl et al. (2015), Pang (2013), Gripenberg (2011), and Dahlbom(1996).

From the point of view of interaction modes, the merging process betweenhumans and ICT can be observed at three different levels:

1. The first level of entanglement corresponds to low-level, direct relationships,including those that rely on invasive and non-invasive forms of interface (as ine.g. brain-computer interfaces). Entanglement at this low level allows for directsensing and control, however, it is also relatively cumbersome in that it requires

sensors to be implanted or worn and is also difficult to operate at conceptuallyhigher levels of command.2. A different form of entanglement is possible at the level of personal informatics

devices, in which communication and interaction occur through (human—butcrucially not only human) sensory modalities. The question of what kind of “communication language” to use (i.e. in terms of message encoding) dependingon the semantics of the information, as well as—among others—the modalityto be used, the application environment, and the user’s individual cognitivecapabilities are strongly relevant to this level of entanglement. It is important

to note that the challenge consists not only in providing effective and ergonomicinterface design, but also in accommodating the transfer of an expanding setof semantic concepts—relevant at large temporal scales, for instance in co-existive smart home and other augmented virtual reality applications—throughthe limited possibilities afforded by human sensory modalities.


36/229


37/229


become entangled with systems of ICT components, new use case requirements,and new possibilities for previously unknown functionalities are discovered. Theaccommodation of such functionalities, in turn, can create new patterns of usage—many of which may then have a recurrent effect on usage patterns, and potentiallyeven on the social organization of the community of users involved in the mergingprocess. This mutual influence between usage patterns and use-case requirementsis creating an open-ended evolution of functionality. From a different perspective,long-term co-evolution between humans and ICT is also enabling the “offline”collection of vast amounts of data which can later be instrumental in developing newapplications (for example, if those applications rely on machine learning techniquesrequiring large amounts of data). As a result, the creation of new functionalities isfacilitated in more than one way through the implicit knowledge that is generatedfrom past interactions. Based on the above, we introduce the term tangleface to

be used instead of the word interface when characterizing ubiquitous human-ICTrelationships. Additionally, we use the term tangleaction instead of interactionwhen such relationships persist through extended periods of time. These notionswill be used often and elaborated from a variety of perspectives in later parts of thisbook.

2.3.2 Levels of Cognitive Capability

From a CogInfoCom perspective, any kind of hardware or software component thatactively collects/stores/understands/communicates data can be seen as a componentwith a set of cognitive capabilities. Whenever users become entangled with a systemof such capabilities, the border between the natural and artificial gradually becomesvague. In other words, it is often the case that there is no longer any objective borderbetween purely human and purely artificial cognitive capabilities. For example, in ascenario where a user controls an industrial robot with one hand using knowledgeobtained from a smartphone held in her other hand, the question immediately arises:

should this interaction be characterized from the perspective of communicationbetween three different entities, or is there benefit in viewing the user and thesmartphone as one entity that is communicating with the robot? The answer to thisquestion is important, if only for the fact that both the robot and the supportingsmartphone application might be designed differently if it is known in advancethat they will used together in this specific scenario, or if the cognitive effectsthat the smartphone application will have on the user—such as limited dexterityand attention, increased capabilities for information access, etc.—are known inadvance. To consider two other examples, the boundary between artificial and

human capabilities would be equally blurred in a scenario where a user’s lower armis augmented through a robot arm that is capable of downloading new “skills” froman ICT network; or in a scenario where a pair of augmented glasses, or an augmentedhelmet is used to provide an industrial operator with real-time information feedscomplementing the task at hand (such technologies are already present in industry,and are on the verge of commercial breakthrough).


38/229

2.3 Implicit and Explicit Assumptions 19

The bottom line is not that one would be philosophically inclined to specify aboundary between entities, but that it is also necessary to specify such boundariesfrom the functional perspective of engineering design. On the one hand, in a domainwhere difficult problems of synthesis can be effectively tackled only by breakingthem down into smaller components and gluing those components together throughsome form of communication once they are complete, the functional boundariesat which this is done can make or break the tractability and sustainability of animplementation. On the other hand, once it is accepted that the boundaries betweenartificial and natural are not as clean as they were a few decades ago, unprecedentedpossibilities emerge for the development of new functionalities—even cognitivecapabilities. Such capabilities can be seen as implemented in the dependenciesbetween components in much the same way as lower-level functionalities arecreated as a result of several different components working appropriately in

mutually specified contexts. This hierarchical dependence among capabilities canbe seen as leading to a hierarchical organization of cognitive capabilities.

The embodied perspective of cognition that is currently favored in the cognitivesciences adopts the view that human cognitive capabilities and human intelligenceare emergent properties which cannot be separated from the physical, biological,neurophysiological and higher-level bases of our mental existence (Deacon 2013).It is also clear that the social and technological context of our interactions with otherhumans and ICT further influences the kinds of mental and physical work that weare able to perform (Hollan et al. 2000; Deacon 2013). Further, analogous emergent

properties can be identified in the functionalities of ICT devices and networks.Although the view that computers are merely symbol processing systems has beenimplicitly accepted for decades by thinkers and technologists of all backgrounds, anemergentist view of computing is now gaining acceptance. Though fundamentallydifferent from living systems it can be argued that all computational systems(apart from purely theoretical constructs such as the Turing Machine) have someform of embodiment, and that furthermore, computation in general has physicalunderpinnings and physical ramifications (Heder 2014). As highlighted by severalauthors, even lexical knowledge such as knowing the derivative of the sine function

or knowing the capital of a country is strongly embodied in the sense that withoutdirect experience in working and manipulating functions, or without being able totravel and experience through our bodies what really constitutes a city, our notionswould be entirely different (Picard 2003a; Heder 2014).

The extension of such notions to human-ICT entanglement and emergentcognitive capabilities can be seen as a natural development. It can be argued thatnot only are new functionalities and cognitive capabilities formed through human-ICT entanglement, but that they can also be seen as higher-order in the sense thatthey are dependent on lower-level foundations. However, this point of view also

makes it possible for such higher-order capabilities to be combined into newerones that are located at still higher levels of hierarchy. This process is illustratedin Fig. 2.1. From a practical point of view, the figure shows that all cognitivecapabilities, however trivial in a human-ICT context, can be analyzed in terms of other capabilities on which they depend. This observation can serve as an important


39/229


human sensory

modalities

artificial sensorymodalities

newer use casesmore data accumulated

newly available tanglefaces

emergent cognitive entities

time

Fig. 2.1 New, higher-level artificial capabilities and emergent cognitive entities are createdthrough time as new use cases are generated through a broadening of artificial sensory modalitiesand the increasing possibility for users to co-evolve with them through extended periods of time

starting point in the design of CogInfoCom systems, especially when combined

with the goal of providing functionality rather than the ability to utilize operationalprocedures (see also Sect. 2.3.4). In this book, the term cognitive entity will be usedwhen describing any synergic combination of humans, devices, infrastructure andenvironment that is identifiable from the perspective of some (high-level) cognitivecapability. Whenever a cognitive capability is considered at a lower, less complexlevel—and pertains directly to a human or an artificial device, the capability is saidto be embodied by a cognitive being, or a cognitive thing, respectively.3 Basedon this terminology, cognitive entities are formed when information on variousrelationships between cognitive beings, cognitive things and their environment are

stored, interpreted and acted upon in identifiable ways. As a result, a new perspectiveautomatically emerges based on which cognitive entities can be analyzed andunderstood in terms of holistic cognitive capabilities without being broken up intoconstituent components. At the same time, the emergence of a new generation of users can also be prognosticated, which we refer to as the generation of cognitiveentities, or generation CE by analogy with the well-known terminology of thegenerations X, Y and Z. It can be argued that members of generation CE, growingup starting from around the year 2010, are unique in the sense that the maturation of their personality and social life unfolds in an environment that is inseparable from

cognitive ICT. The extended cognitive capabilities that ICT provides are merged into

3The analogy with the Internet of Things is clear, when IoT is regarded as a cyberization of thephysical world together with humans, as will be detailed Chaps. 3 and 4.


40/229

2.3 Implicit and Explicit Assumptions 21

the daily experience of this generation, and become irrevocably intertwined with itsexpectations and thinking processes, both at a conscious and subconscious level.Expressed more directly, one can say that ICT cannot be ‘taken away’ from thisgeneration without bringing about significant cognitive effects and psychologicaldiscomfort.

2.3.3 An Emergent Concept of Information

It is important to clarify that the notions of merging, entanglement and levels of cognitive capability also suggest a unique concept of information that is implicitlypresent in CogInfoCom. Importantly, rather than being treated as a commodity that

can be transmitted from one end of a communication line to another, information isseen as an artifact—a functionally relevant physical by-product—that emerges fromembodied patterns of interaction and communication.

In this sense, the design of a CogInfoCom system or application involves notonly the identification of various data and information types that are to be sent tovarious components at certain points in time, but also a broader consideration of how nuanced differences in behavior can lead to functionally relevant by-products,how these by-products cause the communication between cognitive entities toevolve through time, and how this evolution can be directed towards further,

novel functionality in flexibly re-usable ways. Thus, a common past—i.e. one thatis accumulated in goal-independent ways through a progressive human-ICT co-evolution—can be expected to eventually yield increasingly rich, functional modelsof how humans are capable of communicating and operating in ICT settings.

While it may be the case that in an information system, a solution to any of these problems will eventually require suitable data structures for representation andmanipulation at a more “atomic” level, the CogInfoCom perspective neverthelesssuggests that the starting point of design should be the observation and detection of emergent possibilities for novel interpretation leading to novel functionality.

2.3.4 Transitions from Operation to Functionality

As described earlier in Sect. 2.3.1, new human-ICT capabilities are formed basedon new kinds of co-dependence relationships among and between humans and ICT.It can also be observed that the details behind these capabilities (i.e., how theyare implemented) become gradually less important through time—at least from the


41/229

22 2 Definitions, Concepts and Assumptio

cognitive infocommunications

Documents