BOOK REVIEW

N.-M. Klug and H. Stockl (eds): Handbuch Sprache imMultimodalen Kontext

De Gruyter, Berlin/Boston, 2016

Steven Schoonjans1,2

Received: 13 April 2017 /Accepted: 6 June 2017 / Published online: 12 June 2017

� Springer International Publishing AG 2017

Introduction

The study of multimodality, the interaction of words with other layers of expression

in making and conveying meaning, is a major and growing field in present-day

research in different disciplines, including linguistics and media and communication

sciences. In this respect, one could say that the Handbook of Language in

Multimodal Contexts (this is the English title as announced on the publisher’s

website) comes at the right moment. Bringing together 21 contributions in both

German and English, the volume ‘‘outlines the current theoretical-methodological

and empirical situation in research on multimodal texts and communicative actions

[…] from an in principle multi- and transdisciplinary perspective, which does

however have its foundation in linguistics’’ (p. III, my translation). The eventual

goal is ‘‘to take linguistically based multimodality research to a clearer contouring

and a further canonization of its objects, paradigms, notions, and methods’’ (p. VII,

my translation). Indeed, as the editors indicate on page XIII of the introduction, a

terminological ‘‘harmonization’’ is needed, as different notions with not entirely

corresponding denotations and connotations have been used more or less

interchangeably, while the same notion does not necessarily have exactly the same

intension in all contexts in which it is used.

As the book covers 512 pages (including the editors’ introduction) and more than

twenty contributions, it would be impractical to give detailed accounts of each

& Steven Schoonjans

steven.schoonjans@uibk.ac.at

1 Leopold-Franzens-Universitat Innsbruck, Innrain 52d, 6020 Innsbruck, Austria

2 KU Leuven/FWO-Vlaanderen, Leuven, Belgium

123

Corpus Pragmatics (2017) 1:281–292

DOI 10.1007/s41701-017-0020-0

contribution separately. Therefore, in the following, I will rather offer a general

discussion, focusing on the issues that are relevant throughout the book, but still

referring to the individual papers where this is relevant.

Structure of the Book

The 21 contributions to the volume are divided in three groups, representing the

three parts of the volume. The first part, Sprache im Feld multimodaler

Kommunikation (‘language in the field of multimodal communication’), contains

five chapters that offer a kind of frame in which the multimodal phenomena

discussed in the remainder of the volume can be situated and analyzed. While the

first two chapters, by Hartmut Stockl and John A. Bateman, deal with general

theoretical (and to some extent methodological) matters, the following three

chapters present particular areas, topics and perspectives in multimodality research

that are relevant for the studies presented in the other parts of the volume. These are

the role of metaphor as a cognitive mechanism for the relation between modalities

(Constanze Spieß, Chapter 3), the role of typography research (Jurgen Spitzmuller,

chapter 4), and the integration of discourse analysis and visual analysis (Sigrid

Norris, chapter 5). The approach in these three chapters is very different: while

Norris and Spitzmuller illustrate their point by means of truly multimodal examples,

Spieß mainly offers a discussion of the notion of metaphor and the state of the art in

metaphor research, without really going into its multimodal application. Given the

aims and scope of the volume, this is somewhat regretful. Nevertheless such a

general discussion of metaphor is justified, as metaphor indeed is an important

notion in a lot of work on multimodality (including a number of contributions in the

remainder of the volume, especially Charles Forceville’s chapter on pictorial and

multimodal metaphor); hence it is helpful to be offered sufficient background

knowledge on the notion and function of metaphor.

The second part of the volume is entitled Zugriffe auf multimodale Verknupfun-

gen (‘ways of accessing multimodal linkages’) and brings together eight

chapters presenting (non-mutually exclusive) angles and perspectives in research

on multimodal texts. This part of the volume offers a good overview of what may be

the most important and/or promising methods in present-day work on multimodal

texts, each time illustrating the approach and method by means of at least one case

study. The methods and approaches discussed are: social semiotics (Carey Jewitt

and Berit Henriksen, chapter 6), text and discourse semantics (looking, for instance,

at topoi and both inter- and intra-textual relations—Nina-Maria Klug, chapter 7),

verbal-visual semiotics (focusing on the interaction between modes—Winfried

Noth, chapter 8), verbal-visual rhetoric (applied to works of art, especially paintings

and sculptures—Nadia Koch and Thomas Schirren, chapter 9), the study of pictorial

(or: visual) and multimodal metaphor (Charles Forceville, chapter 10), Multimodal

Critical Discourse Analysis (Andrea Mayr, chapter 11), production analysis (i.e., the

study of ‘‘doing multimodality’’ or how multimodal texts are produced—Michael

Klemm, Daniel Perrin and Sascha Michel, chapter 12), and cross-cultural

multimodality research (Michael Klemm, chapter 13).

282 S. Schoonjans

123

Finally, the third part of the volume—Aspekte multimodaler Textsorten (‘aspects

of multimodal text types’)—takes a different point of view and starts off from

multimodal discourse or text types. After a more general chapter (14) in which

Ulrich Schmitz discusses what a typology of multimodal text types could look like

and which challenges are to be dealt with when trying to come up with such a

typology, the seven remaining chapters each present a case study on one multimodal

text type, viz. print advertisements (Terry D. Royce, chapter 15), song (Helen

Andersson and David Machin, chapter 16), news clips (Werner Holly, chapter 17),

websites (Stefan Meier, chapter 18), Facebook pages (Volker Eisenlauer, chap-

ter 19), photo communities such as Flickr (Christina Margrit Siever, chapter 20),

and school textbooks (Jeff Bezemer and Gunther Kress, chapter 21). The chapter on

textbooks is somewhat different from the others in that, by looking at how textbooks

have evolved from the 1930s, it also contains a temporal dimension that is not

present in the other case studies nor among the methods discussed in the second part

of the volume. In this sense, it is suited as a final chapter because it shows how

multimodal text analysis opens up to new areas and dimensions and still has room to

grow as a field of research with a bright future.

General Impression

As should be clear from the previous section, the book presents an overview of

different multimodal text types and different perspectives from which these can be

studied. In this sense, it clearly does what a handbook of this type is expected to do:

not just give a general introduction to the field (first part), but also give an idea of

the multitude of possible research lines within the field (second and third part:

methods and text types to which these can be applied), going beyond established

and traditional topics (to the extent that one can already speak of ‘traditional’ topics

in such a rather recent field of research) and providing some space for new

perspectives as well. An example of this is the diachronic dimension in Bezemer

and Kress’s chapter (21) on textbooks, but also in the second part of the book (the

methods overview), there is some room for newer approaches, in the sense that not

(yet?) all methods presented may be as ‘‘etabliert’’ (‘established’) as the editors

themselves suggest (p. IX of the preface). Indeed, several contributors rightly state

that the approach they are proposing is still rather new. This applies, for instance, to

the cross-cultural comparative approach discussed by Michael Klemm (chapter 13)

and the production-oriented view put forward by Klemm, Perrin and Michel

(chapter 12).

The volume looks carefully edited, not just structure-wise but also when it comes

to its general appearance, with only minor imperfections in language and

formatting. Most disturbing is perhaps that in the chapter on ‘‘doing multimodality’’

(Klemm, Perrin and Michel, chapter 12), the French examples have not been

translated (although the volume has a multilingual set-up in general, French

examples in a German chapter might not be intelligible for some readers) and the

excerpt printed as example 16 (p. 295) does not seem to be the right one (it is a copy

N.-M. Klug and H. Stockl (eds): Handbuch Sprache im… 283

123

of example 14 on page 294, which does not quite fit the discussion of example 16).

Despite these suboptimalities, the authors manage to make their point clear.

Generally speaking, the last point actually holds for all contributions in the

volume: the papers are clear and well-structured and give a good and clear

impression of the approach or text type they are supposed to introduce, although in

some cases the papers would have been even better if fewer points had been left

implicit. In their rhetorical analysis of works of art (chapter 9), for instance, Koch

and Schirren do not offer definitions or explanations for several notions from the

field of rhetoric, thereby assuming more background knowledge in this area than

should be the case in a handbook which does not have rhetoric as its central topic.

Similarly, Sigrid Norris (chapter 5) refrains from indicating who is who in the video

stills going with her examples taken from painting classes. It is only in the

discussion of Figure 7 (p. 137) that it becomes clear that the teacher is the lady in

blue, while in order to assure full understanding of the examples, it would have been

good if this had been clear from the first image (p. 131). Nevertheless, in the end,

the reader in both cases gets a clear view of the point the authors want to make.

Finally, in two papers, one may wonder whether the phenomena and results

discussed do not actually illustrate the role of another factor than the one the authors

are referring to. In his cross-cultural study in chapter 13, Michael Klemm compares

the discussion of the death of Osama Bin Laden in the annual news reviews by an

American (ABC) and a German (ZDF) television channel. The author notes that the

German report is more static and critical, while the American report is more

dynamic and emotional, and relates this to cultural differences. In this particular

case, however, the reader may wonder whether the difference is not also due to a

difference in involvement: the Afghanistan war was a reaction to the 2001 terrorist

attacks on the World Trade Center in New York, so one could say that the

Americans are more directly and more personally involved in this matter than the

Germans, which could also explain the more emotional nature of the American

report. It certainly does not seem unreasonable to assume that cultural differences

do play a major role, but the analysis would have been all the more convincing if the

author had either referred to the involvement matter and discussed why it is not the

only factor at stake, or if he had picked another example in which the involvement is

less likely to play a role.

Similarly, Helen Andersson and David Machin (chapter 16) discuss the

differences in prosody (pitch, sound quality…) in three songs and relate these

differences to the different ‘‘kinds of desire’’ (p. 377) that are expressed in the

songs. However, the songs represent three different genres, so one may wonder to

what extent the prosodic differences do not actually reflect typical features of each

of the three genres rather than of each of the ‘‘kinds of desire’’—although in this

case the situation is more complex, as each ‘‘kind of desire’’ may be typical of a

particular genre. Here as well, it is at least plausible that the ‘‘kind of desire’’ does

indeed influence the prosody, but the analysis would have been more convincing if

the authors had made reference to the potential role of genre features or if they had

made their point using songs from the same genre.

In both of these cases (the role of involvement in news reports and the role of

genre in song prosody), an extra factor at stake may be the scope of a handbook

284 S. Schoonjans

123

chapter, which does not allow to present the results of the studies from which the

data were taken in their entirety. That could have shed a different light on the

questions just raised, but like any genre, a handbook chapter has its restrictions.

Nevertheless, the analyses presented by the authors are at least plausible, and they

definitely show what the kind of analysis concerned could look like or how it could

work, which is also an important (if not the most important) goal of this type of

handbook chapters.

Contents and Topics Discussed

As mentioned in the previous section, the volume offers a good overview of the

areas discussed, in a way that is suited for the genre of a handbook. However, the

selection of topics and areas discussed in the book is somewhat restrictive, leaving

aside at least two kinds of phenomena that could also be considered as part of the

field of ‘‘language in multimodal contexts’’: visuals embedded within the text and

the typical aspects of multimodality of spontaneous speech as realized, for instance,

through gesture, posture, and facial expression.

Embedded visuals are only mentioned briefly and more or less in passing by

Christina Margrit Siever (chapter 20) when she refers to the role of emojis in

reactions to photo community postings (p. 473). However, as she has shown

elsewhere (e.g. Siever 2015), there is a lot to say about these emojis and how they

function within such postings and interact with the verbal layer. In fact, they can be

used both incrementally as a kind of ‘‘downtoners’’ showing the author’s stance

towards the posting and in an embedded way, replacing verbal elements—what is

called their referential use (see also Schlobinski and Watanabe 2003). Both uses are

illustrated in Fig. 1, a Facebook status update by the Belgian biathlete Michael

Rosch about the upcoming Winter Olympics: the winking smiley is an example of

Fig. 1 Emojis on Facebook

N.-M. Klug and H. Stockl (eds): Handbuch Sprache im… 285

123

the downtoning type, while the emojis replacing the words ‘beer’, ‘fries’, and

‘chocolate’ illustrate the referential type.

Siever (p. 472) is of course right in claiming that this use of emojis is a rather

recent phenomenon, so research on this is scarce (although recall that the volume

does contain chapters discussing recent fields and topics). On the other hand, the

referential use of pictures as such is not all that new. Already back in the first half of

the nineteenth century, this technique was applied in so-called ‘rebus letters’ (see

Wietholter 2008:124–125), and it is not uncommon for instance in advertising, as

illustrated by the advertisement for the Red Cross in Fig. 2 (see also Rohen 1981,

among others, for further discussion).

Applying Fricke’s (2012) definition of multimodality, as some contributors

explicitly do, one could admittedly argue that the nonverbal elements in these

examples are actually part of the language and not of a multimodal context, which

could indeed be an argument for considering such cases as going beyond the scope

of the volume (as it is about language in a multimodal context). However, the same

also holds for several modes that are discussed in the volume, such as prosody and

Fig. 2 Multimodal slogan[https://www.pinterest.com/pin/234327986831637731/ (3 April2017)]

286 S. Schoonjans

123

(at least certain aspects of) typography. Furthermore, if the book is concerned with

how language is embedded in multimodal constructs, one would expect to find at

least a reference to the inverse relation, as the two cannot be fully detached from

each other: where you draw the line between multimodal language and its

multimodal context necessarily also influences your view on the relation between

language and context at stake in the volume. Clearly, embedded visuals are not the

focus of the volume, and there is nothing wrong with that, but still the matter could

have been given some more attention to delineate the aims and scope of the volume

more acutely.

The second rather neglected phenomenon in the book is the multimodality of

spontaneous speech: the volume mainly deals with multimodality in written

situations (textbooks, print advertisements, photo communities, and the like). Some

studies do involve the oral modality, but typically these are concerned with genres

such as news reporting and song, that do not contain truly spontaneous speech. A

notable exception is found in Norris’s data from painting classes (chapter 5), but

even there, the focal topic of the analysis is the interaction with some external entity

(the paintings made in class) and not the multimodal nature of the spontaneous

interaction itself. Norris does discuss pointing gestures, but only insofar as they play

a role precisely in referring to the paintings at issue. Hence, apart from the

discussion of prosody in Andersson and Machin’s analysis of song (chapter 16), the

typical aspects of multimodality of spontaneous speech (prosody, gesture, posture,

facial expression, gaze behavior…) are at best mentioned in passing by a few

contributors, but never really discussed in depth. This is all the more striking, given

that several contributors adhere to a broad notion of ‘text’ that also includes spoken

discourses and interactions (most explicitly Stockl in chapter 1, page 19, and

Schmitz in chapter 14, p. 331).1 Admittedly, a discussion of these multimodal

dimensions of spontaneous speech would be a handbook of its own (note that in De

Gruyter’s Handbooks of Linguistics and Communication Science series, there is a

two-volume handbook on gesture alone: Muller et al. 2013, 2014). However, in a

handbook about ‘‘language in multimodal contexts’’ and using such a broad notion

of ‘text’, one would expect to find at least some references to the multimodality of

speech, especially since several of the questions raised in the volume are also

relevant for the multimodal dimensions of speech (e.g. Schmitz’s question in

chapter 14 whether paraverbal layers count as real modes, the issue of modal

affordances mentioned by several authors, the relation between visuals replacing

words as in Figs. 1 and 2 and gestures filling in syntactic slots by replacing rather

than accompanying words, and so on). Once again, one could argue that these

elements are actually part of language itself and not of the context, yet even then it

is remarkable that the multimodality of speech is only given lip service if the

volume is to start from a broad notion of text.

The editors admittedly state in the preface that the book ‘‘cannot offer an

encompassing view of all fields, topics, and areas of application’’ (p. XII, my

translation). This is definitely true even when only considering the part of the field

of multimodality research that the book is covering. Nevertheless, by leaving aside

1 Although Schmitz explicitly refers to authors who restrict ‘text’ to written materials.

N.-M. Klug and H. Stockl (eds): Handbuch Sprache im… 287

123

the phenomena just mentioned, the volume is somewhat one-sided and the topics

and approaches discussed only cover a small part of the broad field suggested in the

title—smaller than justified by the editors’ disclaimer in the preface. Hence, it

would have been good if the editors had been more explicit about this restriction

from the outset—in the preface, or perhaps even already in the title (or a subtitle).

Conceptual and Terminological Issues

One of the aims of the book is the striving for ‘‘a clearer contouring and further

canonization’’ (p. VII, my translation) at the level of terminology, and indeed a

terminological ‘‘harmonization’’ would be a goal worth pursuing. This is shown all

the more clearly when reading the volume, as several (if not most) contributions

contain a section in which definitional issues are addressed. To what extent the

volume actually contributes to a more streamlined and uniform terminology is

unclear, as the individual contributions take different positions on certain matters,

but thereby the volume at least puts the finger on some sore spots where further

unification could be beneficial.

One case in point has been hinted at in the previous section already: what counts

as language? Are the notions verbal and linguistic synonymous or not? This issue is

hardly mentioned explicitly in the volume, but most contributors seem to assume

implicitly that they are, using the labels linguistic and language to refer solely to the

verbal layer of expression. Nina-Maria Klug (chapter 7), for instance, actually uses

the notions para-/nonverbal and para-/nonlinguistic interchangeably without

addressing this explicitly, and similarly, on page 274, Andrea Mayr refers to ,,both

linguistic and non-verbal semiotic resources‘‘, suggesting that what is non-verbal

cannot be linguistic. However, this is not really compatible with Fricke’s (2012)

notion of multimodality that several contributors refer to: to Fricke, ‘language’ is

inherently multimodal; it is more than just the purely verbal layer of spoken or

written (or, for that matter, signed) words and also involves other layers of

expression (in her case especially gesture). At this point, we actually see the

ambiguity McNeill (2005: 21) refers to when it comes to the question whether

gesture is part of language:

First, I use ‘language’ in a technical-linguistic way, to refer to those static

structures of language consisting of grammar, words, etc. In this sense, there is

a ‘language-imagery’ or ‘language-gesture’ dialectic […]. Second, I also use

‘language’ in a traditional nontechnical way, to indicate what it is we know

when we say we ‘know a language’ or what we ‘use’ when we speak, listen,

read, etc. In this way, gesture is ‘part’ of language, implying that language

consists of more than words, sentences, etc., and also includes spontaneous,

speech-synchronized gestures.

As mentioned before, gesture does not play a central role in most studies presented

in the volume, but other para- and nonverbal layers of expression are at least in a

similar position vis-a-vis the verbal level. If, following scholars such as Fricke and

McNeill, we use the term language for the multimodal whole, a terminological

288 S. Schoonjans

123

distinction is called for, for instance by using linguistic to refer to the entire

multimodal ‘matrix code’ (as Fricke calls it) while using verbal to refer only to the

purely verbal layer of (spoken, written, or signed) words. In this sense, nonverbal

does not by definition imply nonlinguistic2: para- and nonverbal layers of expression

such as gesture, prosody, and typography are inherently not verbal but may very

well be considered as linguistic. But even if the terminological distinction is not

made in this way, the bearing of the term language should not be left implicit

(especially in a volume on multimodality), as this does have its implications for

which modes are part of language and which modes language can interact with—

hence, what is part of multimodal language and what is part of the multimodal

context.

Now the reader may wonder why I introduced yet another notion which is not

central to the volume, viz. layers of expression, rather than using the more common

and central notion of mode. This is because I wanted to avoid any possible

implications with regard to another terminological issue, namely what counts as a

mode. This is indeed a question several authors discuss in their contributions—one

more extensively than the other. However, most of the time, we do not actually end

up with a clear definition of mode—which fits in nicely with John A. Bateman’s

observation (p. 38) that ‘‘arguments in favour of maintaining a certain looseness in

definitions of multimodality’’ are not uncommon. However, Bateman argues on the

next page, ‘‘it is only with more precision that such complex phenomena can be

addressed productively.’’ While the question can be raised how far this should be

pushed, it is clear that a certain terminological ‘‘harmonization’’ (re-using the term

the editors propose) could be helpful in aligning different perspectives and facilitate

exchange and discussion among scholars, as indeed several questions receive

different answers throughout the volume.

A nice example of this is the question whether paraverbal layers of expression,

such as prosody, typography, and layout, should be considered as real modes. Ulrich

Schmitz (p. 332) explicitly argues against this view: to him, these layers are always

tied to one of the basic modes (including oral and written language) and accord

them a form of appearance and manifestation. In other words, oral and written

language cannot exist without prosody, typography, and the like, so the latter are not

true modes but mere channels or tools to realize fully-fledged modes such as speech

and writing. On the other hand, while admitting that the verbal can only exist

through the paraverbal layers that embody it, Nina-Maria Klug (p. 168) still speaks

of ‘‘paraverbal sign modalities’’ (my translation). Similarly, though not stating

explicitly if it is to be seen as a true mode, Jurgen Spitzmuller at least points out that

typography is clearly more than ‘‘just the ‘incorporation’ of written language’’ (p.

100, my translation). This actually also takes us back to the previous issue, as

Spitzmuller raises (but does not really answer) the question whether typography

should be seen as a part of language or rather as something different but closely

related to it. However, this question is actually more complex than just deciding

2 Note that, unlike what Bohle (2007) claims, nonverbal, when used in this way, does not necessarily

imply a hierarchical downgrading vis-a-vis the verbal level, and indeed no such hierarchy is intended

here.

N.-M. Klug and H. Stockl (eds): Handbuch Sprache im… 289

123

upon whether language refers to the multimodal whole (Fricke’s matrix code) or

only to the verbal layer: we also have to decide upon whether we consider language

as one mode (as several contributors suggest)—it probably is if it only involves the

verbal layer, but things become less clear if we include paraverbal aspects, let alone

nonverbal resources (so language would be inherently multimodal but still

constitute just one mode). This again takes us to another issue left rather implicit

in the volume, viz. the relation between the notions mode and modality: can we say

that language is one multimodal mode consisting of several modalities? Further-

more, it still remains to be clarified whether spoken language and written language

should not be seen as two separate modes rather than one single mode ‘language’

(cf. John A. Bateman, p. 38). Hence, we are really dealing with a web of intricately

related questions, and resolving the discussion is not easy, as the answer to one

question almost necessarily has implications for other questions.

But even if we managed to come up with a solution to these matters, the

terminological fuzziness would still not be eradicated: notions such as language and

mode/modality are fundamental to the topic of the volume and thus receive attention

in several contributions, but a close look at the volume still reveals other points for

which a terminological ‘‘harmonization’’ could be helpful, especially in view of

increasing exchange and collaboration between fields in multimodality research. An

example would be the triplet modal affordance (Jewitt and Henriksen)—semiotic

potential or reach of mode (Klug)—semantic potential (Eisenlauer). Not only can

the reader only assume to what extent these notions have the same intension, it is

also not entirely clear how they relate to similar notions such as meaning potential

as it is used for instance by Halliday (1978: 27–28) and Langacker (2008: 29).

In cases such as these, it seems reasonable to assume that differences in

terminology may also reflect differences in perspective or framework. However, the

role of perspective and framework goes further than the purely terminological level

and also affects conceptual matters and methods. A case in point discussed in the

volume is the issue of mode hierarchies. On the one hand, there are scholars who

assume that one mode can be dominated by the other: Winfried Noth (chapter 8)

shows how an image can be superior or inferior to the verbal level when it comes to

semiotic potential, and Bezemer and Kress (chapter 21) discuss textbook designs in

which writing or image is ‘‘prior’’ (p. 483, 487) or ‘‘dominant’’ (p. 488). On the

other hand, Christina Margrit Siever (chapter 20, p. 464) refers to the difficulty of

deciding upon such dominance relations and, following Hartmut Stockl (who does

not, however, mention the issue explicitly in his contribution to this volume), rather

proposes to ‘‘assume the general reciprocity of semiotic processes at the interface of

both sign systems and describe it in a differentiated way’’ (my translation). In this

view, describing how the modalities interact within the multimodal whole takes

precedence over determining which modes are more important than others. Of

course, the answer to such a question depends on one’s perspective and aims, and

striving for harmonization in such cases would almost by definition do injustice to at

least one of the perspectives at stake. Strictly speaking, the diversity of perspectives

and frameworks is not a real problem (quite the contrary: it can be an impetus for

further research and shows that the field of multimodality research is ‘alive and

290 S. Schoonjans

123

kicking’)—provided that it is stated clearly which position is taken and how terms

are used.

Summing up, the volume clearly illustrates the point made by the editors in the

preface that research would—or at least: could—benefit from ‘‘a clearer contouring

and further canonization’’ at the level of terminology, as indeed, at several points in

the volume, we are confronted with notions that are defined in different and unclear

ways and, consequently, with different notions referring to related phenomena but

where it is not clear to what extent they can be used interchangeably. Sometimes

these matters are discussed explicitly (as with notions such as language and mode),

while in other cases they remain rather implicit (for instance the modal affordance/

reach of mode/semantic potential case). However, it is not clear to what extent the

volume actually helps in coming to a terminological ‘‘harmonization’’—at best, we

can say that it takes the first step in this process by showing what the issues are, but

a lot of questions still remain without a clear answer. Moreover, as was shown by

the comment on mode hierarchies, a full ‘‘harmonization’’ is perhaps not what we

want, as it would do injustice to the variety of approaches and frameworks that

necessarily implies different perspectives. To some extent, ‘‘harmonizing’’ termi-

nology could be beneficial and contribute to an easier exchange of thoughts and

views among paradigms, but more importantly so, authors should be more explicit

about how they use certain notions and not take for granted that readers

automatically know what is meant exactly. This is something most contributors to

the volume seem to have understood, looking at the number of sections on

terminological and definitional matters throughout the volume.

Conclusion

Taking together all points made above, the general assessment of the volume and

the individual contributions can only be positive. There clearly are some

imperfections, but even the most important shortcoming is not grave enough to

justify a negative critique: the volume only covers a restricted part of the field

opened up by the title, paying limited attention to areas such as the typical

multimodal dimensions of spoken interactions, but at least the areas that are

discussed are presented in an accessible way that is suited for the genre of a

handbook. Hence, the book really does what a handbook is supposed to do:

introduce the field (first part) and give an idea of the multitude of possible research

lines within the field, presenting different possible approaches and frameworks

(second part) and discussing different types of data to which these approaches can

be applied (third part). Furthermore, the volume is not restricted to traditional

studies but also leaves room for newer and more innovative lines of research,

including diachronic and cross-cultural comparisons, and it also points out

terminological and conceptual issues and differences between approaches, which

may hamper exchanges between scholars if one is not aware of them. Hence, despite

the imperfections listed above, the book definitely presents a good resource for

scholars who want to familiarize themselves with this part of the field of

multimodality research.

N.-M. Klug and H. Stockl (eds): Handbuch Sprache im… 291

123

In this respect, the volume comes at the right moment, as indeed there is growing

conscience in linguistics and media and communication sciences that language is

more than just words: it is inherently multimodal and ‘‘if one wants to describe

language as it is used, it cannot be detached from its multimodal context’’ (Nina-

Maria Klug, p. 168, my translation), as every mode and every semiotic resource

involved contributes to the construction and expression of meaning. One may raise

the question whether linguistics has to take all of these para- and nonverbal

dimensions into account, but given the multimodal nature of language, including

them is the only way of studying language in an ‘‘adequate’’ (Schmitt 2005: 23)

way. Furthermore, as Nina-Maria Klug (p. 186, my translation) writes,

With this extension [to include the para- and nonverbal, S.S.], one should not

fear the loss nor the total de-bounding of the linguistic research area. Rather, it

is a chance for a phenomenon-oriented pragmatic linguistics that aims to

describe its object of research in the most holistic way possible. Since

language use is in essence multimodal, communicative properties and

functions of language in use cannot be described in a way that is suited for

the phenomenon when this is done in isolation, but always only through the

interaction with and the contrast vis-a-vis other communicatively exploited

semiotic resources. This opens up a wide and promising area for future

research.

And precisely that is where this volume can contribute.

References

Bohle, U. (2007). Das Wort ergreifen – das Wort ubergeben: Explorative Studie zur Rolle

redebegleitender Gesten in der Organisation des Sprecherwechsels. Berlin: Weidler.

Fricke, E. (2012). Grammatik multimodal: Wie Worter und Gesten zusammenwirken. Berlin: De Gruyter.

Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and

meaning. London: Edward Arnold.

Langacker, R. W. (2008). Cognitive grammar: A basic introduction. Oxford: Oxford University Press.

McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.

Muller, C., Cienki, A., Fricke, E., Ladewig, S. H., McNeill, D., & Bressem, J. (Eds.). (2014). Body—

language—communication: An international handbook on multimodality in human interaction (Vol.

2). Berlin: De Gruyter Mouton.

Muller, C., Cienki, A., Fricke, E., Ladewig, S. H., McNeill, D., & Teßendorf, S. (Eds.). (2013). Body—

language—communication: An international handbook on multimodality in human interaction (Vol.

1). Berlin: De Gruyter Mouton.

Rohen, H. (1981). Bilder statt Worter. Zeitschrift fur Germanistische Linguistik, 9, 308–325.

Schlobinski, P., & Watanabe, M. (2003). SMS-Kommunikation—Deutsch/Japanisch kontrastiv: eine

explorative Studie. http://www.mediensprache.net/networx/networx-31.pdf. Accessed 4 Apr 2017.

Schmitt, R. (2005). Zur multimodalen Struktur von turn-taking. Gesprachsforschung, 6, 17–61.

Siever, C. M. (2015). Multimodale Kommunikation im Social Web: Forschungsansatze und Analysen zu

Text-Bild-Relationen. Frankfurt am Main: Peter Lang.

Wietholter, W. (2008). Ikono-Graphie. In A. Bohnenkamp & W. Wietholter (Eds.), Der Brief – Ereignis

& Objekt (pp. 117–189). Frankfurt am Main: Stroemfeld.

292 S. Schoonjans

123