DOI 10.

1515/text-2014-0007 Text&Talk 2014; 34(3): 231237

Elisabetta Adami and Gunther Kress

Introduction: multimodality, meaning
making, and the issue of text

Elisabetta Adami: Dipartimento di Lingue, Letterature e Culture Moderne, Universit G.

DAnnunzio Chieti-Pescara, Pescara, Italy. E-mail: e.adami@unich.it
Gunther Kress: Department of Culture, Communication and Media, Institute of Education,
University of London, UK. E-mail: g.kress@ioe.ac.uk

Something may be an entire commonplace and yet barely be understood. The con-
temporary media of communication are an instance: used everywhere, by nearly
everyone (even if only in the most rudimentary fashion as in automated check-
outs at the supermarket), and yet they remain just about opaque in terms of their
present social effects and future consequences. In terms of the thematic remit of
this journal, the increasingly insistent presence of multimodal texts is another
such instance: at a different level, though with related profound consequences.
This special issue will not delve into the more far-reaching, more profound,
social consequences; it will confine itself to exploring lesser questions, such as
how do we think about text in a world where multimodal semiotic entities are
beginning to dominate the semiotic landscape? For a journal that has, enshrined
in its title, the aim to expand the notion of text to include talk, what develop-
ments, threats, challenges does the appearance of the notion of multimodality
foreshadow for its field of enquiry?
Here we make a quick list of some of the issues which go to the core of work
represented in this journal; we discuss each one briefly, in turn. The list is meant
neither as a definitive nor as an exhaustive list, but to prompt debate and possibly
further work.
Most of all, the phenomenon of multimodality shifts the center of gravity from
linguistic to semiotic concerns. When all the resources which matter in meaning
are at issue, then the tools of one discipline linguistics and its satellite (sub-)
disciplines can no longer be sufficient to provide satisfactory accounts of the
materials to hand and the questions they pose.
To make this somewhat concrete, consider the screenshot shown in Figure 1.
Assume just for the sake of argument that we wanted to give some account
of this (now entirely usual kind of) semiotic entity we might want to say some-
thing about its genre; maybe about the audience and the kinds of imagined sub-
jectivities of its members; or maybe describe its style; etc. The kinds of tools which
(different kinds of) linguistics offers will not allow us even to begin making any

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
232 Elisabetta Adami and Gunther Kress 

Fig. 1: (Part of) Homepage of the website www.visitlondon.com (accessed 15 December 2013)

inroads. Two alternatives offer themselves at once: we might invent ad hoc cate-
gories, which we bolt onto the linguistics of our choice, and see where we get to;
or we might say: this is not the sort of thing we have an interest in; it is not (a)
text.
Members of a wide and disparate audience who come to this website do make
meaning from and with the resources offered here. They have no difficulty noting
and interpreting in various ways the difference between this web page and the
one shown in the screenshot of Figure 2.
This would introduce a further problem: some web pages, such as the one in
Figure 2, seem amenable, at least in part, to a linguistic analysis of some kind;
while others, like the one in Figure 1, are not. Clearly there are problems here;
turning away from them may not be the perfect response. We can say that the ex-
ample in Figure 1 requires us to take account of the fact that there are images; that
the images unlike the one in Figure 2 are in color; that is, color as well as
image seems to be a resource drawn on in the construction of that semiotic entity.
And of course there is writing. We might need to comment on the varying pro-
portions of writing and image (and color) in the two cases; and we might wish to
hypothesize what might be motivating that differential use of the three modes.
We might further wish to comment on the startlingly different forms of layout of
the two sites a relatively traditional linearity in the one case (Figure 2) and
a relatively strong modularity in the other (Figure 1) and speculate about the
origins and meanings of these different principles of composition: are they effects

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
  Introduction 233

Fig. 2: (Part of) Web page of the Mayor of London, www.london.gov.uk/who-runs-london/mayor

(accessed 17 August 2013)

of different social environments, or of the digital environment, or of both? We

might ask about resources and principles for producing cohesion and coherence
in texts which are based on modular composition and are multimodal in charac-
ter and what follows for conceptions of coherence, socially and semiotically.
Beyond the co-presence of different modes, there is the matter of their dif-
ferent uses and functions: writing is central in one case (Figure 2) and marginal
in the other (Figure 1). If our interest were to remain settled on the two linguistic
modes of speech and writing, we would nevertheless feel a need to reflect about
the changes in function of writing which seem indicated here, and speculate
about the future of the mode of writing in a broad way.
That route takes us beyond description alone, and into exploring connec-
tions between social environments and the respective uses (and functions) of a
mode, or of different modes. Even the two examples used here suggest that we
cannot answer those kinds of questions without considering all the resources
which occur in what is now a larger frame than that of speech and writing alone,
namely of all the modes which are available to members of one social/cultural
group. We have, in taking that step, moved decisively from the one discipline
linguistics concerned with one mode (or the two modes of speech and writ-
ing) to another discipline (social) semiotics and its focus on all resources for
making meaning available for members of social groups as their cultural re-
source. The insights of the various forms of linguistic theorizing and work would

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
234 Elisabetta Adami and Gunther Kress 

remain as a resource to draw on, though they might, given the social semiotic
framing, need to be modified and extended.
As one seemingly slight example, take the realization of URLs, as in the
case here (taken at random from a thread on an article on Syria in The Guardian
newspaper:

http://sabbah.biz/mt/archives/2013/12/15/sy-hershs-chemical-misfire/#utm_source=
feedburner&utm_medium=feed&utm_campaign=Feed%3A+SabbahsBlog+
%28Sabbah+Report%29

We are unable to say whether this is writing in any conventional sense. We are
able to say as any of our readers will agree that such kinds of elements are
entirely usual in contemporary semiotic entities. In an up-to-date theory of text
making it ought to be possible to account for any semiotic entity which the ordi-
nary man and woman in the street encounters constantly, and treats as usual.
There are two further issues which are distinct and closely connected. The
first is that multimodality is an approach is not a theory. It stakes out major
aspects of the domain in which meaning is made. Different theories pose differ-
ent questions within this domain, questions which are central to that theory. The
theory invoked here is social semiotics (Hodge and Kress 1988; Kress 2010; Van
Leeuwen 2005) and, generally speaking, it is used by all the contributors in this
volume. It asks questions, broadly about all aspects of meaning and meaning
making. We are aware of the limits of applicability of that theory; and where we
come to its margins, we are happy to enlist the services of other theories. The
second matter is that meaning is brought into being in particular sites, by those
who make meanings. Some of those sites are those of the new media, so called.
In using two websites as our examples, we have made some slight reference to
such sites already. Yet multimodality as such does not concern itself with sites
of appearance whether these are the material sites of everyday social inter-
action, or the differently material sites of digitally produced and instantiated
media. Quite clearly the potentials and limitations of sites in their material,
technological, social aspects cannot simply be ignored when we think about
meaning. For us, it is social semiotics which supplies hypotheses, descriptions,
frames, and principles here. Multimodal composition as an issue is at once inde-
pendent of sites, and, at the same time, entirely shaped by the characteristics of
the sites.
Above all, we feel there is a strong claim to be made that the issues raised
here and developed in particular ways in the rest of this special volume are issues
which are now central to the domain of work of this journal.

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
  Introduction235

Each of the papers that follow examines a different kind of multimodal

text. These encompass digitally produced texts and transformations of tradi-
tional texts in mass media productions: online videos (Adami) and urban youths
video-based music production (Domingo); transmediated TV series (Moschini)
and new forms of filmmaking (Rowsell); video chats (Sindoni) and PowerPoint
presentations (Zhao, Djonov, and Van Leeuwen).
Focusing on video-based asynchronous communication, Adami discusses in-
teraction practices on YouTube. The paper analyses relatedness in videos linked
as responses to other videos, identifying a wide range of variation. Interactions
include not only remotely related and highly implicit exchanges (the result of
practices of recontextualizing), but also relations hinging on form-as-meaning.
The notion of intertextual relatedness established through form opens to un-
expected uses of the multimodal possibilities of the medium, thus extending the
notion of dialogue.
Discussing another form of digital video production, Domingos ethno-
graphic study examines hip hop music videos produced by a group of UK-based
Filipino young people. It investigates how participants use multimodal resources
to express cultural remix, affiliation, and belonging across different linguistic
and discourse communities. In the participants use, modes embody social and
cultural significance. Combining both online and offline production, their sign
making extends notions of text to include fluid configurations, as is increasingly
the case in contemporary representation.
That issue is discussed further by Moschini in her analysis of recent trans-
formations in TV series, encompassing multimodal intertextuality, webridized
sign making, and transmedial production. These features can hardly be described
in any traditional textual analysis. Such transmedial processes problematize
what constitutes a text, extending sign making across different spaces and as-
signing co-production to different agencies, including fans.
Rowsell observes filmmakers selection of modal resources in designing doc-
umentaries and 3D animations. In her discussion of their use of modes to evoke
emotions in viewers experiencing of texts, the author raises questions concern-
ing the role of embodiment and perception in meaning making. These questions
inevitably become more compelling as forms of engagement increasingly involve
immersive digital environments and the use of mobile and wearable technologies.
Focusing on a different kind of video production, Sindonis paper examines
a new type of synchronous communication, i.e., video chats, where verbal lan-
guage itself is multimodally orchestrated, with functional loads shifting between
speech and writing and with other modal resources being significantly shaped by
the mediums affordances. The paper discusses the issues faced by multimodal

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
236 Elisabetta Adami and Gunther Kress 

analysts in transcribing and analyzing such a multimodally dense interactional

environment.
As a final example of time-based multimodal communication, Zhao, Djonov,
and Van Leeuwens paper introduces a multimodal social semiotic approach to
studying PowerPoint presentations. These are seen as a multidimensional semi-
otic practice combining the softwares design, the multimodal composition of
slideshows, joined with their spoken performance/presentation in the communi-
cative event. The paper problematizes the notion of text in each dimension; it dis-
cusses the limitations of logocentric approaches to their analysis while proposing
a holistic and adaptive multimodal approach, crucial for the understanding of
time-based multimodal productions.
While the papers raise a series of questions on what is to be considered as
text in contemporary sign making, this special issue does not aim to provide
definitive answers, nor does it intend to provide an exhaustive catalogue of con-
temporary forms of multimodal representation. Both tasks are impossible. First,
new uses of multimodal resources appear every day, thanks to the socially driven
pace of technological change and the sign makers creative uses of technologies
for sign making. Second, and maybe more profound, digitally produced and me-
diated materials are likely to be mere changed instances if at a somewhat
general level of older phenomena. So, by adopting a social semiotic perspective
to text, sign making in digital environments might be making more tangible for
researchers and theorists and bring to a point what had always been there
in all multimodal forms of communication.
This special issue intends to prompt debate, and further work. We feel certain
that with the many questions raised about our understanding of text and mean-
ing, it will offer a perspective that enables a look at texts (and, more broadly, at
meaning making) in a relevant and aptly enlarged frame.

References
Hodge, R. & G. Kress. 1988. Social semiotics. Cambridge: Polity.
Kress, G. 2010. Multimodality. A social semiotic approach to contemporary communication.
London: Routledge.
Van Leeuwen, T. 2005. Introducing social semiotics. London: Routledge.

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM
  Introduction237

Bionotes
Elisabetta Adami is a researcher in English Language and Translation at the
Department of Modern Languages, Literature and Cultures at the University G.
DAnnunzio of Chieti-Pescara, Italy. Her research focuses on language, multi-
modal representation and communication in digital environments. Her recent
publications use a social semiotic framework for the analysis of text production
in social media, the affordances of mobile phones (with G. Kress), and the use of
copy and paste.

Gunther Kress is Professor of Semiotics and Education at the Institute of Edu-

cation, University of London. His interests are in meaning (making) and com-
munication in contemporary environments, with two broad aims: to continue
developing a social semiotic theory of multimodal communication; and, in that,
to develop an apt theory of learning and apt means of valuation of learning.
Some of the books along the road are Language as Ideology; Social Semiotics
(both with Bob Hodge); Before Writing: Rethinking the Paths to Literacy; Reading
Images: The Grammar of Graphic Design; Multimodal Discourse: The Modes and
Media of Contemporary Communication (both with Theo van Leeuwen); Literacy
in the New Media Age; Multimodality: A Social Semiotic Approach to Contemporary
Communication.

Authenticated | e.adami@unich.it author's copy

Download Date | 5/21/14 10:55 AM