Collana di sintomatologia delle apocalissi culturali

diretta da Leonardo Terzo

10 5

Nella stessa collana 1. 2. 3. 4. 5. L. Terzo, Pornografia ed episteme. Per una sintomatologia delle apocalissi culturali. B. Berri, Nathaniel Hawthorne. Dal subliminale al trascendentale. S. Monti, Le vicissitudini della corporeit. Anima e anatomia nella narrativa inglese e americana dellOttocento. L. Terzo, Sublimit contemporanee. B. Berri (a cura di), Saggi italiani su Elizabeth Bowen. (Saggi di E. Cotta Ramusino, S. Granata, S. Monti, L. Terzo, C. Marelli, B. Berri, L. Guerra, J. Meddemmen). Cristina Marelli, The Survival of Literature. Remake Practices from Shakespeare to the Graphic Novel. William Shakespeare, Michael Hoffman, Ingmar Bergman, Woody Allen, Art Spiegelman, Marjane Satrapi. L. Terzo (a cura di), Assurdo, paradosso, follia. Samuel Beckett, Oscar Wilde, William Shakespeare. (Saggi di B. Berri, S. Monti, L. Terzo, E. Zuccato). L. Terzo (ed.), Lunatic Giants, (Essays by B. Berri, F. Ceravolo, C. Marelli, S. Monti, L. Terzo, C. Viola). L. Terzo (ed.), La civetta di Venere, (Essays by P. Nerozzi, Hic Sunt Group, L. P. Ellis, L. Goldheim, V. Tavazzani, L. Terzo).

6.

7.

8. 9.

Mara Logaldo

AUGMENTED LINGUISTICS
Language and Communication in the Age of Augmented Reality

Edizione a cura di Arcipelago Edizioni Via Pergolesi, 12 20090 Trezzano sul Naviglio (Milano) info@arcipelagoedizioni.com www.facebook.com/ArcipelagoEdizioni Prima edizione novembre 2012 ISBN 978-88-7695-480-1 Tutti i diritti riservati Ristampe: 7 6 2018 2017 5 2016 4 2015 3 2014 2 2013 1 2012 0

vietata la riproduzione, anche parziale, con qualsiasi mezzo effettuata, compresa la fotocopia, anche ad uso interno o didattico, non autorizzata.

TABLE OF CONTENTS

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. SEMANTIC COMPLEXITY IN AR 1.1. Partial overlapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2. Mixed reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3. Semantic continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4. Reaching beyond: metaphoric connections . . . . . . . . . . . . 1.5. The Semantic Web Layer Cake on and off line . . . . . . . . . 2. SUPERIMPOSITION 2.1. You are a Layar! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. A brief history of superimposition . . . . . . . . . . . . . . . . . . . 2.3 An aesthetics of concealment . . . . . . . . . . . . . . . . . . . . . . 2.4. Interference patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5. Word and image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6. Verbal and visual superimposition . . . . . . . . . . . . . . . . . . . 2.7. Superimposition in cognition and language . . . . . . . . . . . . 2.7.1. Cognitive blending and metaphor . . . . . . . . . . . . . . 2.7.2. Foregrounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8. Superimposition in AR: encrustation vs. integration . . . . . 2.9. Logical categories in mixed environments . . . . . . . . . . . . . 3. THE WORLD IN A TAG 3.1. The map is not the territory . . . . . . . . . . . . . . . . . . . . . . . . 3.2. The end of photography? . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. Framing and browsing the real . . . . . . . . . . . . . . . . . . . . . 3.4. Tagging the world . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5. Verbal tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6. Tag clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7. QR tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8. AR tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27 31 35 39 44 51 55 66 68 72 75 79 81 84 88 93 97 103 105 114 117 121 129 135

3.9. Secret language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10. RFID tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11. Mobile tagging and ostensive communication . . . . . . . . . . 4. BETWEEN TAG AND SCREEN 4.1. Augmented art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Augmented literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3. Augmented advertising . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4. Augmented cinema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5. Augmented bodies, augmented fashion . . . . . . . . . . . . . . . 5. AR IN EDUTAINMENT, INFORMATION, AND AUGMENTED ALTERNATIVE COMMUNICATION (AAC) 5.1. Relevance theory and AR: towards a redefinition of context . . . . . . . . . . . . . . . . . . . . 5.2. Affordance theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3. Locative narratives and locative games . . . . . . . . . . . . . . . 5.4. Augmented learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5. Augmented journalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6. Making reality speak: AR for the language and visually impaired . . . . . . . . . . . . 6. TOWARDS AN AUGMENTED LINGUISTICS 6.1. Testing perceptive limits: how far can we go? . . . . . . . . . . 6.2. In praise of noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3. Conversational noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4. An augmented idea of felicity conditions . . . . . . . . . . . . . 6.4.1. Quantity and quality . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2. Relation and manner . . . . . . . . . . . . . . . . . . . . . . . . 6.5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AFTERWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

138 140 142 147 152 158 168 169

187 189 192 197 202 208

213 220 225 227 229 232 234 237 239

INTRODUCTION

The title of this work, Augmented Linguistics, was originally conceived to suggest, admittedly in a rather provocative and ironic way, the necessity of a general re-thinking of the fundamental assumptions of linguistics in the era of augmented reality (AR). Although I felt rather pleased with the choice of this expression to synthesize the scope of my study, I lay no claims over the invention of the phrase itself: indeed, there were few doubts about its being a neologism which, like many similar ones, exploited the adjective augmented that has become so fashionable since the spreading of AR to indicate the application of this technology to different fields. Also in the association of the adjective augmented with the noun linguistics there seemed to be nothing particularly creative: it could be as predictable as that between augmented and media, or between augmented and art. However, when I typed the phrase in the Google search box to check as was most likely if somebody else had already used this combination of words and explored the subject, much to my puzzlement I noticed that, while there were thousands of webpages containing the expressions augmented media and augmented art, only two of them mentioned the expression augmented linguistics.
As stargazers avatar may say: an exercise in augmented linguistics it was.1 The call for augmented linguistics is growing.2
1 2

<www.fansofrealitytv.com> <www.alternet.org> 7

Augmented Linguistics

Neither website used the phrase with the particular meaning I intended to give to it. However prophetic, the two excerpts mainly signalled a lack of studies in the field. The former, dated 2003, was a comment made by the fan of a reality TV programme during a discussion forum: the exercise in augmented linguistics consisted in the transcription of a dialogue between two participants in the show, with particular reference to the words spoken by one called Amy. From my point of view the only interesting aspect of the remark was the connection between the qualifier augmented and the idea of difficulty, the reference to something so complex that it verged on the cryptic. In fact, this idea will often recur in this work: indeed, one of the peculiar characteristics of AR is the systematic search for extreme stimuli, whose challenging quality, and the ways people respond to them, constitute one of its distinctive traits. This trait but emphasises the use of language in the era of Web 2.0. Suffice it to think about the encryption of information in the QR codes that have been burgeoning in the past few years on labels, billboards, and magazine covers; or the jagged alternation of fonts of different sizes and intensity of colour in tag clouds, where unpopular items may become so faded to be almost invisible to the untrained eye. The second quotation was clearly more focused on linguistic problems. But, if we read the linked page, an article entitled Set Your Speling Free, we discover that by augmented linguistics the author meant only the wished for adoption of a simplified spelling (speling) system that would according to him - better mirror the way English is actually pronounced, and, by shortening several words (for instance, yacht into yot), would also make the language fitter for the restricted writing space of mobile phones and keypads. This aspect, already widely explored by linguists with reference to different forms of display-based texting (Crystal, 2008), could also be of some relevance in a study on AR, space being a fundamental category of augmented reality. The strategic arrangement in space of virtual objects or texts is actually necessary to the very
8

INTRODUCTION

existence of AR. The multimodal notion of framing (Kress van Leeuwen, 1996; van Leeuwen, 2005: 7-19; 277) can certainly be applied also to AR, since it is the result of a process carried out by framing parts of the world within the display of handheld devices and augmenting them with the digital information provided by the online connection. On the other hand, we know that AR technologies tend to solve the problem of space not by looking for new possible ways of cramming writing and pictures into tiny monitors, but rather by studying how to get rid of displays altogether. Although for the time being we have to rely on smartphones and other analogous devices, the ultimate goal of AR technologies is turning the whole world into a 3D screen that may receive and englobe the computer-generated imagery or text ubiquitously (Boulter - Grusin, 1999: 216-9; Hainich, 2009 [2006]: 17).3 While augmented linguistics seems to be, at least so far, a rather unexplored land, Augmented Communication appears as a more popular subject, also among linguists. For more than half a century - the first experiments can be dated back to the 1950s the expression Augmentative and Alternative Communication (AAC) has been commonly used for all the
3 However, by ubiquitous computing Bolter and Grusin mainly mean the placing of several computer screens in strategic places rather than, for instance, the use of near-eye projectors, the exploitation of atmosphere particles for the projection of 3D imagery or the superimposition of computergenerated messages over the real. The discussion requires a reflection upon the differences between virtual and augmented reality (see chapter 1). In order to achieve this, however relative, autonomy from screens including those of smart phones new devices placed closer to the retina are being experimented. As Hainich points out: This technology, getting as near to the personal communicator as anything so far, will surely hit the wall in the near future, simply because there is no acceptable display, and rollout displays or projectors wont do. Long ago I realized this problem, and now technology is ready to solve it. We need to eliminate the screen in favor of a near-eye projector []. Virtual objects, virtual devices will surround us, everywhere. (Hainich, 2009 [2006]: 17).

Augmented Linguistics

devices that make communication easier for people with speech and language impairments due either to congenital causes or acquired conditions. Also AR technologies can be used in this field, particularly to provide sensory substitutions, such as visual cues to the deaf or aural information to the blind. Given the importance of AR in AAC, some of its applications will be investigated in a chapter of this book (. 5.6.). On the other hand, Augmented Communication considered as the result of the use of augmented reality devices as such, that is without specific remedial purpose, has not been thoroughly explored yet. It has certainly been recognized as a new trend in communication which encompasses all media and different contexts, but only since the past year the interest has gone beyond the study of the marketing possibilities of this new technology. Yet, the subject has been pointed at as a compelling one for speculative study, particularly by cognitive linguists. In the closing paragraph of Augmented communication: The communicative potential of the internet, for instance, the author argues that the access to the internet will one day lead to easier communication online in virtually augmented reality (and later in pure virtual reality) than face to face in real reality, through internet-mediated extensions of the mutual cognitive environment (Philip Diderichsen, 2006). Other studies are following, but they are still at an embryonic stage.4 The necessity to study how augmented reality is changing our way of viewing language and communication has been the starting point for this book. Although I do not in the least consider it an exhaustive work on the linguistic aspects connected with AR there are far too many facets to be taken into account and what we are witnessing at present is only the beginning I
The Centre for Research in Media and Cultural Studies of the University of Sunderland has recently launched a call for papers for a publication focused on the influence of Augmented and Mixed Reality in art, history, literature and media studies. The publication, edited by Jay Bolter and Maria Engberg, is due for the beginning of 2014. 10
4

INTRODUCTION

wish it to encompass a wide range of problems connected with the forms of communication and textual organization that are gradually being introduced by AR technologies. One point I would like to make clear is that this is not just one more book on the language of the Web. Research on Internet-based forms of communication from websites to emails, from blogs to chatrooms of several kinds - have been (and are still being) excellently carried out by linguists who constantly monitor the state of the art of the ever-evolving forms of communication of the Net (Aitchinson, 2001; Crystal, 2001, 2008, 2011; Herring, 1996, 2007; Posteguillo, 2003). Being AR an online form of communication, this work will inevitably touch on some of these aspects (such as the use of tag clouds both in webpages and in AR applications, or the employ of Web 2.0 in social networks and in experiments of hybrid narrative designs) but it intends to concentrate chiefly on those computer-mediated texts that are associated with the textual features of augmented reality as distinguished from virtual reality. These are, for instance, mobile computing and browsing and the information made available by augmented reality appliances on handheld devices (i-Phones, Android smartphones, etc.), tag texts, which are common both to online communication and to AR mobile applications, the use of QR tags in magazines and advertising, but also, significantly, in art and storytelling, the use of AR tags to visualise digital information in different kinds of environments. The principal change that is being produced by the new technology is that thanks to the multiplication of computer interfaces the world and the web are increasingly overlapping. However, this does not necessarily involve the virtual in time entirely substituting the real. The idea at the core of AR is that the real and the virtual are not mutually excluding dimensions. Augmented Life is neither Second Life nor Virtual Life. On the one hand Internet communication can no longer remain immersed in the underworld depths of the computer screen,

11

Augmented Linguistics

where it naturally belongs. On the other hand this does not entail the future of digitalization leading to the total substitution of the real world with a virtual one, either in the claustrophobic rooms of MUDs (multi-user dungeons) or in similarly constraining simulations taking place outdoors but made possible thanks to the use of special devices that exclude the subject from the surrounding context (Poster, 2001). The digitalization of data and imagery will not automatically lead to a futuristic dimension in which people wear special goggles and fumble in a virtual space without any coordinates and connections with everyday life. People have often demonstrated that their nature is amphibious. Generally speaking (there are of course some exceptions) we do not like to consider ourselves either as life-prisoners, although under virtual identities, or wired puppets. We do have aspirations to improve (read augment) our lives, but we also love our real selves and our real context. It is very unlikely that we will choose between opposites. We shall more probably opt for intermediate states, being attracted by opposite poles and different experiences at the same time. It is undeniable that with AR the Internet jumps out of the screen to mix up with the real world: tagging and geotagging are doing precisely this. However, even if it were possible to tag every square metre on earth and all the objects it contains, this would not inevitably involve a perfect fusion of the real and the virtual. A necessary condition for the very existence of AR is an only partial overlapping of the two dimensions. Paraphrasing Cleopatras words in Shakespeares tragedy, like Antonios delights, AR applications are dolphin-like, they show their backs above the element they live in. 5 Which is the World Wide Web. With the aid of AR appliances the Internet can easily emerge on this side of the computer interface and, to some extent, be superimposed on the real world. The terms usually associated with AR themselves, such as for instance Supranet,
5

W. Shakespeare, Antony and Cleopatra, Act 5, Scene II, 79. 12

INTRODUCTION

entail the coming to the surface of Internet applications, over the screen and into the world. This extends the mode that characterises most social networks, significantly named Brightkite and Twitter, suggesting that the forms of communicative exchange and we with them should raise above the Web and, birdlike, fly with the wind. The immense ether, with its connotations of freedom and infinity, will hardly lose its mysterious essence, however numerous the devices and strings of texts that inhabit it. Curiously, a fascinating application of AR made possible by Google Sky Map and Android operating system, is that of learning the name of stars and planets by pointing the smartphones in their direction (Furth, 2011: 12). AR demonstrates that we are not looking for totally immersive experiences on either side of the computer interface (Milgram and Kishino, 1994). With reference to Boulter and Grusins milestone book Remediation, we could say that since its publication in 1999 the contemporary subject has increasingly left behind the twin logics based either on a desire for immediacy (Boulter Grusin, 1999: 9) and transparency or on the awareness of an all-pervasive and stratified hypermediation. The search for experiences that are uncompromisingly immersive has been abandoned in favour of incomplete illusions. As Erkki Huhtamo foresaw, Technology is gradually becoming a second nature, a territory both external and internalized, and an object of desire. There is no need to make it transparent any longer, simply because it is not felt to be in contradiction to the authenticity of experience. (Huthamo, 1995: 171; see also Boulter Grusin, 1999: 42). This does not totally obliterate the difference between the real and the virtual but definitely adds a sense of continuity between the two dimensions: their relationship is perceived as metonymic rather than as merely metaphoric and substitutive. As the expression AR itself suggests, the emphasis is on the word reality. A new kind of realism, hybrid and hypermediated, is steadily spreading. In an AR perspective, the Web will still retain its enormous importance, but principally as a reso 13

Augmented Linguistics

nance box of what is taking place in the augmented ecosystem. Once again, my thought goes to tag clouds, which visualize mixed occurrences taking place on both sides of the screen. The Web will most probably increasingly become the storing place of augmented experience, the place where individual mixed existences will be shared. On this account, the most suitable example of an early application of social networks as repositories is the creation of sites that try to weave narrative designs from individual contributions (photos, writings, videos) perceived as belonging to hybrid ecosystems (Pata, 2010). From this perspective the Web can be considered as the place which, at once, triggers, stores, and implements data and emotive responses. But although augmented realism is bound to become the episteme (Foucault, 1966) of our time, it would be incorrect to think that the shift to the new framework will be easy and undisruptive. As history teaches us, the passage from one idea of realism to another always entails a series of problems. In this case, first of all cognitive problems related to visual perception. These are mainly caused by the superimposition of real and virtual objects. Let us take for example a tag superimposed on a building: which is in the foreground? The virtual tag or the real building? Which is in the background? Is the virtual tag embedded in the real building or is the real building embedded in the virtual dimension of the tag (fig. 1)?
Fig. 1. Source: <http://futuremediachange.com/ 2009/12/augmented-reality-for-non-profits/>

According to Hainich, from a merely technical perspective the problem may be solved in two opposite ways: either by making the
14

INTRODUCTION

superimposed device transparent so that the underlying real object may be seen through; or by overlaying a picture of the real object over the picture of the virtual object (Hainich 2009 [2006]: 47).6 However, Hainichs starting point is the search for a seamless integration of virtual objects into everyday life (Hainich 2009 [2006] :27). This, according to me, is not the point with popular uses of AR. We are not looking for a virtual world that looks real. We just do not care about noticing the seams and joints, (Hainich 2009 [2006] :17) probably because, as we have seen, they have already become ontologically permeable categories. It would rather be more interesting to highlight the different paradigm engendered by the superimposition of the two environments, the very possibility of acquiring a sort of stereoscopic vision.7 This stereoscopic vision does not have to lead to a perfectly consistent 3-D perception of a world which incorporates the virtual into itself. Actually, AR experts hardly talk about stereoscopic vision but, rather, about augmented vision (Behringer et al, 1999). The function of augmented vision, one may deduce, is primarily that of making simultaneously sense of the several layers of which augmented reality is composed. Some of these layers are real, some are computer-mediated: what is important to find out is whether, being co-present, they enhance one another rather than obstruct one another, whether communication proceeds from layer to layer in a way which is smooth, without catches or serration. (Memmot, 2000) and how this process may meet felicity conditions (Austin, 1962; Searle, 1969). Hence the starting point of this book is augmented reality as distinguished from both the real environment and the virtual
6 Hainich describes these solutions as follows: 1) to cut out the shape of the real object from the virtual object and the mask display, in order to uncover it; 2) to overlay a picture of the real object, as taken from the position sensor cameras, over the picture of the virtual object. (Hainich, 2009 [2006]: 47). 7 Stereograms seem to provide this possibility of seeing superimposed objects at once, thus acquiring a 3-D perception of the image.

15

Augmented Linguistics

environment. Indeed, experiments in AR have blurred a clearcut distinction between the real and the virtual world, but they have in no way substituted the former with the latter. The computer screen has become a portable, thin, sometimes almost imperceptible membrane allowing a two-way passage of meaning (Hayles, 2005) in a complex, or hybrid, ecosystem, but has not, at least so far, become totally transparent or all-embracing. Urban settings, in particular, present a jigsaw puzzle of screens in which information seems to flow freely, according to a complex system of signs which incorporates the reflection of the onlooker and interacts with it.8 The above remarks seem to concentrate on the visual aspects connected with AR. But, as always happens with visually-based media, all problems connected with the act of seeing are also bound to turn into linguistic problems. For every seeing act is, inevitably, also a model of textualization by which we make sense of the world. The blurring of the boundary between the real and the virtual environment that I have tried to highlight so far also involves the dimension of language. As David Crystal foresaw, the clearcut distinction between sign language and computer-mediated language can no longer be held: [] the Net is only a part of computer-mediated language. Many new technologies are anticipated, which will integrate the Internet with other communication situations, and these will provide the matrix within which further language varieties will develop (Crystal, 2001: 225). The conception of the world as a hybrid ecosystem in which the virtual and the real co-exist and often mix can no longer limit the field of linguistic studies to one dimension of
8 This aspect was discussed by Mauro Carbone (Universit Jean Moulin Lyon3) in a paper entitled The dream of living in a screen: from movie screens to screen-facades. Reflections starting from Forever 21 Times Square giant interactive billboard presented during the international conference on Media City: new spaces new aesthetics organized by the Triennale di Milano (7-9 June 2012).

16

INTRODUCTION

communication focusing, alternatively, on spontaneous speech, on media discourse or on computer-mediated texts. It has to focus on mixed languages in a mixed reality, even contemplating ideas that were until a little time ago inconceivable.
If you had said to me, a few years ago, that it was possible to have a successful conversation while disregarding the standard conventions of turn-taking, logical sequence, time ordering, and the like, I would have been totally dismissive. But the evidence is clear: millions are doing just that (Crystal, 2001: 170).

Crystal is here referring to synchronous and a-synchronous forms of verbal exchange that we may find in chatrooms. Now, ten years later, these impossibilities have multiplied. Mixed realities, intelligible overlapping messages, enriching noise are just a few of the oxymoronic expressions and aporias (Derrida, 1996) underlying the world of language-based communication in our brave augmented world. The experience of AR is so complex that not even a multimodal approach (Kress van Leeuwen, 2001) can thoroughly pay justice to its intricacy. For multimodal linguistics analyses the interaction between different forms of communication both visual and aural ones, pictures, videos, written and spoken texts, in media such as printed magazines, cinema and television, the Internet but not the short-circuit deriving from the superimposition of the modes which concur to create virtual multimodal texts to the modes which concur to create real multimodal texts (in the semiotic sense, that is including both mediated texts and objects belonging to the experiential world alike). On the other hand I think that, in order to analyse the interaction between virtual and real layers, it can be useful to borrow from the multimodal approach some of the categories it utilized to study the relationship between word and image. For instance, the overlap which, in advertising is likely to occur between the fantasy envisaged in the picture and the reality given by the text and the picture(s) of the product (van

17

Augmented Linguistics

Leeuwen, 2005: 12) can be translated in AR in a reflection upon the overlap occurring between the virtual dimension of the tag and the real world. More generally, the notions of anchorage and relay, and the notions of permeability, porosity, and overlap (Barthes, 1977: 40 ff.; van Leeuwen, 2005: 229), which in multimodal linguistics are often adopted in the study of the interaction between the text and the image, will often recur in this text with reference to AR. A fundamental point in common with the multimodal approach is the belief that the distinction between text and visual image can no longer be held. This is particularly evident in the use of typographic signs (van Leeuwen, 2005: 27-9). Tag clouds, for instance, are pictures and texts at the same time (see 3.6.). The complexity in the relationship between the word and the image can be seen in any kind of AR application. In the use of handheld devices, for instance, the interaction between the information framed by the display and the surrounding or embedded - real world is also problematic from a linguistic perspective. As Rodowick finely argued in his essay Reading the Figural or, Philosophy after the New Media (2001) - it is the very status of the image in relation to discourse which has been questioned by the advent of the digital era. The separation between the temporal dimension of language and the spatial dimension of the image can no longer be unambiguous: [] in the era of electronic and digital communication, the figural is increasingly defined as a semiotic regime where the world of things is penetrated by discourse. And, recalling Foucaults reading of the figural as similitude: Here the figural disturbs the collateral relation that defines figure and text into two separate streams, one characterized by simultaneity (repetitionresonance), the other by succession (difference-affirmation) (Rodowick, 2001: xii). As a result, he concludes text was spatialized, thus losing its uniform contours, fixed spacing, and linear sense, and, at the same time, space was textualized (Rodowick, 2001: 3).

18

INTRODUCTION

This anticipates a possible definition of AR: a spatialization of text or a textualization of space. This idea, of course, is not new: since the advent of semiotics, space has always been treated as a text (Sebeok, 1997a). Nonetheless AR shakes the ground of this distinction even more violently and makes the hybridization even more cogent, because it does not identify or substitute text and space, it has them partially overlap. What actually happens is that mediated discourse penetrates the objects that surround us. And, I would add, it is not important whether these objects are virtual or real, because discourse infiltration works in either case, thanks to a technological handling which involves images and text strings alike. All the issues rather clumsily I am afraid illustrated in this introduction will be extensively dealt with, and in a more orderly fashion, in the ensuing chapters of this book. The first chapter will set the framework to the study of language in augmented reality. It will introduce the fundamental concept of AR: the existence of a multi-layered discourse and its way of affecting the world of communication. It will focus on context as a hybrid ecosystem which blends real and virtual objects, paying particular attention to the problematic notion of semantic continuity between digital contents and the actual world. Chapter two will concentrate on the historical, aesthetic and cognitive aspects involved by superimposition. This issue will be first explored with reference to the visual aspects, then in relation to the overlap of verbal and visual layers and, finally, within language itself, proposing an overview on the commonest linguistic phenomena that imply the idea of superimposition, such as cognitive blending, metaphor and foregrounding. Chapter three investigates the process by which the world and the web can be made isomorphous, a condition which is necessary to the very existence of AR. After a survey on the notions of representation, framing and browsing, it will focus on the meanings of tag and some of the most relevant applications

19

Augmented Linguistics

of tagging. Particular emphasis will be given to the tag conceived as a kind of sign that, on the one hand, strongly shakes the Saussurean idea of the arbitrariness of the signifier in relation to the signified (Saussure, 1916), while, on the other hand, it confirms Peirces position which stresses the crucial role of the interpretant (Peirce, 1934). The linguistic and communicative implications of the tagging process will be mainly observed in the kind of text which can best visualize them: the tag cloud. Chapter four explores both the aesthetic and the practical changes brought about by AR in the field of the media, art, literature, and fashion. In art, a blatant example is the session recently dedicated to AR art at the MoMA in New York. In literature, it will be interesting to analyse first the hypertexts which seem to forerun the modes of AR by showing an extensive superimposition of texts and images, such as Memmots From Lexia to Perplexia (2000), and then those that overtly apply AR technologies to the literary text, as in the experiment Between Page and Screen recently carried out at the Otis College of Art and Design in Los Angeles. Chapter five will re-propose the notions of relevance and affordance by applying them to mixed environments, giving examples of the related possible applications of AR in edutainment, information and augmentative alternative communication (AAC), particularly to provide sensory substitution to the visually impaired. Chapter six will revisit some of the main theories of linguistics in augmented reality environments. An important assumption that needs to be re-examined is the notion of noise, particularly in view of the positive value attached to interference, which in AR is no longer seen as an infringement to successful communication but as an augmentation of meaning, as added value. Finally, starting from Grices maxims of quantity, quality, relevance and manner (Grice, 1975), I will concentrate on the changes provoked by AR vis--vis the pragmatic aspects of communication.

20

INTRODUCTION

My attempt cannot but be cross-disciplinary. This means that, at worst, my way of proceeding will be far from systematic, and that, at best, it will open up some new perspectives for further analysis.

21

AUGMENTED LINGUISTICS
Language and Communication in the Age of Augmented Reality

Yet is not language itself taking on new forms in the era of virtual reality and cyberspace? Can one rest with a selfstyled deconstructive stance when the material infrastructure of the sign is being so drastically reconfigured? (Poster, 2001: 125-126). [] it is at least possible that new technologies, increasingly ubiquitous, multi-purpose and natural in terms of their interfaces, will help create a fourth dimension of communication in the same way that writing created a third (Kress van Leeuwen, 2001: 11). [] programmable media provide arbitrarily numerous means to realize, in program and performance, complex relationships between the symbolic realm of language and the world it dwells within, represents and constitutes. To achieve this we require a textuality of complex surfaces, capable of conveying a multi-dimensionality that is commensurate with lived human experience, including the structured culture of human time (Cayley, 2005).

1.

SEMANTIC COMPLEXITY IN AR

1.1. Partial overlapping

It is like Internet, but it is outside the Internet. (Sterling, 2009: 139) a distinction between clear-cut unambiguous boundaries and fuzzy boundaries. But the terms I have used differ overlap in the one case, and permeability in the other. Clearly another, slightly more general term is needed, a term which can encompass the many ways [] in which boundaries can blur and categories overlap. (van Leeuwen, 2005: 19)

According to researchers who have concentrated on the possibilities of AR, it is difficult to support the idea that computergenerated data and imagery will forever be perceived as something that stands out of the real context or is simply superimposed on it. For the time being, for instance, holograms are just fascinating intruders, but they will probably not be recognized as such in the long run, nor will they exhaust all the potential of the medium (Hainich, 2006). One possible question is whether we shall get accustomed to augmented reality as we have got
27

Augmented Linguistics

accustomed to our television sets, whether virtual objects will soon simply become part of the furniture. Maybe they will, even with a vengeance. While the distinction between the real world represented, say, by the room we are in and the images flickering on the television screen can still be felt, even after more than half a century of daily fruition (Fiske, 1987), ubiquitous computing might soon turn digital imagery into something which is part and parcel of the surrounding context (Hainich, 2006). For, however used we have become to television, we know that its images exist only within the circumscribed space of a screen, while AR owes its very definition to its being potentially unframed. As I mentioned in the introduction, ubiquitous computing may be resolved, rather than with the wearing of helmets and goggles, which inevitably fall short of the sophistication and precision of eyes and ears, with the use of tablets that can be compared to portable windows (see, for instance, the project iTacitus, Sterling, 2009: 138-142). This would bridge the gap between the virtual and the real, at least from the point of view of visual perception. Also other systems of visualization such as Mirage TM try to minimize the discrepancy between the real environment and the superimposed recorded image by adopting lightweight, ergonomic and see-through devices. The merging of the virtual and the real actually seems to be the primary aim of these systems of technological visualization, often with useful practical applications, as in medicine, surgery, e-learning and teleconferencing. However, in spite of these functional uses, AR has mainly been surrounded by a halo of futuristic awe, opening in front of us fantastic scenarios: windows with virtual views on the ocean, the possibility of turning a grey sky into a perfectly blue one and add little patches of sunlight to the scenery (Hainich, 2009 [2006]: 140); even the perspective possibility of getting rid of people we do not like has been envisaged.1 Given this general kind of approach,
1

See www.openthefuture.org. 28

1. ~ SEMANTIC COMPLEXITY IN AR

it is not surprising that scholars have described AR by resorting to analogies with science-fiction: while Boulter and Grusin, for instance, refer to the film Strange Days (Boulter Grusin, 1999), others frequently remind the reader that the future applications of AR could transform the world into the setting of one of Gibsons novels (Tagliagambe, 2009: 80). In actual fact, most of the AR applications described so far are just fascinating hypotheses that deliberately go beyond the scope of augmented reality as such. Significantly, the subtitle of Hainichs book is Augmented Media and Beyond (2009 [2006]), thus acknowledging that he is trying to foresee possible future situations rather than simply record what is available on the market today. At least for the time being, AR is far from entailing completely virtual experiences. I am not denying the existence of cyberspace (Gibson, 1984; Levy, 1997): the very perception of space as a mixed dimension made of both real and virtual objects must change our view of the ordinary world. But I intend to consider AR, as I think it actually is now, as a dialectic space, as a hybrid dimension in which the virtual constantly tends to actualization. By actualization, I mean, with Pierre Lvy, a dynamic configuration involving the production of new qualities, a transformation of ideas, a true becoming that feeds the virtual in turn (Lvy, 1998: 25). Or, we could also define it as a borderline on which the tension between the real and the virtual can constantly be felt. As Silvano Tagliagambe writes:
[] rather than going beyond the real world, plunging directly into a virtual reality, we should operate on the borderline between the outside and the inside of the latter, that is, on the interface between virtual reality and ordinary reality. [] a liminal reality, a frontier, a place where different states meet and are mediated. A place dominated by the category of the inbetween. (Tagliagambe, 2009: 88; my translation)

The idea that we live in a hybrid ecosystem has gradually become dominant. Actually, it is not an utterly new concept.
29

Augmented Linguistics

Tagliagambe reminds us that, as early as in the 1920s, the Russian philosophers Vernadskji and Florenskij already claimed that the world should be considered as a totality, a whole, a biosphere in which information is transmitted from every organic and non-organic category to other organic or nonorganic categories. No real gap should be perceived between the geosphere, the living world, and the noosphere, which is the dimension of thought and culture: they all influence one another in a complex mechanism of transformation and translation. This happened well before the discovery of quantum physics (Tagliagambe, 2009: 15). Today the notion has been widened, including all computer-mediated forms of communication; on the other hand, the holistic perspective that was implicit in twentieth-century theories has been left behind in favour of more relativistic ones. Kai Pata, for instance, speaks of a deliberate blending of geographical spaces with collaborative environments (such as blogs, microblogs, social repositories and social networks). She believes that there are no longer any boundaries between the real and the virtual world, actually, that all boundaries have blurred. Among these, the boundary between the subject, the computer, and the augmented world: Augmented reality makes us distributed beings spatially, and activity based (Pata, 2010). This all-pervasive technological space strongly affects cognitive and linguistic processes. Katherine Hayles was, with Pierre Lvy (1994) one of the first philosophers to theorize about the intrinsic relationship between mind and technology: language does not exist apart from its penetration by code, so the subject does not exist apart from the technology that produces it and that it also produces (Hayles, 2005 ). It will by now have become clear that it is not easy to delimit the space of ordinary reality; and that it is impossible not to go somehow beyond it: we actually do so all the time, and in the most disparate ways. Yet, this does not jeopardize the idea that the achievement of a perfectly seamless continuity between

30

1. ~ SEMANTIC COMPLEXITY IN AR

the real and the virtual may not be the primary aim of AR. If this complete integration were actually feasible, it would lead to the obliteration of reality itself in the name of an all-embracing virtualization. And this would mean the end of virtualization, too. Resorting to linguistic categories, we could make about AR the same remark that has been articulated about metaphor: it is difficult to sustain that there is a place in language which is uncompromisingly literal; yet, if we discard the existence of this dimension altogether, we also deny the existence of figurative language: If we deny the literal in language, we deny the possibility of metaphor as well (Kittay, 1987: 20). Similarly, we could say that it is difficult to find a place in the world which is not virtual, yet, if we deny the existence of the real altogether, we also deny the possibility of the virtual.

1.2. Mixed reality

The idea of a continuum existing between the real and the virtual environments, not only due to technology but also as the result of an ontological change in the perception of reality itself, is a far more stimulating approach. We can find this kind of representation in Paul Milgrams and Fumio Kishinos definition of AR (1994), where the continuum is graphically represented as follows (table 1):

Table 1. AR graph by Paul Milgram and Fumio Kishino (1994)

31

Augmented Linguistics

According to this diagram, augmented reality belongs to Mixed Reality. It is therefore a hybrid dimension, a computer-mediated ecosystem spanning between the two poles, though its place is closer to the Real Environment than to Augmented Virtuality, which is closer to the Virtual Environment. In slight contrast with this graph, which suggests the idea of several degrees of reality or virtuality, Ronald Azuma (1997) proposes a general description based on the following three conditions 1) AR combines real and virtual; 2) is interactive in real time; 3) is three-dimensional. Let us consider both positions: in the above graph Augmented Reality is included in the span containing the whole wide category of Mixed Reality. In Azumas definition, emphasis is given not only to the hybrid nature of AR, but also to its being interactive and three-dimensional. Both, however, start from the idea that in AR levels of reality are mixed or combined. This is perfectly consistent, though I would accept the idea of a mixing or combination of levels of reality only if it does not entail a complete confusion of the levels themselves. On the one hand AR is bound to remain a mixed experience (at least in the short and medium terms): one composite context will derive from its extensive application, in which computergenerated texts and images will be perceived neither as a mere excretion nor as belonging integrally to the context. On the other hand this continuum would be better represented as a series of superimposed three (but also two)-dimensional planes rather than by an unbroken line. This diagram (table 2) is more indicative of my perspective, which is based, rather than on the idea of a continuum between the virtual and the real environment, on the belief in the partial overlapping of these dimensions.

32

1. ~ SEMANTIC COMPLEXITY IN AR

Table 2. Mixed reality diagram.

The tension between real and virtual objects should be preserved, if only to keep us interested in what surrounds us. As Ken Perlin remarked in one of his blogs, entitled You cant live in the future for more than five minutes:
[] the human brain is simply not wired to sustain a sense of novelty. Unfortunately, all new things on our event horizon become reduced to the mere normal with astonishing rapidity, and our voracious and fickle appetite for the new and different can all too quickly lead us to consume the very change we wish to enjoy. We eat the future for breakfast, by mid-morning we have indigestion, and by lunchtime we are hungry again. [] Even if you were to build a time machine, put on your silver lam suit, set your flux capacitor to full forward thrust, and emerge two hundred years in the future, you would have only about five minutes to enjoy the sensation, more or less. During that time you might marvel at the wonders of antigravity, the graceful arc of the protective energy dome over your city, the glint of sunlight off the floating skyscrapers in the sky above, or the way your brain tickles from the seamless techno-telepathy that appears to have rendered both TV and the internet obsolete. But after about a minute or so, your brains novelty
33

Augmented Linguistics

normalization filter will begin to kick in. Within three minutes everything around you will start to seem obvious, even prosaic. After five minutes youll once again simply be living in the ordinary present. Yes, it will be a present that contains floating cities, free infinite energy, shimmering holograms you can control with pure thought. But none of that will matter once you get used to it. It will just be normal.2

However open to future possibilities mixed reality may be, we should keep within the bounds of what is technically available today, and, which is even more important, within the bounds of what peoples communicative and technological demands make of AR today. The latter is, I think, an essential point. Because technological innovations are usually accepted only to the extent to which they meet our desires. Producers of AR systems know this, as the following online advertisement shows:
Our mission is to bring augmented reality technology to the user level, and offer complete solutions that are simple to use and do not require knowledge of the underlying aspects of the technology. We want our clients to focus on their own AR applications and reach their goals as fast as possible and with minimum effort. Giving excellent customer service is our priority and we are committed to helping our customers integrate our technology to their projects.3

Some technologically available innovations have disappeared into nothingness or found practical applications much later in time simply because people found them too difficult to use, or because they did not need or like them (Crystal, 2001: 226; Fuhrt, 2011). A crucial example is television: technically available since the 1930s, it became popular only in the 1950s, when the economic and cultural conditions were favourable to
http://blog.kenperlin.com/?m=201001&paged=3>; posted on January 11 , 2010. 3 http://www.arcane-technologies.com/en/
th 2

34

1. ~ SEMANTIC COMPLEXITY IN AR

it. In his last book, Culture and Explosion (1992), Juri Lotman brilliantly summarized this process which involves the turning of revolutionary scientific discoveries (the explosive event) into gradual technological progress: But at the very moment when the explosion releases its internal energy it is changed by the chain of cause and effect and enters into the time of technology. Logical development selects from the explosion those ideas whose time has arrived and which can then be used. (Lotman, 2009 [1992]: 61).

1.3. Semantic continuity

The awareness of the necessity of a semantic continuity between physical and digital data and the importance of a knowledge concerning the users demands has already led to significant experiments within the Internet. Tim Berners-Lee was the first to conceive the idea of a semantic web where data and information should be associated according to complex processes of interpretation building up a network of relationships and cross-references that are far more sophisticated than a mere syntax or key-word-based linking (Berners-Lee et al, 2001). This aim could only be achieved by substituting syntaxbased linking procedures with semantics-sensitive ones. Intelligent agents should be able to recognize the meaning of online texts and therefore direct the user to the needed information, moving from link to link in a logical and coherent way, mapping data according to the classes and concepts belonging to particular domains. Significantly, these agents are not the outcome of experiments in Artificial Intelligence but rather complex applications, such as URI (Uniform Resource Identifiers) and other XLM-based standards such as RDF (Resource Description Framework) and RDF Scheme (Dorati Costantini, undated).

35

Augmented Linguistics

If the problem of defining semantic coordinates is crucial to the Internet, it becomes even more vital in AR, where the connection of different dimensions, of virtual and concrete worlds, makes the need for semantic coherence and cohesion even more urgent. Indeed, AR is activated by the physical context. The augmentation, consisting of digital imagery or strings of texts retrieved from the Internet is the result of the act of pointing at physical objects in the surrounding environment with a handheld device provided with AR technology. Obviously, the information called about by this pointing must be consistent with the existing world. Studies concerning semantic continuity between real and digital data are routinely made by marketing experts who are aware of the importance of context to give users the information they want. The following example, taken from a website, shows the results of research made in the field of Mobile Tourist Information Systems. The description specifically refers to the notion of Context concepts:
For a mobile information system, several aspects of context can be considered [], such as the characteristics of the particular mobile device (storage and screen size) and network (bandwidth and peers), context of the application (requirements in storage, download and display capability), context of the user of the system (e.g., time, location, interests), context of information objects (e.g., location). The handling of the concept data depends on the intended usage: information about the mobile device could be used to adapt the networking mode (to gain efficiency in the system communication) or to adapt the information display (to gain effectiveness in the user communication); []. We believe, that a systems concept of context needs to be open and extensible in order to address various application environments. In general, the concept of context is only pertinent if the system supports context-adaptation or context-awareness. That is, change must be an inherent and explicit concept in the system. Also, the changes should not be

36

1. ~ SEMANTIC COMPLEXITY IN AR

directly predictable; otherwise a simple parameterisation would be more appropriate.4

Context-awareness and context-adaptation are here identified with functionality: knowing the characteristics of the mobile device, the context of the application, the context of the user and of the information objects is a necessary pre-condition for producing effective systems. Another point is that such notion of context needs to be open and extensible, that is, ready to include whatever new contextual elements might come into view. But the most interesting part of this advertisement is the conclusion: context-awareness should not lead to a merely mechanical application of parameters. The author of the texts seems to be conscious of the fact that complete predictability in the whole procedure would take to the antipodes of the information system scope: non-informativity. Indeed, to be effective information needs to contain an element of novelty and unpredictability, otherwise it becomes totally redundant. This, too, was clearly envisaged by linguists (Shannon & Weaver, 1949). On the other hand, a lack of context-awareness (including a knowledge of both the environment and the users intentions) would lead to complete unpredictability, which is also identifiable with uninformativeness. In other words information stems from the balanced interplay between the known and the unknown. This notion has been principally systematized in linguistics resorting to the theme-rheme distinction in sentenceconstruction (Halliday, 1967-8; see also Sperber and Wilson, 1996: 215-217). Technically speaking, in AR spatial and temporal continuity is provided by tags: particularly by Radio Frequency, QR (Quick Response), and AR tags (see chapter 4). All these types of tags contain finder, alignment and timing patterns which

http://www.cs.waikato.ac.nz/~hinze/isdb/publications/hinze_buchanan_ CONTEXTmHCI.pdf 37

Augmented Linguistics

allow the AR device to retrieve semantically-related information. At the same time, systems of geolocalization allow the user to be tracked, thus making the supply of information spatially and temporally related to the users environment. But beside being spatial and temporal the relationship between AR and physical reality should also be semantic (Milgram Kishino, 1994). Obviously, there should be a semantic continuity between AR and the environment: if I am playing chess, for instance, the computer-generated text or imagery should be semantically consistent with the game itself: what will appear in front of me to be moved is most probably a virtual chess piece, not, say, a virtual baseball club or a tennis racket. This aim may be achieved thanks to the pattern markers contained in ARtags. These permit the virtual camera of the computer or of the handheld device to be aligned with real objects in real time by allowing a computer vision algorithm to calculate the camera pose. Thanks to these tags computer-generated imagery can be coherently positioned over the real world, so that we do not feel the incongruence between the 3D graphics, animations, and videos and the surrounding context. Virtual objects should also possess, as far as possible, the physical properties of real objects and behave accordingly. In the section entitled Behavioural Association of the Real and Virtual in the Handbook of Augmented Reality (Fuhrt, 2011) the process is described in the following terms:
We semantically modellise virtual objects by taking into account their physical properties according to the laws of gravity, contact, elasticity, fluidity, etc. so as to enrich the scene. Prior knowledge is used in the real scene and its objects. This functionality defines geometrical and physical interactions between real and virtual objects. For example, we can mention the behaviour and attraction functions of virtual objects with real objects (Fuhrt, 2011: 53).

On the other hand, in particular situations I could see something which has little to do with the surrounding context. AR can aim at creating artificial environments rather than
38

1. ~ SEMANTIC COMPLEXITY IN AR

real(istic) ones. Rather than augment the real world, AR can attempt to create possible alternative worlds. For instance, a beautiful tropical island can be made to appear in front of a grey block of flats to provoke in me the desire to set out on a journey (maybe there is a travel agents just around the corner). Here the category of semantic continuity is overtly called into question. In other words, artificial AR environments can be obtained through a semantic gap produced by the incoherence of the overall meaning of the mixed environment (Fuhrt, 2011: 58). As we shall see, this semantic gap can be studied in terms of metaphoric substitution.

1.4. Reaching beyond: metaphoric connections

The possibility of obtaining additional information about the places of interest (houses, restaurants, companies, etc.) that surround us in a specific geographical area depends on the global positioning system (GPS) provided by our handheld device, which can localize us and supply context-related information ( 2.2.). From this perspective the notion of context becomes fundamental. AR is deeply contextualized; more precisely, it is activated by context. The question therefore is: in what kind of relationship do AR texts and imagery stand to context? And how does this relationship differ from that existing in other forms of mediated communication? The relationship between AR and the surrounding context is spatial and temporal. Its very existence is determined by coordinates that make it happen there and then. In oral communication this relationship is mainly expressed through deictics: The fruit is in the bowl over there, where the over there acquires meaning only if the speakers share the same physical and cultural context. In AR spatial interconnectedness is given by the alignment process made possible by videocameras and

39

Augmented Linguistics

ARtags while the temporal interconnectedness is given by the real time response of the interface.5 Thus in AR contextual salience is, in a sense, less ambiguous than in other kinds of verbal exchange, because the association of real and computergenerated objects is directly (using an old-fashioned term we might say automatically) called upon by the technological device itself. If I point at the Coliseum, for instance, the popup window appearing on my smartphone will most certainly tell me something about the historical and artistic relevance of that monument, rather than about other sites, at least not about sites that have nothing to do with the Coliseum. In other words if I am likely to get the fruit I want by saying that it is over there I will most certainly get what I want with AR, because deictics are physically actualized by the use of my handheld device: I physically6 point at things. Yet, the process is not as simple as that. AR is based on indexicality: on the one hand it is true that AR is activated by pointing a handheld device at an object, thus establishing a connection between discourse and the world; on the other hand, AR shares with the forms of indexicality we find in natural language the ambiguous relationship existing between discourse and the world. As Roderick, interpreting Lyotards thought, argues:
[] the eye is in the word because there is no articulation without the appeal to an outside constituted as a visibility where objects are designated in space, as well as spatialization that resides at the heart of discourse as an unconscious forcedesire. [] Lyotard finds that figure resides in discourse as

5 A discourse apart should be made on the content of AR texts and imagery. Time can make them slightly obsolete if they are not constantly updated. For instance, a pop-up could tell me that I am in front of an Indian restaurant while the site has now become a private home. 6 I am sticking to the present situation. Researchers are studying ways of activating AR through eye-blinking or even with thought.

40

1. ~ SEMANTIC COMPLEXITY IN AR

the intractable opacity of the visible [] plasticity and desire, an extensive horizon. Indexicality means that discourse is shot through with the visible. [] the nonc must point beyond its borders to objects positioned in space with respect to it. It is plunged into a textual space that surrounds it, and it is riddled from within by deictic holes whose function is to indicate positionality in space (here/there) and in time (now/then). [] Nonetheless there is a negativity of a special type, an opening in space between eye and object as a kind of moving frame that is formal or formalizing. Indexicality gives us a formed space. (Rodowick, 2001 : 6)

In the effort to make sense of reality, indexicality creates a network of cross-references that give unity to otherwise unrelated objects, simply by connecting the eye to the world and signalling (i.e. phatically stating) the connection. On this account the semiotic interpretation of deictics is far more effective in dealing with the complexity of the process (Greimas, 1968), but it is not the only applicable one. Indeed in AR semantic continuity is difficult to define. Of course the problem is also cogent in other forms of languagebased communication and is definitely not new. Metonymies, for instance, do stand in a semantic relationship. Collocation is another linguistic category based on the idea of continuity between the elements of an utterance (Firth, 1957). This continuity is given either by the code itself (as a matter of fact) or by linguistic habit (a gorgeous day). But what would we make of metaphors, for instance, especially of far-fetched ones, in which the relationship between the tenor and the vehicle is not immediately detectable (Richards, 1936)? This problem surfaced also with reference to the Internet. In a recent issue of Wired (Priolo Maggiato, 2009: 54 ff.), which stressed the importance of context-aware search engines, the authors underlined that most of the experiments carried out in this direction are based not only on the study of logical associations, but also on the possibility that users might reason indirectly and metaphorically.
41

Augmented Linguistics

To widen the perspective of this research by adding elements that can throw light on this problematic aspect which invests also digital contents, I may resort to other ways by which spatialization and indexicalization are actualized in computer-mediated texts. Let us think about hypertexts. A parallel has been made between hypertexts and suburbs on the basis of the concept of linkage to distant items. According to American thinker David Kolb, just as chunks of suburban buildings acquire a meaning from their linkages to distant sites, so hypertext chunks owe their meaningfulness to their being connected to other chunks, even if they are not immediately obvious or visible:
I want to make, but also qualify, a parallel between suburbs and hypertexts, as a way of emphasizing that the meaning of a given chunk of suburban building or real estate usually depends on its linkages to distant items. The basic comparison with hypertext a non-linear assemblage of chunks of text with links among them is that the form of the text is not the same as the form visible on any one page or screen. It reaches beyond, just as form of the suburb is not the same as the immediately visible spatial connections. Immediate architectural form is not the same as the place form of suburban locations, because they reach out beyond the local horizon, and form wholes and networks that are not architecturally obvious. We are not sure how to express this linkage architecturally, and most suburban architectural types celebrate isolation rather than connection. [] This parallel is useful, because the armature of links in a hypertext creates a spatiality that has more complex interrelations and dimensions than the linear one-thing-after-another of physical space, or of pages in a novel.7

It reaches beyond, just as form of the suburb is not the same as the immediately visible spatial connections. But isnt this precisely the definition of metaphor, particularly of the far7

www.dkolb.org/sprawlingplaces/suburbsa.html. My italics. 42

1. ~ SEMANTIC COMPLEXITY IN AR

fetched metaphor? If, pillaging Shakespeare (Sonnet 18), I say to my husband you are a summer day what semantic relationship connects my husband to the weather condition he is figuratively identified with? Rhetoricians speak of the existence of a tertium comparationis which unites the two semantic fields (Richards, 1936): in the above example the human being and the weather situation could be semantically connected by the attributes of brightness, lovingness and light-heartedness. However, the criteria by which this term is chosen among thousands to represent certain attributes are not self-evident. An interesting question might be whether AR contents can also work indirectly, that is following metaphorical forms of reasoning and distant connections. In other words, whether AR is prone to metaphorical uses. I think it is, and in two possible ways: 1) either by a deliberate choice of AR users, when they want to generate virtual imagery or texts that are not directly related to the context but are the result of their own desires or 2) when AR texts and images are proposed/imposed to the user as in virtual forms of advertising, anticipating or provoking the users needs and desires. Literally speaking, as I am likely to get the fruit I want by saying that it is over there, I will most certainly get what I want with AR, because deictics are actualized by the use of my handheld device, either through digitalization or through eyetwitching movements. But AR like the Internet could also be designed so as to react to metaphorical thinking by responding to peoples cognitive complexity or playing with it for commercial advantage. In the real world I can say that my colleague is beating about the bush and the sentence would be accepted as perfectly plausible even if I uttered it during a business meeting on the top floor of a New York skyscraper: because everybody would understand that I am speaking figuratively. In AR a bush will hardly be aligned to the furniture of my office by simply pointing at the physical context. Unless I (or other people for me) want to play with virtual imagery by

43

Augmented Linguistics

placing a virtual bush in my city office, thus provoking a semantic gap produced by the incoherence of the overall meaning of the mixed environment (Fuhrt, 2011: 58). But we could also interpret the incoherence (or have it interpreted) as metaphorical thinking and adapt AR to our association of ideas and to our dreams, thus attributing to objects other-than-literal meanings. Or, more radically, we could conclude that metaphors are all-pervasively and inextricably interwoven with the perception of semantic spaces. In the end, as Lotman stated:
The problem of the intersection of semantic spaces is complicated by the fact that the circles we draw on paper represent a particular visual metaphor rather than a precise model of the object. [] In the distribution of these intersections across the entire space of a language, so-called linguistic metaphors are generated. (Lotman, 2009 [1992]: 19).

Semantic continuity must come to terms with all this: not only space is invested with cultural and symbolic meaning; in cognition, in its translation into language and then during communicative processes it goes through stages of modellisation that are inseparable from metaphorisation, so that it is hardly a datum which is given once and for all.

1.5. The Semantic Web Layer Cake on and off line

The same complexity in the activation of linking procedures is at work both inside and outside the Web, both within an entirely digital context and within mixed dimensions involving the interaction of the virtual with the real. This would also explain why researchers encounter serious technical problems in finding software that takes into account the inevitable semantic gaps existing between real and computer-generated (or computer-processed) objects (Behringer et al, 1999; Hainich, 2006). The mapping of objects from reality in the
44

1. ~ SEMANTIC COMPLEXITY IN AR

model (as in ARtags) can minimize this gap but not obliterate it entirely. The difficulty of defining context univocally is due to the complex processes by which we perceive and make sense of the world. As Lotman remarked, semantic spaces are linked to individual consciousness (Lotman, 2009 [1992]: 19), although they may partly intersect. As Kai Pata echoes, place is not a uniquely physical category: it is something that is meaningful for me, it is related to some people, emotions, actions (Pata, 2010). Thus considered, also space in hybrid ecosystems (such as AR) is better defined as an ontospace which contains, concurrently, a semantic and a pragmatic dimension (table 3).

Table 3. Ontology Dimensions Map. Source: <http://accuracyandaesthetics.com/?p=370>

The idea of a Semantic Web (Tim Berners-Lee, 2001) is inseparable from the acknowledgement of the existence of an ontological level. But can this ontospace be measured or defined? As experience is inextricably entangled with this dimension, and therefore, as the graphs show (tables 3 and 4), the pragmat 45

Augmented Linguistics

ic aspects are inseparable from the semantic ones, it would be much more fruitful to achieve environments that are capable of reacting to peoples communicative needs. Experiments in the direction of the creation of reactive environments have actually already been made. Among these, an interesting one aimed at defining a semantic topology which takes into account all possible implications of hybrid environments has been recently carried out by Pierre Lvy in his IEML project (2010). But difficulties remain: successful communication can be made probable but never guaranteed (Sperber and Wilson, 1986: 17). AR ultimately replicates the same pragmatic problems involved by every communicative act. As we shall see in Chapter 6, communicative intent, cooperative principle, and inferential strategies are at work also here (Grice, 1975). AR systems should be as good in applying inferential strategies to messages as people are in normal interaction. These strategies appear to be inseparable from the awareness that communication is a multi-layered dimension, a layer cake, as the following graph representing the dynamics of the Semantic Web shows (table 4).

Table 4. Semantic Web Layer Cake. Source: <http://www.semanticfocus.com/blog/entry/title/introduction-to-the-semantic-web-vision-and-technologies-part-2-foundations/> 46

1. ~ SEMANTIC COMPLEXITY IN AR

Though graphically conceived in different ways the two graphs (tables 3 and 4) show a similar interest towards those specific categories that play a vital role in communicative processes. 1. Code or standard: RDF, XML Query Language, XML Schema, Uri, Iri, Unicode. Grammar rules, vocabulary, syntax, sentence structures and models; encryption; formal categories (expression, design methods, granularity, etc.), Namespace*; 2. Modality: rigour, validity, strictness, trust, proof, norms, authority, signature, governance; 3. Context: domain, Namespace (*it is also known as context because the correct operative meaning of the name depends on context) 4. Pragmatic aspects: shared beliefs, ontology, knowledge; content as related to the users encyclopedia and emotions (things, stuff, relationships), intended use <-> functional domain; 5. Logical and cognitive patterns: reasoning, schema, models. All these categories work together to produce meaning. This happens both in natural and in computer-mediated communication: form and content, rules belonging to code, cognitive patterns, modality, conversational implicatures, shared knowledge, etc., all contribute to the creation of meaning. While the first graph visualizes the process as a set system, the second one figures it out as a stratified structure whose layers are all necessary to the making of sense. Indeed, the awareness of the multi-layeredness of discourse is vital not only to the Web but to cognitive processes and communication in any context of situation. When we communicate we act according to mental frames (Goffman, 1974), knowledge schemas and interactive frames which are both prior to the actual verbal exchange and inherent to it (Tannen, 1993). As a result, different planes of talk are integrated (Shiffrin, 1994:
47

Augmented Linguistics

21). Communication is therefore based on a multi-layeredness which is both semantic and pragmatic; it is a multi-layered framework of framing, in which the various facets of situation can be seen in qualitatively different types of overlapping notions of frames (Johnson, 2008). On this point, semiologists and linguistis seem to agree. As Lotman stated:
Semiotic space appears before us as the multi-layered intersection of various texts, which are woven together in a specific layer characterised by complex internal relationships and variable degrees of translatability and spaces of untranslatability. The layer of reality is located underneath this textual layer the kind of reality that is organized by a multiplicity of languages and has a hierarchical relationship with them.

The existence of superimposed levels in cognitive and consequently on communicative processes is similarly stressed by cognitivists. As Sperber and Wilson remark: While grammars neutralise the differences between dissimilar experiences, cognition and memory superimpose differences even on common experiences (1986:16). The idea that communication is a stratified or multi-layered dimension is of course crucial also in a multimodal perspective. In Kress and van Leeuwens words:
[] communicative practices are seen as multi-layered and include, at the very least, discursive practices, production practices and interpretive practices, while they may also include design practices and/or distribution practices. [] each of these layers contributes to meaning. [] we assume that meaning is made everywhere, in every layer, in phonology and in grammar/syntax. In any mode all realisational elements are available for the making of signs (Kress van Leeuwen, 2001: 111).

AR is just one more facet added to the situational frame and as such, it partly overlaps and interacts with all the other facets
48

1. ~ SEMANTIC COMPLEXITY IN AR

belonging to the context in which communication takes place. If the theory of framing is applicable to any kind of spontaneous or mediated communication, it is even more apt to describe the modes of AR, which is by definition a hypermediated dimension based on a stratification of levels. Not only those that, as we have seen, pertain to cognition and communication in general, but also those resulting from the different strata or levels of reality involved in this particular medium: the virtual, the real, and all the layers in-between. Finally it is important to highlight that while in the Internet, at least so far, the necessity to make choices between a plethora of potential meanings has been signaled by the presence of disambiguation pages (Crystal, 2001: 210) in AR, by contrast, the ambiguity becomes part and parcel of the experience, so that making choices seems to be no longer required by the medium. This further complicates the problem. It is therefore on the notion of superimposition and on other phenomena founded on overlapping processes that we shall concentrate in the ensuing section.

49