An exploration of a semiotic model of interaction through interactive media

Shaleph O'Neill The HCI Group, School of Computing, Napier University, Edinburgh, EH10 5DT, +44 0131 455 2699, s.oneil@napier.ac.uk David Benyon The HCI Group, School of Co mputing, Napier University, Edinburgh, EH10 5DT, +44 0131 455 2699, d.benyon@napier.ac.uk
1. Using a qualitative methodology, is it possible to find evidence to support a theoretical semiotic model of interaction? What other elements can be found in the data to develop this model so that it is more effective at describing interaction from this semiotic point of view? Can we begin to move away from purely denotational meanings in a semiotic evaluation and start to consider the richer world of connotations during interaction?

1 Introduction
Semiotics has been called the mathematics of the arts because it deals with concepts that are relevant to so many different disciplines; cultural studies, literature, art, anthropology and so on. It has also been applied to HCI where Benyon (Benyon 2001) has argued that it could form the basis of a 'new HCI. Semiotics central critical focus on the signs, messages and texts that are used to create meanings within different domains make it a prime candidate for inclusion in HCI theory. Over the past ten years or so a number of experts have conducted research into Semiotics in relation to HCI. Most notable among them are Peter Bogh Andersen (Andersen 1993, Andersen 2001, Andersen 1990) and the members of the semiotics-engineering group (SERG) (Barbosa, et al. 1999, deSouza, et al. 2001a, deSouza, et al. 2001b, deSouza, et al. 2000, Prates, et al. 2000a, Prates, et al. 2000b). However as technology continues to advance and the diversity of interaction continues to develop computer semiotics, as it is, has not kept pace. Like most HCI research, comp uter semiotics has focused on traditional ideas of interaction. A re-evaluation of this approach needs to be conducted in new technological domains, taking into account the already existing spectrum of semiotic theory. Semiotics as a whole has a rich history of critiquing a wide variety of domains and as such it offers HCI a unique perspective and insight into areas such as virtual environments, CSCW and ubiquitous computing. The aim of research from this perspective is to offer accounts of these new domains, which may extend our concepts of analysis and design moving towards a semiotics of interaction.

2.

3.

4. What, if anything, can this semiotic model of

interaction say about the notion of presence in virtual environments, given that interaction in a virtual world is a mediated experience?

3 Towards A Semiotic Model

The focus of our early research into semiotics has taken a broad view of many strands of semiotic theory. In our first paper (O'Neill, et al. 2002) we focused on the work of Umberto Eco (Eco 1976) in relation to a semiotic analysis of mobile phone interfaces. Starting from the SERG perspective that interfaces can be considered to be a one shot message from designer to user (Prates, et al. 2000b), We applied Ecos revised KF model as a tool by which to analyse interactions looking at the meanings associated with the signs within the interface. The revised KF model is built around the notion that meanings can be extrapolated from signs as either denotations or connotations that are dependent on the context and circumstances in which the signs are encountered. The revised KF model then is a dynamic tool that looks at the way the meanings of signs change depending on where they are encountered. Applying these ideas to mobile phone interfaces uncovered how the meanings of individual signs were dependent on the context provided by the concurrent and sequential signs in the interface (Andersen, 1990). This early work showed that the KF model is useful as an analytical tool for studying the intricacies of

2 Position
Semiotics concern with the nature and use of signs is a good place to approach HCI from because some of the central concerns of HCI parallel those already present in semiotics. The notion of the sender and reader in semiotics is not dissimilar to the notion of designer and user in HCI. From this perspective and other research around semiotic ideas in HCI a number of questions have emerged that we are interested in looking at.

interaction at the interface and that semiotics in general can be applied to interface concepts. However, although its level of detail is very useful for looking at interface problems it is a particularly cumbersome tool to use for looking at complex interfaces, capturing nothing of the human aspects of interaction in general. Clearly a more general model of interaction with much more scope across different domains could be a useful contribution from semiotics to HCI.

knowledge of codes is the well that the semiotic process draws from to allow signification to take place. Ecos revised KF model then, is really a dynamic model of the semiotic process in relation to the Umwelt (Eco 1976).

3.3 The Perception/Action Loop

Contained within Uexkulls conception of the Umwelt is a model of the relationship between organism and environment, which is a perception/action model. Again it is very similar to those proposed in HCI by cognitive psychology. This is very useful because it frames these ideas in semiotic terms that allow us to build an integrated model of interaction from a semiotic perspective. The fundamental difference between Uexkulls perception/action model and the cognitive perspective is that Uexkull characterises its operation in terms of signs rather than in terms of processing raw sensory data. This is an important shift in perspective that considers these signs as phenomena which are encountered during interaction which provide possibilities for interaction rather than the goal, task, action, evaluation approach originated by the ecological psychologist Gibson and subsequently extended and applied to HCI in a major strand of Normans work (Norman 1998).

3.1 Sequential and concurrent syntagms

Peter Bogh Andersens notions of concurrent and sequential syntagms (Andersen 1990) provide an insight into HCI by abstracting a point of view direct from structural semiotics. By drawing on semiotic concepts from Theatre and Dance, Andersen f ocuses on the notions of the sequence of events in relation to the actors and props present on the stage. For Andersen computer based signs exist as two-dimensional objects that occupy both sequential and concurrent planes. During interaction computer based signs occupy a place in the interface, which is relative to other signs on the screen. As they are interacted with they are brought into relation temporally to other signs in the interface that occur as a result of system response. Andersen proposes a model here that looks at the process of interaction based on the notion that it takes place through the manipulation of the signs within an interface over a period of time. What is unique in this description of interaction is that it can be viewed as a kind of pseudo discourse that takes place between the computer and the user in terms of the meanings each one can attribute to the signs as they are activated during the interaction. These ideas provide the backbone for our interaction models as they are evident in various domains such as theatre, dance, cinema and wayfinding (Andersen 1990) (Passini 1992) to name but a few.

3.4 Information artefacts

According to Andersen information artefacts are the different types of signs that make up an interface. These are the buttons, graphics, words and such like that he categorised in his book Computer Semiotics (Andersen 1990). Since then however, as pointed out earlier, many new forms have come to be included in an interface to the point where we now have the new media Metalanguage (Manovich 2001). So the information artefacts in our model are considered to be all of the elements that now go into an interface which constitute the beginnings of this new metalanguage.

3.2 The Umwelt

Jacob Von Uexkulls conception of the Umwelt (Allot 1994, Allot 1992, Deely 2001, Kull 1998, Sebeok 1979) is built upon the unique notion that all significations take place within the bounds of firstly, our genetic codes in terms of hereditary aspects of species, and secondly, the social codes within which we live as aspects of our environment into which we become indoctrinated as we develop and grow as people. There can be no signification outside these constraints because they are what give us a) the need to communicate and b) the means by which to do it. The Umwelt then is effectively the mass of knowledge that we carry around with us into every interaction, which has been formed and continues to form as a result of those interactions. This is an important idea in our model for two reasons. Firstly it takes the place of existing cognitive models in HCI in representing knowledge and memory. Secondly, It is clearly linked to ideas about semiosis and connotation because this

3.5 The Model

The model (Figure 1) works like this: Uexkulls perception/action loop encounters and manipulates phenomena or information artefacts (Benyon 2000) that exist in an interface. This activity, which produces sequential chains from the concurrent chains of the system throughout an interaction, is a sense making process that occurs in relation to aspects of denotation, connotation and meta-language. In other words our semiotic model of interaction describes an interface/information space as a number of signs brought together to form a text. This text is experienced as a pseudo-discourse, mediated through the concurrent and sequential chains of the system. This results in the production of meaning (interpretation) by the user who acts through the messages of the text as if it were a medium.

Fig 1. A semiotic model of interaction through a computer Medium

5 Future work
In relation to our future work we are focusing on the development of this model as a tool for exploring interaction in Virtual Environments. As part of the Benogo project (Arnspang 2002) work is scheduled to take place that will look at the concept of presence in interactive mediated environments. It is hoped that this model, and the methods associated with it will bring some insight into this type of mediated interaction from a semiotic point of view. There is an interesting phenomenological perspective to presence that might link the notion of the Umwelt in this model to the notion of the embodied mind as proposed by Lakoff and Johnson (Lakoff, Johnson 1999). In their work the utterances of individual subjects uncover the metaphors by which they understand meanings. This is similar to the semiotic processes in the Umwelt and might help us to understand more about the part connotations play in interaction. Lakoff and Johnson provide a number of categories of metaphor that they consider as being active in the mind. Further development of a method to explore the notion of the Umwelt in this model might do well to include some of this research, particularly in relation to a sense of presence in mediated environments.

Allot R (1992) The Motor Theory of Language: Origin and function. In: Winderval J (ed.) Language Origin: A Multidisciplinary Approach. Kluner Academic Publishers, Dordrecht Andersen PB (1993) A semiiotic approach to programming. In: Peter bogh Andersen BH, Jens F. Jensen (ed.) The Computer as Medium. Cambridge University Press, Aarhus (pp 16-67) Andersen Pb (2001) What Semiotics can and cannot do for HCI. Knowledge-Based Systems Volume 14: 419-424. Andersen PB (1990) A Theory of Computer Semiotics. Cambridge University Press, Cambridge Arnspang J, Benyon, D. R., Fahle, M. W., Granum, E., Madsen, C. W., Pajdla, T., Peleg, S., Smyth, M., Turner, P., Turner, S. and Weinshall, D. (2002) An Investigation Into Virtual Representations Of Real Places. In Proceedings of PRESENCE 2002:. Barbosa DJ, Prates R, deSouza CS (1999) Direct and Indirect user-to-developer messages through communicability evaluation. Representational Support for User Developer Communication workshop, INTERACT'99. Benyon D (2000) Beyond the Metaphor of Navigation in Information Space. CHI2000

References
Allot R (1994) Language and the origin of semiosis. In: Noth W (ed.) Origins of Semiosis: sign Evolution in Nature and Culture. Morton de Gruyter, Berlin

Benyon D (2001) The new HCI? Navigation of information space. Knowledge-Based Systems Volume 14: 425-430. Deely J (2001) Umwelts Semiootika osakonna kodulehekulg. Semiotika 134 special volume about Jakob von Uexkull: 125-135 deSouza TS, Barbosa SDJ, Prates RO (2001a) A semiotic engineering approach to user interface design. Knowledge-Based Systems Volume 14: 461465 deSouza S, Barbosa SDJ, Prates RO (2001b) A Semiotic Engineering Approach to User Interface Design. Knowledge-Based Systems 14: 461-465 deSouza S, Prates R, Carey T (2000) Missing and Declining Affordances: Are these Appropriate Concepts? Journal of the Brazilian Computer Society vol.6 Eco U (1976) A theory of Semiotics. Indiana University Press, Indiana Kull K (1998) On semiosis, Umwelt, and semiosphere. Semiotica vol. 120: 299-310

Lakoff G, Johnson M (1999) Philosophy of the Flesh Manovich L (2001) The Language of New Media. MIT Press Norman D (1998) The Psychology of Everyday Things. MIT Press, London O'Neill S, Benyon DR, Turner SR (2002) Semiotics and Interaction Analysis. ECCE 11. Passini R (1992) Wayfinding in Architechture. Reinhold, New York Prates O, deSouza CS, Barbosa S (2000a) A case Study for Evaluating Interface Design through Communicability. ACM Designing Interactive Systems, DIS'2000. Prates O, deSouza CS, Barbosa S (2000b) A method for evaluating the communicability of User Interfaces. Interactions: 31-38 Sebeok TA (1979) The sign and its Masters. University of Texas Press, Austin