Corso di Interazione Uomo Macchina

Federico Cabitza

CONCETTI AVANZATI
Corso di laurea in Informatica
Università degli Studi di Milano-Bicocca
Dipartimento di Informatica, Sistemistica e Comunicazione
Sistemi e interfacce d’uso
Users

2
Sistemi e interfacce d’uso
Users Developers

3
Sistemi e interfacce d’uso

Objectivist

Subjectivist /
Constructivist

4
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)

5
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)
❑ in particolare non pensano di essere messi in comunicazione con gli sviluppatori tramite esse.

6
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)
❑ in particolare non pensano di essere messi in comunicazione con gli sviluppatori tramite esse.
2) Invece i programmatori dovrebbero ragionare diversamente: le macchine sono il loro strumento di comunicazione con gli
utenti (Semiotic Engineering).

7
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)
❑ in particolare non pensano di essere messi in comunicazione con gli sviluppatori tramite esse.
2) Invece i programmatori dovrebbero ragionare diversamente: le macchine sono il loro strumento di comunicazione con gli
utenti (Semiotic Engineering).
❑ Ma perché mai gli sviluppatori dovrebbero comunicare con gli utenti?
8
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)
❑ in particolare non pensano di essere messi in comunicazione con gli sviluppatori tramite esse.
2) Invece i programmatori dovrebbero ragionare diversamente: le macchine sono il loro strumento di comunicazione con gli
utenti (Semiotic Engineering).
❑ Ma perché mai gli sviluppatori dovrebbero comunicare con gli utenti?
3) Per convincerli a fare qualcosa! Captology! (Computers As
9
Persuasive Tecnology)
Sistemi e interfacce d’uso

I concetti avanzati in una slide

1) Gli esseri umani sono esseri sociali (psicologia) e tendono a considerare le macchine come degli attori sociali (teoria
CASA, Computers as Social Actors)
❑ in particolare non pensano di essere messi in comunicazione con gli sviluppatori tramite esse.
2) Invece i programmatori dovrebbero ragionare diversamente: le macchine sono il loro strumento di comunicazione con gli
utenti (Semiotic Engineering).
❑ Ma perché mai gli sviluppatori dovrebbero comunicare con gli utenti?
3) Per convincerli a fare qualcosa! Captology! (Computers As
10
Persuasive Tecnology)
 Psicologia QB

Based on: Carlos Han (2019) 7 Principal Psychological Phenomena in UX Design.

https://blog.prototypr.io/7-principal-psychological-phenomena-in-ux-design-1104e09fc974. Archived at: http://archive.is/cMcbd
Good job!
 Good job!

Unintended use!
Address the needs!

Maslow’s Hierarchy of Needs

Maslow’s Hierarchy of Needs
 Lussuria Gola

Avarizia/Avidità Accidia Ira

Invidia Superbia
8 mechanisms to be aware of
1. RIFLESSO PAVLOVIANO
1. RIFLESSO PAVLOVIANO

Attenzione, questo è un dark pattern!

2. EFFETTO PLACEBO (CF. FEEDBACK)
3. EFFETTO CAMALEONTE/SPECCHIO
4. EFFETTO ALONE (HALO EFFECT)

Beauty pays off

5. VON RESTORFF EFFECT (ISOLATION EFFECT)

Notifications signals must stand out from the background

6. PRIMING (ORDER BIAS / SERIAL POSITION EFFECT)
7. MERE EXPOSURE EFFECT (FAMILIARITY BIAS)

(splash screens, usual UIs)

8. REWARD OR REINFORCEMENT
CASA theory, Computer as Social Actors.

28
CASA theory, Computer as Social Actors.

Chi non l’ha mai fatto?

Bisbigliare alla lavatrice,
imprecare contro il proprio
computer, parlare al
telefono mentre squilla…

29
CASA theory, Computer as Social Actors
Esperimenti degli anni 90 e 2000 hanno
dimostrato che gli utenti sono indotti a
comportarsi come se le macchine fossero umane,
anche se sanno benissimo che non hanno
coscienza o non agiscono mosse da motivazioni
tipicamente umane.

30
CASA theory, Computer as Social Actors
Esperimenti degli anni 90 e 2000 hanno
dimostrato che gli utenti sono indotti a
comportarsi come se le macchine fossero umane,
anche se sanno benissimo che non hanno
coscienza o non agiscono mosse da motivazioni
tipicamente umane.

L’attribuzione di attitudini, intenzioni e disposizioni

umane alle macchine (e agli oggetti in particolare)
si chiama ethopoeia (attribuzione di
comportamenti umani alle cose, diversa
dall'antropomorfizzazione, che riguarda invece la
forma).
31
CASA theory, Computer as Social Actors
Esperimenti degli anni 90 e 2000 hanno L’ethopoeia è cosa diversa dalla
antropomorfizzazione.
dimostrato che gli utenti sono indotti a Si può attivare con qualsiasi
comportarsi come se le macchine fossero umane, tipo di oggetto antropomorfo.
anche se sanno benissimo che non hanno
coscienza o non agiscono mosse da motivazioni
tipicamente umane.

L’attribuzione di attitudini, intenzioni e disposizioni

umane alle macchine (e agli oggetti in particolare)
si chiama ethopoeia (attribuzione di
comportamenti umani alle cose, diversa
dall'antropomorfizzazione, che riguarda invece la
forma).
32
CASA theory, Computer as Social Actors

Basta poco per indurre l’ethopoeia:

(in ordine di efficacia)

1) messaggi in linguaggio testuale (words for

outputs)
2) risposte basate su input predefiniti multipli
(interactivity)
3) attribuzione di ruoli tipicamente umani (ad
es. assistente)
4) utilizzare voci umanoidi (cf HER)
5) forse, utilizzare volti umanoidi?

33
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Reciprocità nella cortesia;
✓ Reazione a personalità
diverse
(dominante/remissivo)
✓ Adulazione;
✓ Stereotipi legati al genere;
✓ Senso di appartenenza;
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Reciprocità nella cortesia;
Applicazione di tutoring & testing + somministrazione auto-
valutazione.
Dopo il compito, i soggetti intervistati sulla qualità
dell’applcazione.
Tre condizioni sperimentali:
1) Questionario somministrato sullo stesso PC;
2) Questionario somministrato su con carta e penna
3) Questionario somministrato su un altro PC.
Le risposte erano significativamente più positive e omogenee
nella condizione 1

35
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Reciprocità nella cortesia;
Applicazione di tutoring & testing + somministrazione auto-
valutazione.
Dopo il compito, i soggetti intervistati sulla qualità
dell’applcazione.
Tre condizioni sperimentali:
1) Questionario somministrato sullo stesso PC; Implicazioni per il design:
2) Questionario somministrato su con carta e penna 1) La valutazione di una applicazione (NON) andrebbe mai fatta
3) Questionario somministrato su un altro PC. nel contesto della medesima applicazione…
Le risposte erano significativamente più positive e omogenee
2) Gli utenti si aspettano educazione e cortesia: i messaggi di
nella condizione 1 errore devono essere educati e non troppo bruschi!
3) Collegato al precedente: attenzione alla semplice traduzione.
36
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Reazione a diverse personalità:
il simile apprezza il simile.

37
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Reazione a diverse personalità:
il simile apprezza il simile.
1. Gli utenti sono in grado di identificare messaggi che indicano
dominanza (“devi procedere più velocemente, forza!”) o
remissività (“forse dovresti andare più velocemente, che ne
dici?”).
2. Gli utenti sono in grado di avvertire somiglianze tra la
personalità dell’applicazione e la loro.
3. Gli utenti preferiscono interagire con computer che esibiscono
la stessa tipologia di personalità.
4. Programmi denotati come di informazione sono stati
considerati più “informativi” degli stessi programmi denotati
come “di informazione e intrattenimento”. 38
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Adulazione

39
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Adulazione
Condizione sperimentale: una app di indovinelli.
Tre tipi di feedback a 3 campioni di utenti diversi:
1) generico: “OK, passa ora al prossimo indovinello”.
2) “Grande, sembri avere una dote non comune nel risolvere gli
indovinelli!”
3) Idem ma ai soggetti è stato detto il programma non sa quale
sia la risposta giusta o che fornisce un feedback entusiasta a
prescindere.
Risultati: i soggetti nelle condizioni 2 e 3 (elogio e adulazione) si
sono sentiti più felici alla fine del task e più soddisfatti
dell’esperienza d’uso risptto alla 1. Non è stata trovata differenza
significativa tra la 2 e la 3! 40
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Adulazione
Condizione sperimentale: una app di indovinelli.
Tre tipi di feedback a 3 campioni di utenti diversi:
1) generico: “OK, passa ora al prossimo indovinello”.
2) “Grande, sembri avere una dote non comune nel risolvere gli
indovinelli!”
3) Idem ma ai soggetti è stato detto il programma non sa quale
Implicazioni per il design:
sia la risposta giusta o che fornisce un feedback entusiasta a
Molti sistemi sono normalmente severi o neutri nel loro feedback.
prescindere.
Potrebbe essere una buona idea addolcire il feedback, anche
Risultati: i soggetti nelle condizioni 2 e 3 (elogio e adulazione) si
sfiorando la piaggeria, soprattutto nel caso di applicazioni tutorial
sono sentiti più felici alla fine del task e più soddisfatti
o perla formazione, in quelle per aumentare la creatività o che
dell’esperienza d’uso risptto alla 1. Non è stata trovata differenza
impongono compiti ripetitivi e noiosi.
significativa tra la 2 e la 3! 41
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Stereotipi di genere

42
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Stereotipi di genere
1. Applicazioni che hanno usato voci femminili per impartire
istruzioni in tono perentorio (dominanza) sono state valutate
più negativamente delle stesse applicazioni che usavano voci
maschili.
2. Applicazioni di tutoring che impiegavano voci maschili sono
state valutate come più “competenti” delle stesse applicazioni
con voci femminili.
3. Applicazioni che davano suggerimenti in tema di amore e
rapporti umani con voci femminili sono state considerate
migliori delle stesse applicazioni con voci maschii.

43
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Stereotipi di genere
1. Applicazioni che hanno usato voci femminili per impartire
istruzioni in tono perentorio (dominanza) sono state valutate
più negativamente delle stesse applicazioni che usavano voci
maschili.
2. Applicazioni di tutoring che impiegavano voci maschili sono
state valutate come più “competenti” delle stesse applicazioni
Implicazioni per il design:
con voci femminili.
Purtroppo i pregiudizi sono meccanismi umani potentissimi. E non
3. Applicazioni che davano suggerimenti in tema di amore e
si limitano solo al genere (anche età, accento straniero o
rapporti umani con voci femminili sono state considerate
fortemente connotato regionalmente): bisogna tenerne conto o
migliori delle stesse applicazioni con voci maschii.
sfidarli apertamente?
La scelta di una voce o di un avatar/assistente virtual per il
proprio
44
applicativo è una scelta molto delicata.
CASA theory, Computer as Social Actors.

Alcuni comportamenti sociali

✓ Stereotipi di genere
1. Applicazioni che hanno usato voci femminili per impartire
istruzioni in tono perentorio (dominanza) sono state valutate
più negativamente delle stesse applicazioni che usavano voci
maschili.
2. Applicazioni di tutoring che impiegavano voci maschili sono
state valutate come più “competenti” delle stesse applicazioni
Implicazioni per il design:
con voci femminili.
Purtroppo i pregiudizi sono meccanismi umani potentissimi. E non
3. Applicazioni che davano suggerimenti in tema di amore e
si limitano solo al genere (anche età, accento straniero o
rapporti umani con voci femminili sono state considerate
fortemente connotato regionalmente): bisogna tenerne conto o
migliori delle stesse applicazioni con voci maschii.
sfidarli apertamente?
La scelta di una voce o di un avatar/assistente virtual per il
proprio
45
applicativo è una scelta molto delicata.
CASA theory, Computer as Social Actors

46
CASA theory, Computer as Social Actors

47
CASA theory, Computer as Social Actors

48
CASA theory, Computer as Social Actors
CASA theory, Computer as Social Actors

Il comportamento
sociale è rivolto al
progettista /
programmatore?
CASA theory, Computer as Social Actors

Il comportamento
sociale è rivolto al
progettista /
programmatore?
Esperimento: a metà campione di utenti è stato ricordato
che usando un applicativo stavano lavorando con una
macchina; all’altra metà è stato fatto notare che stessero
in realtà interagendo a distanza e asincronamente con
dei programmatori. Pur usando lo stesso applicativo, gli
utenti del primo campione trovarono questo più user-
friendly e congeniale a loro.
Sistemi e interfacce d’uso

52
INTRODUCTION TO SEMIOTICS
For HCI students…

by Federico Cabitza

Largely based on Daniel Chandler book

Semiotics: The Basics (2008)
Claude Shannon, 1949

CODING DECODING
OR
 Or…

Cf. Aberrant Decoding (U. Eco)

 Signs?

Informatics
Computation

Signification
Communication
 Semiotics?

The shortest definition: it is “the study of signs”.

 Semiotics?

The shortest definition: it is “the study of signs”.

Contemporary semioticians study

signs not in isolation but as part of
semiotic ‘sign-systems’.

They study how meanings are made

and how reality is represented.
 Semiotics?

The shortest definition: it is “the study of signs”.

Contemporary semioticians study

signs not in isolation but as part of
semiotic ‘sign-systems’.

They study how meanings are made

and how reality is represented.

Semiotics: the study of how

meaning is built and of
meaningful communication.
 Semiotics?

Semiotics can be divided in three ambits:

Syntactic: relations between

signs in formal structures.

Semantics: relations between signs

and what they refer to, what they
mean.

Pragmatics: relations
between signs and
their effects.
 Semiotics?

Semiotics can be divided in three ambits:

Syntactic: relations between

signs in formal structures.

Semantics: relations between signs

and what they refer to, what they
mean.

Pragmatics: relations
between signs and
their effects.
 Semiotics?

Semiotics can be divided in three ambits:

Syntactic: relations between

signs in formal structures.

Semantics: relations between signs

and what they refer to, what they
mean.

Pragmatics: relations
between signs and
their effects.
 Semiotics?

Semiotics can be divided in three ambits:

Syntactic: relations between

signs in formal structures.

Semantics: relations between signs

and what they refer to, what they
mean.

Pragmatics: relations
between signs and
their effects.
 Semiotics?

Semiotics can be divided in three ambits:

Syntactic: relations between

signs in formal structures.

Semantics: relations between signs

and what they refer to, what they
mean.

Pragmatics: relations
between signs and
their effects.
 Signs?

‘Signs’ are everywhere in everyday life, such as road

signs, pub signs, visual signs, but they also include words,
sounds and ‘body language’...

...odors, flavors, acts or objects.

 Signs?

‘Signs’ are everywhere in everyday life, such as road

signs, pub signs, visual signs, but they also include words,
sounds and ‘body language’...

...odors, flavors, acts or objects.

These things have no intrinsic meaning and become
signs only when we invest them with meaning.
 Signs?

‘Signs’ are everywhere in everyday life, such as road

signs, pub signs: visual signs, but they also include words,
sounds and ‘body language’...

...odors, flavors, acts or objects.

Signs regard
These thingsperceivable things meaning
have no intrinsic that signify
andsomething
become
(have a meaning) in a social context.
signs only when we invest them with meaning.
Swiss linguist Ferdinand de Saussure (1857–1913)
Signs?

‘Signs’ are everywhere

"It is . . . in everyday
possible to life, such as road
signs, pub signs:conceive of a science
visual signs, but they also include words,
which studies the role
sounds and ‘body language’...
of signs as part of social
life."
 Signs?

‘Signs’ are everywhere in everyday life, such as road

signs, pub signs:‘We
visualthink only
signs, but they also include words,
in signs’
sounds and ‘body language’...
American logician Charles Sanders Peirce
(Peirce 1931–58, 2.302)
 Signs?

‘Nothing is a sign unless it is interpreted as a sign’

(Peirce 2.172).

Anything can be a sign as long as someone interprets

it as ‘signifying’ something – referring to or standing for
something other than itself.

We interpret things as signs largely unconsciously by

relating them to familiar systems of conventions. It is
this meaningful use of signs which is at the heart of
the concerns of semiotics.
 Signs?

‘Nothing is a sign unless it is interpreted as a sign’

(Peirce 2.172).

Anything can be a sign as long as someone interprets

it as ‘signifying’ something – referring to or standing for
something other than itself.

We interpret things as signs largely unconsciously by

relating them to familiar systems of conventions. It is
this meaningful use of signs which is at the heart of
the concerns of semiotics.
A sign?
 Sign?
A correlation of differences, the whole that results from the
association of the signifier with the signified, which are
inseparable as the two sides of a piece of paper (Saussure)

signified

signification

signifier
 Sign?
A correlation of differences, the whole that results from the
association of the signifier with the signified, which are
inseparable as the two sides of a piece of paper (Saussure)

signified

In the Saussure’s model

the two parts of a sign
signification consist of a ‘sign vehicle’
(‘signifier’,signifiant), and
its meaning, the ‘signified’
(signifié).
signifier
 Sign?
A correlation of differences, the whole that results from the
association of the signifier with the signified, which are
inseparable as the two sides of a piece of paper (Saussure)

signified

In the Saussure’s model

the two parts of a sign
signification consist of a ‘sign vehicle’
(‘signifier’,signifiant),
The signifier is now and
its meaning,
commonly interpreted asthe
the‘signified’
(signifié).
material (or physical)
signifier perceptible form of the sign
– it is something which can
be seen, heard, touched,
smelled or tasted.
 Sign?
The sign is what emerges
A correlation of differences,
from the interaction from the whole that results from the
these two, i.e., anything that
association
can be of the signifier with the signified, which are
used to communicate
inseparable as cf.
(or to tell a lie, the two sides of a piece of paper (Saussure)
Eco).

signified

In the Saussure’s model

the two parts of a sign
signification consist of a ‘sign vehicle’
(‘signifier’,signifiant), and
its meaning, the ‘signified’
(signifié).
signifier
 Sign?
For me signs refer primarily to each other. No sign
makes sense on its own but only in relation to other
signs. Both signifier and signified are purely relational
entities.

signified

signification

signifier
 Sign?
What I refer to as the ‘value’ of a sign depends on its
relations with other signs within the system.
A sign has no ‘absolute’ value independent of this context.

signified

signification

signifier
 Sign?
What I refer to as the ‘value’ of a sign depends on its
relations with other signs within the system.
A sign has no ‘absolute’ value independent of this context.

signified
The Saussurean legacy of
the relationity and
arbitrariness of signs:
signification
the relationship between
the signifier and the
signifier signified is conventional
– dependent on social and
cultural conventions
which have to be learned.
 Sign?
A sign is something which
stands to somebody for
something in some respect or
capacity.

sign
 Sign?
Peirce offered a triadic (three-part) model consisting of:
1. The representamen: the form which the sign takes (not necessarily material, though usually
interpreted as such) – called by some theorists the ‘sign vehicle’.
2. An interpretant: not an interpreter but rather the sense made of the sign.
3. An object: something beyond the sign to which it refers (a referent).

interpretant

sign

representamen object
 Sign?

The two models: not really an extension, rather a specialization…

signified

sign

signifier
 Sign?
To qualify
Peirce offered a triadicas(three-part)
a sign, all three
model consisting of:
elements are essential. The
1. The representamen: the form which sign the sign takes (not necessarily material, though usually
is a unityas
interpreted ofsuch)
what–iscalled
represented
by some theorists the ‘sign vehicle’.
(the object), how it is
2. An interpretant: not an interpreter but rather the sense made of the sign.
represented (the
3. An object: something beyond the sign to which it refers (a referent).
representamen) and how it is
interpreted (the interpretant).
interpretant

sign

representamen object
 Sign?
The object is not
Peirce offered a triadic (three-part) model consisting of: confined to physical
1. The representamen: the form which the sign takes (not necessarily material,
things ,and itthough
can usually
interpreted as such) – called by some theorists the ‘sign vehicle’.include abstract
2. An interpretant: not an interpreter but rather the sense made of the sign.
concepts, events and
3. An object: something beyond the sign to which it refers (a referent).
even fictional
entities (e.g.,
interpretant equality). Equality in
mathematics is a
different object than
‘equality before the
law’. The interpretant
is _your _idea of
equality!

sign

representamen object
 Sign?
“the meaning of a representation can be
nothing but a representation“.

interpretant

(unlimited) semeiosis

sign

representamen object
 Sign?
The interaction between
“the meaning of a representationthecan be
representamen, the
nothing but a representation“. object and the
interpretant is referred
to by Peirce as ‘semeiosis’
interpretant

(unlimited) semeiosis

sign

representamen object
 Sign?
Semiosis is a process where we apply abductive reasoning (abduction): we
generate hypotheses to explain (i. e. to interpret) significant elements of
reality around us and we test them against evidence. If the hypothesis is
confirmed this is accepted as a general principle valid for all of the similar
signs. Otherwise we revise our beliefs, continuously and on-the-go.
interpretant

semiosis

representamen object
 Sign?
Semiosis is a process where we apply abductive reasoning (abduction): we
generate hypotheses to explain (i. e. to interpret) significant elements of
reality around us and we test them against evidence. If the hypothesis is
confirmed this is accepted as a general principle valid for all of the similar
signs. Otherwise we revise our beliefs, continuously and on-the-go.
interpretant

Abduction explains the “paradox of the active

user”: the fact that users never read the manual
and can end up by wasting more time (in the long
semiosis
run) in misunderstanding how to use a system
rather than devoting some time to Read The F…
Manual. However, the user abducts the “right”
representamen object
behavior of the application even if from a
programmer’s point of view the users’ reasoning
and interpretation is totally wrong: the users
believe that they are right.
Jakob Nielsen
 Sign?
Semiosis is a process where we apply abductive reasoning (abduction): we
generate hypotheses to explain (i. e. to interpret) significant elements of
reality around
…sous we
and we test them
cannot against evidence. If the hypothesis is
allow
confirmed thisengineers
is accepted to
as abuild
general principle valid for all of the similar
signs. Otherwise we revise our beliefs, continuously and on-the-go.
products for an idealized
interpretant
rational user when real
humans are irrational: we
must design for the way
Abduction explains the “paradox of the active
users actually behave.
user”: the fact that users never read the manual
and can end up by wasting more time (in the long
semiosis
run) in misunderstanding how to use a system
rather than devoting some time to Read The F…
Manual. However, the user abducts the “right”
representamen object
behavior of the application even if from a
programmer’s point of view the users’ reasoning
and interpretation is totally wrong: the users
believe that they are right.
Jakob Nielsen
 Sign?

interpretant

semiosis

representamen object
Semiotic Triangle
 Semiotic Triangle Jakobson

“concept”
“sense”
“reference” Peirce
“meaning”
“content” “signified”
“interpretant”
“signatum”

Saussere

“signifier”

“symbol” “signans” “sign” “Object”

“sign’s vehicle” “referent”
“expression” “token” “designatum”
“representation”
“representamen”
Semiotic Triangle

to hint at the unity of marks,

concepts and real objects to make
up what we call “signs”…
Cf. Floyd Merrell
 Semiotic Triangle
WHATEVER!!
The signifier or
representamen is the
form in which the sign
appears (such as the
spoken or written form
of a word) whereas the
sign is the whole
meaningful ensemble.
 Semiotic Triangle

An example from éidos

the human
studies

skhéma morphé
 Semiotic Triangle

The idea of
An example from éidos «rabbitness»
the human
studies

skhéma morphé
 Semiotic Triangle

An example from
the everyday
8
experience
 Semiotic Triangle

An example from
the machine
8
learning domain

Convolution + non-linearity +
max pooling + classification

Convolution + non-linearity
 Semiotic Triangle
The idea of the designer to
convey a “closing” affordance,
the expectation of the user that
by clicking the X the window will
close.
An example from
the HCI field.

a computational procedure that

causes a well-determined state
of the system
 Semiotic Triangle
The history of X sign to close
windows:
https://medium.com/re-form/x-
to-close-417936dfc0dc
(archived:
http://archive.is/25vvW)

Batsu!
 Semiotic Triangle

Where would
YOU put data and
information?
Semiotic Triangle

Information?

Data?
 Semiotic Triangle

Information? Data?
Semiotic Triangle

Information?

Data?
 Semiotic Triangle

No right answer
exists.
Just try to justify
yours to others!
Become more aware
of your own ideas
about ideas!!!
Semiotic Triangle
My own two cents: if it’s written,
on any support, or produced for
communication and archival
purposes: it’s a
rapresentamen/data. If it’s an
interpretation of the
rapresentamen, it’s interpretant.
Semeiosis is the process in which
one gets informed, as well as
his/her thought and actions, and
possibly moved toward some
object (either material or
abstract, like a purpose)
DOG
 How many different types of sign do you see in this slide?

DOG
 How many different types of sign do you see in this slide?

3!

DOG
Icon, Symbol, Index
 Icon, Symbol, Index

1. Symbol/symbolic: a mode in which the signifier does not resemble the signified
but which is fundamentally arbitrary or purely conventional – so that this
relationship must be agreed upon and learned: e.g. language in general (plus
specific languages, alphabetical letters, punctuation marks, words, phrases and
sentences), numbers, morse code, traffic lights, national flags.
 Icon, Symbol, Index

In 1830 railroad companies adopted a lighted means to let train engineers know when to
stop or go, with different lighted colors representing different actions. They chose red as
the color for stop, it is thought, because red has for centuries been used to indicate danger
(the color of blood and severe injury?). It is also the color with longest wavelength.
For the other colors, they chose white as the color for go and green as the color for caution.
 Icon, Symbol, Index

The choice of a white light for go turned out to cause a lot of problems.
In an incident in 1914 a red lens fell out of its holder leaving the white light behind it
exposed. This ended with a train running a “stop” signal and crashing into another train.
 Icon, Symbol, Index

Thus, the railroad decided to change it so the green light meant go and a caution “yellow”
was chosen, primarily because the color is so distinct from the other two colors used and is
the most visible in case of fog.
 Icon, Symbol, Index

Taken from https://goo.gl/n05yQp

In 1920 in Detroit Michigan, a policeman named William L. Potts proposed to adopt the
three-color traffic signal used in the rairoad system also for city traffic. To avoid confusion
(and minimize accidents), in 1935 the Federal Highway Administration created “The Manual
on Uniform Traffic Control Devices.” which set uniform standards for all traffic signals, road
signs, and pavement markings.
 Icon, Symbol, Index

Italy
 Icon, Symbol, Index

Netherlands
 Icon, Symbol, Index

For1. me, a symbol isa ‘a

Symbol/symbolic: sign
mode which
in which therefers
signifierto the
does notobject
resemblethat it
the signified
denotes byisvirtue
but which of a law,
fundamentally usually
arbitrary an association
or purely conventional –of general
so that this ideas,
which operates
relationship mustto
be cause the and
agreed upon symbol toe.g.
learned: belanguage
interpreted as(plus
in general
referring to that object’ and we interpret symbols according to ‘a
specific languages, alphabetical letters, punctuation marks, words, phrases and
rule’ or ‘a habitual connection’
sentences), numbers, morse code, traffic lights, national flags.
 Icon, Symbol, Index

2. Icon/iconic: a mode in which the signifier is perceived as resembling or imitating

the signified (recognizably looking, sounding, feeling, tasting or smelling like it) –
being similar in possessing some of its qualities: e.g. a portrait, a cartoon,
a scale-model, onomatopoeia, metaphors, realistic sounds, imitative gestures: i.e.,
icons, despite the name, are not always only visual…
 Icon, Symbol, Index

3. Index/indexical: a mode in which the signifier is not arbitrary but is directly

connected in some way (physically or causally) to the signified (regardless of
intention) – this link can be observed or inferred: e.g. ‘natural signs’ (smoke, thunder/lightening,
footprints, echoes, non-synthetic odours and flavours), medical symptoms (pain, a rash, pulse-rate), measuring instruments
‘signals’ (a knock on a door, a phone ringing), pointers (a pointing ‘index’ finger, a directional signpost), recordings (a
photograph, a film, video or television shot, an audiorecorded voice), personal ‘trademarks’ (handwriting, catchphrases).
Icon, Symbol, Index
Icon, Symbol, Index
Iconicity is based on (at least
perceived) ‘resemblance’ and
similarity; indexicality is based
on (at least perceived) ‘direct
connection’ and contiguity.
Icon, Symbol, Index

It is easy to slip into referring to

Peirce’s three forms as ‘types of
signs’, but they are not necessarily
mutually exclusive: a sign can be
an icon, a symbol and an index, or
any combination.
Icon, Symbol, Index

A map is indexical in pointing to the

locations of things, iconic in
representing the directional relations
and distances between landmarks, and
symbolic in using conventional symbols
(the significance of which must be
learned).
Icon, Symbol, Index

This traffic light is

symbolic re the choice of
the colors, iconic re the
choice of the shapes…
Semiotic Engineering
 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.
Clarisse de Souza
 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

For SE, the designers of interactive software are active participants in the
communication that takes place through user interfaces.
They communicate their design vision to users by means of interface signs
like words, icons, graphical layout, sounds, and interface controls like
buttons, links, and dropdown lists.
 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

For SE, the users unfold and

interpret this message as they
interact with the system.
 Semiotic Engineering
It is a semiotic theory of human-computer interaction (HCI)
that sees HCI as computer-mediated communication
between designers and users at interaction time.

HUMAN-COMPUTER INTERACTION  COMPUTER-MEDIATED HUMAN COMMUNICATION

In other words, the
communication between users
and systems is in fact part of a
metacommunication process;
part of the communication
process initiated by designers
about how, when, where and
why to communicate with the
system they have designed.
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users
what the software does, how it
can be used, why, and so on.

This is the concept of designer’s deputy: “a portion of software, which carries explanations; help
information, confirmation dialogs, error messages and warnings”.
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users
what the software does, how it
can be used, why, and so on.

This is the concept of designer’s deputy: “a portion of software, which carries explanations; help
information, confirmation dialogs, error messages and warnings”.

Through their deputy, designers are

telling users:

➢ what they know about the users;

➢ how they have responded to the
users’ needs or aspirations;
➢ what values are encoded in their
response (why and how does the
system improve the users’ lives).
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users Mind that Clarisse is
what the software does, how it saying that you will
can be used, why, and so on. anyway tell users
these things, either
willing or not.
So do it consciously,
This is the concept of designer’s deputy: “a portion of software,
andwhich
well!carries explanations; help
information, confirmation dialogs, error messages and warnings”.

Through their deputy, designers are

telling users:

➢ what they know about the users;

➢ how they have responded to the
users’ needs or aspirations;
➢ what values are encoded in their
response (why and how does the
system improve the users’ lives).
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users

How?
what the software does, how it
can be used, why, and so on.

This is the concept of designer’s deputy: “a portion of software, which carries explanations; help
information, confirmation dialogs, error messages and warnings”.

Through their deputy, designers are

telling users:

➢ what they know about the users;

➢ how they have responded to the
users’ needs or aspirations;
➢ what values are encoded in their
response (why and how does the
system improve the users’ lives).
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users
what the software does, how it
can be used, why, and so on.
Many things!
For instance, with
opportune and apt visual
This is the concept of designer’s deputy: “a portion of software, which carries explanations; help
information, confirmation dialogs, error messages and warnings”. affordance and
constraints; with the
controls (press buttons,
Through their deputy, designers are slide controls, dials), the
telling users: name of the commands,
their relative position; the
➢ what they know about the users; language used in the
➢ how they have responded to the alerts, tips and contextual
users’ needs or aspirations; help boxes, etc...
➢ what values are encoded in their
response (why and how does the
system improve the users’ lives).
Semiotic Inspection Method
Semiotic Inspection Method
Semiotic Inspection Method
Semiotic Inspection Method
Semiotic Inspection Method
Taken from Clarisse de Souza. Semiotics. The Encyclopedia of HCI. Interaction Design Foundation.
URL: https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/semiotics. Last Accessed on the 10th of March 2017.

An example of SE analysis
Archived at: http://archive.is/3JWqC

MS Word Open Office Writer

 An example of SE analysis
1) document content;
2) document manipulation
and authoring;
3) mail preparation;
4) interface customization
and extensions;

MS Word Open Office Writer

 An example of SE analysis
1) document content;
2) document manipulation
and authoring;
3) mail preparation;
4) interface customization
and extensions;

1) tools related to
document content;
2) document structure;
3) inclusion of media files;
4) bibliography handling;
5) mail preparation;
6) data field manipulation;
7) interface customization
and extension
MS Word Open Office Writer
 An example of SE analysis
1) document content;
2) document manipulation
and authoring;
3) mail preparation;
4) interface customization
and extensions;

LOOK AT WORDING, ORDERING, GROUPING

1) tools related to
document content;
2) document structure;
3) inclusion of media files;
4) bibliography handling;
5) mail preparation;
6) data field manipulation;
7) interface customization
and extension
MS Word Open Office Writer
 An example of SE analysis
The designers of both
applications are telling their
users that tools applying to
document content are top of
the list, and that ‘Spelling and
Grammar’ tools are the most
important ones.

MS Word Open Office Writer

 An example of SE analysis
Whereas in MS Word names
1)tend
document content;
to express objects (with
2) document manipulation
the exception of ‘customize’), in
and authoring;
OO Writer the designers’
3) mail preparation;
4)communication alludes to
interface customization
agents:
and ‘Mail Merge Wizard’ and
extensions
‘Extension Manager’.

In these two cases, the

communication is clearer (more
precise) in OO than in MS Word,
which may achieve the effect of
encouraging novice users into
experimenting new features
(since a wizard and a manager
will be there to help them).

MS Word Open Office Writer

 An example of SE analysis
MS Word’s designers
1)communicate
document content;
very clearly their
2) document manipulation
emphasis on group collaboration.
and authoring;
They are telling users that
3) mail preparation;
4)document preparation is or can
interface customization
be aextensions
and group activity, requiring such
things as shared workspaces and
online communication

OO Writer’s designers, by
contrast, do not talk about
computer-supported
collaboration. The
communication of their design
vision at this point portrays an
individual user, working on the
details of his/her document.
MS Word Open Office Writer
 An example of SE analysis
By communicating that they
1) document
expect theircontent;
users to need a
2) document manipulation
bibliographic database, OO
and authoring;
Writer’s designers give us the
3) mail preparation;
impression that they have worked
4) interface customization
for
and meticulous
extensions users engaged in
longer-term tasks.

By contrast, MS Word designers

give us the impression that they
have worked for busy users,
engaged in broader-context
activities of which document
preparation is only a part.

MS Word Open Office Writer

 An example of SE analysis
A semiotic analysis
can directly help us
explore how
technology
influences problem-
solving strategies
and knowledge
discovery.

MS Word Open Office Writer

Another example of SE analysis
Another example of SE analysis
Communicative Breakdowns during Interaction

The distinction between the language user’s intent (the illocutionary

act) and the resulting effect of language use (the perlocutionary act)
helped the study of language and communication.

The five basic classes of speech acts proposed by Searle are:

1. assertives (speech acts that commit the speaker to the truth of what is
being said);
2. directives (speech acts that aim at causing the hearer to do something);
3. declaratives (speech acts that change the status of the world by virtue
of what is said, by whom and to whom);
4. commissives (speech acts that commit the speaker to taking some
particular course of action in the future);
5. expressive (speech acts that aim at drawing the hearer’s attention to
the speaker’s psychological state or attitude).
Communicative Breakdowns during Interaction

HCI encompasses speech acts performed by the user

and the designer’s deputy.

Both produce illocutions where expression, content,

and intent bring about some effect on the state of
affairs—perlocution.

When perlocution is completely consistent with

illocution—that is, when the effects achieved by what is
said fully coincide with what was meant to be the
case—communication is totally successful.
Communicative Breakdowns during Interaction
The designer’s (good) questions The user’s (bad) questions
The designer’s heuristics The user’s breakdowns
The designer’s heuristics The user’s breakdowns
The designer’s heuristics The user’s breakdowns
 Since designers
cannot be
personally present
when a user
interacts with
software,
they have to represent themselves
in the interface and tell the users
what the software does, how it
can be used, why, and so on.

This is the concept of designer’s deputy: “a portion of software, which carries explanations; help
information, confirmation dialogs, error messages and warnings”.

Through their deputy, designers are Two final remarks.

telling users:
The first is the ‘sandafloor’
➢ what they know about the users; attitude, that is the trial-and-
➢ how they have responded to the error approach that takes the
users’ needs or aspirations;
users in the evaluation loop.
➢ what values are encoded in their
response (why and how does the
system improve the users’ lives). The second is…
A cultural shift.
From...
A cultural shift.
To...
 It means to see your job as

A cultural shift.
much as software development,
as idea growing!
(in the heads of the users)
To...
 Persuasive computing, or
A cultural shift. captology, is the study of
To... computers as persuasive
technologies. It includes the
design, analysis, and theory of
computer technologies created
to change attitudes and
behaviors (first of all of use of
the technology itself).
La mia ultima sigaretta
Fitness tracker
Game of War
Cash Royale

Does it matter?
Persuasive computing?
Persuasive computing?
Anything new?
Persuasive computing?
Anything new?
Persuasive computing?

The Turing’s test is persuasive computing!

Persuasive computing?

The Turing’s test is persuasive computing!

The best (imho) movies on AI are all about persuasive computing!

Blade Runner (1982)
Ex Machina (2015)
2001: A Space Odyssey (1968)
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

For purposes of captology, persuasion is defined as the attempt
to change (or affect/condition) attitudes or behaviors or both.
It’s not coercion (coercizione) nor deception (inganno)
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

Affect/condition

Coercizione/inganno

Intenzionalità, semiotic
Macrosuasion: il engineering
sistema vuole motivare
e persuadere.
Microsuasion:
funzionalità specifiche
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

Un medium è (dopo McLuhan) qualsiasi tecnologia attraverso esseri umani

informano altri esseri umani o comunicano con loro.
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“computer don't (yet) read subtle cues from
people, but they do offer emotional cues, which
can be applied to persuade. This imbalance puts
humans at a relative disadvantage. We are
emotional beings, especially when it comes to
issues of influence. We expect ethical
persuasion to include elements of empathy and
reciprocity. But when dealing with interactive
technology, there is no emotional reciprocity.”

2002!!
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“the study showed that people

observed a common social dynamic,
the rule of reciprocity; they repaid a
favor that a computer had done for
them.””
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

(…because Fogg extrapolates competence from trust)

CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“Credibility makes
PERSUASION persuasion possible”
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“If one dimension of credibility is
strong while the other dimension
is unknown, the computing
product still may be perceived as
credible, due to the "halo effect"
(if one virtue [e.g., likability] is
evident, another virtue maybe
PERSUASION assumed, rightly or wrongly)”
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“ “principle of near perfection:

computing technology will be
more persuasive if it never
commits what users perceive as
errors.” ”
PERSUASION
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“ “principle of near perfection:

computing technology will be
more persuasive if it never
commits what users perceive as
errors.” ”
CASA theory, Computer as Social Actors

CAPTOLOGY, Computer As Persuasive Tools

“My own view is that the use of
emotions in persuasive technology
is unethical or ethically questionable
only when its intent is to exploit
users or when itpreys on people's
naturally strong reactions to
negative emotions or threatening
information expressed by others”

Ethical Unethical Ethical Unethical Ethical Unethical

Praiseworthy Responsible Not Responsible Not Responsible

and at fault responsible and at fault responsible and at fault
 “One of the crucial differences introduced by semiotic engineering in
the HCI picture is thus, that HCI designers are present at interaction

* De Souza, C. S. (2005). Semiotic engineering: bringing designers and users together at interaction time. Interacting with Computers, 17(3), 317-341.
time. That is, unlike more traditional views where HCI is viewed as
communication between users and systems, in our view designers are
communicating with users at interaction time through a specifically
designed space of communicative exchanges that users can have with
the system”. *
 The system as a
bottle-with-a-message-inside…
Designers send
users a one-shot
message. The
message’s
representamen
(i.e. the perceptible
sign that stands for
the message) is the
The system as a
system itself (i.e.,
bottle-with-a-message-inside…
the interface and
the related /
controlled
behaviors) .
The one-shot message (in the bottle):

Here is my understanding of who you are, what I’ve learned

you want or need to do, in which preferred ways, and why.
This is the system that I have therefore designed for you, and
this is the way you can or should use it in order to fulfill a
range of purposes that fall within this vision.
Here is my understanding of who you are, what I’ve learned
you want or need to do, in which preferred ways, and why.
This is the system that I have therefore designed for you, and
this is the way you can or should use it in order to fulfill a
range of purposes that fall within this vision
It doesn’t matter if
your vision is either
objectivistic or
subjectivistic,
just…
It doesn’t matter if
your vision is either
objectivistic or
subjectivistic,
Remember: willing or not, your
just… values, stances and viewpoints
will be necessarily reflected
in your design and given
(imposed?) to the users!
Reflect on this!