Transforming the Enculturated Mind: Sensory Substitution and Complementarity

Mirko Farina
farinamirko@gmail.com

Cognitive Science ARC Centre of Excellence in Cognition and its Disoders

Complementarity in a Nutshell
Complementarity defences of EMT argue that many of the kinds of cognition humans excel at can only be accomplished by brains working together with a body that directly manipulates and acts on the world I take SSDs as my empirical case study to explore and illustrate the ramifications of Complementarity

Todays Talk
In the first part of this talk, I quickly look at the issue of whether there is truly substitution (either visual or tactile) or not

I argue that there is no real substitution but rather cognitive and perceptual supplementation
I finish up by relating my conclusions to the idea of Complementarity

So, the Idea

Sensory substitution basically means to transform the characteristics of one sensory modality into stimuli of another sensory modality The principles of sensory substitution have been formulated by Bach-y-Rita, who conducted experiments with the potential of the skin as a medium for transmitting pictorial material

Visual-to-Tactile Substitution Devices

Visual-to-Auditory Substitution Systems

SSDs raise many interesting philosophical questions

What type of perceptual experience the practiced user of a sensory substitution device can be said to undergo?

Keeleys Dedication
Keeley believes that possessing an authentic sensory modality involves the acquisition of a genuine, wired-up (dedicated) sense organ

This organ has to be phylogenetically developed to facilitate survival with respect to an identifiable class of phenomena

Dedication and SSD perception

It is true that through the coupling with the device the impaired user receives visual information about the world, but she discriminates such stimuli behaviourally, via a tactile/auditory capacity! Providing blind individuals with an SSD doesnt suffice to endow them with a sensory modality they did not have before.

For Keeley, the eyes are necessary for seeing and nothing can see that doesnt have a sense that has evolved for detecting properties via light. The coupling with the SSD only allows the agent to jerry-rig a sensory system dedicated to the reception of mechanical distortion (his skin) into one capable of providing him with generally reliable information about the electromagnetic spectrum. [Keeley (2002),p.20]. SSD provides the impaired users with information but only via a dedicated (tactile/auditory) channel that has already evolved to detect properties in the world.

Spatial Encoding & Cognitive Inference

Prinz and Block also accept that the SSD perceiver has experiences with spatial significance but they deny that this spatial significance is visual in character

They concede that SSDs encode spatial contents but argue that this only enables the visually impaired to use some features of the proximal stimulus to make cognitive inferences on the basis of dedicated neural pathways

Is it rather Vision?
Hurley and No have argued that after substantial training and adaptation the phenomenology of the perception obtained through the coupling with an SSD switches from tactile/auditory to visual

The nature of SSD perception

Depends on the amount of sensorimotor contingencies that this acquired perception shares with natural vision The more the user masters the device, the more invariants her acquired perception shares with vision. The more invariants the acquired perception shares with natural vision the more it resembles it.

Is it TRULY substitution?
I argue it is not !

It is a supplementation

Neither Touch nor Vision

Auvray et al. (2007) the conveyed qualitative experience is not automatically associated to either audition/touch or vision but rather reported to occur as something entirely new, whose nature was essentially task-dependent

tactile sensation persists over time AFTER TRAINING - veridical representations of things out there in a three-dimensional space
This new type of experience doesnt entirely qualify as tactile nor exclusively as visual, but possesses both components

Quasi Vision?
Through the coupling with SSDs the visually impaired gets a mode of access to the world that depends vertically on pre-existing modes of perception whilst nevertheless counting as something entirely new This new mode of access emerges from users preexisting sensory modalities, and its novelty is determined by the fact that it no longer aligns with them

So SSD perception, stands at a new level above the pre-existing perceptual modalities and its various sensory divisions

Something new: but what exactly?

If the phenomenology of SSD perception doesnt stay in one modality but exploits the pre-existing senses to give us something new, couldnt we just speculate that SSD perception, in giving the visually impaired something new, blends vision with hearing or touch?

Artificial Synaesthesia?
Synaesthesia is condition in which stimulation of one sensory or cognitive pathway leads to automatic and unintentional occurrences in a second sensory or cognitive pathway.

Synaesthesia is often described as a merging of the senses, a cross-modal union of different sensory modalities!

The Neuroscience Behind Synaesthesia

To explain the neurocognitive mechanisms that characterise this phenomenon Cohen Kadosh, Walsh, and Henik have suggested that synaesthesia is due to disinhibition or unmasking of signals between or within brain areas

The Neuroscience behind SSDs (1/2)

Amedi et al. (2007) have shown that the LOtv is involved in shape extraction/recognition from visual-to-auditory soundscapes Interestingly, this is reported to trigger cross-modal experiences where auditory/tactile stimulations are combined with visual elaboration

The Neuroscience behind SSDs (2/2)

Kupers and colleagues (2011) have recently favoured an account of cross-modal plasticity in SSD users that involves disinhibition of existing pathways over a view that prescribes cortical reorganisation

If cross-modal plasticity in SSD users is explained in terms of disinhibition and this form of disinhibition or unmasking also characterises the phenomenon of synaesthesia, couldnt we propose that a form of artificially induced synaesthesia can occur in SSD perception?

A Report:
Monochrome artificially induced synaesthesia in certain frequencies of sound...The thing I experience is not in color, is in my mind's eye, and can be very distracting. The shapes are consistent and can be reproduced by the same sound. It is almost as if you had a computer with two monitors running simultaneously different pictures, one was a very grey blurred version of the real world, and the other was a pure grey background with a big semi-circular light grey arc on it, and sometimes you switched your attention between both. [Ward & Meijer (2010),p.497-498].

Stability over time: a hallmark of Synaestetic Experience !

Interestingly, subjects claim to see sounds even when not wearing the device.

Their brain has internalized the vOICe rules for mapping between hearing and vision and these rules are deployed, by virtue of mental imagery, both when the device is worn and when it is not

A Long Journey Back Home .

This novel sensorimotor coupling triggers an experience that is quite original SSDs systematically transform the sensory experience of the impaired, by providing a novel perceptual modality that compensates for loss or impaired sensory channel Mind Enhancing Tools, at least for the visually impaired [Clark (2003)]

Cognitive Reengineering
Auvray &Myin (2009)have argued that, such devices should not be understood as merely external stand-ins for already existing purely internal processes ... but rather taken to transform cognition and perception in a qualitative way [Auvray & Myin (2009), p.1051]; in a way that would otherwise be precluded to the impaired non-SSD user

Through learning in fact, they get factored and integrated into the impaired users perceptual processing and become a different but complementary part of the machinery that realises her cognitive capacity

SSDs are therefore an example of cognitive and perceptual transformation, achieved via acquisition of embodied expertise

Thus SSDs, via plasticity, provide the visually impaired with the means for expanding perception towards new horizons SSD perception isnt a mere substitution but rather an addition, a supplementation or better a complement

The result of this complement is a biotechnological synthesis that entails the creation of a new space of coupling between a human being and the world

Special Thanks to:

John Sutton Richard Menary Julian Kiverstein Andy Clark Greg Downey Malika Auvray Jack Loomis Peter Meijer