Both The vOICe and the PSVA are auditory displays developed for blind people, with the aim of conveying live visual information from a head-worn camera through complex sounds. The effectiveness of both approaches will to a large extent depend on the (largely unknown) neural plasticity of the human brain as required to master a new cross-modal mapping in order to mentally invert the image to sound mapping of the respective devices and approaches. Neuroscience research based on fMRI, PET, MEG and rTMS techniques has shown that the human brain can adapt and for instance cause the occipital cortex ("visual cortex") and visual association areas to respond to auditory stimuli, but the functional significance of this in relation to auditory displays as visual prostheses remains to be established. Research into brain plasticity and the use of the PSVA is performed under F.R.S.M. Grant 3.4505.04  "Neural activation study of sensory substitution and cerebral plasticity in the early blind human". Results of this study may in part also apply to The vOICe.

The vOICe was developed by Peter Meijer in The Netherlands, while the PSVA was later developed at the Université Catholique de Louvain (UCL - GREN, Catholic University of Louvain-la-Neuve) in Belgium by

October 2004 After trying the  PSVA demo material, one early blind female user of The vOICe commented "From what i could tell, the PSVA is very musical... to me... albeit kind of atonal, so it's hard to associate that with any kind of visual thing. With The vOICe, there are no notes or chords to distract you."

Capelle et al. in the group of Claude Veraart with later evaluation studies performed by  Laurent Renier and others. The image-to-sound mappings employed are different though. The vOICe uses an approximate inverse spectrographic mapping involving a scanning approach, whereas the PSVA uses a direct non-scanning mapping. The vOICe typically offers (technically) a soundscape image resolution on the order of 4000 pixels for a one-second sound scan of a given camera view, while the PSVA offers up to a 124 pixel resolution for a given view, leaving (manual) scanning for further detail up to the user. Although the PSVA in a sense gives more direct real-time feedback, the required manual scanning to compensate for its lower resolution makes that the actual time needed for fully analyzing an image or scene can be significantly larger than with The vOICe. In order to allow for benchmarking the respective approaches, a number of visual test patterns are provided on this page. These were modelled after the test patterns as given in the publications relating to the PSVA (see the references from Patricia Arno and Christian Capelle et al. below, notably the reference and  online demonstration on "Auditory coding of visual patterns for the blind").

You can download and unzip the 25 visual test patterns (as used with the PSVA) contained in sse.zip (5K zip file). The resulting 15 by 15 pixel images s01.gif through s25.gif can be easily imported for sonification into The vOICe for Windows via its File menu or by pressing Control O. Multiple images can be selected by keeping the Control or Shift key pressed while selecting files in the file requester, and The vOICe will then cycle through the selected set of images. Alternatively, you can also download and import the animated GIF image sse.gif into The vOICe for Windows to hear all visual patterns.

You will likely find that once you are only slightly familiar with the sounds of these test patterns, you can rather easily identify each of them in about a second, using The vOICe. This appears to be significantly faster than the time needed by users of the PSVA to identify the test patterns, in part because of the active exploration involved in pattern identification via the PSVA - although it may in part also be attributed to the experimental methodology, requiring the PSVA users to use random movements to allow for subtracting cerebral activity related to hand movements in the PET brain scan studies. Of course the final choice or preference is up to blind users.

General findings about crossmodal binding between pitch and elevation, as employed by The vOICe, are discussed in K. K. Evans and A. Treisman,  ``Crossmodal binding of audio-visual correspondent features,'' Journal of Vision, Vol. 5, No. 8, p. 874a, 2005, using congruent and incongruent bimodal stimuli in response time measurements. This indicates that the use of pitch for height by The vOICe is not an "arbitrary" encoding, but an encoding that is well-rooted in human physiology. Similarly, the temporal ventriloquism effects discussed in J. J. Stekelenburg and J. Vroomen,  ``An event-related potential investigation of the time-course of temporal ventriloquism,'' Neuroreport, Vol. 16, No. 6 pp. 641-644, April 25, 2005, lends further physiological support to The vOICe's time-domain multiplexing with lateral position (horizontal position) mapped to time and stereo position.

For more information about brain implants, see the brain implant page, and for still other sensory substitution (vision substitution and synthetic vision) approaches for the blind, see the sensory substitution (SENSUB) page.

References:

P. Arno, C. Capelle, M.-C. Wanet-Defalque, M. Catalan-Ahumada and Claude Veraart,  ``Auditory coding of visual patterns for the blind,'' Perception, vol. 28, no. 8, pp. 1013-1029, 1999. Mapping demonstration available  online.

P. Arno, A. G. De Volder, A. Vanlierde, M.-C. Wanet-Defalque, E. Streel, A. Robert, S. Sanabria-Bohórquez and C. Veraart, ``Occipital Activation by Pattern Recognition in the Early Blind Using Auditory Substitution for Vision,'' NeuroImage, vol. 13, pp. 632-645, 2001.

P. Arno, A. Vanlierde, E. Streel, M.-C. Wanet-Defalque, S. Sanabria-Bohorquez, C. Veraart, ``Auditory substitution of vision: pattern recognition by the blind,'' Applied Cognitive Psychology, Vol. 15, No. 5 , pp. 509 - 519, 2001. Available  online (PDF file).

D. Baril,  ``Entendre ce qu’on voit - Les aveugles «recyclent» leur cortex visuel pour d’autres tâches,'' University of Montreal iForum, Vol. 38, No. 23, March 8, 2004.

D. Brown, T. Macpherson and J. Ward, ``Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device,'' Perception, vol. 40, no. 9, pp. 1120–1135, 2011. Abstract available  online.

C. Capelle, C. Trullemans, P. Arno and C. Veraart, ``A Real-Time Experimental Prototype for Enhancement of Vision Rehabilitation Using Auditory Substitution,'' IEEE Trans. Biom. Eng., Vol. 45, pp. 1279-1293, Oct. 1998.

O. Collignon, M. Lassonde, F. Lepore, M. C. Vasilescu, D. Bastien, D. Tranduy and C. Veraart, ``Functional involvement of occipital cortex for sound processing in early blind subjects using a sensory substitution prosthesis,'' poster presentation at the 34th Annual Meeting of the Society for Neuroscience (SfN) in San Diego, USA, Sunday October 24, 2004.

O. Collignon, M. Lassonde, F. Lepore, D. Bastien and C. Veraart, ``Functional Cerebral Reorganization for Auditory Spatial Processing and Auditory Substitution of Vision in Early Blind Subjects,'' Cerebral Cortex, March 31, 2006. Available online (PDF file).

O. Collignon, G. Vandewalle, P. Voss, G. Albouy, G. Charbonneau, M. Lassonde and F. Lepore, ``Functional specialization for auditory-spatial processing in the occipital cortex of congenitally blind humans,'' PNAS, Vol. 108, No. 11, pp. 4435-4440, 2011. Available  online (PDF file).

O. Collignon, P. Voss, M. Lassonde and F. Lepore, ``Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects,'' Experimental Brain Research, Vol. 192, January 2009, pp. 343 - 358. Abstract available  online.

A. De Volder, O. Collignon, C. Poirier, L. Renier. D. T. Duy, A. Vanlierde and C. Veraart,  ``Neural activation study of sensory substitution and cerebral plasticity in the early blind human,'' 2004-2007 project description ("Etude, par l'activation neurale, de la substitution sensorielle et de la substitution sensorielle et de la plasticité cérébrale chez le non-voyant précoce").

A. G. De Volder, H. Toyama, Y. Kimura, M. Kiyosawa, H. Nakano, A. Vanlierde, M. C. Wanet-Defalque, M. Mishina, K. Oda, K. Ishiwata, and M. Senda, ``Auditory triggered mental imagery of shape involves visual association areas in early blind humans,'' NeuroImage, Vol. 14, pp. 129-139, 2001.

C. Poirier, O. Collignon, C. Scheiber, L. Renier, A. Vanlierde, D. Tranduy, C. Veraart and A. G. De Volder, ``Auditory motion perception activates visual motion areas in early blind subjects,'' NeuroImage Vol. 15, No. 31, pp. 279-285, May 2006.

C. Poirier, O. Collignon, C. Scheiber, L. Renier, A. Vanlierde, D. Tranduy, C. Veraart and A. G. De Volder, ``Specific activation of the V5 brain area by auditory motion processing: an fMRI study,'' Cognitive Brain Research, Vol. 25, pp. 650–658, 2005.

C. C. Poirier, A. G. De Volder and C. Scheiber, ``Visual-like perception through an auditory-for-visual substitution prosthesis,'' poster presentation at the 12th Annual Meeting of the Organization for Human Brain Mapping (HBM 2006) in Florence, Italy, June 11-15, 2006.

C. C. Poirier, A. G. De Volder and C. Scheiber, ``What neuroimaging tells us about sensory substitution,'' Neuroscience & Biobehavioral Reviews, Vol. 31, No. 7, pp. 1064–1070, 2007. Abstract available  online.

C. Poirier, A. G. De Volder, D. Tranduy and C. Scheiber, ``Neural changes in the ventral and dorsal visual streams during pattern recognition learning,'' Neurobiology of Learning and Memory, Vol. 85, No. 1, pp. 36-43, January 2006.

C. Poirier, A. De Volder, D. Tranduy and C. Scheiber, ``Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects,'' Neuropsychologia, Vol. 45, No. 5, pp. 1108-1121, 2007. Abstract available  online.

C. Poirier, M.-A. Richard, D. T. Duy and C. Veraart, ``Assessment of Sensory Substitution Prosthesis - Potentialities in Minimalist Conditions of Learning,'' Applied Cognitive Psychology, Vol. 20, No. 4, pp. 447-460, 2006.

L. Renier, R. Bruyer and A. G. De Volder, ``Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision,'' Perception & Psychophysics Vol. 68, No. 4, pp. 535-542, May 2006.

L. Renier, O. Collignon, C. Poirier, D. Tranduy, A. Vanlierde, A. Bol, C. Veraart and A. G. De Volder, ``Cross-modal activation of visual cortex during depth perception using auditory substitution of vision,'' NeuroImage, Vol. 26, pp. 573-580, 2005.

L. Renier, O. Collignon, D. Tranduy, C. Poirier, A. Vanlierde, C. Veraart and A. De Volder, ``Visual cortex activation in early blind and sighted subjects using an auditory visual substitution device to perceive depth,'' poster abstract TU 364, 10th Annual Meeting of the Organization for Human Brain Mapping (HBM2004), Budapest, Hungary, June 13-17, 2004.

L. Renier and A. G. De Volder, ``Cognitive and brain mechanisms in sensory substitution of vision: A contribution to the study of human perception,'' Journal of Integrative Neuroscience, Vol. 4, No. 4, pp. 489-503, December 2005.

L. Renier and A. G. De Volder, ``Vision substitution and depth perception: Early blind subjects experience visual perspective through their ears,'' Disability & Rehabilitation: Assistive Technology, Vol. 0, No. 0, pp. 1-9, 2010. Abstract available  online.

L. Renier, C. Laloyaux, O. Collignon, D. Tranduy, A. Vanlierde, R. Bruyer, A. G. De Volder, ``The Ponzo illusion using auditory substitution of vision in sighted and early blind subjects, Perception, Vol. 34, pp. 857-867, 2005.

L. Renier, C. Laloyaux, O. Collignon, D. Tranduy, A. Vanlierde, A. G. De Volder,  ``Perception of visual illusions with a sensory substitution system,'' Proc. 26th European Conference on Visual Perception (ECVP 2003), Paris, France, September 1 - 5, 2003.

C. Veraart, F. Duret, M. Brelén, M. Oozeer and J. Delbeke, ``Vision rehabilitation in the case of blindness,'' Expert Review of Medical Devices (Review), Vol. 1, No. 1, pp. 139 - 153, 2004. Full article available  online (PDF file).

---

A. Amedi, F. Bermpohl, J. Camprodon, S. Fox, L. Merabet, P. Meijer and A. Pascual-Leone, ``Neural correlates of visual-to-auditory sensory substitution in proficient blind users,'' poster presentation at the 12th Annual Meeting of the Cognitive Neuroscience Society (CNS 2005) in New York, USA, April 11, 2005, and at the 57th Annual Meeting of the American Academy of Neurology (AAN 2005), Miami Beach, Florida, USA, April 10 and 12, 2005.

A. Amedi, F. Bermpohl, J. Camprodon, L. Merabet, P. Meijer and A. Pascual-Leone, ``LO is a meta-modal operator for shape: fMRI study using auditory-to-visual sensory substitution,'' poster presentation at the 12th Annual Meeting of the Organization for Human Brain Mapping (HBM 2006) in Florence, Italy, June 11-15, 2006. Abstract available online.

A. Amedi, J. Camprodon, L. Merabet, P. Meijer and A. Pascual-Leone, ``Towards closing the gap between visual neuroprostheses and sight restoration: Insights from studying vision, cross-modal plasticity and sensory substitution,'' [Abstract]. Journal of Vision, Vol. 6, No. 13, 12a, 2006. Abstract available  online.

A. Amedi, W. Stern, J. A. Camprodon, F. Bermpohl, L. Merabet, S. Rotman, C. Hemond, P. Meijer and A. Pascual-Leone, ``Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex,'' Nature Neuroscience, Vol. 10, No. 6, pp. 687 - 689, June 2007. Abstract available online.

A. Amedi, W. Stern, J. A. Camprodon, F. Bermpohl, L. Merabet, P. Meijer and A. Pascual-Leone, ``Extracting shape and location information conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex and dorsal visual stream respectively in blind and sighted individuals,'' poster presentation at the 8th Annual Meeting of the International Multisensory Research Forum (IMRF 2007), Sydney, Australia, July 5-7, 2007. Abstract available online.

A. Amedi, W. Stern, L. Merabet, E. Striem, U. Hertz, P. Meijer and A. Pascual-Leone, ``Audio-visual integration for objects, location and low-level dynamic stimuli: novel insights from studying sensory substitution and topographical mapping,'' poster presentation at the 9th Annual Meeting of the International Multisensory Research Forum (IMRF 2008), Hamburg, Germany, July 16-19, 2008. Abstract available online.

A. Amedi, W. Stern, E. Striem, U. Hertz, P. Meijer and A. Pascual-Leone, ``A what/where visual-to-auditory sensory substitution fMRI study: Can blind and sighted hear shapes and locations in the visual cortex?,'' presentation at the 31st European Conference on Visual Perception (ECVP 2008), Utrecht, The Netherlands, August 24-28, 2008. Abstract available online.

M. Auvray, ``Immersion et Perception Spatiale - L’exemple des dispositifs de substitution sensorielle,'' Ph.D. thesis, Laboratory for Experimental Psychology of the University of Paris 5, Paris, France, 2004. Available online (PDF file, 4MB).

M. Auvray, S. Hanneton and J. K. O'Regan, ``Localisation and form recognition in sensory substitution,'' poster presentation at 26th European Conference on Visual Perception (ECVP 2003), Paris, France, September 1 - 5, 2003.  Abstract and poster (PDF file) available online.

M. Auvray, S. Hanneton and J. K. O'Regan, ``Learning to perceive with a visuo-auditory substitution system: Localisation and object recognition with ‘The vOICe’,'' Perception, vol. 36, no. 3, pp. 416–430, 2007. Abstract available  online.

M. Auvray, S. Hanneton, C. Lenay and K. O'Regan, ``There is something out there: distal attribution in sensory substitution, twenty years later,'' Journal of Integrative Neuroscience, Vol. 4, No. 4, pp. 505-521, December 2005.

M. Auvray and J. K. O'Regan, ``Voir avec les oreilles,'' Dossier Pour La Science, April 2003, pp. 30-35. Available online (PDF file).

P. D. Fletcher, ``Seeing with Sound: A Journey into Sight,'' invited presentation at the Tucson 2002 conference on Consciousness in Tucson, Arizona, USA, Monday April 8, 2002.

O. Gapenne and P. Gaussier, ``Suppléance perceptive et interface: une introduction,'' Arobase, Vol. 1, pp. 1-7, 2005.

J.-K. Kim, ``Sensory substitution learning using auditory input: Behavioral and neural correlates,'' PhD thesis, Montreal Neurological Institute, McGill University, Montreal, 2011. Available  online.

J.-K. Kim and R. J. Zatorre, ``Visual-to-auditory substitution learning: Behavioral findings and neural correlates,'' poster presentation at the 7th Annual Meeting of the International Multisensory Research Forum (IMRF 2006), Dublin, Ireland, June 18 - 21, 2006. Abstract available  online.

J.-K. Kim and R. J. Zatorre, ``Generalized learning of visual-to-auditory substitution in sighted individuals,'' Brain Research, Vol. 242, pp. 263-275, 2008. Available  online (PDF file).

J.-K. Kim and R. J. Zatorre, ``Can you hear shapes you touch,'' Experimental Brain Research, Vol. 202, 2010, pp. 747–754. Available  online (PDF file).

J.-K. Kim and R. J. Zatorre, ``Tactile-auditory shape learning engages the Lateral Occipital Complex,'' Journal of Neuroscience, Vol. 31, No. 21, pp. 7848-7856, May 2011. Available  online.

A. Jacomuzzi and N. Bruno, ``Perceiving occlusion through auditory-visual substitution,'' Cognitive Processing, Vol. 7 (Suppl. 1), pp. S128–S130, 2006. Abstract available  online. Also presented at ICSC 2006 (3rd International Conference on Spatial Cognition), Rome and Perugia, September 2006.

P. B. L. Meijer, ``An Experimental System for Auditory Image Representations,'' IEEE Trans. Biom. Eng., vol. 39, No. 2, pp. 112-121, Feb 1992. Reprinted in the 1993 IMIA Yearbook of Medical Informatics, pp. 291-300.

P. B. L. Meijer, ``Cross-Modal Sensory Streams,'' Conference Abstracts and Applications, ACM SIGGRAPH 98, p. 184, as part of an invited panel presentation and demonstration of The vOICe for Windows at SIGGRAPH '98, Orlando, Florida, 1998.

P. B. L. Meijer, ``Seeing with Sound for the Blind: Is it Vision?,'' invited lecture at the VSPA conference on Consciousness at the University of Amsterdam, Amsterdam, The Netherlands, June 1, 2001.

P. B. L. Meijer, ``Seeing with Sound: Wearable Computing for the Blind,'' invited presentation at NIC2001 (Nordic Interactive Conference), Copenhagen, Denmark, Thursday November 1, 2001.

P. B. L. Meijer, ``Seeing with Sound for the Blind: Is it Vision?,'' invited presentation at the Tucson 2002 conference on Consciousness in Tucson, Arizona, USA, Monday April 8, 2002.

L. Merabet, D. Poggel, W. Stern, E. Bhatt, C. Hemond, S. Maguire, P. Meijer and A. Pascual-Leone, ``Activation of visual cortex using crossmodal retinotopic mapping,'' poster presentation at the 14th Annual Meeting of the Organization for Human Brain Mapping (HBM 2008) in Melbourne, Australia, June 15-19, 2008. Abstract available online.

L. B. Merabet, L. Battelli, S. Obretenova, S. Maguire, P. Meijer and A. Pascual-Leone, ``Functional recruitment of visual cortex for sound encoded object identification in the blind,'' Neuroreport, Vol. 20, No. 2, pp. 132-138, January 2009. Abstract available online.

B. Pollok, I. Schnitzler, T. Mierdorf, P. Stoerig and A. Schnitzler, ``Image-to-sound conversion: experience-induced plasticity in auditory cortex of blindfolded adults,'' Experimental Brain Research, Vol. 167, No. 2, November 2005, pp. 287 - 291. Abstract available  online.

M. J. Proulx, P. Stoerig, E. Ludowig and I. Knoll, ``Seeing 'Where' through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution,'' PLoS ONE, Vol. 3, No. 3, March 2008, e1840. Available  online.

S. Pennec, ``Monte le son, je vois pas bien...,'' introductory article on Cuk.ch about The vOICe, in French, December 17, 2004.

Z. Reynolds and B. Glenney, ``Interactive Training for Sensory Substitution Devices,'' Asia-Pacific Computing and Philosophy (AP-CAP 2009) in Tokyo, Japan, October 1-2, 2009. Available  online.

P. Stoerig, E. Ludowig, P. B. L. Meijer and A. Pascual-Leone, ``Seeing through the ears?,'' poster presentation at the 4th Forum of European Neuroscience (FENS Forum 2004) in Lisbon, Portugal, Wednesday July 14, 2004.

P. Stoerig, E. Ludowig, T. Mierdorf, A. Oros-Peusquens, J. N. Shah, P. B. Meijer and A. Pascual-Leone, ``Seeing through the ears? Identification of images converted to sounds improves with practice,'' poster presentation at the 34th Annual Meeting of the Society for Neuroscience (SfN 2004) in San Diego, USA, Sunday October 24, 2004.

E. Striem-Amit, L. Cohen, S. Dehaene and A. Amedi, ``Sensory substitution selectively activates the visual word form area in the blind,'' Neuron, Vol. 76, No. 3, pp. 640-652, 2012. Abstract available  online.

E. Striem-Amit, O. Dakwar, U. Hertz, P. Meijer, W. Stern, A. Pascual-Leone and Amir Amedi, ``The neural network of sensory-substitution object shape recognition,'' Functional Neurology, Rehabilitation, and Ergonomics, Vol. 1. No. 2, pp. 271-278, 2011. Available  online.

E. Striem-Amit, M. Guendelman and A. Amedi, ``'Visual' Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution,'' PLoS ONE, Vol. 7, No. 3, March 2012, e33136. Available  online.

J. Ward and P. Meijer, ``Visual Experiences in the Blind induced by an Auditory Sensory Substitution Device,’’ Consciousness and Cognition, Vol. 19, No. 1, pp. 492-500, March 2010. Abstract available  online.

"The vOICe" transforme des images vidéo en sons. Les images captées par une webcam (ou un téléphone mobile avec caméra) sont scannées de la gauche vers la droite et transformées en informations sonores en fonction de leur emplacement et de leur luminosité. C'est une technologie d'aide pour les aveugles et malvoyants.

Original camera images (left) and spectrographic reconstructions from The vOICe soundscapes (right).

The vOICe for Windows translates images from your PC camera (webcam) into sounds that you hear via your stereo headphones, thereby targetting auditory synthetic vision for the totally blind by means of a wearable computer. The vOICe for Windows has been available since January 1998. The vOICe approach was originally published in the IEEE Transactions on Biomedical Engineering, Vol. 39, No. 2, pp. 112-121, Feb 1992: P.B.L. Meijer, ``An Experimental System for Auditory Image Representations.'' This paper was next selected for reprint in the 1993 IMIA Yearbook of Medical Informatics, pp. 291-300. Awarded U.S. Patent 5097326, on an "image-audio transformation system", filed July 27, 1990: ``In a device for converting visual images into representative sound information especially for visually handicapped persons an image processing unit is provided with a pipelined architecture with a high level of parallelisum. An image is scanned in sequential vertical scanlines and the acoustical representatives of the scanlines are produced in real time. Each scanline acoustical representation is formed by sinusoidal contributions from each pixel in the scanline, the frequency of the contribution being determined by the position of the pixel in the scanline and the amplitude of the contribution being determined by the brightness of the pixel.''

Common misspelling: PVSA for PSVA.