The vOICe at Tucson 2002

Bach-y-Rita, Paul, Tactile sensory substitution in blind subjects. 186

Meijer, Peter,  Seeing with sound for the blind: Is it vision?  MP3 format 187  (slides in 2.4 MB  PDF file)

Fletcher, Pat,  Seeing with sound: A journey into sight.  MP3 format 188  

Sur, Mriganka, Rewiring cortex. 121

Noë, Alva, Neural plasticity and consciousness: An enactive approach. 120

Seeing with sound for the blind: Is it vision?
By Peter Meijer

It is technically possible to convert video into audio while preserving a significant amount of visual information in the resulting - usually highly complex - sounds. If blind people can learn to mentally reconstruct the visual content of live camera views, as now carried by these sounds or "soundscapes", then they might be able to "see" with sound. The prerequisite of getting visual information into the brain of blind people can nowadays be fulfilled, but it remains to be established to what extent this will amount to "vision" after prolonged immersive usage to allow the brain to adapt as far as it can.

The subject currently brings the frontiers in many technical and scientific disciplines together, e.g., computer science, neuroscience, psychology and philosophy. The latest personal computer technology nowadays allows for easy (home) assembly of wearable vision systems for totally blind users by using only affordable and mature mass market components such as webcams, subnotebook PCs and stereo headphones. PCs are now fast enough to handle real-time video to audio conversion in software, as with "The vOICe Learning Edition", available as a free download from the Internet. Neuroscience research on crossmodal plasticity has in recent years shown that the "visual" (occipital) cortex of blind people can be activated by sound, thus making it more plausible that a camera-based auditory display, acting as a crossmodal sensory bypass, may indeed induce or restore a level of vision. However, at a fundamental level, the distinction between seeing and hearing has become totally blurred by the artificially created crossmodal mapping. Is the qualification "vision" or "hearing" determined mostly by the physical carrier of the information, or rather by the information content? We conjecture that it is the latter. Still, even if a blind person would functionally behave much like a sighted person in various "visual" tasks, and even if his or her "visual" cortex would provably become functionally involved (recruited), one could still argue and insist that the brain is "only" doing much extended and very sophisticated auditory processing to make good use of the camera sounds.

So we may indeed need to resort to (third-person statistics of) first-person accounts of subjective experiences in order to answer questions relating to important qualia, as in asking "Is it vision?", or perhaps rather asking "Does it 'feel' like vision?", or asking to what extent it is like normal vision, to what extent it is like hearing, and to what extent it is like neither. Therefore, the two presentations on "seeing with sound" at this conference will address various ("third-person") scientific issues as well as preliminary first-person experiences reported by a blind user of the "seeing with sound" technology.

More information, as well as free evaluation software for use with a webcam and stereo headphones, is available from the website http://www.seeingwithsound.com

Seeing with sound: A journey into sight
By Pat Fletcher (a blind user of The vOICe)

I first became aware of the Seeing with Sound project in 1999. I was primarily interested in the program’s capability to determine color. I wanted a means by which I could identify the colors of the clothing in my wardrobe. did not realize at that time the full potential of the program nor how it would eventually so dramatically impact my life.

Initially beginning with a desktop, stereo headset and a scanner, I started to explore the many different features incorporated in the vOICe program. The soundscapes were unusually strange and confusing. It was like trying to discern the contents of a photograph using only your fingertips.

Eventually through experimenting by placing objects on the scan bed and comparing the soundscapes to the object in hand, I learn to relate and decode the information. Still the process was very slow and time consuming. I often stood on the threshold of giving up trying to comprehend these allusive sounding specters of sight teasing my mind’s eyes.

A true breakthrough began when I could leave the limiting boundaries of my desktop behind and use the program in a mobile environment. Clad in my armor of a notebook stuffed in a backpack, a close talk headset, and a web cam mounted on the brim of a ball cap, I took my first timid forays on to the pathways of becoming sighted.

I shall never forget the shock and joy of first glimpsing down my hallway and seeing blinds hanging on the window. Such a simple sight, yet after twenty years of blindness I was over whelmed. "It works - This program really works! I can see!" I spent the next following days and months re-discovering my new world. I had forgotten how intricately designed with detailed lines and patterns the world is created.

The evolution of my equipment from those early days is an on going process. The awkwardly built ball cap has been replaced with sophisticated hatcams, fanny packs, and video glasses built by security companies. I explored the world of "covert spy cams" in order to improve the cameras and conceal their presence. I have encountered the public’s curiosity and hostility despite explaining the meaning behind my wearing of this OD attire. It is a struggle to balance on the fine line of society’s acceptance and the wearing of the equipment.

Finally to address the question, " Is this vision?" Yes, vision, but with limitations. I presently can’t discern the finest details of a soundscape/image. I see only in black and white. If I am placed in an unfamiliar environment it is a struggle to decipher the meaning of the view till I am acquainted with the surroundings. My sight is constrained within the limitations of the hardware, the software, and my own abilities. As I cast my eyes toward TOMORROW’S horizons for the Seeing with Sound program, I can’t help but wonder where the journey into sight will lead.

The complete presentation by Pat Fletcher is available online as 16 Kbps streaming  RealOne/RealPlayer audio and 32 Kbps streaming  MP3 audio (23 minutes, 28 seconds).

With visual objects observed as auditory objects via soundscapes from The vOICe, the situation is different, and the answer would be ``Yes'': it is certainly possible to hear (and understand) the roundedness of a sphere as compared to the more sudden sound transitions observed with the squareness of an upright cube. What does that mean? Could it mean that seeing-with-sound is for early-blind people actually closer to vision than having biological eyesight surgically restored? It is certainly closer to vision than the phenomenon called "blindsight" in that the visual content as encoded in sound can be consciously accessed. See also the sound samples in the Powerpoint file  "molyneux.ppt" (750 K).