Let's make Vision Accessible

Scanning for soundscape Meijer's system [...] consists of a video camera that takes a picture which is converted into a digitised image made up of 64 by 64 pixels. But then the image is converted into sounds by a computer, following two simple rules. First, pixels situated ``high'' in the picture are converted into high tones; those that are low are converted into low tones. Secondly, the brighter the pixel, the louder the sound. So a bright pixel near the top of the pixel grid would be high-pitched and loud. If you were to ``hear'' a picture with Meijer's device, you wouldn't hear the whole image instantly: rather you would hear a column at a time, from left to right. A bright, diagonal line stretching upward to the right produces a loud ``ooiieep'' sound and another stretching downward to the right makes the opposite sound - ``eeiioop.'' After one entire scan, which takes about a second, the scan begins again. If the image changes, so will the next pattern of sound. [...] ``Blind people have their cane, which is a very useful thing,'' says Meijer. ``But they don't have the ability to detect buildings from a distance, or to recognize buildings they have encountered before. I hope a system like this would help with orientation in particular.'' Rosie Mestel, New Scientist, June 4, 1994, pp. 20-23.

Background

Synthetic vision by hearing sonified pictures

Auditory display for synthetic vision The vOICe hardware prototype Originally, a hardware prototype, nick-named The vOICe (OIC? Oh I see!), was developed to prove the technical feasibility of the underlying concepts. It starts by subdividing an electronic photograph into 64 rows and 64 columns, giving 4096 pixels. Shading is reduced to 16 levels of grey. Column by column, the image is then translated into sound. The top pixel in a column gives a high pitch, and the bottom pixel a low pitch. An intermediate position gives an intermediate pitch. The grey-level is expressed by loudness. All pixels in a single column are heard simultaneously, much like a musical chord. With subsequent image columns, these chords change according to the pixel brightness distribution within the column. In this way, the image content is translated into sound by scanning through all 64 columns, from left to right. Finally, a click marks the beginning of a new image. Typically, fresh electronic photographs are taken and converted into sound at one-second intervals. Of course, there is a lot more to be said about technical issues relating to this general image-to-sound mapping for artificial vision. The vOICe software Very important for a good mapping is the concept of preservation of simplicity and similarity: simple images should sound simple, and simple shifts in position should lead to correspondingly simple perceptual changes in the sounds. Complexity should not arise from the mapping itself, but only from the complexity of the content of the image being mapped into sound! Furthermore, the mapping should not only allow one to distinguish different image sounds and to learn to associate a given set of auditory patterns with their visual counterparts, but also to analyze and generalize them for situations that were not in the training examples. The vOICe hardware did not become available as a product. However, in the photograph on the left, a software version of The vOICe is being used, running on a fast Pentium notebook PC (inside the shoulder bag) with a PC camera and headphones. This software is available and now lets you experience The vOICe yourself using The vOICe for Windows for Microsoft Windows. Try it!

Show me more

• Literature: P.B.L. Meijer, ``An Experimental System for Auditory Image Representations,'' IEEE Transactions on Biomedical Engineering, Vol. 39, No. 2, pp. 112-121, Feb 1992. The paper was selected for reprint in the 1993 IMIA Yearbook of Medical Informatics, pp. 291-300.

• To make it easy to grasp the simplicity and generality of The vOICe mapping principles without requiring a camera, you can try the on-line interactive demonstration given by The vOICe Java auditory display applet: you can draw an image and simultaneously hear its corresponding soundscape

  Examples

  being synthesized on-the-fly!

  Individual bright dots simply sound like short beeps, with the pitch and time-after-click indicating the position of each dot in the image. The sound of a straight bright line on a dark background, running from the bottom left to the upper right, is heard as a single tone steadily increasing in pitch, until the pitch jumps down after the click that separates subsequent images. Two bright lines give rise to two tones of varying pitch, etc. In general, arbitrary images are converted into soundscapes.

  However, for the best results you should try The vOICe for Windows. It allows you to use a regular PC camera to interactively explore the full-stereo soundscapes of real-life images from your own environment! From the perspective of wearable computing technology, one can say that The vOICe for Windows provides a low-cost implementation of the concept of a ``wearable camera,'' while bringing so-called augmented reality to those who need it most for lack of direct perception of the visual reality.

• The image-to-sound mapping is related to spectrographic representations as used in speech analysis, and yields similar sounds as used in research on auditory profile analysis. To prove that the soundscapes still preserved enough image information, spectral analyses were performed to map sounds back to images, as illustrated below:

  Original image

  Mapping the above photograph of a parked car into sound, and then mapping the resulting soundscape back into an image via spectrographic analysis gives: Spectrogram of one second of sound from The vOICe This visual reconstruction proves that much of the original image was preserved in the soundscape generated by The vOICe: otherwise it would have been impossible to get a recognizable spectrogram.

  From image to sound and back

Recent highlights and key events from the history of The vOICe are listed on the highlights and events page.

Low vision is now within reach, but...