Recommended reading:
Search for paradise: A patient's account of the artificial vision experiment
(Amazon: autobiography by Jens Naumann, 2012; alternatively get it from
Xlibris).
Also recommended is to listen to the interview with Jens Naumann about his former Dobelle brain implant for vision and his current use
of The vOICe to "see" in the NCBI
"Technology Podcast Episode 62: July 2017" (MP3 file URL).
In recent years, progress is being made towards sensory substitution
devices for the blind. In the long run, there could be the possibility of brain implants.
A brain implant or cortical implant provides visual input from a camera directly to the brain
via electrodes in contact with the visual cortex at the backside of the head. A computer is
used to process the sensory streams, as is typical for a brain-computer interface (BCI).
Initial results have been obtained by pioneering investigators like
William Dobelle (web archive),
for instance with his blind volunteer named Jerry, and later on with "Jens" (Jens Naumann, at the time
a blind farmer living near Napanee, Ontario, Canada), and
it may now be interesting to try and compare the current status of "bionic vision" through brain implant
technology with the current status of The vOICe auditory display technology.
|
``The device was pretty functional for as long as you could baby it and doctor it,''
Naumann recalls. But after Dobelle's death in 2004, Naumann could not maintain it.
The implant is still in his brain, but infections forced removal of the connectors
placed in his skull to connect the external camera.
![]() |
Now for some more facts, figures and notes:
Both the Dobelle brain implant and The vOICe auditory display make use of truly visual input, normally using a live video feed from a small camera.
![]() ![]() |
|
It has been demonstrated in some studies that to a sighted person, image resolution of some 32 by 32 pixels is (more than) enough to get easily recognizable images. See for instance the reference at the end of this web page to an article on pixelized vision by Richard Normann et al., which suggested a lower limit of around 625 pixels. Similarly, a study by Angélica Pérez Fornos suggested a minimum of 400–500 pixels for reading text, with less than a factor two further reduction in case of real-time visual feedback. Thus a 1000 pixels should do for many purposes, but some 64 pixels (for instance arranged in an 8 by 8 matrix) or less rarely gives recognizable images to a sighted person, so we cannot expect this to be any better in an alternative display that is likely to be much more crude than what Nature normally provides us with.
The effect of image resolution is further illustrated with the images below, where a photograph of a parked car has been pixelized to 4 by 4, 8 by 8, 12 by 12, 16 by 16, 32 by 32, 64 by 64 and 128 by 128 pixels, respectively. The images here still include shading, while some implants may give little more than on/off signals per pixel or phosphene.
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
(4 × 4) |
(8 × 8) |
(12 × 12) |
(16 × 16) |
(32 × 32) |
(64 × 64) |
(128 × 128) |
Visitors may download these images and next import them one by one into
The vOICe for Windows via its File menu,
or by pressing Control O to get to its file requester. For your convenience, the image
set is available for download as a 28K zip file cartest.zip
or as an animated GIF image, or you can listen to the
Visual resolution with The vOICe
video clip on YouTube.
Thus you can try it yourself and listen for any noticeable differences - or lack
thereof - as you pick different image resolutions:
See also the visual acuity in sensory substitution page. With The vOICe display, pixels (voicels) are all nicely ordered as in a bitmap, while it has been found that this is not the case with the cortical implants as developed and tried so far. Permutation of pixels (phosphenes) in a "stars-in-the-sky" like image perception has been reported. Of course, computer processing may be applied to undo these permutations as they occur, after the perceived image map has been systematically characterized. The stimulus independent flicker in the phosphene image map, as elicited by the electrodes, forms another yet unresolved problem with the brain implant approach, because it does not allow for shading. The vOICe incorporates shading naturally through smooth changes in loudness.
![]() |
![]() |
Listen to 44K WAV stereo soundscape (one second) or 88K WAV slow motion (two seconds) |
Via the above links to audio samples, you will hear the equivalent sounds for an example image showing a single period of a sine wave curve going up, down, and then up again, as well as ten little squares sounding as little noise bursts at various positions within the soundscape sample. The sound samples were synthesized with The vOICe for Windows seeing-with-sound software. Pitch always indicates vertical position, and stereo panning from left to right indicates horizontal position of any visual items. How well would a brain implant handle such an image?
Note: your audio player is best set to auto-repeat in order to hear the soundscape again and again for easier mental analysis.As a further illustration, we will take one of the simple example images that Dobelle discusses in his ASAIO paper on what has been dubbed the "Dobelle Eye": his "tumbling E" shows a capital "E" rotated 90 degrees to the left. This looks as if the capital "E" is sort of lying on its back, with the three "legs" pointing up. A typical time per image frame is about one second with image enhancement, both as reported about Dobelle's brain implant and in using The vOICe auditory display. However, the tunnel vision resulting from the narrow field of view of the brain implant makes that Jerry in addition needs to slowly scan around the shape of the letter "E" to identify it as the tumbled "E", thereby taking much more time than is needed with The vOICe.
The image on the right and image link give a demonstration of what The vOICe for Windows seeing-with-sound software makes of such an image. Clicking the image or activating the image link gives the corresponding one-second soundscape (44K file size).
![]() "Tumbling E" (44K WAV) |
Using The vOICe for Windows, one can import images via its file menu or by pressing Control O to get to the file requester. In this manner, or by using the built-in Internet sonification browser (Control U), one can for instance also download and import the following other variations of the tumbling "E" image set to hear out their differences:
![]() |
![]() |
![]() |
![]() |
Again, the use of inverse video F5 is recommended for these black-on-white images.
Clearly, at the current state-of-the-art, The vOICe offers a far higher image resolution than the brain implant, and it remains to be seen whether or when future brain implants can and will catch up with that as a visual prosthesis. The contact area between the electrode array and the visual cortex can probably be increased for a larger field of view, to alleviate some of the tunnel vision effect. However, increasing pixel density is likely to be much harder. Effective electrode density is for instance already limited today by the effects
| |
![]() |
The following appears similar to what a late-blinded user of The vOICe noted in saying
``Most people want that magic bullet giving instant sight.
While this does occur over time, it is a developed thing.''
On CBC, January 2003,
" |
![]() |
The implementation of both the Dobelle approach and The vOICe approach involves a fast notebook PC as a key part of the brain-computer interface (BCI), both apply or support image processing such as negative video, contrast enhancement as well as edge enhancement, and both can run up to 8 frames per second. The vOICe software includes support for enhanced depth perception through binocular vision when using a stereo camera. However, with The vOICe, a higher frame rate goes in a trade-off with resolution, being a consequence of the frequency-time uncertainty relation in using audible sound. The resolution trade-off with higher frame rates might in principle be less severe with a brain implant, but this is not yet known or at least not published, so perhaps effective resolution also goes down with frame rate for the brain implant. Note that even at 8 frames per second, the effective resolution offered by The vOICe may still be comparable to or above what the brain implant currently gives at one frame per second.
The vOICe approach obviously requires more or less normal hearing, so it cannot
be applied for deaf-blind people (unless in an experimental combination with
a cochlear implant), whereas a cortical implant may in principle prove applicable
for deaf-blind people. The vOICe also has the disadvantage that it interferes, or
may interfere, with normal hearing. The brain implant on the other hand has the
disadvantage that it is invasive, requiring head surgery with a risk of infection,
and with many uncertainties about the long-term effects of implanted electrodes.
One paper reports about cortical implant experiments performed at the NIH saying
``A volunteer at NIH died of infection after pulling at the external side of his
implant.'' (Daniel Wagenaar,
"Cortical stimulation for the evocation of visual perception",
term paper CNS247: Cerebral cortex, R. Andersen, Caltech, 2004.)
Or as one blind user of The vOICe put it: ``I call it my non-intrusive way to sight. Believe me if you have submitted to the many claims of surgery to regain sight you would understand this term. The Voice gives sight and does not hurt the body!'' A visual cortex brain implant cannot work for people who are blind due to damage in that brain area, be it due to traumatic brain injury or due to other causes, while The vOICe might conceivably still be applicable in such cases. The visual cortex brain implant probably also does not work for congenitally blind or early-blind people, because their visual cortex is organized differently. In fact, some would-be candidates for a Dobelle brain implant turned to The vOICe for this reason. The vOICe offers better chances here because of its partial reliance on auditory processing for parsing image content. The vOICe is more like an "explant" than an implant, and such a non-invasive external sensory bypass makes it much cheaper while it uses only proven, mature and technically reliable mass-market hardware. The hardware need not even be in touch with the body (using sound as the "wireless" connection to the body), it is perfectly biocompatible. Moreover, The vOICe's low-cost technology is available today to everyone for personal and non-commercial evaluation and exploration. It is up to the blind user to decide what is preferred.
Whether it will be or become easier to understand images from a brain implant or from The vOICe is currently not known. The demonstrations given so far with the Dobelle brain
Legend: (+)+ = (very) good, (-)- = (very) bad |
||
Cortical implant | The vOICe | |
Resolution | + (100 to 1,000 pixels) | ++ (1,000 to 10,000 pixels) |
Field of view | - (10 to 20 degrees) | ++ (60 to 135 degrees) |
Frame rate | ++ (4 to 8 fps) | + (1 to 4 fps) |
Shading | -- (phosphenes on/off only) | ++ (16 shades of loudness) |
Perception as light | + (flashing phosphenes) | -/+ (initially as sound) |
Sensory interference | ++ | - (some sound masking) |
Expense | -- ($100,000) | ++ ($500) |
Surgical risks | - (infection, stroke, ...) | ++ (non-invasive) |
Biocompatibility | - (corrosion, ...) | ++ (non-invasive) |
For early blind | -- (non-visual cortex) | ++ (if normal hearing) |
For late-blind | ++ (if normal visual cortex) | ++ (if normal hearing) |
Apart from the brain implant, there is the very promising development of ocular implants, specifically retinal implants for those whose blindness results only from malfunctioning of the retina, e.g., due to retinal degenerative diseases like Retinitis Pigmentosa (RP) or Age-related Macular Degeneration (AMD). There is also the option of using high resolution non-visual displays, such as sonar-based mobility devices. Various relevant links for these alternatives can be found on the external links page.
In June 2002, William Dobelle, John Antunes, Domingos Coiteiro, John Girvin and Kenneth Smith
reported about eight people who received implants on a commercial basis, from the Dobelle Institute
in Lisbon, Portugal: Jens Nauman, Dennis, Gerald, Kenneth, Marina, Edmundo,
Klaus Faron, and Keith Theobald. An information and fundraising website at
YouTube).
Also, she could only use the device for about one hour a day.
Cheri Robertson died September 24, 2015.
Other blind persons who sought a Dobelle brain implant included
Joe Lazzaro had planned to receive an implant at a later stage.
William Dobelle
died in October 2004
from complications of diabetes. Stony Brook University and
Avery Biomedical Devices Inc. would reportedly continue development of the Dobelle brain implant.
An approach similar to the Dobelle brain implant was pursued in a European research project named
CORTIVIS, which aimed
to demonstrate the feasibility of an electrode array implant to make an interface with the visual cortex
|
There is definitely a need for independent, systematic and unprejudiced benchmarking to move beyond the hype about "cybersenses", and compare the merits and prospects of the various artificial vision, vision substitution and synthetic vision technologies that are currently under development. Many of the above approaches show market potential, but market predictions may easily be defeated by unforeseen setbacks in clinical trials or by competition from alternative approaches, including the non-invasive approaches. Sensory substitution currently has a head start, possibly soon followed by the first generations of retinal implants.
Back to the Future
Some 15 years after receiving a Dobelle brain implant, Jens Naumann started using The vOICe. Below you find a few short video clips, recorded in 2017, that show him using The vOICe, with The vOICe for Android running on VISION-800 smart glasses.
This probably makes him the most qualified person in the world for comparing the pros and cons of brain implants and sensory substitution for vision from a blind end user perspective.
See also The vOICe Training Manual.
References:
W. H. Dobelle, M. G. Mladejovsky and J. P. Girvin,
``Artificial Vision for the Blind: Electrical Stimulation of Visual
Cortex Offers Hope for a Functional Prosthesis,''
Science, vol. 183 pp. 440-444, February 1, 1974.
W. H. Dobelle, ``Artificial Vision for the Blind by Connecting a Television Camera
to the Visual Cortex,'' ASAIO Journal (American Society for Artificial
Internal Organs), January - February 2000. Available
W. H. Dobelle, ``Artificial Vision/Human Implantation Program,'' presentation
at ASAIO 2002 (48th Annual Conference of the American Society
for Artificial Internal Organs), New York, June 13, 2002.
J. Dowling, W. Boles and A. J. Maeder ``Mobility Assessment Using Simulated Artificial Human Vision,''
1st IEEE Workshop on Computer Vision Applications for the Visually Impaired (CVAVI 2005),
in conjunction with the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2005),
San Diego, USA, June 2005.
E. Fernandez, F. Pelayo, P. Ahnelt, J. Ammermüller and R. A. Normann,
``Cortical visual neuroprostheses for the blind,'' in a special issue of
Restorative Neurology and Neuroscience on ``Neural chips and neural protheses'', 2004.
P. D. Fletcher, ``Seeing with Sound: A Journey into Sight,'' invited presentation at
the Tucson 2002 conference on Consciousness in Tucson,
Arizona, USA, April 8, 2002.
A. P. Fornos, J. Sommerhalder, B. Rappaz, A. B. Safran and M. Pelizzone,
``Simulation of artificial vision, III: do the spatial or temporal characteristics of stimulus pixelization really matter?''
Investigative Ophthalmology and Visual Science, Vol 46, pp. 3906-3912, 2005. Available
T. Kujala, K. Alho and R. Näätänen, ``Cross-modal reorganization of human cortical
functions [Review],'' Trends in Neurosciences, vol. 23, pp.115-120, 2000.
R. B. McDonald,
P. B. L. Meijer, ``An Experimental System for Auditory Image Representations,''
IEEE Transactions on Biomedical Engineering, Vol. 39, No. 2,
pp. 112-121, Feb 1992. Reprinted in the 1993 IMIA Yearbook of Medical
Informatics, pp. 291-300. Electronic version of full paper available
online.
P. B. L. Meijer, ``Cross-Modal Sensory Streams,'' invited presentation and demonstration
at SIGGRAPH 98 in Orlando, Florida, USA, July 19-24, 1998.
Conference Abstracts and Applications, ACM SIGGRAPH 98, 1998, p. 184.
P. B. L. Meijer, ``Seeing with Sound for the Blind: Is it Vision?,'' invited presentation
at the Tucson 2002 conference on Consciousness in Tucson,
Arizona, USA, April 8, 2002.
J. Naumann,
P. Stoerig, E. Ludowig, T. Mierdorf, A. Oros-Peusquens, J. N. Shah, P. B. Meijer and A. Pascual-Leone,
``Seeing through the ears? Identification of images converted to sounds improves with practice,''
poster presentation at
B. Thirion, E. Duchesnay, E. Hubbard, J. Dubois, J.B. Poline, D. Lebihan and S. Dehaene,
``Inverse retinotopy: Inferring the visual content of images from brain activation patterns,''
NeuroImage, Vol. 33, No. 4, pp. 1104-1116, 2006. Abstract available
D. A. Wagenaar, ``Cortical stimulation for the evocation of visual perception,''
term paper for CNS247: Cerebral cortex, R. Andersen, Caltech, 2004. Available
A. Amedi, F. Bermpohl, J. Camprodon, S. Fox, L. Merabet, P. Meijer and A. Pascual-Leone,
``Neural correlates of visual-to-auditory sensory substitution in proficient blind users,''
poster presentation at CNS 2005 (12th Annual Meeting of the Cognitive Neuroscience Society)
in New York, USA, April 11, 2005, and at the 57th Annual Meeting of the
American Academy of Neurology (AAN 2005), Miami Beach, Florida, USA, April 10 and 12, 2005.
K. Cha, K. W. Horch and R. A. Normann ``Mobility performance with a pixelized
vision system,'' Vision Research, vol. 32, pp. 1367-1372, 1992.
Jens Naumann discussing The vOICe sensory substitution at
t=4415.
online (web archive).
online (PDF file).
``My brain implant for bionic vision: the first trial of artificial sight for the blind,'' Amazon (July 2019).
``Search for paradise: A patient's account of the artificial vision experiment,'' Xlibris Corporation (August 2012).
online.
online (PDF file).
For other useful literature, see recent publications about artificial vision, phosphenes and brain-computer interfaces.
Common blind misspellings: dobel, dobell, debell, dobb, vioce, VOIC, vOISe, voyce, voyc.
William H. Dobelle (1941-2004)
Vision of the Future, Los Angeles Times, September 16, 2002