Accessible VRML/X3D for the Blind

Safe orientation and mobility training in 3D virtual environments!
Experience for yourself visual perspective, occlusion, and more!

The vOICe technology even supports access to 3D game worlds

The vOICe for Windows « The vOICe Home Page
« The vOICe for Windows

This web page is obsolete and will no longer be maintained.

VRML and its successor X3D are languages for building virtual 3-dimensional environments,
Short VRML/X3D movie (100K WMV)
often called "worlds". Sighted people can navigate these worlds using a VRML or X3D browser. Until now, purely visual virtual worlds have been inaccessible to blind people. This has changed! When using the screen sonification option of The vOICe for Windows seeing-with-sound software in combination with a VRML/X3D browser, totally blind people can now independently experience and navigate VRML/X3D scenes. This page explains how yet another accessibility barrier has been taken down. An example is included of a purely visual VRML/X3D environment that you as a blind visitor can (learn to) navigate. Once again this will demonstrate how The vOICe technology makes vision accessible, through audio rendering of arbitrary images for sound-controlled synthetic vision. Moreover, with The vOICe auditory display approach, no expensive special hardware nor changes to mainstream browsers are needed for access to virtual worlds:

It is Now available to you.  Free of charge!

Several MP3 audio samples on this page further illustrate the principles, but it is highly recommended to install the required software and explore interactively how the soundscape scenery around you changes as you move through a virtual environment. There is no experience like an immersive experience. Try it!

Later on, similar approaches for VRML/X3D sonification may also become available for mobile phones, for possible use in conjunction with The vOICe MIDlet and The vOICe for Android.

Virtual reality mobility training

Example of VRML/X3D scenery sonified by The vOICe: 4 soundscapes in 64K MP3 audio sample
The vOICe window while sounding VRML/X3D
Seeing with Sound - The vOICe

In order to view the VRML/X3D world available on this page, you first need to install a VRML 2.0 or  X3D compliant viewer plugin such as  Flux Player from Media Machines  - unless you already have that. Note that Flux Player requires clicking inside the Flux Player window (best done when maximimized) to make it responsive to the arrow keys for turning a scene or object: normal tabbing or Alt-tabbing to bring the player application or plug-in into focus does not suffice.

Secondly, you need The vOICe for Windows software. Once you have all that, you can proceed with the following recipe for hearing VRML/X3D worlds such as the one given on this page. First read and perhaps copy to another document all steps of the procedure such that you will remember or can find what to do after leaving this web page to enter the VRML/X3D world. The order of the steps may be important to avoid error messages on audio conlicts.

Procedure

What about 3D Games for the Blind?
Technical feasibility of accessing non-VRML/X3D 3D game environments
Non-VRML/X3D gaming: Using largely the same procedure, you can also access and hear the visual scenery in most of the latest 3D games. Just start your favourite DirectX or OpenGL 3D graphics game after applying steps 2 and 3.

Notes: Full accessibility or playability is not implied or guaranteed. Some games will fail to cooperate with The vOICe sonification. Visual contrast in 3D graphics games is often poor, resulting in low-contrast soundscapes. Use of a multichannel soundcard or equivalent is recommended.

The vOICe window: would in reality remain hidden
Click to hear The vOICe Half-Life MP3 soundscape (18K)
Once all of this works as described, you can start navigating around the VRML/X3D scene, using the arrow keys on your keyboard for full interactivity. Arrow-up moves forward, and arrow-down moves backward. The shift key can be used for more speed. The arrow-left key turns you several degrees to the left, while the arrow-right key turns you several degrees to the right Keeping the keys pressed will allow for more or less continuous motion. Apart from these basic arrow key controls, some viewers may support several other keyboard shortcuts that can be useful, and below we outline a few keyboard controls of the older  Cosmo Player viewer. For instance, pressing the delete or backspace key may undo your last movement. Pressing Shift Home may fly you back to the initial entry viewpoint. Pressing the page down key may automatically move to the next predefined viewpoint, if any, while page up may return to the currently selected (predefined) viewpoint. Alt arrow left and Alt arrow right may slide you sideways without changing the viewing direction, and control arrow up and control arrow down may let your viewing direction tilt up and down, respectively. Again, all these extra features, as described for Cosmo Player, vary with the type of viewer and may not be supported or may involve other keyboard shortcuts.

After the following brief description of The vOICe image to sound mapping, we will continue with a more complete overview of the VRML/X3D scene, such that you know what to look around for in this virtual reality training example.

Simple line and dot images The vOICe's synthetic vision mapping sounds the VRML/X3D scene as displayed on the screen by mapping brightness to loudness, elevation to pitch and lateral position to time in each image scan (using stereo panning for enhanced perception). Thus the greyscale content of any VRML/X3D scene can be rendered in sound. If your sound card allows for multiple simultaneous audio streams, you will not only hear The vOICe's soundscapes, but objects will also speak their names whenever you touch them. Objects thereby double as their own talking signs. As long as you do not touch or bump into objects, only the purely visual content of the soundscapes will guide you through the visual environment. This is the true equivalent of general vision, because it does not involve any tagging of visual objects, and will work for all existing VRML/X3D worlds on the web, including those that were not designed or adapted for accessibility!

The entry scene upon entering the VRML/X3D demo shows a view of a road running towards the horizon. The grey road surface gives a soft noise. The road has a wide dark grey verge, sounding as a still softer noise. Visual perspective makes the road sound like a big triangle with the point at the top: the so-called vanishing point at the far end of the road. Two fast rhythms correspond to two rows of pillars alongside the road. There are ten pillars bordering on the left side of the road and ten pillars bordering on the right side of the road, thus forming two rows that run in the same forward direction as the road. The dark grey road verge extends well beyond the left and right of these rows of pillars, but the lighter grey road surface does not. As you move forward (arrow-up or shift arrow-up), you get closer to the two rows of pillars, making them sound "bigger". Next, you will move in between the rows of pillars, and if you continue, you will end up on an empty stretch of road as the pillars move out of view behind you.

If you are currently just reading this web page, you can get some idea of the soundscapes beforehand by listening to an MP3 audio sample containing four soundscapes. If you have a media player installed, try the four soundscapes link to load the 64K MP3 audio sample, showing the VRML/X3D entry view, a viewpoint closer to the pillars, and two viewpoints resulting from turning some 30 degrees to the right while moving forward. In the fourth soundscape, one pillar is particularly nearby. Neglect the continuous high pitched sounds in the soundscapes: these just sound the menu bar at the top side of the browser window. Similarly, the lowest pitched sounds are from some VRML/X3D viewer controls showing near the bottom of the screen. These sounds too should be neglected because the visual controls are not part of the VRML/X3D scene.

Assuming again that you were actually navigating the scene, having moved beyond the rows of pillars, you can turn around with the left or right arrow keys (if supported by your viewer), and if you do a 180 degree turn, you will hear the pillars again that were behind you. Moreover, there is a bright wall behind the pillars, sounding as a band of rather loud noise. This wall was actually right behind you when you first entered the scene, but you could only have heard it by turning around: no eyes in the back of your head.
Perpendicular VRML view on wall with gate
A sideways VRML view on wall with gate
Now with the pillars and the wall in view, move forward such that you pass the pillars again, and listen for the gate in middle of the wall. By the way, the wall runs perpendicular to the road. Readers of this web page can try a 18K MP3 soundscape of a perpendicular view on the wall with its gate, in which you can clearly hear the interruption of the wall sound by the gate. Another 18K MP3 soundscape shows a sideways view on the wall with the gate, looking at the gate from a position somewhat to the right side of the gate. The soundscape at the latter viewpoint is rather similar to the demonstration sound as discussed on the "Bright Wall with Gate" web page. Continue to head for the gate, adjusting direction with left and right arrow keys as appropriate, and moving forward with the arrow-up key. If you do this carefully, you will not bump into the wall, but manage to move through the gate. At that point, you will see a big sphere on your left and a big cube on your right. These objects had been occluded by the wall until you passed the gate to move to the other side of the wall. You can head for the sphere or the cube, move around them, turn around to see the wall from the other side, and so on. You are entirely on your own now, but you cannot be hurt! This is a major advantage of orientation and mobility in virtual environments like these. Without risk, and without special hardware, you can now learn all about visual perspective, occlusion, parallax, shading, and so on and so forth. Orientation and mobility experts and others are invited to build other VRML/X3D worlds to arrive at a library of rich training environments, for instance with various kinds of visual landmarks for learning to see with sound.

The Molyneux problem revisited using The vOICe
Molyneux problem: sphere in VRML/X3D The seventeenth-century philosopher William Molyneux, whose wife was blind, asked his friend John Locke whether a man born blind, in case he recovered his sight, would be able to tell a cube from a sphere by sight alone, with previous sensory experience with cube and sphere limited to touch. Molyneux thought the answer was ``No'', and Locke agreed (in ``An Essay Concerning Human Understanding'').

With visual objects observed via soundscapes from The vOICe, the situation is different, and the answer would be ``Yes'': it is certainly possible to hear the roundedness of a sphere as compared to the more sudden sound transitions observed with an upright cube. Just try this with the soundscapes of the VRML/X3D view of a cube and a sphere as described on this page. See also the Powerpoint file  "molyneux.ppt" (750 K).

Molyneux problem: cube in VRML/X3D
 

A totally different interactive VRML/X3D scene, showing the planet Saturn with its rings, is given near the bottom of the planet Saturn page, while mental rotation test objects are discussed on the Shepard-Metzler page. The same basic approach as outlined above for accessing VRML 2.0, VRML97 or  Web3D's X3D can also be used with Ajax3D (formerly at www.ajax3d.org), Google  SketchUp, Macromedia  Shockwave 3D and Cycore  Cult3D, as well as with scalable vector graphics (SVG, sample file molyneux.svg). It can also be applied in multiplayer 3D worlds like  Second Life (cf.  Eduverse for a virtual education focus in the metaverse).

See also the interactive online 3D Maze example, and the Blind drawing with Logo page for programming your own 2D and 3D shapes.

Other related projects include  Making VRML Accessible at the Open Virtual Reality Testbed (OVRT) of the National Institute of Standards and Technology (NIST). For general information about VRML, visit the  Web3D Consortium website. A publication on the use of visual-to-auditory sensory substitution by blind people to access visual virtual environments is the paper  Perception of graphical virtual environments by blind users via sensory substitution" by Shachar Maidenbaum et al. in PLoS ONE 11(2), February 2016, e0147501.

Copyright © 1996 - 2024 Peter B.L. Meijer