Remote Control of The vOICe

Software developers can use the C/C++ source code on this page to make their application audio-compatible with The vOICe for Windows, as well as to interface with for instance lightweight indoor and outdoor eye-tracking devices to support active vision through the use of eye saccades with The vOICe auditory display. The experience of seeing with sound may prove more compelling, and more visual, through the use of an eye-tracker that controls the camera view that is rendered into sound. It amounts to a non-invasive alternative for a camera implant in the eye, and may also be applied in experimental cross-modal cognitive load balancing systems that aim at augmented cognition through sensory channel mode switching.

To simulate the auditory equivalent of saccades for blind users having good functional control over their eye movements, one can think of using The vOICe in combination with the  SMI Gazewear from  SensoMotoric Instruments (SMI), the EyeLink II (ELII) from  SR Research or the head-mounted eye tracking systems from  Engineering Systems Technologies (EST),  Applied Science Laboratories (ASL, "Mobile Eye"),  Arrington Research ("EyeFrame"),  Positive Science, LLC, or  EyeTracking, Inc. (ETI). Related research is the  Itap (Information Technology for Active Perception) project at the Institute for Neuro- and Bioinformatics at the University of Luebeck in Germany, by Erhardt Barth and Thomas Martinetz. Such an approach lends further support to the position of The vOICe as a non-invasive vision technology for the totally blind, as compared to invasive technogies such as retinal implants and brain implants, although a hybrid approach with camera implants for The vOICe (intraocular camera) is another possibility. Moreover, gaze-directed use of The vOICe by means of an eye-tracker may prove of use to people with central vision loss, for instance caused by age-related macular degeneration (AMD or ARMD), by giving them detailed auditory feedback on what is in the blind central part of their eye's view, using a zoomed-in (or telescope) view. For sighted people too, it might be an interesting experience to simultaneously hear what they are looking at or searching for, for instance while searching the web with Google ( Eye-tracking studies: more than meets the eye).

Alternatively, one can think of a head-mounted device (HMD) that uses Electroencephalography (EEG) for measuring the neural activity of the brain, Elecromyography (EMG) for measuring electrical signals from muscle contractions, Electrooculography (EOG) for measuring the electrical potential of the skin around the eyes, or a tongue drive system for measuring tongue movements. Such alternative approaches may for instance suit blind people whose eyes were enucleated for medical reasons, such that conventional eye-tracking for gaze direction is not applicable. In this context one may check out the non-invasive neural sensor and signal processing technology of  NeuroSky to control operation of The vOICe software.  Emotiv Systems is working on a brainwave detection helmet in what they call "Project Epoc", which could control The vOICe via the Emotiv Development Kit.

Eye-tracking in combination with The vOICe could also serve to make Google's proposed use of eye-tracking for unlocking augmented reality glasses accessible for totally blind users of augmented reality glasses (see  Unlocking a screen using eye tracking information and The vOICe for Android).

Real-time sensory substitution for the blind?

A preliminary investigation of a combination of The vOICe with eye-tracking, using the software interface described on this web page, was performed in 2004 by Simon Vogt of the Institute for Neuro- and Bioinformatics at the University Luebeck in Germany (reported in Forschungsbericht Universität zu Lübeck, Juli 2004, and documented in his September 17, 2004 Bachelor's thesis titled "Active seeing with sound - optimizing auditive image perception by using eye movements").

Command sequences via vOICeCommand() like "Speed*4", "Zoom*4" and "Saccade 25 75" (or combinations of "Home", "Left", Right", "Up", "Down") can be used to quickly associate newly tracked eye positions with corresponding changes in the absolute positions in a zoomed auditory view, thereby possibly enhancing the visual experience in seeing with sound. A special lens or mirror may be used to maintain some peripheral vision in addition to the zoomed "foveal" view, since The vOICe does not yet support this in software.

Note: remote control commands issued via the vOICeCommand() function are available only to registered users of The vOICe. However, the (un)muting via vOICeSound() for audio compatibility does not require registration. Also, registration is no longer needed as of The vOICe for Windows version 2.01 and later.

/* vOICeControl.c, version 1.01. (c) 1998 - 2003 Peter B.L. Meijer. * For use with The vOICe for Windows v1.50 or later. * * Source code for "remote control" of The vOICe for Windows, * using WM_COPYDATA messages to claim and release the audio device * or apply other supported remote control commands. No lib's or * dll's or other proprietary binary material is involved here! * Note: no Unicode is used, so if necessary change your project's * configuration properties to use the multi-byte character set. * * REGISTRATION REQUIRED? * The (un)muting via vOICeSound() does not require registration, * but all the commands issued via vOICeCommand() are available * only to registered users of The vOICe. * * APPLICATION EXAMPLES: * For instance, a screen reader or a talking GPS system could * turn The vOICe sound OFF by calling vOICeSound(hWnd, FALSE), * next speak one or several sentences, and when done turn * The vOICe sound ON again via a call to vOICeSound(hWnd, TRUE). * * With registered users, third-party eye-tracking software could * couple the eye saccades of blind users to corresponding view * changes in The vOICe auditory display using vOICeCommand() calls. * Similarly, EMG or EEG signals could be used. * * HOW TO USE: * Copy the declarations and definitions of the vOICeDone global * variable as well as the vOICeSound(), vOICeCommand() and * vOICeConfirm() functions into your application's source code, * and call the vOICeConfirm() when a WM_COPYDATA event occurs. * Basically that's all there is to it! * * FREE USE: * You may freely use and adapt this source code to make your * own software application compatible with The vOICe software. * Please report problems or other issues back to the author, * and give appropriate credit for the use of this interface. */ #include <windows.h> /******************* START ********************* * Functions for sending messages to The vOICe * ***********************************************/ // Copy this into your application's source code // Global variable BOOL vOICeDone = FALSE; /* Function prototypes, in case you need these */ // BOOL vOICeSound(HWND hWnd, BOOL Sound); // BOOL vOICeConfirm(HWND hWnd, HWND hWndSrc, PCOPYDATASTRUCT pcds); // BOOL vOICeCommand(char *cmd); /* Function to turn The vOICe sound ON or OFF. * Returns FALSE if a time-out error occurs. */ BOOL vOICeSound(HWND hWnd, BOOL Sound) { static char *message; static COPYDATASTRUCT copydata; int time_out = 0, TIME_OUT = 100; HWND hvOICeWnd = NULL; // The vOICe target window is named "The vOICe" hvOICeWnd = FindWindow("The vOICe", NULL); // Just return OK if The vOICe is not running: nothing to [un]mute if (hvOICeWnd == NULL) return TRUE; // Start or stop sound generation by The vOICe if (Sound) message = "STARTSOUND"; else message = "STOPSOUND"; // Fill the struct copydata.dwData = (DWORD) hWnd; copydata.cbData = lstrlen(message) + 1; copydata.lpData = message; // The vOICe still needs to know what you are about to ask vOICeDone = FALSE; // Now tell The vOICe what you want... SendMessage(hvOICeWnd, WM_COPYDATA, (WPARAM) hWnd, (LPARAM) &copydata); // Wait until The vOICe confirms that is has indeed processed your request. // (E.g., a STOPSOUND request takes some time to release the audio device.) while (!vOICeDone && time_out < TIME_OUT) { Sleep(100); time_out++; } // The vOICe *is* running, but nevertheless it does not respond??? if (time_out == TIME_OUT) return FALSE; // Error situation return TRUE; } /* The vOICe will return a message after it processed your request: * put a call to this function in your WM_COPYDATA event handler. */ BOOL vOICeConfirm(HWND hWnd, HWND hWndSrc, PCOPYDATASTRUCT pcds) { if (strcmp((LPSTR)pcds->lpData, "The vOICe: Done!") == 0) vOICeDone = TRUE; return TRUE; } BOOL vOICeCommand(HWND hWnd, char *cmd) { int time_out = 0, TIME_OUT = 100; HWND hvOICeWnd = NULL; static COPYDATASTRUCT copydata; // The vOICe target window is named "The vOICe" hvOICeWnd = FindWindow("The vOICe", NULL); // Just return OK if The vOICe is not running if (hvOICeWnd == NULL) return TRUE; copydata.dwData = (DWORD) hWnd; copydata.cbData = lstrlen(cmd) + 1; copydata.lpData = cmd; vOICeDone = FALSE; SendMessage(hvOICeWnd, WM_COPYDATA, (WPARAM) hWnd, (LPARAM) &copydata); while (!vOICeDone && time_out < TIME_OUT) { Sleep(100); time_out++; } if (time_out == TIME_OUT) return FALSE; // Error situation return TRUE; } /******************** END **********************/ /****************************************************************** * The remainder of this example defines a dummy application that * * demonstrates remote control over The vOICe when a mouse button * * or key is pressed * ******************************************************************/ // Note: you don't need to include this part in your own application: // you will just need to put calls to vOICeSound() or vOICeCommand() // in your application, as well as a call to vOICeConfirm() upon a // WM_COPYDATA event. #define AppName "vOICeControl" // Process any messages LRESULT CALLBACK myWindowProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam) { switch(uMsg) { case WM_LBUTTONDOWN: // Left mouse click: turn The vOICe sound OFF vOICeSound(hWnd, FALSE); // Should always return TRUE: not checked here. break; case WM_RBUTTONDOWN: // Right mouse click: turn The vOICe sound ON vOICeSound(hWnd, TRUE); // Should always return TRUE: not checked here. break; case WM_CHAR: switch (wParam) { case 'z': vOICeCommand(hWnd, "Zoom*2" ); break; // Press 'z' for zoom by 2 case 'Z': vOICeCommand(hWnd, "Zoom*1" ); break; // Press 'Z' to turn off zoom case 'i': vOICeCommand(hWnd, "Inverse"); break; // Press 'i' for inverse video case 'I': vOICeCommand(hWnd, "Positive"); break; // Press 'I' for normal video case 'r': vOICeCommand(hWnd, "Reset" ); break; // Press 'r' // ... } break; case WM_KEYDOWN: switch (wParam) { // Home key and left, right, up and down arrow keys case VK_HOME : vOICeCommand(hWnd, "Home" ); break; case VK_LEFT : vOICeCommand(hWnd, "Left" ); break; case VK_RIGHT: vOICeCommand(hWnd, "Right"); break; case VK_UP : vOICeCommand(hWnd, "Up" ); break; case VK_DOWN : vOICeCommand(hWnd, "Down" ); break; // ... } break; case WM_COPYDATA: vOICeConfirm(hWnd, (HWND) wParam, (PCOPYDATASTRUCT) lParam); break; // case ... // ... case WM_CLOSE: case WM_DESTROY: PostQuitMessage(0); break; default: return DefWindowProc(hWnd, uMsg, wParam, lParam); } return 0L; } int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow) { HWND hWnd; MSG msg; WNDCLASS wc = { CS_DBLCLKS, myWindowProc, 0, 0, hInstance, LoadIcon(NULL, IDI_APPLICATION), LoadCursor(NULL, IDC_ARROW), (HBRUSH) (COLOR_WINDOW + 1), NULL, AppName }; RegisterClass(&wc); hWnd = CreateWindowEx(0L, AppName, AppName ": Click left or right!", WS_OVERLAPPEDWINDOW, 100, 100, 320, 200, HWND_DESKTOP, 0, hInstance, NULL); ShowWindow(hWnd, nCmdShow); UpdateWindow(hWnd); while (GetMessage(&msg, 0, 0, 0)) { TranslateMessage(&msg); DispatchMessage(&msg); } return msg.wParam; }

Note: in case your application is 16-bit rather than 32-bit, you will have to first define the WM_COPYDATA and COPYDATASTRUCT yourself as

The executable of the above example is for demonstration purposes available as voicecontrol.exe. Just run it at the same time as The vOICe for Windows and click the left and right mouse button within the (blank) window to mute and unmute The vOICe.

In the following table an overview is given of the commands currently supported via calls to the vOICeCommand() function:

 
For some combinations with other GUI applications, you may wish to turn off The vOICe's automatic CPU priority switching by going to the menu Edit | Soundscape Preferences | Advanced button, and unchecking the "HIPRIO" checkbox to obtain a more smooth overall GUI response.

Pranav Lal created "vOICeCmd", a command-line interface for use of the above functionality, available for free via the GitHub project page  https://github.com/pranavlal/vcmd/ (GPL license, with  Manual). See also his  Technology Esoterica blog post about this utility.