This system and method are intended for assisting instrument placement for procedures in which there is significant knowledge of the context and extents of the target and access for a procedure in computer memory before the execution. In general, this concerns procedures in which anatomical geometry has been rendered before the operation and is registered to the patient. The lesion is localized in the data, and there is some optimal trajectory and/or manipulation of the instrument with respect to the lesion (for example, tumor stereotaxis) or some placement of some anatomical object such as a bone fragment (for example, craniofacial reconstruction). We further specify that this procedure plan is of such a complexity or precision that it cannot be fully remembered or known by the surgeon during the operation without intermittent recourse to the prerecorded trajectory (i.e. some mnemonic is needed). The positional and orientational states of objects are dependent upon prior placements and therefore cannot be planned except in terms of some set of constraints, such as proportions (i.e. a number of consecutive measurements are needed). In the former case, we define as a procedure requiring an insertion trajectory, and the latter as a measurement task.

In addition to the standard methods for acquiring patient-specific data and generating detailed 3D models our system comprises:

• A memory for storing one or more surgical target paths (usually a straight line).
• Sensors for sensing instrument placement by the surgeon, and the patient’s position and orientation.
• Audio feedback for advising the surgeon in real-time based upon a comparison of the prescribed surgical path and the sensed surgical execution progress.

A surgical target path is expressed in terms of two or more spatial coordinates, the values of which are indicative of the desired position and velocity of the instrument used in the surgical procedure along the surgical target path. This further comprises a way for translating values of the two or more spatial coordinates obtained from the measurement into corresponding values of two or more coordinates of an audio space. In particular, each of the two or more coordinates of the audio space may correspond to an audio theme recognizable by the surgeon. This, for example, this can be a consonant harmonic structure, such as a major triad, each tone of which corresponds to values along a specific spatial coordinate. Spatial coordinates are broadly considered as positional (x-y-z) or angular coordinate, velocity, acceleration, torques, etc.

In addition, this system further comprises means for automatically selecting different coordinates of the audio space based upon the surgical target path and the sensed surgical execution, so that a surgeon can know how close the hand held instrument is moving to a desired path simply by listening to the audio feedback. Notably, unlike visual systems that require full attention from the human for certain periods of time, an operating surgeon can correct his movements virtually without a distraction. Naturally, this feedback can further be supplemented by a corresponding visual feedback means for advising the surgeon.

A number of methodologies have emerged for employing audio feedback in computer and embedded applications. We define these as audio gages, mimetic audio, symbolic audio, audio for positional guidance, and musical instrument interfaces.

The most basic methodology is to use a simple monophonic signal, modulated in some way, as an alternative to readouts and gages for industrial equipment. This has proven useful in critical situations such as an airplane cockpit when the operator is incapable of comprehending in a single glance the states of all the readouts, perhaps some gages need not even be considered except when they reach some critical value. Audio feedback provides a very effective way to overcome this visual overload resulting from cluttered or complex display systems. Some approaches have included warning systems for civil aircraft and audio feedback systems for medical equipment^22,23.

A mimetic form of audio feedback quite frequently figures in computer games and environment simulations. For example, when the user’s avatar or cursor interacts with an object within the simulated environment, a sampled audio clip is triggered. These sounds are intended to represent or mimic sounds that would result from real physical interactions, such as a car impacting a brick wall. In this methodology sound is used more as a redundancy measure to give credence to the graphical simulation, than as a parallel informational channel.

A common methodology applies audio feedback in the symbolic manner of icons in graphical user interfaces. With this approach programmatically generated music and sampled audio clips function as auditory icons, so-called "earcons", which may be manipulated using a mouse or other 2D controller within an audio desktop space⁹. A more primitive example of this methodology is the use of a sound scheme in a graphical desktop environment such as Microsoft Windows 95 where audio clips or synthesized sounds are triggered as the result of actions such as clicking upon an icon or a system error^10,11,13.

The musical instrument interface methodology is based, in part, on the understanding that the interface design methodology employed by musical instruments may serve as a model for systems which aim to provide positional guidance using of audio feedback. Musicians who play variable pitch instruments such as the violin, the trombone, or the Theremin control the acoustical aspects of their performance by varying their hand position relative to the instrument body. Sensitivity to position measurable to fractions of a millimeter is necessary for certain notes to sound correctly upon an instrument such as the violin³¹.

The musical instrument interface presents one or more axes of control. The violin, for example, presents at least three axes of manipulation. Firstly, the musician’s fingering upon the string controls gross pitch. Secondly, another axis perpendicular to the string, parallel to the horizontal plane of the instrument body controls a smaller range of pitch called bend. Lastly, the other axis perpendicular to the string and parallel to the plane of the instrument varies amplitude and the spectral composition of the sound (via bowing). The latter two axes are small in comparison to the string axis, but are important in precisely shaping the resulting sound. There are further other axes of control but these three serve to illustrate the feasibility of this design methodology. It is worth noting that a trained violinist can accurately, precisely and repeatably place his/her hand in the same position in order to create a particular sound. This facility is also well illustrated by the classic electronic musical instrument known as the Theremin, invented in 1928 by Leon Theremin. The Theremin is unique in the fact that the musician controls pitch and amplitude by moving his/her hands in the air, relative to two antennae¹².

This surgical navigation methodology, tactical audio, is based on a conceptual inversion of this paradigm: instead of using position to control sound; sound is used to provide feedback as to position in manual placement tasks.

Design methodologies using tactical audio – which we define as an audio feedback expedient for a achieving a goal, such as for providing spatial guidance for placement tasks - are of a more speculative nature despite the observation that they are similar in concept to the gage and musical instrument methodologies. These systems employ simple algorithms that translate values of two or more spatial coordinates into corresponding values of two or more coordinates of an audio space. Some experimental applications have included a 3D auditory ‘visualization’ system used for simulating spacecraft maintenance tasks in zero gravity. This system uses frequency beat interference between two sinusoids as a means for providing feedback for properly positioning a circuit board in a training simulation³². Other related applications have included a 3D visualization system using spatialized audio²¹.

There are a number of obstacles to developing a feasible methodology for tactical audio user interfaces. For example, numerous psychoacoustic phenomena - especially the nonlinearity of human hearing³⁴, and the problem of determining a frame of reference for the user. The problem of determining reference frames can be stated as the decision as to the appropriate mapping of virtual space to the user's coordinate system, and to the real world.

The use of reference frame mapping in interface design is actually quite common. Consider how computer users have no trouble overcoming the positional translations which occur between a mouse which must be manipulated upon the horizontal plane of a table top, and the corresponding cursor which is projected upon a plane perpendicular to the table top. The interface has a surprising intuitiveness despite the fact that the axes may be reversed and offset, and the magnitude of movement scaled. The interface is simple and consistent; the expected outcome of shifting the mouse does not diverge too significantly from the actual movement of the cursor. See figure 2.

Mapping geometry into sound places a larger challenge upon the cognitive faculties of the user than the simple transformations of the mouse-to-cursor interface. This involves what could be described as transmodal mappings instead of homomodal mappings - translating one or more axes within the same modal space. In order for the system to present a coherent interface to the user, the interface designer must determine which dimensions of the initial modality are to be mapped to the resulting modality. For systems requiring such a high degree of usability as systems intended for use in the operating room, the chosen mapping scheme must be intuitive to the surgeon. Perceptual issues are important if the transformation is desired to be as lossless as possible due to the differing, even incomparable perceptual resolutions, ranges, nonlinearities, etc.^5,29. We believe it is possible to overcome these obstacles. Some of the mapping schemes we have prototyped are described below.

We assembled an experimental audio feedback system using commercially available hardware and a software system we wrote for 3D modeling, surgical procedure planning, real-time instrument tracking and audio generation. Prototype software algorithms for generating audio feedback as a function of instrument position relative to a preplanned trajectory were designed and implemented.

The hardware system was based upon a Pentium PC attached to a specialized high-performance audio engine, the Lake DSP Huron Digital Audio Convolution Workstation. The Huron is a rack-mounting industrial system fitted with a combination ISA bus and proprietary Huron bus backplane allowing standard ISA bus boards as well as Huron DSP and I/O boards to be installed¹⁵. The PC was inserted into an ISA slot. A Polhemus Insidetrak electromagnetic position tracking system was inserted into an ISA slot on the PC board. The PC ran a control system which acquired data from the Insidetrak, compared this data with the surgical plan, and then according to whatever specific audio feedback algorithm was implemented, sent commands to the DSP programs running upon the Huron. Audio output from the Huron was amplified and fed through a pair of professional grade Beyer Dynamic headphones. This assemblage is depicted in figure 3.

For clarity, we refer to the subsystems in this and related figures using the following labeling scheme: (A) sensor subsystem, (B) computational subsystem, (C) sound synthesis and filtration subsystem, (D) amplification and public address subsystem.

Deserving special attention here is the means by which audio signals and filtration of those signals are accomplished. The audio synthesis and filtration programs embodying the core of our system are implemented using Lake’s Huron DSP board. The DSP board is a high performance, multi-processor audio DSP engine. The DSP board interfaces with the Huron bus. This bus is a proprietary 256-channel 24-bit Time Division Multiplexed (TDM) audio bus that provides low-latency communication to I/O boards and other DSP boards. The DSP board is based upon a configuration of four 40 MHz. Motorola D5P56002 processors. Each processor may read from or write to any of the TDM channels forming the Huron Bus. The onboard (internal) bus supplies additional 512 TDM channels. In all there are 768 TDM channels available for interprocessor communications. The external bus permits installation of up to 20 boards, providing significant audio DSP capacity. On-board DRAM is arranged permitting accesses anywhere within a 102-word page at zero wait state (0ws.). This allows algorithms to run at maximum speed¹⁶. Because of the great speed and versatility of the DSP56002 processor, which features 24-bit word memory²⁰, providing a total of 1Mwords (3Mb.) DRAM and 128kwords (384kb.) SRAM, we found a configuration of 2 DSP boards (8 processors) sufficient for running many of the basic audio feedback algorithms described below.

An I/O carrier board permitted interfacing analog and digital audio signals from the outside world with the system, via an industrial 1U unit with XLR connectors.

In the interest of brevity, we will discuss only the real-time component of the software system. Software implemented upon the hardware system previously described consisted of two main segments: PC executables and DSP executables. An overall memory map for this system, depicting the segmentation and residence of these executables on a particular hardware device, is depicted in figure 6.

Figure 6: Software system memory map.

In overview, the PC executables acquired data from the tracking device(s), compared it with the preoperative plan, and used the resulting error function via some specific audio feedback algorithm to control the DSP program. We depict this overall function in figure 7.

On the DSP end, the basis of our synthesis approach is a wave-table lookup oscillator program running on the Motorola DSP56002 DSP chipset. The wave-table is scanned by means of an index that is incremented at each sample period. This industry-standard approach permits the generation of arbitrary frequency sine waves using interpolation to reduce distortion²⁸. Sound synthesis is implemented using banks of these sine-wave oscillators. These oscillator banks served as the building blocks for creating synthesizers using additive synthesis as well as other synthesis techniques. These synthesizers are controlled by the PC host, using the specific audio feedback algorithm to control the various parameters.

In addition to sound synthesis, we employ a subsystem for filtering or spatializing the signals output from the oscillator banks. This subsystem is based upon Lake’s proprietary sound field simulation and auralization software. To create a consistent and realistic simulation this system employs either a two or three stage filtering process. The first stage of the filtering process involves the processing of head-related localization effects. As the head turns, this system smoothly and rapidly switches in the FIR filter coefficients that implement the head-related transfer functions (HRTFs) for the corresponding azimuth and elevation. These filters are typically short (of the order of 256-4096 taps). The second stage adds to the sound additional localization information from the early reflections in the space. These filters are around 3000-4000 taps in length. The third stage of the process incorporates environment reverberation and late reflections from the space. These filters are of the order of 60-256,000 taps in length. The resulting signals are decoded to headphones using a binaural decoder. Simulations using this subsystem provide a high degree of realism through the use of proven acoustic modeling methods, in conjunction with Lake DSP^’s long, low-latency convolution technology, giving total control of the locations of multiple sound sources and listeners within a virtual acoustic space¹⁷.

As we discussed earlier, the intuitiveness, in short the usability of the user interface embodied by the audio feedback algorithm will make or break the system. In the course of the engineering process we implemented a variety of audio feedback algorithms. We will use these prototypes at a later date to perform formal usability tests in order to determine which factors and which specific approach holds the most promise for audio user interface design for surgical navigation systems. We discuss some of these design approaches.

This approach produces an intuitive interface, but there are certain drawbacks. For example, it is obvious that there are solutions for which it would appear that the cursor triad were approximately in harmony with the reference set, yet in terms of the actual coordinates, the positioning would be in error, for example, if:

It becomes obvious that there are many other similarly deceptive combinations. Some scheme for excluding these ambiguities must be devised if this approach is to achieve an acceptable level of usability.

The use of 3D spatialization is similar to the beat interference method in that one or more coordinates of some function of instrument position are mapped to one or more coordinates of a generalized audio space. In this case instrument position mapped to 3D audio space about the user’s head. This approach cannot stand alone as an interface but can be employed as a redundancy measure and extension to any other algorithms. The particular strengths of the Huron hardware with respect to audio digital signal processing¹⁶, in accordance with nature of 3D localization in human hearing, permits such an approach using our system. It is worth mentioning here how this is possible.

Although human beings are usually considered sight-dependent, auditory cues play an important role in our ability to comprehend the surrounding environment. From a physiological standpoint, sound waves are conducted to the inner ear via a mechanical linkage between membranes and small bones. In the inner ear, the spectral components of a particular sound are extracted in a manner similar to Fourier decomposition²⁴. This decomposition is facilitated by the involute form of the bony labyrinth. Amplitude for a particular partial is sensed on the part of fluid compression/decompression-induced friction or stress upon bunches of stereocilia on the organ of Corti by the tectorial membrane. This friction triggers neurons traveling through the auditory nerve to the cerebral cortex. These impulses are interpreted by the brain as sounds composed of partials of a particular pitch and intensity⁵.

The ability to localize sound sources in 3D space around the listener’s head is a function of intensity and phase differences between the signals from the left and right ears. The impact of the shape of each individual's head and external ear, or pinnae, on the reflected sound waves received by the inner ear is crucial for sound localization. The pinna has a significant influence on shaping the spectral envelope of incident sound. This spectral shaping is dependent upon the 3D origin of the sound source with respect to the listener’s head and pinnae. The auditory cortex determines 3D spatial position from the unique signature the pinnae place upon the acoustic pressure wave. The interference characteristics of head and pinnae shape on the transference of sound to the ear canals is a function that can be modeled and employed by an audio convolution algorithm to simulate the placement of sounds in 3D space. In practice, speaker arrays and sensitive miniature microphones inserted into the ear canal make it possible to derive a set of HRTFs. Four other parameters, in addition to the parameters of interaural time delay, head shadow, pinna response, and shoulder echoes, comprise the Head-Related Transfer Function (HRTF). These include head motion, vision, intensity, and response caused by the local acoustical environment¹.

This approach concerns the requirements of taking measurements intraoperatively, such as for craniofacial reconstruction. On-the-fly positioning or measurement tasks are simplified. Instead of using hardware calipers, rulers and other measurement devices, all manner of measurements may be taken and recorded using an audio feedback system, a stylus or speech recognition system and a footswitch. With caliper measurements, taken between two points in 3D space, the surgeon samples an initial point upon the anatomy by placing the stylus in the desired location, for example at nasion in figure 10, and then activating a footswitch or speaking some command such as "origin".

This 3D location forms the origin of a spherical gradient of sound events propagated in 3D space. As the surgeon moves the stylus through this sound field, sound events are triggered at the passing of each concentric measurement increment. For example, each millimeter increment triggers an audible click, and each centimeter increment triggers a speech synthesizer to speak the radius in centimeters from the origin. See figure 11.

When the desired radius has been located, its position may be recorded by use of some input device, either speech or switch, or the sound field may be turned off. For complex measurement tasks, such as required for minimizing multiple skull fractures or placing and wiring multiple bone fragments, measurements may be automatically accumulated, and labeled. In this way the surgeon might simply say the word "nasion" and the feedback system would automatically propagate a sound field around that location.

Of particular concern in the use of audio user interfaces for surgery is the problem of generating useful audio representations of 3D volume objects. What is needed is something akin to rendering or raytracing in graphics. The technique of granular synthesis and wave terrain synthesis have useful applications in this area because they facilitate the generation variable density clouds of sound particles and synthesis of audio waveforms from 3D surfaces. These approaches could be applied to generating audio renderings of 3D volumes.

In a granular system, sound is viewed in terms of both wavelike properties and particulate properties. Complex waveforms are synthesized by aggregating quanta of sound. These grains last less than 100 milliseconds, which is within the minimum perceivable limit for discrimination of duration, amplitude, and frequency²⁶. In this sense granular synthesis is similar to an inversion of standard wavelet theory - using wavelets, or so called grains to synthesize complex waveforms instead of decomposing them¹⁴. A 3D volume propagated with grains is analogous to a 3D wavelet decomposition space^6,7.

Each grain is shaped by an amplitude envelope. This envelope can vary in configuration from Gaussian to nonlinear functions. It is possible to implement granular synthesis using a sine wave oscillator controlled by an envelope generator, but there are some performance issues because much of the control - specifying the grain envelope - is passed to the host CPU.

Granularity proves to be a useful model for synthesizing complex waveforms necessary for representing 3D anatomical volumes. A particular anatomical object 3D volume may be propagated or "colored" with a particular type and density of sound grain (see figure 13). Penetration of this object would result in the generation of a unique spectrum. This spectrum would change as the instrument passes through density gradients. Because each grain is mapped to a particular region in 3D space, changes in velocity of the stylus as it passes through this field would also result in modification of the generated waveform.

Wave-terrain synthesis is another approach that may be applied to audio rendering of 3D volumes, or more specifically, 3D surfaces. This technique proceeds from the principle of wave-table lookup synthesis. It is possible to extend the basic principle of wave-table scanning as implemented in a sine wave oscillator to the scanning of 3D anatomical object surfaces for the purpose of generating an audio representation of such objects.

In implementing wave terrain synthesis in an audio feedback system for surgery, the rigid body angle of the instrument is measured with respect to the surface of the anatomical object. This angle defines a normal to a surface region to be sampled from the anatomical object as a wave terrain. This surface region is scanned using a periodic function – elliptical or otherwise. This scanning generates a stream of waveform values that are directed to the DSP for sonification. See figure 14.

The signal generated depends on both the wave terrain and the scanning trajectory. The trajectory may take any number of forms, such as a straight linear trajectory, or an elliptical function. When the trajectory is a periodic function, the resulting waveform exhibits a static spectrum. This spectrum will be relatively homogeneous as the instrument passes across homogeneous surfaces, but will vary significantly upon passing across a surface in which the surface function changes abruptly. This change in spectrum can assist a user attempting to comprehend or "visualize" the surface terrain of an object.

Current approaches for haptic and acoustic simulation focus upon either haptic or audio simulation as entirely separate modeling and simulation processes. The resulting simulation systems proceed from haptic or acoustic-biased approaches to the engineering process, instead of proceeding from knowledge of the geometry and physical nature of objects. These simulation systems are unconvincing in their realism.

In an extension to the sonification of 3D volumes, specifically the wave terrain approach, both haptic and acoustical feedback would be derived from a single general geometric and physical model. Deriving haptic and acoustic feedback from a general physical model will result in more realistic simulations, because these two sensory modalities, while perceived through different physiological apparatus, proceed, in the natural world, from the physical properties of materials. Such a direct correspondence between physical interactions with objects, and their resulting haptic and acoustic effects, will result in a greater degree of immersion for the user in simulated environments.

Material descriptions, which formerly only described geometric and optical, or possibly haptic parameters will be revised to include parameters such as 3D physical surface texture (as opposed to 2D texture or "bump" maps), material physical properties. Graphical, haptic and acoustic subsystems will read these parameters and render them to the appropriate display hardware (e.g. heads up display, phantom, and headphones).

For surgical simulation incorporating haptic feedback this approach provides a greatly increased sense of realism. For example, when a scalpel scrapes against bone, the resulting haptic forces and acoustic waveforms will be simulated correctly, directly in proportion to the user interaction forces, as opposed to using discrete sampled sound clips or simple haptic models. Such an approach also opens up the exciting possibility for enhancing, amplifying, or otherwise processing subtle haptic and acoustic bandwidths out of the range of perception in the natural world. For example, in simulating the sound and reaction forces of a biopsy needle passing through subtly different tissue types, the resulting feedback would be scaled to fall within perceptible ranges. Rotating the needle, extending and retracting the biopsy core, and applying differential pressure would all result in noticeable changes in the simulated reaction forces on the biopsy needle, and sounds generated by the procedure. This feedback could be used as an aid to help surgeons better understand the interplay of forces between instrument and tissue, resulting in more precise cutting and placement tasks.

During operation a haptic subsystem would analyze general object physical properties and user interaction forces, and based upon these parameters return haptic force feedback data to a haptic display device (e.g. a Sensable Devices Phantom). This would include various functions for enhancing or amplifying the simulated reaction forces.

The acoustic subsystem analyzes object properties and user input forces, and returns waveform data to an audio-rendering device (e.g. a Lake DSP Huron). The 3D physical texture map (identical to that used for the haptic simulation) is sampled over time, and is implemented as a wave terrain. As the cursor scans or otherwise interacts with this wave terrain over time, a unique waveform is generated which bears the signature of the interaction forces. See figure 15.