SBIR Phase II Proposal for
research to be conducted for:
DARPA
- Defense Advanced Research Projects Agency
3701 North
Fairfax Drive, Arlington, Virginia 22203-1714
c/o Richard M. Satava, M.D. FACS
D.S.O./Biomedical
Technology
Developing an Audio-Enabled
Intraoperative Computer-
assisted Surgical System
Project Period: 18 months, from 4/97 to 12/98.
Amount Requested: $749,968.38
Principle Investigator:
Daniel B. Karron, Ph.D. e-mail: karron@casi.net
President and Chief
Technical Officer
Computer Aided Surgery, Inc. (C.A.S.I.)
Other Investigators:
Kristen Wegner, B.A. e-mail: wegner@casi.net
Research
Associate
Computer Aided Surgery, Inc. (C.A.S.I.)
Adam J. Flisser, M.D.
Medical Resident
New York University Medical Center
Werner K. Doyle, M.D.
Department of Neurosurgery
Hospital for Joint Diseases/Orthopedic Institute
Nolan Karp, M.D.
Institute of
Reconstructive Plastic Surgery
New York University Medical Center
Gene Grossi, M.D.
Department of
Cardiothoracic Surgery
New York University Medical Center
Officer To Whom Award Documents Should Be
Mailed:
Daniel B. Karron, Ph.D. e-mail:
karron@casi.net
President and Chief
Technical Officer
Computer Aided Surgery, Inc. (C.A.S.I.)
300 East 33rd
Street, Suite 4N
New York, NY 10016
Table of Contents
1. Abstract
2. Introduction
3. Background
3.1. Medical Imaging
3.2. Preoperative Planning
3.3. Telesurgery
3.4. Virtual Reality Technology and Surgery
3.5. Medical Education
3.6. Problems in imaging
3.7. Problems With Endoscopy
3.8. Problems in system design
3.9. Limitations of VR technology
3.10. Simulator Sickness
3.11. System Design Requirements
3.12. Physiological and Cognitive Basis of Audition
3.13. Spatialized Audio in VR
3.14. Earcons, Audio/Musical User Interfaces
3.15. Previous Use of 3D Aural Feedback in VR Applications4. Virtual Audio and Medicine
5. Avantages of Incorporating Audio in Medical VR
6. Discussion of Technical Issues
7. Scope of the Proposal
8. Technical Approach
8.3.1. Reference & Pursuit: Feedback for One Dimension
8.3.2. Musical Structural Functions
8.3.3. Multitimbral System: Feedback for Three Dimensions
8.3.4. Spatialization: Feedback for Three Dimensions
8.3.5. Volume Sonification & the Soundscape Paradigm8.4. Medical Implementation of this Technology
9. Work Plan/Statement of Work
9.1. Phase 1: Low Level Construction
9.2. Phase 2: Systems Integration
9.3. Phase 3: Aural User Interface & GUI
9.4. Phase 4: Mock Clinical Trials & Reengineering10. Deliverables
11. Commercialization Strategy
11.1. Product
11.2. Customers & Estimated Market
11.3. Capitalization Requirements
11.4. Company Marketing Expertise
11.5. Competitors12. Proposal Costs
13. Key Personnel: Curriculum Vitae
13.3. Werner K. Doyle, M.D. (CV In Brief)
13.3.1. Selected Publications
13.4. Nolan Karp, M.D.
13.4.1. Education
13.4.2. Residency Training
13.4.3. Appointments
13.4.4. Board Certification
13.4.5. Awards
13.4.6. Grants
13.4.7. Publications
13.4.8. Abstract Presentations13.5. Adam J. Flisser
13.5.1. Education
13.5.2. Research and Work Experience
13.5.3. Publications14. Facilities & Equipment
15. References
Current visual display technology for interactive model-guided surgery has not yet developed an easy and intuitive way to help guide a surgeon’s hand and instrument along a prescribed 3D trajectory with demonstrated usability for extended periods. This goal is a central requirement of successful virtual reality applications that aim to assist surgeons in the operating room. In contrast, the potential to translate positional data into audio feedback is a technological innovation that is comparatively unexamined in the wide range of available methodologies for virtual reality medical devices. Musicians demonstrate extremely fine positional control of their hands and instruments on the basis of aural feedback. Surgeons can potentially be trained to do the same, except with the reverse intent: instead of using position to control sound, as in the case of a musical instrument, we believe it is possible to use sound in order to provide feedback as to the precise manual positioning of a surgical instrument. We will build a training simulator to test these suppositions.
Advances in surgical techniques, instrumentation, computer processing/imaging have opened more possibilities, and made more information available to surgeons than ever before. Minimally invasive techniques have allowed surgeons to treat internal regions without the traumatic overexposure and dissection formerly required. Magnetic resonance imaging (MRI) and computer-aided tomography (CT) have allowed surgeons to see with greater clarity vital anatomy than palpation and x-rays ever permitted. Despite these advantages, which are profound even when compared with the technological milieu of less than fifty years ago, the manual dexterity and ability of the surgeon to assimilate information has remained essentially unchanged. Medical imaging supplies a plethora of data from which is difficult and time consuming to filter out relevant structures. Endoscopes interpose a complicated non-intuitive interface between the surgeon and patient. Intraoperative navigation in general, and adhering to a pre-planned surgical trajectory, in particular, can be problematic.
The emerging field of virtual reality purports to solve the new problems engendered by the misplacement of vital medical information by creating integrated and intuitive interfaces which will allow surgeons to focus upon performing operations on their patients rather than on their instrumentation. The tentative results of this unprecedented exercise in interface design have been titillating if less than actually usable in the real surgical world. The overwhelming demands made by virtual reality systems on the surgeon’s visual faculties have contributed, perhaps, more problems than they were intended to solve; the surgeon is now obliged to contend with irritating head-mounted display equipment, and lagging, confusing imagery overlaid over what once was a clear and unencumbered - albeit less informative - view of the patient operative field. The question of whether these systems, at the present technological stage of development, can actually make any improvement - both in terms of investment return, and improved quality of care - over established medical technology is hotly debated.
Despite these drawbacks intraoperative computer-assisted surgical systems still hold great promise. Systems that can overcome some of the fundamental processing and interface design problems will enable surgeons to treat their patients more rapidly and with greater precision than before, taking full advantage of medical imagery and vital signs as available. For civilian and combat emergency medical procedures, where a few golden minutes can mean the difference between life and death, when surgeons are forced to make split-second decisions and must utilize available data to its fullest, such systems will prove invaluable. In navigation-critical procedures, such as stereotactic neurosurgery, such systems will contribute greatly by allowing surgeons to plan and adhere to nonlinear trajectories, thereby sparing, or reducing iatrogenic damage to, eloquent regions of the brain.
In order to overcome some of the situational awareness interface design problems previously encountered in computer-integrated surgical systems, we propose developing interface techniques using sound, in addition to graphics, in order to provide rich and informative intraoperative feedback to the surgeon. We believe that, by taking some of the bandwidth out of the visual display, the wealth of situational information available to surgeons in this information age may be intelligently inserted into the surgeon’s awareness when and where it is most needed, where it will become an angel of mercy for the patient instead of an encumbrance and headache for the surgeon.
Since the advent of the endoscopic age of surgical intervention in the mid-1980s, vast and rapid changes in the field of surgery have made familiarity with minimally-invasive techniques the sine qua non of the modern surgeon. In fields ranging from combat casualty care to neurosurgery, surgeons are driven to accept the new endoscopic methodologies from several sources. Patients demand the newer procedures, which are specifically designed to minimize trauma and recovery time, and insurance companies and employers push the medical establishment toward endoscopic operations for financial reasons related to decreased hospital stays and lost productivity.
What is obvious even to the casual observer, however, is that the new endoscopic techniques strain the talents of the surgeon in several areas. The comparative difficulty of operating through a two-centimeter incision, using rigid instruments and a video screen to replace the surgeon’s tactile sense, experienced hands, and vision, clearly presents new challenges for the medical professional. Fortunately, the surgeon is not alone on the surgical battlefield of the 1990s. Ongoing research in computer-assisted surgery continues to produce technological developments that are invaluable to the modern surgeon. As computing power continues to increase, innovative techniques designed to aid the surgeon in the operating room will revolutionize the standards for surgical care. It is a time of great change in the medical world, in which technological innovation has a considerably increased significance: "never before have surgeons been so dependent on technology, and never before has there been such a need for cooperative and open-minded surgeons…[who] must be able to exploit technologies without losing control over what they do."
Computers themselves have long been in use in the medical world, and their utility in the operating room and pre-operative period predates the advent of laparoscopy by decades. Imaging systems used in radiology and radiotherapy and computer-generated coordinate systems used for stereotactic neurosurgery and arthroscopic surgery are just two of the most obvious examples of how computers have been successfully integrated into the practice of medicine: "the incorporation of computer-based medical imaging data into stereotactic techniques helped surgeons localize lesions by point-in-space methods…convenient to use in an operating room environment." More recently, computer models are being used in many fields to help surgeons plan operative approaches. The application of computer technology to medical education, both at the level of medical school and residency and beyond, is fast becoming an indispensable aid for the training of new doctors.
Among these revolutionary developments is a new type of computer assistance: the development of virtual reality technology in itself, and its application to the medical field. While machines that simulate or model aspects of the real world have been in use for most of the past century, from the rudimentary flight simulators of the 1930s to the total-immersion virtual environments being developed today, recently technology has become available to generate physical models that are increasingly realistic, haptic simulations that reproduce actual physical feedback, graphical displays that begin to approximate actual vision, at speeds that give these developments some practical utility in the operating room as well as significant value in the pre- and post-operative stages of surgical therapy. The goal of many researchers, both within the medical establishment and elsewhere, is to develop the extremely powerful capabilities of virtual reality technology in order to make it an essential and empowering part of the modern medical world.
One of the distinguishing characteristics of this effort to develop virtual reality applications for medical use is the fact that they are predominantly visually-driven, and concentrate on video imaging, three-dimensional modeling, and visual enhancement, often at the expense of other potential areas of development. While there is considerable potential in the development of head-mounted displays that will enhance the surgeon’s vision, instruments that will provide tactile feedback, and remotely-operated devices that can make the concept of telesurgery a clinical reality, there exist other powerful technological applications that remain untested in the field of virtual medicine and computer-assisted surgery: "the current portal into this information rich arena is the video monitor," but "in the future other display technologies such as head-mounted displays (HMD), video-glasses, holograms, palmtop computers, or other mobile ubiquitous devices may be the interface."
In fact, current visual display technology for interactive model-directed surgery has not yet developed an easy and intuitive way to help guide a surgeon’s instrument along a proscribed 3D trajectory with demonstrated usability for extended periods. This goal is a central requirement of successful virtual reality applications that aim to assist surgeons in the operating room. Common shortcomings in the visual applications include slow frame refresh rates (resulting from the inadequacy of current processing power to generate 3D images from massive amounts of volume data at a suitable speed); poor stereoscopic vision, focusing, and depth perception; and "simulator sickness" caused by a time disparity between expected and perceived visual images.
In contrast, the potential to translate positional data into audio feedback is a technological innovation that is comparatively unexamined in the wide range of available methodologies for virtual reality medical devices. Musicians demonstrate extremely fine positional control of their hands and instruments on the basis of aural feedback. Surgeons can potentially be trained to do the same, except with the reverse intent: instead of using position to control sound, as in the case of a musical instrument, we believe it is possible to use sound in order to provide feedback as to the position of a surgical instrument relative to a pre-planned trajectory, pre-placed tags or cues, or anatomical models.
This approach has several advantages over using real-time 3D graphics animation alone. By providing both new and redundant spatial information in parallel, using a different, yet relatively under-exploited sensory modality, it is theoretically possible to maintain a higher bandwidth of information transferal to the operator, thereby avoiding visual desensitization as a result of visual overload. Additionally, music/audio adhering to the MIDI protocol potentially has significantly lower bandwidth requirements than 3D real-time graphics. Thus, a successful implementation of sound feedback used in conjunction with a simplified visual display could reduce graphics processing delays (eliminating the problem of flicker-induced interference with picture) and the attendant problem of long duration simulator sickness. The problem is to discover an aesthetic and ergonomic balance; an aspect crucial to the efficacy of virtual reality system designs.
The incorporation of these new technologies, including in particular the use of sonification and aural feedback as an integral part of surgical planning, modeling, simulation, and guidance, will have an extraordinary impact on the state-of-the-art in computer-assisted surgery. Video-driven virtual reality is the darling technology of the VR developer, and alternatives, especially in the area of aural feedback for visual guidance, remain largely undeveloped. Only recently have researchers begun to pursue these avenues, although "there is a growing awareness that a human factors input is critical to help drive further technological development. Important issues will include assessment of the amount of necessary resolution, assessment of the necessary field of view (including appropriate binocular overlap), and assessment of the importance of, to date, relatively unexplored aspects of VR (such as 3D sound and tactile and force feedback) in the creation of convincing immersive environments." In a medical paradigm where each bit of information, when properly delivered to the doctor, can make a crucial difference in patient treatment and outcome, we cannot afford to neglect the possibilities for improving the transfer of data to the physician.
These new technologies bring their own shortcomings and challenges to the medical world, just as the advent of endoscopic surgery did. The challenge of revolutionizing the surgical world is made more difficult by the unyielding nature of the demands of surgery. The operating room is notoriously inflexible in certain ways. For example,
In most cases, the surgeon is working under enormous pressures of time, action, and reaction, because of medical reasons. The surgeon must concentrate primarily on the operating field. In addition, his or her hands must not leave the sterile area. Interactions with the non-sterile equipment must be performed through sterile barriers (e.g., flexible plastic tubes) using foot-operated switches, speech input, or dialogue (additional human-human interface!) with a person assisting in the nonsterile area. At the same time, the surgeon’s work requires extreme concentration, dexterity of hands, and eye-hand coordination, as well as 3D thinking and orientation. It represents mental and visually guided fine sensorimotor work on a high level. Any additional interactive load with complex technical processes must be avoided.
The immediate future requires the development of innovative answers to the technical questions raised by the new technologies. Finally, while the lure of virtual reality technologies as potential solutions to the challenges of surgery is immense, it must be remembered that "there is not a valid reason why they should by used by industry or commerce. They, like any other technological aid, must prove their worth. We must demonstrate that through their use, a task is easier, quicker, or less tiring; or the end product is cheaper, more efficient, more reliable, and so on." The particular limitations of laparoscopy, graphical displays, computer processors, feedback technology, and virtual reality itself must be addressed, if the next generation of medical science is truly to become the age of virtual surgery.
The use of computers for 3D-imaging "rapidly arose in the 1980s owing to several advances in computer science, in particular increased hardware computational speed and improved graphics software." This development enhanced the ability of the surgeon to integrate two-dimensional imaging information into the three-dimensional conception necessary for accurate surgical planning and execution. 3D computer reconstruction, formerly too processor-intensive and time-consuming, is now available for the use of reconstructive cranio-facial surgeons and neurosurgeons who require accurate three-dimensional representations of the pathologies they seek to correct operatively. As it can be a crucial part of pre-surgical planning, the ability of generated 3D images to portray "precise definition of a lesion’s various components, such as neoplastic tissue, edema, necrosis, and hemorrhage…along with the relationship of critical surrounding structures" or the precise position of bones or soft tissues, is an invaluable asset to the surgeon. Instead of having to integrate individual cross-sections generated by CT or MRI scans, or extrapolate from two-dimensional radiographs or ultrasound images in order to gain a three-dimensional perspective of the patient’s anatomy, the surgeon can now refer to highly-sophisticated volumetric renderings of the patient that represent the patient’s actual appearance in three dimensions.
The developments in the field of stereotactic neurosurgery are a specific example of how advances in computer technology have translated into direct advantages for the practicing surgeon, as "advances in computer capacity and speed and reduction of cost" have increased the availability of the technology, while at the same time making it possible for "stereotaxis [to] progress from point-in-space stereotactic procedures to volumetric procedures" based on the power of the new computers. A volume-based approach to surgery enables both the identification of tumor areas in three dimensions as well as the representation of instrument position in three dimensions relative to the operative field.
In addition, in the post-operative period, 3D imaging can be used for the assessment of surgical outcomes. While at present, "assessment of postoperative results in craniofacial surgery often is qualitative, consisting primarily of [the] comparison of preoperative [to] postoperative photographs," the use of computer-driven image comparisons can add a less subjective comparison: "quantitative analysis of the results of craniofacial surgery…[including] such uncontrollable factors as growth in children, resorption of bone grafts, and the tendency of moved segments of bone to return to their original positions." Thus, the integration of computers into the operating room and into surgical practice open new dimensions for the surgeon and the patient.
Another way in which computers can revolutionize the practice of surgery is in the area of presurgical modeling and planning. The development of high-fidelity computer models of patients’ biophysical data will allow the surgeon to plan a preoperative course based on the actual characteristics of the patient. In addition, sophisticated deformable models that react to surgical intervention the way actual tissues would allow prediction of post-operative results, and in the future may even allow pre-operative "rehearsal" of an operation. At present, the "surgeon planning a complex operation takes advantage of drawings and photographs, montages, radiologic data, and CAD tools." The addition of the powerful tool of computer-assisted preoperative planning will have a demonstrable effect on the risks and unknowns in the operating room, and in this area alone is an important supplement to the surgeon’s own skills and discretion.
In neurosurgery, "surgical approaches to subcortical lesions should be through nonessential brain tissue parallel to white matter projections and in the least invasive manner available." The use of computer-modeling and 3D imaging in this area is of clear benefit. In the future, to goal is to have computer models that will be capable of incorporating "all preoperative databases…fused into a uniform computer matrix that corresponds to the individual patient’s brain."
In summary, it is apparent that from the perspective of altering future trends in the practice of medicine, the computer can revolutionize research efforts, surgical practice in the office, and operative procedures. However, the significant advantages to be gained from integrating computer imaging and enhanced visual modeling are not the only possible directions currently under exploration in the medical world.
The use of computers in surgery is clearly not limited to their awesome capability to present detailed images of the body in three dimensions or their capacity to provide realistic and dynamic models of a patient’s actual anatomy. In this context, modern endoscopic techniques represent the most primitive form of a new aspect of surgery: telesurgery, defined as "the extension of human inspection and manipulation capability to remote or otherwise inaccessible locations." In the case of laparoscopy, the "otherwise inaccessible location" is the interior of the abdomen, reached with telescopic instruments through minimal incisions. At present, the state of robotic technology is such that we may be years from the advent of a truly functional ‘surgical robot,’ but computers can be of immense assistance in a telesurgical context, even at the more primitive level of endoscopic procedures. For example, in retinal surgery, "the surgeon’s hand and finger movements must be scaled down kinematically in both displacement and force." The use of computer-controlled or -modulated equipment to modify the natural movements of the surgical operator fulfills this function. Telesurgery, when eventually combined with the sophisticated image databases, will be added to the arsenal of modern technical assistance in the operating room. Thus, the ultimate goal of telesurgical development is for "in any endoscopic, laparoscopic, or other telescopic surgery or microsurgery where a video image is the primary display, the motor actions of the surgeon will be transformed into forces and motions on an endoscope or on surgical instruments directly." In this way, the potential success of endoscopic telesurgery is dependent on the evolution of imaging technology and techniques, a dependency that may have serious consequences in the advancement of virtual surgery as a discipline.
Virtual Reality Technology and Surgery
In computer-assisted surgery systems designed to take advantage of the potential of virtual reality, the user is immersed in a ‘virtual world’ that models the real world to a varying degree, realized in real-time when possible. A typical tool for this variety of virtual reality application is the Head-Mounted Display, a visual display system worn on the user’s head usually as goggles or a visor. The user either sees only what is being fed into the HMD from outside inputs, such as video cameras or animations generated by the computer, or the user may see the real world with computer-generated images superimposed on the natural viewing field, so-called "augmented reality." The HMD is thus a potentially very powerful tool for drawing a user into the virtual world and immersing the surgeon in the computer-assisted reality that is the goal of computer-assisted surgery technology. Currently, HMD systems are incorporated into a number of different types of VR projects, including, for example, an ultrasound viewing system developed at the University of North Carolina, which "displays multiple 2D medical ultrasound images overlaid on real-world images." This system, which "combines real-world images captured by head-mounted TV cameras with synthetic [ultrasound] images generated to correspond with the real-world images," uses the HMD as the modality to deliver the virtual environment. In theory, when the technologies underlying the use of HMD interfaces and 3D real-time imaging become more developed, it will be possible to immerse the surgeon in an augmented reality filled with useful data, much in the way that military pilots receive visual data superimposed on their flight visors: "doctors and surgeons are already used to using image data derived from 3D scanners; however, imaging the medical benefits if a surgeon could see the patient through a lightweight HMD which integrated 3D graphic data derived from other sources…thus providing the medical practitioner with pseudo X-ray vision." Again, however, the HMD/video emphasis of this technology will be limited by the state-of-the-art in 3D imaging technology.
There is great potential for computers to be used in the area of medical education, at various levels including medical school, residency training, and post-graduate education. Computer models that can be interacted with and manipulated at the convenience of the user, repeatedly, and in a consistent way can augment the teaching of anatomy, physiology, and virtually all other subjects in the medical school curriculum. For example, a product such as the "3D Brain Atlas" supplements the classical methods of teaching neuroanatomy and neurophysiology: it "is based on an MRI data set wherein every voxel has interactively been labeled with an attribute, describing its membership to an anatomic or functional constituent of the brain. It allows dissection and surgical training on the computer screen." In a different type of application of virtual reality application, "it is also possible to increase viewing magnification so that an object can be enlarged and examined more closely, and it is possible to overlay or ‘map’ additional information onto the real cadaver." Under development include virtual colonoscopy models that are intended to surpass the ability of physical models to train endoscope operators. Without analysis in great detail, it is obvious that almost all of the above-mentioned technological developments and methodologies will have equal application to medical education. When surgeons themselves will have the opportunity to simulate operations in three dimensions or in an immersive virtual environment, so will their junior colleagues and students.
The genesis of most virtual reality applications and computer modeling has as its source the immersion into a virtual world or the rendering of previously impossible-to-synthesize images; as such, the use of visual display technology has been a central part of this area of development. Indeed, as previously noted, the preponderance of current research efforts in the area of integrating virtual reality and surgery is in the area of imaging and video-driven simulation. Unfortunately for the researchers leading the efforts in this field, the use of graphical displays already faces grave challenges of a technological nature - challenges that will retard the development and application of the new technologies considerably. For example, in 3D imaging of most types of MRI, CT, or ultrasound data, "the resolution of segmented images is limited by the resolution of the original gray-scale data, and…improvements in this area will require improved scanner technologies and parameter optimization." Even more ominously, the limitations of current computer processors are being tested by the newest image-display applications: "current workstations are not yet able to deliver 3D images fast enough. For the future, it is certainly desirable to be able to interact with the workstation in real time, instead of just looking at static images or precalculated movies."
The utility of imaging and coordinate systems for surgery and surgical planning is also limited by the ability to translate coordinates on an image to the operating field itself: "unless there can be translation of the coordinate system of the imaging database to the real-world environment of the patient on the operating table, all the plans will be qualitative only. This would represent a tragic waste of the precise quantitative information that imaging databases provide."
Finally, even where the processor capacity does exist to produce the kind of images that are desirable and useful, such as in the last decade of computer-assisted stereotactic neurosurgery, "capacious high-speed computers of the type required for efficient image processing were too expensive for the average neurosurgical group, which did not anticipate using the technology often enough to justify the cost." Already, the new problems associated with the development of these new technologies threaten to overwhelm the potential benefits.
Endoscopic operations, while better in many ways for the patient, present serious difficulties at many levels of the procedure, from the simply mechanical to the more sophisticated in nature. At the most fundamental level of difficulty, the actual mechanics of the operation, it has been observed that "there is a high contribution of critical static working postures observable during endoscopic operations …[that] are caused by a disadvantageous arrangement of the equipment in the operating theater and the patient’s positioning…Particularly, the assisting surgeon is additionally stressed by static holding work…caused by the lack of suitable support devices and intelligent holder systems." The physical challenges of endoscopy are thus added to the image-related shortcomings. The design of the endoscopic systems themselves are not ideal for the purpose of use as surgical instruments; in effect, the utility of the technology has been undermined by a failure to consider the operator-stress of using the devices themselves. This is in addition to the understandable challenge of performing an operation remotely when formerly one had been trained to operate with full exposure and ability to manipulate organs and tissues with one’s own hands instead of the intervening telescopic instrument.
On another level, endoscopy itself introduces sensory challenges that complicate the nature of an operation. For example, visual feedback during an operation performed either with mono- or stereo-videoendoscopy, is not ideal: "achieving satisfactory depth perception is one of the most difficult problems of telesurgery or, indeed, teleoperation of any kind." Although this implies that stereoscopic endoscopy is not in itself a sufficient form of guidance feedback, it may paradoxically be a minimum requirement for many types of procedures. In microsurgery, for example, "stereo imaging is crucial…[as it] improves the ability of the viewer to filter out near-field objects in the image system while concentrating on the far field." In contrast, in other types of procedures, the importance of stereoscopic vision "has not been proved. With experience, it is possible to use monocular vision to perform even complex tasks such as suture placement and knot tying…[using cues such as] shadowing effects and the relative sizes of anatomic landmarks." While in daily activity, use of stereo vision is supplemented by the ability of the individual to use parallax and shadowing cues as well as the relative sizes of viewed objects to determine the relative depth location of objects in the visual field, the interposition of a video apparatus between the eyes of the surgeon and the targets of his or her vision reduces the available cognitive input and seriously undermines the crucial ability of the surgeon to perceive depth. Solutions to these problems, including the development of stereo laparoscopy and frame-buffered alternate sampling of two separate cameras on one monitor, are at present rudimentary and insufficient.
The limited field of vision caused by narrow fields of focus in endoscopic intervention can also be difficult to correct. While surgeons performing open procedures can simply adjust their perspective by moving, a perspective change in endoscopic surgery often involves removing the endoscope and re-inserting it into a different port, a time-consuming and inefficient task. The use of newly-developed flexible endoscopes to address this problem, however, "is not as easy or intuitive as using the traditional rigid telescope, because the camera operator cannot simply point it where he or she wishes to look but must turn various dials on the instrument to adjust the direction in which the steerable tip is looking." In addition, "distorted images and ineffectual color transmission…light intensity…[and] variable illumination of the surgical site" all contribute to the danger and stress of endoscopic operations.
In another sensory area, the lack of tactile feedback also undermines the ability of the experienced surgeon, by eliminating a significant part of what guides the surgeon during the course of an operation: "the palpable pulsation of a blood vessel thus normally prevents its accidental opening." While "there are many occasions in performing remote manipulation when touch would be useful…touch has not been highly developed or much used in remote handling." This is in part due to the lack of available technology for the reproduction of the highly sophisticated human sense of touch. At present, computer aids to surgery are simply not capable of reproducing the tactile sense in a particularly useful manner.
A common theme in this litany of inadequacies is that increasing the remoteness of the surgeon from the operative field and increasing his or her dependence of visual technology creates significant problems in feedback quality and usability. Whether the feedback is strictly visual (lighting, contrast, field of view, stereoscopy and depth perception), or tactile (inability of the surgeon to touch the operative site or to get force feedback from the instruments), the interposition of technology that imperfectly models the real world or ineptly transmits its valuable data to the surgeon significantly decreases the applicability of new technology in the medical arena. Solutions to these feedback problems have the potential to salvage a great deal of utility out of presently inadequate technologies.
At the level of system design failure, "spatial misarrangement of endoscope and video monitor may result in problems of hand-eye coordination caused by incompatibilities of the vectors of movement in the hand-eye coordinate systems. This can be addressed as the incompatibility of the manipulating space and the visual-perceptual space." Again, the improper attention to the design of the human-machine interface at the level of the video monitor and visual feedback, necessary because of the interposition of technological devices between the patient and the surgeon, adversely affects the ability of the surgeon to complete a successful procedure. Appropriate system design is the next higher-order task in the development of effective computer-surgical systems, as the limitations of the technologies themselves should not be further exacerbated by the interposition of difficult to use interfaces between the user and the tools: "the second major problem is the design of a user interface that is suitable for the clinician. Currently, there are numerous rather technical parameters" that intervene between the intentions of the surgeon and the realization of those intentions by the assisting devices. This emphasizes the point that not only should the technology used in the operating room be useful in itself, but it most also be intuitively useable in order to be effective.
The challenges of system design within the virtual reality application itself also require specific attention. While it may be obvious that the external system interface, including the technical equipment, positioning of components, and the design of the human-system interaction must be appropriately configured, it should not be overlooked that when immersed in the virtual environment itself, the user may face an entirely different yet equally vital arrangement of icons, tools, and interfaces that need to be examined for ease of use and flexibility:
While there is a great deal of information available from the human factors community, much of this information is unknown to the designers of virtual reality applications. Because virtual reality is currently operating in both a far less defined user environment and very much at (or beyond) the edge of available technology, virtual reality development is much more sensitive to human factors issues than [is] conventional computer graphics…Thus it is incumbent upon the designer of virtual reality applications to either become acquainted with appropriate human factors knowledge or to involve a human factors expert in the design process.
Failure to anticipate this type of system design problem can handicap even the most successful innovation in virtual reality through the introduction of unintended obstacles that will undermine the advantages of the virtual system.
The use of virtual reality in computer-assisted surgical systems, while arguably the most important area of research the development of surgical technology, is limited in many serious ways by the present avenues of exploration, much in the way that the limitations of graphical interfaces and imaging technology have slowed development at other points in the process.
To begin with, animation of a surgical display in real-time, the holy grail of virtual reality surgical applications, must be displayed at a speed greater than 30 frames per second, in order to prevent flickering of images detectable to the eye. Unfortunately, "real-time rendering places tight constraints on the simulation in terms of the complexity of the models. As the model becomes more complex, more computer power is needed to calculate the simulation and display the results." Once again, constraints of bandwidth an processor speed limit the potential of the technology to be of use in the operating room.
The head-mounted displays used in virtual reality engines, designed to immerse the user in a virtual visual field either captured by remote cameras or generated by the computer itself, are "limited primarily with respect to size and weight of the device (producing encumbrance and fatigue), picture resolution (number of pixels), stereopsis…and the time lag and smoothness of the servo-mechanism that drives the remote camera to follow head movements." Other challenges include "tracking…real-world cameras accurately and generating the correct synthetic images to model the views of the cameras."
Furthermore, while the advantages of "augmented reality" HMD systems are obvious, as they allow the user to have real structures and anatomy as visual reference points, while supplementing them with useful information from ultrasound overlays to vital statistics, they suffer from the same classic problems of other sophisticated imaging systems: "lag in image generation and tracking is noticeable in all head-mounted displays, but it is dramatically accentuated with see-though HMD. The ‘live video’ of the observer’s surroundings moves appropriately during any head movement but the synthetic image overlay lags behind." Thus, the advantage of "augmented reality" HMD in this case adds a specific drawback in addition to the generalized problems of the HMD technology itself.
Performance constraints of interactive virtual reality applications show that "the frame rate of the visual display must be greater than about ten frames per second in order for us to perceive changes in the visual images as continuous motion rather than as a series of still images…If the lag time approaches 0.5 s, however, direct manipulation becomes very slow and essentially unusable." This is particularly alarming when the device is intended to act as a surgical aid. Of particular concern when using virtual reality visualization in the context of medical applications is the fact that "sampling below [the rate of three to four times the highest frequency of motion] introduces temporal aliasing effects which can lead to very misleading perception of object motion…[and] if the lag time is longer than the period of the sample, the visual feedback provided to the user about, for example, the user’s hand position will be out of date with respect to the position of the object." The implications for virtual surgery are potentially disastrous: it is clearly not acceptable for the surgeon to be limited by the demands of the technology in his or her speed of motion during an operation. The goal, after all, is for the technology to remove limitations rather than impose them.
Perhaps the most disturbing limitation of immersive virtual reality systems is their potential to wreak havoc with the user’s natural cognitive reaction to sensory input. When immersed in a virtual environment, particularly one that is simulating a significant degree of motion, the user can potentially experience a syndrome known as ‘simulator sickness,’ that stems from the disparity in perceived and expected motion cues that is a consequence of imperfect visual modeling, lag times, and absence of vestibular feedback associated with real-world motion. This syndrome is exacerbated by the failure of "accurate synchronization of motion cues with visual cues. Here, we expect that forces induced by motion coincide exactly with the motion cues encoded in the real-time images." When this expectation is not met, the physiologic consequences can be severe.
When there is a disparity of even minor proportions between the brain’s expected sensory input and the brain’s perceived input, the conflict between these sets of information causes a significant physiologic response. The characteristics of this response are well-described, and include "a warning period of nausea, sweating, loss of equilibrium, pallor, eye strain and a general feeling of being unwell." Following this period, "if this cannot be controlled by conscious action, the vomiting center of the medulla is excited and vomiting occurs within 3 to 5 minutes." This ‘simulator sickness’ is common across all different types of visually-driven virtual reality applications, but may have been best described by researchers studying military flight simulators: "Using the report of at least one characteristic motion sickness related symptom as the criterion for concern, incidences of symptoms of motion sickness for the different simulators [used by the US Navy] ranged from 10% to 60%," and studies performed by the US Army (Gower et al 1988) and by the United Kingdom’s RAF (Chappelow 1988), reported similar findings.
In addition, the duration of this sickness can often last well beyond the period of immersion in the simulator itself: "some pilots who spend long periods training inside a simulator suffer from motion sickness. In some cases, it is so severe that affected personnel are restricted from flying or driving for at least 24 hours until they have completely recovered from symptoms such as sweating, pallor, vertigo and disorientation. The induced nausea has no one cause, but is often attributed to conflict between visual and vestibular information."
Once again, we see that the limitations of imaging technology and visual-based simulation present a difficult problem that is fundamental to the technology itself: "Most modern VR systems still exhibit latency in image generation, tracking and computing physical simulations, therefore there are plenty of opportunities for conflict." Even less dramatic side-effects of this syndrome are related to this problem: "The second side effect of VR immersion is asthenopia (eye-strain). The resulting eyestrain could be caused by the constant refocusing of the eyes to the different images in the virtual world. Additionally, eye-strain may be caused by a distortion in the expected and actual perception. This could be caused by several factors such as a distorted or flickering picture."
These problems can potentially be ameliorated by the development of more realistic simulations and interfaces: "Of critical importance to the theory is the conflict of information received by the senses: in this case a conflict between the visual system, which suggests body movement, and the vestibular system, which suggests a more static body position. This explanation of sickness in VR would suggests that facilitating more natural methods of movement in virtual environments may alleviate symptoms by bringing the vestibular cues more into alignment with the visual cues and hence reducing the sensory conflict." One of the advantages in the development of non-visual-based virtual reality applications, for surgery and in other fields, is this potential to reduce potential side-effects.
Even developments that seem intuitively useful may be difficult to incorporate into the medical world, which like the military paradigm, has a particularly rigorous and inflexible requirement of quality and usability for its associated technologies. Like professionals in other fields, "physicians tend to be suspicious of things they do not understand…and there is a general reluctance to change procedures that work." This underlines the crucial necessity of developing system-operator interfaces that are intuitively simple to use as well as clearly effective or useful. Even when there are virtual reality applications that can potentially change the face of medicine, if they are poorly designed or difficult to use, they will stand little chance of acceptance in the broader medical community.
The shortcomings of current computer technology and of the present approaches to computer-assisted surgery are therefore serious problems requiring well-designed solutions. Fortunately, a field of study exists for the very purpose of designing novel approaches to human-machine interactions. In particular, the new challenges of integrating evolving technologies such as virtual reality engines with the more traditional skills of surgery require a well-organized and systematic effort at system design.
The science of ergonomics involves analyzing the various components of a proposed human-machine system and determining the most favorable way to incorporate its disparate elements, taking into consideration the strengths of the human operator (flexibility, intelligence) and his or her physical and mental demands, while also maximizing the utility of the target technology, whether it is telescopic, aural, tactile, mechanical, or some combination of these. In the medical paradigm, this means that "the aim is to reduce the stress and strain for the medical staff and patient, and, at the same time, to increase the efficiency and safety of the system by means of implementing technical support devices with high-performance user interfaces."
In the field of medical ergonomics, there are additional unique requirements for appropriate system design. Medical human-machine interfaces incorporate a system element that is absent in the most common variety of human-machine systems: another human being, the patient. Not only must the workings of the system be appropriately designed for the maximum comfort and efficiency of the operators and the maximum utility of the machinery, but the needs of the patient must also be satisfied, and with the highest priority. In fact, the presence of the patient is what necessitates the existence of the system in the first place, and as a result the demands placed on the system by the presence of the patient may be considered paramount. These "special" demands require in particular that the attention of medical personnel not be distracted by the tasks of interacting with the machinery, for example, in order that they may devote more of their energy to the care of the patient: "The patient and his or her illness are very complex system components and can even be considered objects that cannot be changed or adapted in any way. The ergonomic design and optimization must focus on the technical process, to relieve the medical staff of avoidable activities and to design technical support devices that do not require the users’ attention intrinsically."
In endoscopic surgery, devices that could assist the surgeon in controlling unwieldy flexible endoscopes with increased degrees of freedom would be useful: "what would help in this situation is the use of a computer to relieve the human operator of the burden of having to work out the detailed control movements needed to position the tip to obtain the desired view." What is certain is that "whether complex teleoperation technology will be fully accepted in the operating room depends on whether improved performance can outweigh the disadvantages of greater cost and complexity."
Finally, the goal of virtual reality system design will include the construction of the virtual and real environment itself, incorporating an understanding of the new technologies and their requirements:
The great change in medicine and surgery…requires an operating room and hospital that is not only worthy of this advanced technology but capable of supporting it. An entirely new environment must be created based upon radically different concepts and the implementation of surgical and minimally invasive (or even non-invasive) therapies. Entirely new space configurations, the use of smart materials and intelligent equipment, and the integration of information infrastructure, knowledge-based decision support, imaging systems and advanced therapeutic modalities will be required to support the new generation of interventional therapists. In order to afford the widest possible opportunities to assess the impact of current and future technologies, the OR will be entirely planned using VR, giving architects, hospital administrators, surgeons, anaesthesiologists, operating room personnel and other key individuals the opportunity to ‘test’ the SRF before building it.
The design of the system at a fundamental level will therefore support the new technology. This will be of crucial importance when we consider the potential for a technological solution to much of the intrinsic failure of virtual reality and video imaging to date: the incorporation of virtual audio techniques of localization and guidance. In a specifically-designed environment, the potential applications and power of aural feedback could be greatly enhanced.
Physiological and Cognitive Basis of Audition
While human beings are generally considered sight-dependent creatures, there is no disputing the importance of auditory cues in our ability to relate to the environment. At the physiologic level, sound waves are transmitted to the inner ear via a system of mechanical interaction between membranes, small bones, and channels containing a fluid medium. In the inner ear, the sound waves of particular frequencies are deflected in such a way as to disturb the position of hair cells that trigger neuronal connections traveling through the auditory nerve to the cerebral cortex. These impulses are interpreted by the brain as sounds of a particular pitch and intensity.
The sense of hearing includes the ability to locate sound sources in three-dimensional space: "It was Lord Rayleigh (1907) who proposed that we localized sound sources by exploiting intensity and phase differences between the signals from the left and right ears." Moreover, the impact of each individual’s head-shape and external ear on the reflected sound waves received by the inner ear is crucial for sound localization: "Research by Shaw (1974) demonstrated that the pinna has a significant influence on shaping the spectral envelope of incident sound. Furthermore, this spectral shaping is dependent upon the spatial origin of the sound source. Thus the brain learns to extract spatial information from the unique ‘earprint’ the pinnae impress upon the incoming pressure waves." Each individual therefore receives the sound waves generated by an auditory source in a slightly different way, and then, using cues based on phase and intensity differences and the information derived from the impact of one’s pinnae and head on the sound waves, can localize the sound source in three dimensions, including azimuth, elevation, and distance from the listener.
More specific investigation of what factors influence sound localization has added four other parameters in addition to the factors of interaural time delay, head shadow, pinna response, and shoulder echoes that comprise the "Head-Related Transfer Function." They include head motion, vision, intensity, and early echo response and reverberation caused by local acoustics.
The particular interference characteristics of an individual’s head-shape and pinnae on the transfer of sound waves to the ear canals is a measurable function that has generated one approach to virtual sound modeling. Various techniques involving speaker arrays and sensitive miniature microphones inserted into the ear canal make it possible to derive an individual’s "Head-Related Transfer Functions (HRTFs)," which actually include the impact of the head, shoulders, and external ear on the incoming sound waves. Once the characteristics of particular sound waves that the listener localizes to a specific point is known, these sound waves can potentially be reproduced artificially, in order to give the listener the impression that the sound source is located in a specific place, whatever the location of the speakers generating the actual sound waves.
However, incorporating sound into a virtual reality application can be accomplished in a number of vastly different ways, with widely different intentions. At the most fundamental level, immersive virtual reality applications are given increased validity and realism when they make use of natural-seeming audio effects, even when such effects are not that closely tied to the visual environment, as they exploit the natural cognitive tendency of the listener to associate logically-associated sensory inputs: "Although a loudspeaker may be displaced from the actual location of a visual image on a television or movie screen, we can easily imagine the sound as coming from an actor’s mouth or from a passing car. This is an example of what is termed visual capture; the location of the visual image ‘captures’ the location derived from audio cues." These effects can be as simple as the triggering of an unmodified pre-stored audio sound when the user acts in a particular way in the virtual environment. They need not be highly sophisticated aural effects calculated for each particular user in order to have a significant effect on the quality of the virtual environment: "A New York Times interviewer, writing on a simulation of a waterfall…described how ‘the blurry white sheet’ that was meant to simulate a waterfall through a $30,000 helmet-mounted display seemed more real and convincing with the addition of the spatialized sound of the water." In this particular case, the addition of a comparatively cheap and easy to incorporate technology, generally appreciable by any user, considerably improved the overall impression of the simulation.
The vast potential of aural feedback to improve the quality of virtual reality systems is clearly at present largely underutilized, and, in the field of medical applications, virtually untested: "high-resolution color graphic hardware and software have been around longer on personal computers than the audio equivalent, CD-quality two-channel digital sound." This potential, however, should be obvious even to the observer unfamiliar with the state of virtual reality technology. Simply stated, "one might be able to work more effectively within a VR environment if actions were accompanied by appropriate sounds that seemingly emit from their proper locations, in the same way that texture mapping is used to improve the quality of shaded images." At the most basic level, therefore, pursuing the potential applications of audio technology for virtual reality is a fruitful avenue of research.
The benefits of the more sophisticated types of audio feedback in virtual reality are potentially much greater: "Although digitized sound samples play an important role in these systems, it is the ability to shape a waveform and adjust features such as pitch, timbre, amplitude, phase and decay that make it an important technology for VR." While as previously noted, digital imaging technology, volume-rendering, and visual display technology are currently stretching the limitations of currently-available computer processing speed and memory, in the case of audio technology, "current generation hardware is already capable of supplying binaural signals that model the attenuation of pressure waves entering the user’s ear canals, and thus simulate the way our ears influence perceived sounds in the real world."
Earcons, Audio/Musical User Interfaces
Using audio feedback in computers is by no means a new idea. The various beeping noises accompanying error messages in computer applications have been around almost as long as the PC. Fairly sophisticated implementations of auditory icons were described as early as 1982 by S. Bly and 1986 by W.W. Gaver and D. Sumikawa et al. Subsequent work by E.M. Wenzel et al, and Brewster, etc., has established a theoretical foundation for the field. There have come to be a number of established methodologies for using sound in VR: to reinforce other senses (almost universally visual) in environment simulations, for symbolic or iconographic representation of objects in integrated desktop type applications, for positional feedback of a cursor, or for sonification of 3-D volume data.
Sound has been employed redundantly in environment simulations by reinforcing other sensory elements in the simulation such as graphical or haptic, just as Foley effects (incidental sound effects) are used in cinematography to reinforce the impact of the moving image. Symbolic sound structures have been used in the manner of icons in Musical User Interfaces (MUI’s) developed for the blind. With this approach musical sound, sampled noise and speech forms "earcons" which may be manipulated within a three dimensional audio "desktop" virtual space. Other non-VR approaches include warning systems for civil aircraft. and audio feedback systems for medical equipment.
A variety of applications which provide positional guidance using audio feedback have also been developed. Some of these include:
The methods generally used by these systems are simple algorithms which convert a single dimension or a few dimensions of relative position data into corresponding acoustical dimensions. E.M. Wenzel describes using the frequency beat interference between two sound sources as the means for providing feedback for properly positioning a circuit board in a NASA astronaut training simulation. With the incorporation of synthetic speech, complex earcons and environmental noise in some of these systems, the level of feedback can become quite sophisticated. The application of such techniques in a medical VR system could have significant benefits.
Previous Use of 3D Aural Feedback in VR Applications
Previous research in the field of virtual audio showed small gains and difficult-to-apply technology. Particularly notable is the effort to produce an audio-guided cursor positioning application: "the results obtained from the experiments on this issue are not very encouraging. As it is concluded from the localization experiments that subjects primarily used the changes in the perceived sounds, better results in cursor position determination with the SAGA system can probably be obtained if some form of motion is incorporated, for instance by the use of moving sound sources or by allowing the user to rotate the cursor."
One of the specific advantages of aural feedback incorporated into medical applications concerns the need of the surgeon to maintain an extraordinary degree of focus on the desired task. In this context, information supplied by the virtual reality system must be easily available without distracting the operator from his or her activity. Durand Begault, in 3D Sound for Virtual Reality and Multimedia, writes that "perhaps the largest area for future development of integrated, spatial audio displays are in high-stress applications where the sound environment as a whole has historically never been considered as a single problem." Using the example of airline cockpits, Begault notes that "with an audio display, the focus of attention between virtual sound sources can be switched at will; vision, on the other hand, requires eye or head movement." This has clear cross-application to the demands of the surgeon involved in the intense concentration of the surgical procedure. In the operating room, conditions in many ways are analogous to the commercial airline cockpit described by Begault, particularly in the audio dimension:
…sounds come from many places and at different volume levels, depending on their function…auditory warnings can come from a variety of different locations…[and]warning signals tend to reflect mechanical alarms…rather than using sounds whose aural content better matches the semantic content of the situation. The reasons for this state of auditory chaos are that each audio subsystem is designed separately, the conventional methods for auditory display have been maintained in spite of technological improvement, and there has been a lack of applications research into advanced auditory displays.
In the context of systems design research, therefore, the development of an operating room environment that is sympathetic to the aural feedback technology could have immense benefits for the medical community. Research that could provide ways to correct the shortcomings and oversights of this aspect of system design is overdue and essential.
The little audio feedback that does exist in virtual reality applications and in everyday audio feedback systems suffers from serious shortcomings that can potentially be corrected much more simply than increased computing power for visual displays can be invented. Certain obvious shortcomings of current audio interfaces include "(1) the sounds are frequently repetitive, loud, and simple, without any way for the user to change the sound; (2) the sounds are generally used as alarms, instead of as carriers of a variety of information; (3) the quality of the sound is low, compared to the CD-entertainment standard, due to the limits of hardware specified within the design; and (4) sound usually contributes little to the ability to use the interface effectively, partially due to the lack of anything but the simplest type of sensor input." The minor steps taken in the area of aural feedback in virtual reality set goals and define areas of concern, but have little in the way of practical impact, especially in the medical field, and the developments we intend in this area will address these particular problems.
Advantages of Incorporating Audio in Medical VR
There are a number of significant advantages to incorporating 3D sound into virtual reality applications for medical use. To begin with, "auditory perception of human beings is omni-directional, as opposed to vision: we can hear sounds emitting from any direction, whereas we can only see what is in front of us. Second, spatial sound can also provide information about the distance of a sound source. These observations imply that the ‘display area’ of a computer would no longer have to be limited to the physical size of the screen used: objects, events, or whatever it is we want the user to be aware of, could be located anywhere in space, not just on the screen."
In an already-complicated HMD troubled by latency, the incorporation of audio information could significantly enhance the utility of the technology to the surgeon: "Because the eye channel is saturated by inputs, the ear can contribute to the input bandwidth through its capability of displaying a sonic variable." In terms of system design, the availability of the aural dimension for the transmission of vital data means that information about instrument position could be overlaid aurally to complement what visual data is available to the surgeon.
The fact that aural feedback avoids many of the shortcomings of the visual-feedback modality makes it an increasingly attractive area of exploration. The latency and flicker-induced vertigo almost universal in 3D visual applications are not a factor in 3D audio, because the amount of processing power needed to generate the audio signals is less than that needed for real-time visual display and volumetric rendering. The simulator sickness produced by immersive visual displays is also not a factor.
Discussion of Technical Issues
The main challenge to the successful development of a VR audio system lies in developing a methodology for transforming the instrument and anatomical spatial data into audio feedback that is optimally intuitive, information-rich, ergonomic and economical in terms of the learning curve for users. Numerous psychoacoustic phenomena, especially the nonlinearity of human hearing, as well as the problem of determining frames of reference for users must be taken into account in order to avoid giving ambiguous or confusing - ultimately useless - feedback. These potential problems can only be solved by building a flexible development platform, and incorporating input from practicing physicians (potential end-users) throughout the entire design process. The utility of such a system will ultimately be dependent upon how effectively information is mapped between modalities; visual geometry mapped to audio in a synesthetic paradigm (synesthesia: from Greek, syn = together + aisthesis = perception; defined as a "…physical experience of a cross-modal association.").
The exploitation of information mapping (which we define as a process whereby a manifold originating in one modality or dimension is transposed to, or is assigned to correspond to, another manifold in another modality or dimension) in interface design is more common than one might imagine. Consider how computer users have no difficulty overcoming the positional translations which occur between a horizontal table-manipulated mouse, and a vertical screen-projected cursor. The interface is surprisingly intuitive because, despite the fact that the y and z axes may be reversed, x offset +/-n depending upon the placement of the mouse (and the handedness of the user) - possibly even the magnitude of movement scaled, the correspondence is simple and consistent. The expected outcome of shifting the mouse in x, z does not diverge too significantly from the actual outcome.
Information mapping in a synesthetic paradigm is a greater leap of the imagination than the simple transformations and scalings of the computer mouse to monitor/cursor interface. Instead of translating one or more dimensions into another within a homomodal space, synesthesia involves transmodal, mappings, in our case visually and positionally based geometry into a corresponding aural space.
A synesthetic mapping algorithm may be thought of as being akin to general signal processing functions. We define a synesthetic function: ¦synes: Modality1[] ® Modality2[] . It is the task of the interface designer to determine which dimensions of the former modality may be mapped to the latter resulting in the greatest intuitiveness of the interface. Perceptual issues here become keenly important if the transformation is desired to be relatively lossless. Between the visual and auditory physiognomies there are differing, even incomparable, perceptual resolutions (minimum perceptual differences), perceptual ranges, nonlinearities of response (e.g. frequency/intensity response in hearing), auditory perceptual illusions such as the aural ‘cone of confusion’ (front-back reversal in 3D localization of sound sources), similarly, illusions of perspective such as the Necker cube in graphics:
A surgical trajectory planning system and sonification algorithm programming environment will be used to design a set of real-time operating procedures. The sonification algorithms will utilize the data on hand from the modeling, planning, and tracking system to generate sonic structures which will change in azimuth, elevation, attenuation, fundamental frequency, duration, spectral distribution, etc., all due to changes in the position and orientation of a space tracking sensor or multiple sensors with respect to the prescribed trajectory and ultimately an MRI/CT derived model.
This system will ultimately be used to investigate instrument position sonification in a user interface that is both intelligible and practical; a form of applied music. We will evaluate the incorporation of different properties of sound (fundamental frequency, spectral distribution, etc.) within an information-rich soundscape paradigm (we use the term soundscape here to refer to an articulated 3D audio world). For example, the feedback system could be programmed to yield harmonic consonances or dissonances based upon the instrument’s divergence or adherence to the prescribed target trajectory. Collisions of the instrument with anatomic obstacles/structures such as bone or nerves might be similarly indicated. We describe some basic implementations below (see: §8.3).
Because experimentation in the unexplored territory of this nascent field of ‘surgical sonification’ presents a decision tree of approaches, we plan to construct a modular, flexible system that will give us leeway to test multiple approaches; from simple monophonic monaural feedback schemes to ray-traced, Doppler-shifted soundscapes.
Results from our previous feasibility study indicate that we need significantly more processing power in order to yield the functionality we believe is necessary for a viable audio-enabled virtual reality surgical system. While we were able to design fairly complex sonification algorithms, we could not execute them in real time due to the limited computational power of the Macintosh PowerPC platform, and the fact that our system was built on top of an already latency-prone, beleaguered kernel (Opcode MAX). This had an inhibiting effect upon our research because we could only extrapolate the utility of the algorithms we designed, not perceive their effects directly. This observation, coupled with our interest in implementing more sophisticated sonification algorithms, and both integrating a new IRIX-based sonification system with a real-time 3D graphics display, and testing user performance with only a single modality of feedback necessitates a more substantial investment in hardware.
The overall system includes two subsystems: 1) the preoperative planning system, and 2) the real-time system, each of which, in turn, consists of a number of subsystems. The preoperative planning subsystem consists of a single mid-range graphics workstation. This platform will run custom software to handle the processing/model extraction from large CT/MRI databases, a surgical trajectory planning program, and a programming environment for designing the sonification algorithms.
Processes on the real-time subsystem will be most computationally demanding. We would like to be able to have position tracking, complex real-time sonification as well as 3D graphics running at 30-60fps in order to observe how these sensory modalities may enhance the viability of an audio-enabled virtual reality surgical system, as well as to test operator performance at a number of different levels using the different modalities (mainly graphics and audio, but also haptic) alone. The real-time subsystem consists of a position tracking device, and a low-end multiple CPU Onyx which will simultaneously power the audio and graphics subsystems. The graphics subsystem consists of a transformation and rendering engine driving some form of display device. This will consist of either a HUD or polarized glasses and a high-resolution monitor in 3-D mode. The setup will be determined by the context of the operation: procedure, O.R. geometry, etc., and ultimately user preference. We will determine, for future implementation, whether also tracking the position of the operator and/or the patient is necessary.
The audio subsystem consists of a high-end multichannel SiliconGraphics audio board. In addition, because we are still considering the utility of MIDI based audio control/generation - despite the specification’s great limitations, we will implement multiple MIDI controlled modular polyphonic synthesizers as an alternative low-end system. The digital output of either of these systems will be run through 3D spatialization hardware (an HRTF based convolution engine and DAC).
For a virtual audio system to be feasible within the acoustical environment of the O.R. we need to take into consideration incidental noise (i.e. surgical instrument noise, etc.) and interpersonal communication which is necessary for the operation to take place. Some experimentation will be made into the use/placing of microphones to capture O.R. sounds, and how they should be processed (compressed, equalized, etc.) and then mixed by a computer-controlled mixing console into the operator’s virtual audio environment. Audio output may be provided to the surgeon via a number of different sound production devices: inserted earphones, over-the-ear headphones, stereophonic/quadraphonic speaker systems or a single parabolic ceiling-mounted speaker. The configuration of such a setup will be determined by the spatialization technique we use (speaker-based transaural filtering or earphone-based user-specific HRTF), and again, user preference.
When we asked a number of surgeons how they would feel about wearing insert earphones or headphones, polarized goggles or HUD’s, they were quite adamant that they would prefer to avoid any further encumbrance in addition to what they are already obliged to wear in the O.R. This strong opinion indicates that we should make a serious attempt at devising a system that uses only video monitors and speaker systems. If this pared-down system proves unworkable we will proceed with more immersive and encumbering body-mounted equipment.
Our overall system software will be built using a number of commercially available toolkits as well as much custom-written code. For the preoperative subsystem, trajectory planning will be accomplished using a simplified 3D visualization and modeling program we will develop using SGI Inventor/Performer and the G.E. Visualization Tool Kit (VTK). Image processing will be carried out using a version of Dr. Dan Karron’s SpiderWeb Algorithm . The SpiderWeb Algorithm extracts models from CT/MRI databases using an implementation of Digital Morse Theory. This software will allow the operator to specify a volume which will encompass the entire procedure. Subsequent planning will be based upon the resultant output model dataset. Because all objects within the specified volume are tree-indexed, the operator can prune out structures of interest and plan operations on them.
Trajectory planning will be accomplished using a 3D visualization and drawing program using alternatively a Polhemus 3Draw stylus, a trackball, a mouse as input devices, or spline functions and point coordinates, which will allow extremely precise planning/modeling of spline space curves, obstacle thresholds, superimposed on the model database and CT/MRI image. In addition, the surgeon will be able to model or generate acceptable error envelopes around the trajectory, tag interesting/critical regions such as major vessels, bone, nerves, tumors, specify lighting, texture-mapping, colors and visual formatting in general to be realized by the real-time system.
The sonification algorithms will be designed using a visual programming environment we will develop using freeware synthesis and DSP libraries such as RTF and Ptolemy, and an IRIX-based IRCAM version of Max. The graphics software for the real-time subsystem will be a no-frills 3D binocular renderer built using SGI Inventor/Performer and VTK. The sonification engine will consist of custom software to execute the planned algorithmic audio control directives, which will drive the digital signal processing hardware.
Following are brief sketches of some very simple approaches for sonifying position data which we will use as a point of departure for some of our research in developing sonification algorithms.
Reference & Pursuit: Feedback for One Dimension
This is the simplest implementation. Two sound sources are used to represent the degree of divergence of the instrument from the pre-planned trajectory. A sound source, a, represents the trajectory, and, b, the state of the instrument. Divergence of the instrument from the trajectory in y spatial dimension, yerr, converts into an increase or decrease in frequency of b from a. The frequency of a functions as a point of origin and is fixed. The operator corrects for divergence of the instrument from the trajectory in y dimension by closing the frequency gap between a and b. We depict this relation musically in Figure 2.
Figure 1: Simple reference & pursuit relationship.
Figure 2: Frequency b diverges
from a.
We have found, through our Phase I research, that this technique yields a fairly intuitive interface, however it results in a phenomenon we call "rubbing." In order to anticipate curve changes in the trajectory the operator "rubs" the instrument rapidly back and forth the across the trajectory and error envelope (see: Figure 3). The large amount of information thus obtained allows the operator to better understand the precise shape of the trajectory curve, yet results in a widening of the error envelope, and ultimately of the required operative space. Future implementations of this technique must take this phenomenon into consideration.
Figure 3: Typical user error in 2D: ‘rubbing’ across the trajectory.
In addition to the basic audio primitives such as frequency, etc., musical informatics presents a large number of high-level combinatorial structural functions which are open to us for consideration in developing a surgical sonification system. These will allow significant flexibility in designing earcon-like structures and sonification procedures. Some of these function classes are:
Algorithmic procedures employing some of these high level functions could impart a greater listenability and coherence to the feedback system overall, as opposed to a purely low-level based approach to which the ear might desensitized. They will also allow the communication of multiple high-level modalities to the operator.
Multitimbral System: Feedback for Three Dimensions
In order to extend a system into three dimensions we may employ the simple reference and pursuit frequency model illustrated above using one {a, b} sound source set for x, y and z of the error trajectory. To facilitate intelligibility, and to insure that sound source sets X, Y, Z do not overlap in a confusion of pitches, each sound set is given a different timbre and a different reference frequency. The spectral distributions, or timbres, of these sets, TX , TY and TZ must be contrasting in order to avoid ambiguities between sets (i.e. the operator should not confuse which sound set represents x, y or z). The reference frequencies are chosen so that, when the instrument is in the proper position (relative to the trajectory) the user perceives a memorable, consonant harmonic structure. This relation is denoted:
X = { a0, b0 = a0
+ ¦synes(xerr)
} Î TX
Y = { a1, b1 = a1 + ¦synes(yerr)
} Î TY
Z = { a2, b2 = a2 + ¦synes(zerr)
} Î TZ
where it is stipulated that:
a0 ¹ a1 ¹ a2 and TX ¹ TY ¹ TZ
Figure 4 graphically depicts this system.
Figure 4: Multitimbral system: three dimensions of error.
In the SBIR Phase 1 we discovered that a degree of clarity was lost in the simulation when the varying frequency sources would cross the reference frequency, despite belonging to timbre set. To correct for this we must specify that no frequency source may be within the frequency range (or tessitura) of another set, also, that it may not cross those frequencies which, although they may physically lie within the tessitura of its set, may belong as reference frequencies to other sets. Therefore we must exclude foreign reference frequencies thus:
{ a1, a2 } Ï X
{ a0, a2 } Ï Y
{ a0, a1 } Ï Z
We notate this relationship in Figure 5.
Figure 5: Notation of XYZ Triad Relation.
Lines connecting to the filled noteheads define the tessituras of source sets, X, Y and Z, where the empty noteheads denote the reference frequency source (pitch). The little boxes mark where ‘forbidden’ frequencies lie. In musical terms, the reference structure outlines a major triad in second inversion, with
We might correct b0 of X in the case of transgression by shifting it up slightly:
if (( b0 = (a0 + ¦synes( xerr ))) == ( a1 || a2 || b1 || b2 )){
b0++ ; // increment by minimal perceptual frequency difference
}
Spatialization: Feedback for Three Dimensions
Techniques using spatialization (3-D localization) of sound sources may be used to sonify instrument and trajectory position. A simple transformation, as in the proceeding examples, may be used to convert real position into virtual, localized position - real instrument movement converting to movement of the sound source in virtual space. The problem here is with reference frames - which to choose, and the inherent ambiguities of human auditory localization; the minimum perceptible audible movement angle being greater than 3-5º.
A possible scheme might involve mapping the operator’s frame of reference to the position and orientation of the instrument. This would be similar to the endoscope. As the instrument diverges from the trajectory, an earcon representing the trajectory moves out in space from the operator’s perceptual origin. Error is corrected by the operator trying to align the trajectory earcon with his/her perceptual origin. Figure 6 illustrates the operator’s head-related coordinate system.
Figure 6: Head-Centered Cursor and Spatialized Trajectory Earcon.
Volume Sonification & the Soundscape Paradigm
By the term soundscape we mean an articulated 3D audio world where sounds emanate not from dimensionless points in space, but as from the surface of virtual 3D volume objects. Sound is absorbed, reflected and refracted by occluding objects as in the real world. Moving objects Doppler shift. Collisions between objects (a virtual surgical instrument and a bone fragment for instance) cause a predetermined change in the fundamental frequency, spectral distribution and amplitude of the sounds emanating from the objects to occur. Presently research in this area is still highly experimental, and computationally demanding, however, we believe that this avenue of exploration would be fruitful for an audio-enabled surgical system because it is not only capable of representing extremely complicated structures and interactions, but our everyday lives are spent immersed in complex soundscapes. Such a system could be highly intuitive for the user. Follows is a brief description of a scenario we would like to implement.
Prior to the real-time execution of the procedure the model database is run through a sonification compiler. The compiler descends the object tree and applies a combinatorial algorithm to each object group and each individual object within the tree. Timbre models are generated and assigned to objects and object groups on the basis of their structural, topological complexity, relative volume, density and texture (derived from the original CT/MRI dataset). Timbre models are built from a finite number of possible signal processing objects using rules defined by the system designer, which permit only useful, listenable timbres to be generated. The timbre models are not static ‘canned’ sounds, but change dynamically with interaction, in real-time like the human voice or natural musical instruments. Before compiling, the surgeon might tag an interesting anatomical region such as a tumor, and on output, the resulting timbre models of such tagged objects would sound more interesting or unique than the others. This approach is analogous to texture mapping.
In execution, movement of the surgical instrument updates the 3D position of all of the sound sources. A soundscape rendering algorithm executes the sound synthesis timbre models, back-clips occluded faces and objects, calculates for refraction around, and transmission of sound through objects, ray-traces the virtual world, and convolves the final auditory scene output using the operator’s head-related transfer functions. With the addition of purely symbolic structures, such a system could maintain a very broad transmission of multimodal information to the operator.
Medical Implementation of this Technology
Werner Doyle, M.D., who is a practicing neurosurgeon will oversee this aspect of the project. With his consultation we will explore the application of audio virtual reality techniques to neurooncological surgery and epilepsy surgery.
Epilepsy surgery presents a fairly complicated scenario because the surgeon not only needs guidance for navigating through the patient’s anatomy, but requires information as to the location of the physiological boundaries of eleptogenic foci; boundaries which are normally invisible to the surgeon as he operates. Being able to know where he is relative to these boundaries would allow the surgeon to place subdural electrode arrays with greater precision and accuracy.
Nolan Karp, M.D., Assistant Professor of Plastic Surgery at NYU School of Medicine, will direct the application of this technology to maxillofacial surgery.
The function of the surgeon in maxillofacial surgery is to determine where malpositioned bones are supposed to be, using a normative database adjusted for sex, age, height, etc., or CT/MRI images taken prior to the trauma, and, using that information, to plate them into the correct position. Extremely precise positioning of the bones is very difficult to achieve. Prior approaches to solving this problem have proven very elaborate, and ultimately as untenable for the average surgeon as for military field surgeons who must regularly treat complex facial fractures. Using our positional feedback system a surgeon would be able to correctly reposition malpositioned bones by clamping them to, and manipulating them using, a 6 degree-of-freedom space tracking device. The system would indicate whether the bone in question was near, or in its prescribed position and orientation. In a telesurgical scenario, a surgeon could remotely direct a field medic, or less experienced personnel, how to reduce fractures.
For purposes of documentation the entire project is partitioned into 4 overlapping project periods, each lasting approximately 5 months. This will not be rigidly demonstrated, but serves to represent the logical division of tasks for this project. During the first period we will construct the hardware basics of the system, including the device drivers, multithread process managers. During the second period we will begin to flesh out the various subsystems. During the third period we will begin to have a moderately functional system, we will start experimenting in real-time with different sonification algorithms, and graphics rendering approaches. For the fourth period we will implement a functional real-time system in a mock O.R., where we will test different sonification approaches with the assistance of NYUMC medical/ surgical faculty.
Phase 1: Low Level Construction
Phase 3: Aural User Interface & GUI
Phase 4: Mock Clinical Trials & Reengineering
The first product incorporating this technology will be a published and licensed specification application programming interface (API). This will enable various implementers on various platforms to achieve uniform results and adhere to a common published standard. The API will be available for use in our proprietary computer-aided surgical hardware/software system, or may be embedded in third-party simulation applications.
The real-time graphics API will contain specifications for modeling and managing 3-D volumes, planning trajectories, placing reference tags, delineating obstacles, specifying 3-D visual formatting. The audio API will contain specifications for building libraries of digital signal processing modules, designing sonification algorithms, and will include a number of pre-designed sonification systems for positional feedback. Additionally we will include specifications for device drivers for various input devices, a real-time process manager, a geometry manager, real-time 3-D graphics rendering tools, and a sonification algorithm execution environment.
We will sell and license a hardware package for audio-based positional feedback in surgical virtual reality systems as well as sonification in a wide variety of different applications. The hardware implementation will consist of guidelines and specifications for a preoperative surgical planning system, and an operating room virtual audio system setup.
NOTE: See also: CASI Marketing.
Customers for this product will include hospitals, clinics, medical schools, universities and research/engineering organizations involved in clinical practice, research in advanced surgical technology, telemanipulation, human factors and interface design.
There are a lot of intangible benefits of our system to the market in reduced rehabilitation costs, reduced stroke potential, faster recovery time, fewer malpractice claims, reduced insurance costs as well as greater patient acceptance. Additionally, our system may potentially allow hospitals to make more efficient use of scare surgical talent using avatar mentoring.
We estimate there are fifty million dollars of eligible surgeries done yearly in the market. Roughly twenty-five million dollars worth of these surgeries may consider our system as a viable option. If, in implementation, our system saves roughly one third the cost of the surgery, our first year of commercial release should be able to save approximately nine million dollars. We estimate this equates to roughly two to three million dollars the first year in gross business.
We estimate the amount of capital required to bring our product to market at 5 million dollars over three years. We intend to raise it by public stock offering. Using this capital we will build a number of demonstration systems. These will be placed in magnet institutions which we will cultivate, and where we will fund live demonstration surgeries using our technology. Additionally, we will fund training fellowships and maintain an Internet website devoted to surgical virtual reality.
P.I. is widely published and trained by some of the leading pioneers in the field. His experience and reputation is a key to building support in the market. Additionally we will recruit a senior marketing executive from the biomedical virtual reality industry as part of the Phase III commercialization.
There are no competitors. There is no other group publicly pursuing this niche. Our aggressive R&D will lead to immediate market dominance.
Table 1
Table 2
Table 3
Table 4
Key Personnel: Curriculum Vitae
Werner K. Doyle, M.D. (CV In Brief)
Doyle, W.K., "Interactive Image-Directed Epilepsy Surgery (Rudimentary Virtual Reality in Neurosurgery) in Interactive Technology and the New Paradigm for Healthcare," Ed. By Satava, R.M., Morgan, K., Sieburg, H.B., Matheus, R., Christensen, J.P., Medicine Meets Virtual Reality Proceedings, IOS Press, 1995, p91-100.
Doyle, W.K. "Rudimentary Virtual Reality in Neurosurgery: Interactive Image-Directed Epilepsy Surgery," Journal of Medicine and Virtual Reality, 1: 46-53, 1995.
Doyle, W.K. "Interactive Image Guided Neurosurgery in Epilepsy," American Association of Neurological Surgeons Annual Meeting, 1995.
Doyle, W.K. "Comparison of Interactive Image-Directed Techniques and Ultrasound for Intraoperative Localization of Lesions in Epilepsy Surgery" Congress of Neurological Surgeons Annual Meeting, October 1995.
Doyle, W.K. "Interactive Image-Directed Surgical Techniques Reduce Craniotomy Size for Temporal Lobectomies and Improve Placement of Subdural Strip Electrodes," Congress of Neurological Surgeons Annual Meeting, October 1995.
Doyle, W.K., Pacia, S., Perrine, K., Devinsky, O. "Preliminary Experience with Interactive Image Directed Techniques for Functional Mapping and Electrographic Localization During Epilepsy Surgery," American Epilepsy Society Meeting, 1995; Abstract and Platform.
Doyle, W.K. "Low End Image-Directed Neurosurgery: Update on Rudimentary Augmented Reality Used in Epilepsy Surgery," Health Care in the Information Age, Ed. By Sieburg, H., Weghorst, S., Morgan, K., Burke, VA: IOS Press and Ohmsha, 1996.
The American Society of Maxillofacial Surgery Award for the best clinical paper in Plastic and Reconstructive Surgery for the year 1990: McCarthy, J.G., Karp, N.S., LaTrenta, G.S., Thorne, C.H.M.: The effect of early frontoorbital advancement on frontal sinus development and forehead esthetics.
First Place for the outstanding paper delivered before the Section of Plastic and Reconstructive Surgery of the New York Academy of Medicine and the New York Regional Society of Plastic and Reconstructive Surgery: Bone lengthening in the canine mandible. March 5, 1990.
First Prize Residents Research Competition, Institute of Reconstructive Plastic Surgery, New York University Medical Center. Best basic science paper: Bone lengthening in the craniofacial skeleton. May 30, 1989.
Plastic Surgery Educational Foundation: An ultrastructual and histologic analysis of the tissue responsible for bone lengthening in membranous bone. 1989
Plastic Surgery Educational Foundation: Multi-dimensional distraction osteogenesis in the canine zygoma. 1992
Vayo, M., Lipowsky, H., Karp, N., Schmalzer, E., Chien, S.: A model of microvascular oxygen transport in sickle cell disease. Microvasc. Res., 30:195-206, 1985.
Roses, D.F., Karp, N.S., Sudarsky, L.A., Valensi, Q.J.,2 Rosen, R.J., Blum, M.: Primary hyperparathyroidism associated with two enlarged parathyroid glands. Arch. Surg., 124:1261-1265, 1989.
Karp, N.S., Boyd, A., DePan, H.J., Harris, M.N., Roses, D.F.: Thoracotomy for Metastatic Melanoma of the Lung. Surgery, 107:256-261, 1990.
Karp, N.S., Thorne, C.H.M., McCarthy, J.G., Sissons, H.A.,: Bone lengthening in the craniofacial skeleton. Ann. plast. Surg., 24:231-237, 1990.
Karp, N.S., Lamparello, P., Ranson, J.H.C.: Total pancreatectomy with celiac artery occlusion. N. Y. State Jour. Med., 90:416-418, 1990.
McCarthy, J.G., Karp, N.S., LaTrenta, G.S., Thorne, C.H.: The effect of early frontoorbital advancement on frontal sinus development and forehead esthetics. Plast. Reconstr. Surg., 86:1078-1084, 1990.
Rusinek, H., Karp, N., Cutting, C.B.: Three-dimensional rendering of medical images: surface and volume approaches. J. Digital Imag., 3:81-88, 1990.
Roses, D. F. , Karp, N. S. , Oratz, R., et al.: Survival with regional and distant metastases from cutaneous malignant melanoma. Surg., Gynec. Obst., 172:262-268, 1991.
McCarthy, J.G., Schreiber, J.S., Karp, N.S., Thorne, C.H., Grayson, B.H.: Lengthening of the human mandible by gradual distraction. Plast. Reconstr. Surg., 89:1-8, 1992.
Roses, D.F, Mitnick, J., Harris, M.N., Kaplon, R., Karp, N.S., Vazquez, M., Dubin, N.: The risk of carcinoma in wire localization biopsies for mammographically detected clustered microcalcifications. Surgery, 110:877-886,1991.
Zide, B.M., Karp, N.S.: Maximizing gain from rectangular tissue expanders. Plast. Reconstr. Surg., 90:500-504, 1992.
Karp, N.S., McCarthy, J.G., Schreiber, J.S., Sissons, H.A., Thorne, C.H.M.: Membranous bone lengthening: A serial histologic study. Ann. Plast. Surg., 29:2-7, 1992.
Rapaport, D.P., Breitbart, A.S., Karp, N.S., Siebert, J.W.: Successful microvascular replantation of a completely amputated ear. Microsurg., 14:312-314, 1993.
Karp, N.S., Kasabian, A.K., Siebert, J.W., Eidelman, Y., Colen, S.: Microvascular free flap salvage of the diabetic foot: A five year experience. Plast. Reconstr. Surg., 94:834-840, 1994.
Glat, P.M., Staffenberg, D.A., Karp, N.S., Holliday, R.A., Steiner, G., McCarthy, J.G.: Multidimensional distraction osteogenesis: The canine zygoma. Plast. Reconstr. Surg., 94:753-758, 1994.
Denk, M.J., Longaker, M.T., Basner, A.L., Glat, P.M., Karp, N.S., Kasabian, A.K.: Microsurgical reconstruction of the lower extremity using the 3M microvascular coupling device in venous anastomoses. Ann. Plast Surg., 35:601-606, 1995.
Bass, L.S., Benacquista, T., Kasabian, A.K., Karp, N.S.: Endoscopic rectus abdominus harvest: Balloon dissection in the fascial plane. Ann. Plast. Surg., 34:274-280, 1995.
Choe, E.I., Kasabian A.K., Kolker, A.R., Karp, N. S ., Zhang, L., Bass, L.S., Nardi, M., Josephson, G., Karpatkin, M.: Thrombocytosis after major lower extremity trauma: Mechanism and possible role in free flap failure. Ann.Plast. Surg., 36:489-494, 1996.
Benacquista, T., Kasabian, A.K., Karp, N.S.: The fate of lower extremities with failed free flaps. Accepted for publication Plast. Reconstr. Surg.
Side, B.M., Karp, N.S.: Surgery for upper lip lesions: Reconstruction of smaller defects. Submitted for publication.
Kolker, A.R., Gottlieb, J.J., Karp, N.S., Kasabian, A.K.: The fate of free flap microanastomosis distal to the zone of injury in lower extremity trauma. Accepted for publication Plast. Reconstr. Surg.
Karp, N.S., Thorne, C.H.M., McCarthy, J.G.: Bone lengthening in the canine mandible. Plastic Surgery Research Council, April 24,1989, Atlanta, Georgia.:~
Karp, N.S., Boyd, A.D., DePan, H.J., Harris, M.N., Roses, D.F.: Thoracotomy for metastatic melanoma of the lung. New York Society for Thoracic Surgery, May 18, 1989, New York, N.Y.
Karp, N. S ., Thorne, C. H. M., McCarthy, J. G.: Bone expansion in the craniofacial skeleton. Thirteenth International Conference on External Fixation, May 27, 1989, Rochester Minn.
Karp, N.S., Schreiber, J.S., Thorne, C.H.M., McCarthy, J.G.: Membranous bone lengthening: A serial histologic study. Plastic Surgery Research Council, April 19, 1990, Washington, D. C.
Karp, N.S., Schreiber, J.S., Thorne, C.H.M., McCarthy, J.G.: Membranous bone lengthening: A serial histologic study. American Society of Plastic and Reconstructive Surgery, Oct. 23, 1990, Boston, Mass.
Karp, N.S., Zide, B.M.: The optimal use of rectangular tissue expanders. American Society of Plastic and Reconstructive Surgery, October 21-25, 1990, Boston, Mass.
Karp, N.S., Kasabian, A.K., Siebert, J., Eidelman, Y., Colen, S.: Microvascular free flap salvage of the diabetic foot: A five year experience. American Society for Reconstructive Microsurgery, Nov. 9, 1992, Scottsdale, Az.
564 1st Avenue
New York, NY 10016
Phone: (212) 685-4364
"Heparin Bonding of Bypass Circuits Reduces Cytokine Release During Cardiopulmonary Bypass." Steinberg, B.M.; Grossi, E.A.; Schwartz, D.S.; McLoughlin, D.E., Aguinaga, M.; Bizekis, C., Greenwald, J.; Flisser, A.; Spencer, F.C.; Galloway, A.C.; Colvin, S. Annals of Thoracic Surgery. 60(3): 525-9, 1995 September.
Computer Aided Surgery, Inc. (C.A.S.I.) is a new corporation founded in 1995 by Dr. Dan Karron. C.A.S.I. has a 1,000sq ft. research laboratory located in mid-town Manhattan, adjacent to the New York University Medical Center. Our address.
Computer equipment includes one high-end SiliconGraphics Indigo2 workstation, two mid-range Indigo’s, two high-end PC’s, a Macintosh PowerPC and a SGI Iris 80GT. Peripherals include two Polhemus space tracking devices (3Draw and 3Space), CD ROM’s, JAZ and ZIP external storage devices, laser printers, scanners. Networking: 10BT routers, high-speed ISDN, dedicated modem lines. Audio equipment: Mackie 1202 VLZ mixing console, Nakamichi RE-2 amplifier, Acoustic Research 4’ wall-mounted speakers, Korg O5R-W synthesis module, Roland SCP-55 synthesis card, various microphones and headsets.
Ballantyne, G.H. "Laparoscopic-assisted colorectal surgery: review of results in 752 patients." Gastroenterologist. 3(1): 75-89. March 1995.
Bargar, R. "Model-based interactive sound for an immersive virtual environment" In SIGGRAPH 1994.
Barkun, J.S.; Caro, J.J.; Barkun, A.N.; and Trindade, E. "Cost-effectiveness of laparoscopic and mini-cholecystectomy in a prospective randomized trial." Surgical Endoscopy. 9(11): 1221-4. November 1995.
Bajura, Michael; Fuchs, Henry; and Ohbuchi, Ryutarou. "Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery Within the Patient." Computer Integrated Surgery.
Begault, Durand R. 3D Sound for Virtual Reality and Multimedia. Boston: Academic Press, Inc. 1994.
Burdea, Grigore C. Force and Touch Feedback for Virtual Reality. New York: John Wiley & Sons, Inc.1996.
Bryson, S. "Design and Implementation of VR Applications." In Virtual Reality Applications.
Bloom, B.S.; Fendrick, A.M.; and McLane, C. "Predicting effects of minimal invasive therapy." Health Policy. 35(2): 179-87. February 1996.
Bly, S. "Presenting Information In Sound," Proceedings of the CHI ’82 Conference on Human Factors in Computing Systems. New York: The Association for Computing Machinery.
Brewster, S.A. Providing a Structured Method for Integrating Non-Speech Audio into Human-Computer Interfaces. Ph.D. Thesis, University of York, U.K., 1994.
Charles, S. "Dexterity Enhancement for Surgery." Computer Integrated Surgery.
Chappelow, J.W. "Simulator Sickness in the Royal Air Force: A Survey." AGARD Conference Proceedings 433. Motion Cues In Flight Simulation And Simulator Induced Sickness. 1988.
Chorafas, D.N., and Steinmann, H. Virtual Reality: Practical Applications In Business And Industry. Upper Saddle River: Prentice Hall PTR. 1996.
Computer Graphics. Proceedings of SIGGRAPH 96 (New Orleans, LA, August 4-9, 1996). In Computer Graphics Proceedings, Annual Conference Series, 1996, ACM SIGGRAPH.
Cutting, C.B., Bookstein, F.L., Taylor, R.H., "Applications of Simulation, Morphometrics, and Robotics in Craniofacial Surgery," in Computer Integrated Surgery, p. 641-671.
Cytowic, R.E. "Synesthesia: Phenomenology And Neuropsychology A Review of Current Knowledge." Sydney: Psyche, 2(10). 1995.
Earnshaw, R.A., Vince, J.A., and Jones, H. Virtual Reality Applications. San Diego: Academic Press, Inc. 1995.
Edwards, A.D.N., "Soundtrack: An Auditory Interface for Blind Users," Human Computer Interaction, 4(1), 1989.
Gardner, W.G., "Transaural 3-D Audio." Technical Report No.342. M.I.T. Media Laboratory Perceptual Computing Section. 1995.
Gaver, W.W. "Auditory Icons: Using Sound in Computer Interfaces." Human-Computer Interaction, 2, 167-177.
Gehring, B. "Attitude Indicator." (United States Patent 4,774,515) Washington, D.C.: Commisioner of Patents and Trademarks.
Glinsky, A.V. The Theremin in the Emergence of Electronic Music. Ph.D. Thesis, School of Education, Health, Nursing and Arts Professions. New York University, New York, NY. 1992.
Gobel, Martin; Muller, Heinrich; and Urban, Bodo (eds.) Visualization in Scientific Computing. Austria: Springer-Verlag Wien. 1995.
Gower, D.W., Lilienthal, M.G., Kennedy, R.S., and Fowlkes, J.E. "Simulator Sickness in US Army and Navy fixed- and rotary-wing flight simulators." AGARD Conference Proceedings 433. Motion Cues in Flight Simulation and Simulator Induced Sickness. 1988.
Gupta, Subhas C., Wantland, Cyril A., and Klein, Scott A. "Cyberpathology: Medical Concerns of VR Applications." The Journal of Medicine and Virtual Reality. 1(2): 34-37.
Karron, D.B. and Cox, J. "Extracting 3D objects from volume data using digital morse theory." In N. Ayache, editor. Computer Vision, Virtual Reality, and Robotics in Medicine. New York: Springer-Verlag. 1995.
Kellog, W.N. "Sonar System for the Blind," Science, 137: 399-404, 1962.
Kelly, Patrick J. "Computer-Assisted Neurosurgery: Needs and Opportunities." Computer Integrated Surgery.
Kikinis, Ron; Gleason, P. Langham; and Jolesz, Ferenc A. "Surgical planning using computer-assisted three-dimensional reconstructions." In Computer Integrated Surgery.
Langreth, Robert. "Hypermusic." Popular Science. 249(4): 59-95. October 1995.
Louie B., Guy J., Quinn M., and Reid R. "Reducing surgical length of stay: Quantifying the impact." Healthcare Management Forum. 8(4): 29-35. Winter 1995.
Mulder, J.D., and Dooijes, E.H. "Spatial Audio in Graphical Applications." Visualization in Scientific Computing.
Patterson, R.D., "Guidelines for Auditory Warning Systems on Civil Aircraft." (Report No. 82017). London, U.K.: Civil Aviation Authority, 1982.
Patterson, R.D., "Alarm Sounds for Medical Equipment in Intensive Care Areas and Operating Theaters," Report No. AC598, Institute for Sound and Vibration Research, University of Southampton, U.K.
Pommert, Andreas; Riemer, Martin; Schiemann, Thomas; Schubert, Rainer; Tiede, Ulf; and Hohne, Karl Heinz. "Three-dimensional imaging in medicine: methods and applications." In Computer Integrated Surgery.
Puckette, M. "Combining Event and Signal Processing in the MAX Graphical Programming Environment." Computer Music Journal. 15(3): 68-77. 1991.
Rau, Gunter; Radermacher, Klaus; Thull, Bernhard; and Von Pichler, Cletus. "Aspects of Ergonomic System Design Applied to Medical Work Systems." In Computer Integrated Surgery.
Regan, E.C. "Human Factors in Immersive VR." In Virtual Reality Applications.
Robb, Richard A., and Dennis P. Hanson. "The ANALYZE Software System for visualization and analysis in surgery simulation." In Computer Integrated Surgery.
Rosen, Joseph M.; Lasko-Harvill, Ann; and Satava, Richard. "Virtual Reality and Surgery." In Computer Integrated Surgery.
Satava, Richard M. "Virtual Reality for the Physician of the 21st Century." In Virtual Reality Applications.
Schippers, E.; and Schumpelick, V. "Requirements and Possibilities of Computer-Assisted Endoscopic Surgery." Computer Integrated Surgery.
Sheridan, Thomas B. "Human Factors in Telesurgery." Computer Integrated Surgery.
Strauss, S. "Cybersickness: The side effects of virtual reality." Technology Review. 98(5): 12-14. July, 1995.
Strybel, T., Manligas, C. and Perrott, D., "Minimum Audible Movement Angle as a Function of the Azimuth and Elevation of the Source," Human Factors, 34(3), p. 267-275.
Stulberg, S. David; and Kienzle, Thomas C., III. "Computer- and Robot-Assisted Orthopaedic Surgery." Computer Integrated Surgery.
Sumikawa, D.; Blattner, D.; Joy, K. and Greenberg, R. "Guidelines for the Syntactic Design of Audio Cues in Computer Interfaces." Lawrence Livermore National Laboratory, 1986.
Taylor, Russell H.; Lavallee, Stephane; Burdea, Grigore C.; and Mosges, Ralph; editors. Computer Integrated Surgery. Cambridge: The MIT Press, 1996.
Tendick, Frank; Jennings, Russell W.; Tharp, Gregory; and Stark, Lawrence. "Perception and Manipulation Problems in Endoscopic Surgery." Computer Integrated Surgery.
Thalman, Daniel. "Applications of Virtual Humans in Virtual Reality." Virtual Reality Applications.
Traverso, L.W., and Hargrave, K. "A prospective cost analysis of laparoscopic cholecystectomy." American Journal of Surgery. 169(5): 503-6. May 1995.
Treat, Michael R. "A Surgeon’s Perspective on the Difficulties of Laparoscopic Surgery." Computer Integrated Surgery.
Vandenbergh, H.C.; Wilson, T.; Adams, S.E.; and Inglis, M.J. "Laparoscopic cholecystectomy: its impact on national health economics." Medical Journal of Australia. 162(11): 587-90. June 5, 1995.
Vanderheiden, G.C. "Nonvisual Alternative Display Techniques for Output from Graphics-Based Computers." Journal of Visual Impairment and Blindness, 1989.
Vince, John. Virtual Reality Systems. Workingham, England: ACM Press, 1995.
Waters, Keith. "Synthetic Muscular Contraction on Facial Tissue Derived from Computerized Tomography Data." In Computer Integrated Surgery.
Wenzel, E.M.; Fisher, S.; Stone, P.K., and Foster, S.H. "A System for Three-Dimensional Acoustic "Visualization" in a Virtual Environment Workstation." Proceedings of Visualization ’90 Conference. New York, NY: IEEE Press. 1990.
World, Linda. "The Reality of Cybersickness." IEEE Computer Graphics and Applications. 15(5): 95. September, 1995.1
Zwicker, E. and Fastl, H. Psychoacoustics: Facts and Models. New York, NY: Springer-Verlag, 1990.
