Physicians in virtual environments — multimodal human–computer interaction

Modern tomography technologies like CT or MRI produce high-quality scans of the human anatomy. While conventional computer-aided image analysis falls back upon editing tomograms layer by layer, virtual environments offer enhanced visualization, image analysis and manipulation of the three-dimensional data sets. In this paper, the application of multimodal, user-oriented human–computer interaction is presented, facilitating and accelerating work with the tomographical data of individual patients. Hand gesture recognition is a major component of the interface, completed by speech understanding and further units like a 6-DOF mouse or acoustic feedback. Three-dimensional image segmentation, virtual bronchoscopy and virtual angioscopy are typical examples that illustrate the benefits of virtual environments for the realm of medicine.