Towards Immersive Human-Machine Interfaces: Sensing, Computing and Displaying

It is of great promise to achieve immersive experience with display-based human-machine interfaces (HMIs) for various end devices ranging from smart phones, PADs to various types of spatial computing devices. This paper reviews the key technologies and challenges in the three key steps of sensing, computing, and displaying for achieving immersive HMI systems. Development of various sensing technologies enables accurate recognition of input information from human. Meanwhile, sensing and reconstruction of the surrounding environment, including 3D scene and other related information, is also key. To overcome the constraints of energy and hardware, near-/in-sensor computing and edge-cloud collaborative computing can be adopted to minimize the requirement of local computational resources. As the final step, the system must be able to display a updated scene with visual and other related information in real-time for the user to perceive with immersive experience. In addition to continuous efforts on key components, system-level design and new heterogenous integration or packaging techniques for physically realizing compact “sense-compute-display” systems with less data transmission and more efficient computation are demanded.