IEEE Journal of Selected Topics in Signal Processing

Structured light techniques are widely used for depth sensing. In this paper, we propose a single shot dual-frequency structured light based method to achieve dense depth in dynamic scenes. The projected pattern is a mixture of two different periodical waves whose phases are related to the change of color and intensity respectively, which can avoid the requirement of Fourier spectra separation in ... View full abstract»

Phase shifting profilometry (PSP) and Fourier transform profilometry (FTP) are two well-known fringe analysis methods for 3D sensing. PSP offers high accuracy but requires multiple images; FTP uses a single image but is limited in its accuracy. In this paper, we propose a novel Fourier-assisted phase shifting (FAPS) method for accurate dynamic 3D sensing. Our key observation is that the motion vul... View full abstract»

3D modeling of building architecture from mobile scanning is a rapidly advancing field. These models are used in virtual reality, gaming, navigation, and simulation applications. State-of-the-art scanning produces accurate point-clouds of building interiors containing hundreds of millions of points. This paper presents several scalable surface reconstruction techniques to generate watertight meshe... View full abstract»

An adaptive range image coding algorithm for the geometry compression of large-scale 3D point clouds (LS3DPCs) is proposed in this work. A terrestrial laser scanner generates an LS3DPC by measuring the radial distances of objects in a real world scene, which can be mapped into a range image. In general, the range image exhibits different characteristics from an ordinary luminance or color image, a... View full abstract»

An inference-based multiview depth image enhancement algorithm is introduced and investigated in this paper. Multiview depth imagery plays a pivotal role in free-viewpoint television. This technology requires high-quality virtual view synthesis to enable viewers to move freely in a dynamic real world scene. Depth imagery of different viewpoints is used to synthesize an arbitrary number of novel vi... View full abstract»

Foreground extraction from video stream is an important component in many multimedia applications. By exploiting commodity RGBD cameras, we could further extract dynamic foreground objects with 3D information in real-time, thereby enabling new forms of multimedia applications such as 3D telepresence. However, one critical problem with existing methods for real-time foreground extraction is tempora... View full abstract»

Applications involving indirect interpersonal communication, such as collaborative design/assembly/exploration of physical objects, can benefit strongly from the transmission of contact-based haptic media, in addition to the more traditional audiovisual media. Inclusion of haptic media has been shown to improve immersiveness, task performance, and the overall experience of task execution. While se... View full abstract»

We study the scenario of multicasting multi-view video content, recorded in the video plus depth format, to a collection of heterogeneous clients featuring Internet access links of diverse packet loss and transmission bandwidth values. We design a popularity-aware joint source-channel coding optimization framework that allocates source and channel coding rates to the captured content, such that th... View full abstract»

Several multiview video coding standards have been developed to efficiently compress images from different camera views capturing the same scene by exploiting the spatial, the temporal and the interview correlations. However, the compressed texture and depth data have typically many interview coding dependencies, which may not suit interactive multiview video streaming (IMVS) systems, where the us... View full abstract»

We study forward error correction codes for low-delay, real-time streaming communication over packet erasure channels. Our encoder operates on a stream of source packets in a sequential fashion, and the decoder must output each packet in the source stream within a fixed delay. We consider a class of practical channel models with correlated erasures and introduce new “streaming codes”... View full abstract»

With the arrival of auto-stereoscopic 3D displays for mobile devices, and emergence of more 3D content, there is much anticipation for 3D mobile multimedia experiences, including 3D display gaming. Simultaneously, with the emergence of cloud computing, more mobile applications are being developed to take advantage of the elastic cloud resources. In this paper, we explore the possibility of develop... View full abstract»

One of the most challenging ongoing issues in the field of 3D visual research is how to interpret human 3D perception over virtual 3D space between the human eye and a 3D display. When a human being perceives a 3D structure, the brain classifies the scene into the binocular or monocular vision region depending on the availability of binocular depth perception in the unit of a certain region (coars... View full abstract»

This paper presents an immersive telepresence system for entertainment and meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as ea... View full abstract»

Supporting a wide set of linked non-verbal resources remains an evergreen challenge for communication technology, limiting effectiveness in many applications. Interpersonal distance, gaze, posture and facial expression, are interpreted together to manage and add meaning to most conversations. Yet today's technologies favor some above others. This induces confusion in conversations, and is believed... View full abstract»