Loading [MathJax]/extensions/MathMenu.js
Example-Based Facial Animation of Virtual Reality Avatars Using Auto-Regressive Neural Networks | IEEE Journals & Magazine | IEEE Xplore

Example-Based Facial Animation of Virtual Reality Avatars Using Auto-Regressive Neural Networks


Abstract:

This article presents a hybrid animation approach that combines example-based and neural animation methods to create a simple, yet powerful animation regime for human fac...Show More

Abstract:

This article presents a hybrid animation approach that combines example-based and neural animation methods to create a simple, yet powerful animation regime for human faces. Example-based methods usually employ a database of prerecorded sequences that are concatenated or looped in order to synthesize novel animations. In contrast to this traditional example-based approach, we introduce a light-weight auto-regressive network to transform our animation-database into a parametric model. During training, our network learns the dynamics of facial expressions, which enables the replay of annotated sequences from our animation database as well as their seamless concatenation in new order. This representation is especially useful for the synthesis of visual speech, where coarticulation creates interdependencies between adjacent visemes, which affects their appearance. Instead of creating an exhaustive database that contains all viseme variants, we use our animation-network to predict the correct appearance. This allows realistic synthesis of novel facial animation sequences like visual-speech but also general facial expressions in an example-based manner.
Published in: IEEE Computer Graphics and Applications ( Volume: 41, Issue: 4, 01 July-Aug. 2021)
Page(s): 52 - 63
Date of Publication: 23 March 2021

ISSN Information:

PubMed ID: 33755560

Funding Agency:

Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Wolfgang Paier is currently a Research Associate with Fraunhofer HHI, Berlin, Germany. His research interests include 3-D scene analysis, and image- as well as learning-based approaches for facial performance capture, animation, and editing. He received the B.S. degree and the M.Sc. degree in computer science from the Technical University of Graz, Graz, Austria, in 2009 and the Free University of Berlin, Berlin, Germany, ...Show More
Wolfgang Paier is currently a Research Associate with Fraunhofer HHI, Berlin, Germany. His research interests include 3-D scene analysis, and image- as well as learning-based approaches for facial performance capture, animation, and editing. He received the B.S. degree and the M.Sc. degree in computer science from the Technical University of Graz, Graz, Austria, in 2009 and the Free University of Berlin, Berlin, Germany, ...View more
Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Anna Hilsmann is currently heading the Computer Vision & Graphics Group, Fraunhofer HHI, Berlin, Germany. Her main research interests cover 3-D image and video analysis, model-based deformable tracking and 3-D reconstruction, as well as image- and video-based rendering, animation, and editing. She received the Dipl.-Ing. degree in electrical engineering and information technology from RWTH Aachen, Aachen, Germany, in 2006...Show More
Anna Hilsmann is currently heading the Computer Vision & Graphics Group, Fraunhofer HHI, Berlin, Germany. Her main research interests cover 3-D image and video analysis, model-based deformable tracking and 3-D reconstruction, as well as image- and video-based rendering, animation, and editing. She received the Dipl.-Ing. degree in electrical engineering and information technology from RWTH Aachen, Aachen, Germany, in 2006...View more
Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Peter Eisert is currently a Professor of visual computing with Humboldt University, Berlin, Germany and heads the Vision & Imaging Technologies Department, Fraunhofer HHI, Berlin, Germany. He has authored or coauthored more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as on the editorial board of the Journal of Visual Communication and...Show More
Peter Eisert is currently a Professor of visual computing with Humboldt University, Berlin, Germany and heads the Vision & Imaging Technologies Department, Fraunhofer HHI, Berlin, Germany. He has authored or coauthored more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as on the editorial board of the Journal of Visual Communication and...View more

Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Wolfgang Paier is currently a Research Associate with Fraunhofer HHI, Berlin, Germany. His research interests include 3-D scene analysis, and image- as well as learning-based approaches for facial performance capture, animation, and editing. He received the B.S. degree and the M.Sc. degree in computer science from the Technical University of Graz, Graz, Austria, in 2009 and the Free University of Berlin, Berlin, Germany, in 2013, respectively. He is the corresponding author of this article. Contact him at wolfgang.paier@hhi.fraunhofer.de
Wolfgang Paier is currently a Research Associate with Fraunhofer HHI, Berlin, Germany. His research interests include 3-D scene analysis, and image- as well as learning-based approaches for facial performance capture, animation, and editing. He received the B.S. degree and the M.Sc. degree in computer science from the Technical University of Graz, Graz, Austria, in 2009 and the Free University of Berlin, Berlin, Germany, in 2013, respectively. He is the corresponding author of this article. Contact him at wolfgang.paier@hhi.fraunhofer.deView more
Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Anna Hilsmann is currently heading the Computer Vision & Graphics Group, Fraunhofer HHI, Berlin, Germany. Her main research interests cover 3-D image and video analysis, model-based deformable tracking and 3-D reconstruction, as well as image- and video-based rendering, animation, and editing. She received the Dipl.-Ing. degree in electrical engineering and information technology from RWTH Aachen, Aachen, Germany, in 2006 and the Dr.-Ing. degree in computer science from HU Berlin, Berlin, Germany, in 2014. Contact her at anna.hilsmann@hhi.fraunhofer.de.
Anna Hilsmann is currently heading the Computer Vision & Graphics Group, Fraunhofer HHI, Berlin, Germany. Her main research interests cover 3-D image and video analysis, model-based deformable tracking and 3-D reconstruction, as well as image- and video-based rendering, animation, and editing. She received the Dipl.-Ing. degree in electrical engineering and information technology from RWTH Aachen, Aachen, Germany, in 2006 and the Dr.-Ing. degree in computer science from HU Berlin, Berlin, Germany, in 2014. Contact her at anna.hilsmann@hhi.fraunhofer.de.View more
Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut HHI, Berlin, Germany
Peter Eisert is currently a Professor of visual computing with Humboldt University, Berlin, Germany and heads the Vision & Imaging Technologies Department, Fraunhofer HHI, Berlin, Germany. He has authored or coauthored more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as on the editorial board of the Journal of Visual Communication and Image Representation. His research interests include 3-D image analysis and synthesis, face processing, image-based rendering, deep learning, computer vision, and computer graphics. He received the Dr.-Ing. degree from the University Erlangen, Erlangen, Germany, in 2000 and was a Postdoctoral Fellow with Stanford University. Contact him at peter.eisert@hhi.fraunhofer.de.
Peter Eisert is currently a Professor of visual computing with Humboldt University, Berlin, Germany and heads the Vision & Imaging Technologies Department, Fraunhofer HHI, Berlin, Germany. He has authored or coauthored more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as on the editorial board of the Journal of Visual Communication and Image Representation. His research interests include 3-D image analysis and synthesis, face processing, image-based rendering, deep learning, computer vision, and computer graphics. He received the Dr.-Ing. degree from the University Erlangen, Erlangen, Germany, in 2000 and was a Postdoctoral Fellow with Stanford University. Contact him at peter.eisert@hhi.fraunhofer.de.View more

References

References is not available for this document.