Proposed Shape-from-Template method architecture. The inputs of the networks are coloured in blue, the learning-based algorithms are in green meanwhile the non-learning o...
Abstract:
We propose WS-DeepSfT, a novel deep learning-based approach to the Shape-from-Template (SfT) problem, which aims at reconstructing the 3D shape of a deformable object fro...Show MoreMetadata
Abstract:
We propose WS-DeepSfT, a novel deep learning-based approach to the Shape-from-Template (SfT) problem, which aims at reconstructing the 3D shape of a deformable object from a single RGB image and a template. WS-DeepSfT addresses the limitations of existing SfT techniques by combining a weakly-supervised deep neural network (DNN) for registration and a classical As-Rigid-As-Possible (ARAP) algorithm for 3D reconstruction. Unlike previous deep learning-based SfT methods, which require extensive synthetic data and depth sensors for training, WS-DeepSfT only requires regular RGB video of the deforming object and a segmentation mask to discriminate the object from the background. The registration model is trained without synthetic data, using videos where the object undergoes deformations, while ARAP does not require training and infers the 3D shape in real-time with minimal overhead. We show that WS-DeepSfT outperforms the state-of-the-art, in both accuracy and robustness, without requiring depth sensors or synthetic data generation. WS-DeepSfT thus offers a robust, efficient, and scalable approach to SfT, bringing it closer to applications such as augmented reality.
Proposed Shape-from-Template method architecture. The inputs of the networks are coloured in blue, the learning-based algorithms are in green meanwhile the non-learning o...
Published in: IEEE Access ( Volume: 13)
Funding Agency:
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Sara Luengo-Sanchez received the B.S. and M.S. degrees in telecommunications engineering from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2018 and 2020, respectively, where she is currently pursuing the Ph.D. degree in artificial intelligence.
She has been a member with the GEINTRA Research Group, since 2016. She held the University Professor Training Scholarship (FPU) from the Spanish Government, from 2...Show More
Sara Luengo-Sanchez received the B.S. and M.S. degrees in telecommunications engineering from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2018 and 2020, respectively, where she is currently pursuing the Ph.D. degree in artificial intelligence.
She has been a member with the GEINTRA Research Group, since 2016. She held the University Professor Training Scholarship (FPU) from the Spanish Government, from 2...View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
David Fuentes-Jimenez received the B.S. degree in electronics engineering, the M.S. degree in electrical engineering, and the Ph.D. degree in artificial intelligence from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2016, 2018, and 2021, respectively.
He was a Research Scientist with Natural Vox, S.A.U., from 2021 to 2023, specializing in face and voice biometry. Since 2023, he has been an Assistant Profe...Show More
David Fuentes-Jimenez received the B.S. degree in electronics engineering, the M.S. degree in electrical engineering, and the Ph.D. degree in artificial intelligence from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2016, 2018, and 2021, respectively.
He was a Research Scientist with Natural Vox, S.A.U., from 2021 to 2023, specializing in face and voice biometry. Since 2023, he has been an Assistant Profe...View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Cristina Losada-Gutierrez received the B.S. degree in telecommunications engineering and the Ph.D. degree in advanced intelligent systems electronics from the University of Alcalá (UAH), Alcala de Henares, Madrid, Spain, in 2004 and 2010, respectively.
Since 2011, she has been a Professor with the Electronic Department, University of Alcalá, and a member with the GEINTRA Research Group. She has been the Assistant Director ...Show More
Cristina Losada-Gutierrez received the B.S. degree in telecommunications engineering and the Ph.D. degree in advanced intelligent systems electronics from the University of Alcalá (UAH), Alcala de Henares, Madrid, Spain, in 2004 and 2010, respectively.
Since 2011, she has been a Professor with the Electronic Department, University of Alcalá, and a member with the GEINTRA Research Group. She has been the Assistant Director ...View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Daniel Pizarro received the B.S. and Ph.D. degrees from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2003 and 2008, respectively.
He was an Assistant Professor and a member with the GEINTRA Research Group, Department of Electronics, University of Alcalá, from 2005 to 2012. From 2012 to 2014, he joined the Advanced Laparoscopy and Computer Vision Group (AlCoV) and was an Associate Professor with the Univer...Show More
Daniel Pizarro received the B.S. and Ph.D. degrees from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2003 and 2008, respectively.
He was an Assistant Professor and a member with the GEINTRA Research Group, Department of Electronics, University of Alcalá, from 2005 to 2012. From 2012 to 2014, he joined the Advanced Laparoscopy and Computer Vision Group (AlCoV) and was an Associate Professor with the Univer...View more
EnCoV, Clermont-Ferrand, France
Adrien Bartoli received the Ph.D. degree from the Perception Group, Inria Grenoble, Grenoble, France, in 2003, and the Habilitation (H.D.R.) degree from Université Blaise Pascal, Clermont-Ferrand, France, in 2008.
He started as a Postdoctoral Researcher with the Visual Geometry Group, University of Oxford, under Andrew Zisserman, in 2004. He joined the CNRS Research Scientist of Institut Pascal, in 2004, where he led ComSe...Show More
Adrien Bartoli received the Ph.D. degree from the Perception Group, Inria Grenoble, Grenoble, France, in 2003, and the Habilitation (H.D.R.) degree from Université Blaise Pascal, Clermont-Ferrand, France, in 2008.
He started as a Postdoctoral Researcher with the Visual Geometry Group, University of Oxford, under Andrew Zisserman, in 2004. He joined the CNRS Research Scientist of Institut Pascal, in 2004, where he led ComSe...View more
Contents Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Sara Luengo-Sanchez received the B.S. and M.S. degrees in telecommunications engineering from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2018 and 2020, respectively, where she is currently pursuing the Ph.D. degree in artificial intelligence.
She has been a member with the GEINTRA Research Group, since 2016. She held the University Professor Training Scholarship (FPU) from the Spanish Government, from 2019 until 2023. Her research interests include computer vision, people detection and re-identification, image registration, and 3D object reconstruction.
Ms. Luengo-Sanchez was awarded an excellence prize for her studies by the Madrid Regional Government, in 2016. She received the José Ramón López Villares and the CEI Intelligent Energy Prices, in 2019. She was granted the IBM prize for the best academic record from the Official College of Telecommunication Engineers (COIT) and the Spanish Association of Telecommunication Engineers (AEIT), in 2021.
Sara Luengo-Sanchez received the B.S. and M.S. degrees in telecommunications engineering from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2018 and 2020, respectively, where she is currently pursuing the Ph.D. degree in artificial intelligence.
She has been a member with the GEINTRA Research Group, since 2016. She held the University Professor Training Scholarship (FPU) from the Spanish Government, from 2019 until 2023. Her research interests include computer vision, people detection and re-identification, image registration, and 3D object reconstruction.
Ms. Luengo-Sanchez was awarded an excellence prize for her studies by the Madrid Regional Government, in 2016. She received the José Ramón López Villares and the CEI Intelligent Energy Prices, in 2019. She was granted the IBM prize for the best academic record from the Official College of Telecommunication Engineers (COIT) and the Spanish Association of Telecommunication Engineers (AEIT), in 2021.View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
David Fuentes-Jimenez received the B.S. degree in electronics engineering, the M.S. degree in electrical engineering, and the Ph.D. degree in artificial intelligence from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2016, 2018, and 2021, respectively.
He was a Research Scientist with Natural Vox, S.A.U., from 2021 to 2023, specializing in face and voice biometry. Since 2023, he has been an Assistant Professor with the University of Alcalá and a member with the GEINTRA Research Group. He held the Researcher Training Scholarship (FPI) from the University of Alcalá, from 2018 until 2021. His research interests include computer vision, 3D object reconstruction, and people detection with depth cameras.
Dr. Fuentes-Jimenez has been a Nova Talent Member, since 2022. He received the CEI Intelligent Energy, in 2020.
David Fuentes-Jimenez received the B.S. degree in electronics engineering, the M.S. degree in electrical engineering, and the Ph.D. degree in artificial intelligence from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2016, 2018, and 2021, respectively.
He was a Research Scientist with Natural Vox, S.A.U., from 2021 to 2023, specializing in face and voice biometry. Since 2023, he has been an Assistant Professor with the University of Alcalá and a member with the GEINTRA Research Group. He held the Researcher Training Scholarship (FPI) from the University of Alcalá, from 2018 until 2021. His research interests include computer vision, 3D object reconstruction, and people detection with depth cameras.
Dr. Fuentes-Jimenez has been a Nova Talent Member, since 2022. He received the CEI Intelligent Energy, in 2020.View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Cristina Losada-Gutierrez received the B.S. degree in telecommunications engineering and the Ph.D. degree in advanced intelligent systems electronics from the University of Alcalá (UAH), Alcala de Henares, Madrid, Spain, in 2004 and 2010, respectively.
Since 2011, she has been a Professor with the Electronic Department, University of Alcalá, and a member with the GEINTRA Research Group. She has been the Assistant Director of the Electronic Department, since 2022. Her research interests include computer vision and sensor systems applied to intelligent spaces and human behavior analysis.
Cristina Losada-Gutierrez received the B.S. degree in telecommunications engineering and the Ph.D. degree in advanced intelligent systems electronics from the University of Alcalá (UAH), Alcala de Henares, Madrid, Spain, in 2004 and 2010, respectively.
Since 2011, she has been a Professor with the Electronic Department, University of Alcalá, and a member with the GEINTRA Research Group. She has been the Assistant Director of the Electronic Department, since 2022. Her research interests include computer vision and sensor systems applied to intelligent spaces and human behavior analysis.View more
Department of Electronics, Universidad de Alcalá (UAH), Alcala de Henares, Spain
Daniel Pizarro received the B.S. and Ph.D. degrees from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2003 and 2008, respectively.
He was an Assistant Professor and a member with the GEINTRA Research Group, Department of Electronics, University of Alcalá, from 2005 to 2012. From 2012 to 2014, he joined the Advanced Laparoscopy and Computer Vision Group (AlCoV) and was an Associate Professor with the Université d’Auvergne, Clermont-Ferrand, France. Since 2015, he has been a Professor with the University of Alcalá and a member with the GEINTRA Research Group, which has been leading, since 2023. He is currently an Associate Member with the Endoscopy and Computer Vision (ENCoV) Research Group. His research interests include image registration, rigid and non-rigid reconstruction, and its applications to minimally invasive surgery.
Dr. Pizarro was awarded the Young Researcher Prize in the area of experimental sciences, in 2015, and the prize for excellence in Ph.D. supervision from the University of Alcalá, in 2018.
Daniel Pizarro received the B.S. and Ph.D. degrees from the University of Alcalá, Alcala de Henares, Madrid, Spain, in 2003 and 2008, respectively.
He was an Assistant Professor and a member with the GEINTRA Research Group, Department of Electronics, University of Alcalá, from 2005 to 2012. From 2012 to 2014, he joined the Advanced Laparoscopy and Computer Vision Group (AlCoV) and was an Associate Professor with the Université d’Auvergne, Clermont-Ferrand, France. Since 2015, he has been a Professor with the University of Alcalá and a member with the GEINTRA Research Group, which has been leading, since 2023. He is currently an Associate Member with the Endoscopy and Computer Vision (ENCoV) Research Group. His research interests include image registration, rigid and non-rigid reconstruction, and its applications to minimally invasive surgery.
Dr. Pizarro was awarded the Young Researcher Prize in the area of experimental sciences, in 2015, and the prize for excellence in Ph.D. supervision from the University of Alcalá, in 2018.View more
EnCoV, Clermont-Ferrand, France
Adrien Bartoli received the Ph.D. degree from the Perception Group, Inria Grenoble, Grenoble, France, in 2003, and the Habilitation (H.D.R.) degree from Université Blaise Pascal, Clermont-Ferrand, France, in 2008.
He started as a Postdoctoral Researcher with the Visual Geometry Group, University of Oxford, under Andrew Zisserman, in 2004. He joined the CNRS Research Scientist of Institut Pascal, in 2004, where he led ComSee, the Computer Vision Research Group. He stayed as a Visiting Professor with DIKU, University of Copenhagen, Copenhagen, Denmark, from 2006 to 2009. He has been a Professor of computer science with Université Clermont Auvergne, Clermont-Ferrand, since 2009. He is currently the Chief Scientific Officer of SurgAR and leads the Endoscopy and Computer Vision (ENCoV) Research Group. His research interests include image registration and Shape-from-X in rigid and non-rigid environments, and its applications to computer-aided medical interventions, particularly endoscopy.
Dr. Bartoli has been a member of the Institut Universitaire de France, since 2016. He received the Grenoble-INP Ph.D. Thesis Prize, in 2004, the CNRS Médaille de Bronze, in 2008, and the Research Prize from Université d’Auvergne, in 2016. He held an ERC Consolidator Grant, from 2013 to 2018, and an ERC Proof-of-Concept Grant, from 2018 to 2019. He is on the editorial board of International Journal of Computer Vision (IJCV) and Journal of Artificial Intelligence Research (JAIR).
Adrien Bartoli received the Ph.D. degree from the Perception Group, Inria Grenoble, Grenoble, France, in 2003, and the Habilitation (H.D.R.) degree from Université Blaise Pascal, Clermont-Ferrand, France, in 2008.
He started as a Postdoctoral Researcher with the Visual Geometry Group, University of Oxford, under Andrew Zisserman, in 2004. He joined the CNRS Research Scientist of Institut Pascal, in 2004, where he led ComSee, the Computer Vision Research Group. He stayed as a Visiting Professor with DIKU, University of Copenhagen, Copenhagen, Denmark, from 2006 to 2009. He has been a Professor of computer science with Université Clermont Auvergne, Clermont-Ferrand, since 2009. He is currently the Chief Scientific Officer of SurgAR and leads the Endoscopy and Computer Vision (ENCoV) Research Group. His research interests include image registration and Shape-from-X in rigid and non-rigid environments, and its applications to computer-aided medical interventions, particularly endoscopy.
Dr. Bartoli has been a member of the Institut Universitaire de France, since 2016. He received the Grenoble-INP Ph.D. Thesis Prize, in 2004, the CNRS Médaille de Bronze, in 2008, and the Research Prize from Université d’Auvergne, in 2016. He held an ERC Consolidator Grant, from 2013 to 2018, and an ERC Proof-of-Concept Grant, from 2018 to 2019. He is on the editorial board of International Journal of Computer Vision (IJCV) and Journal of Artificial Intelligence Research (JAIR).View more