Loading [MathJax]/extensions/MathMenu.js
Building Emotional Machines: Recognizing Image Emotions Through Deep Neural Networks | IEEE Journals & Magazine | IEEE Xplore

Building Emotional Machines: Recognizing Image Emotions Through Deep Neural Networks


Abstract:

An image is a very effective tool for conveying emotions. Many researchers have investigated emotions in images by using various features extracted from images. In this p...Show More

Abstract:

An image is a very effective tool for conveying emotions. Many researchers have investigated emotions in images by using various features extracted from images. In this paper, we focus on two high-level features, the object and the background, and assume that the semantic information in images is a good cue for predicting emotions. An object is one of the most important elements that define an image, and we discover through experiments that there is a high correlation between the objects and emotions in images in most cases. Even with the same object, there may be slight differences in emotion due to different backgrounds, and we use the semantic information of the background to improve the prediction performance. By combining the different levels of features, we build an emotion-based feedforward deep neural network that produces the emotion values of a given image. The output emotion values in our framework are continuous values in two-dimensional space (valence and arousal), which are more effective than using a small number of emotion categories to describe emotions. Experiments confirm the effectiveness of our network in predicting the emotions of images.
Published in: IEEE Transactions on Multimedia ( Volume: 20, Issue: 11, November 2018)
Page(s): 2980 - 2992
Date of Publication: 20 April 2018

ISSN Information:

Funding Agency:

Author image of Hye-Rin Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Hye-Rin Kim received the B.S. degree in computer engineering from Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include image processing, affective computing, and machine learning.
Hye-Rin Kim received the B.S. degree in computer engineering from Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include image processing, affective computing, and machine learning.View more
Author image of Yeong-Seok Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D. degree in computer science at the Department of Computer Science. His research interests include computer graphics, image processing and machine learning.
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D. degree in computer science at the Department of Computer Science. His research interests include computer graphics, image processing and machine learning.View more
Author image of Seon Joo Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include computer vision, com...Show More
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include computer vision, com...View more
Author image of In-Kwon Lee
Department of Computer Science, Yonsei University, Seoul, South Korea
In-Kwon Lee received the B.S. degree in computer science from Yonsei University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a Professor with the Department of Computer Science, Yonsei University. His research interests include computer graphics, HCI and music...Show More
In-Kwon Lee received the B.S. degree in computer science from Yonsei University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a Professor with the Department of Computer Science, Yonsei University. His research interests include computer graphics, HCI and music...View more

Author image of Hye-Rin Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Hye-Rin Kim received the B.S. degree in computer engineering from Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include image processing, affective computing, and machine learning.
Hye-Rin Kim received the B.S. degree in computer engineering from Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include image processing, affective computing, and machine learning.View more
Author image of Yeong-Seok Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D. degree in computer science at the Department of Computer Science. His research interests include computer graphics, image processing and machine learning.
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D. degree in computer science at the Department of Computer Science. His research interests include computer graphics, image processing and machine learning.View more
Author image of Seon Joo Kim
Department of Computer Science, Yonsei University, Seoul, South Korea
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include computer vision, computer graphics/computational photography, and machine learning. He was an Area Chair for ICCV 2017, WACV 2015/2017. He serves regularly as the program committee members for CVPR, ECCV, and ICCV.
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include computer vision, computer graphics/computational photography, and machine learning. He was an Area Chair for ICCV 2017, WACV 2015/2017. He serves regularly as the program committee members for CVPR, ECCV, and ICCV.View more
Author image of In-Kwon Lee
Department of Computer Science, Yonsei University, Seoul, South Korea
In-Kwon Lee received the B.S. degree in computer science from Yonsei University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a Professor with the Department of Computer Science, Yonsei University. His research interests include computer graphics, HCI and music technology.
In-Kwon Lee received the B.S. degree in computer science from Yonsei University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a Professor with the Department of Computer Science, Yonsei University. His research interests include computer graphics, HCI and music technology.View more

Contact IEEE to Subscribe

References

References is not available for this document.