Abstract:
An image is a very effective tool for conveying emotions. Many researchers have investigated emotions in images by using various features extracted from images. In this p...Show MoreMetadata
Abstract:
An image is a very effective tool for conveying emotions. Many researchers have investigated emotions in images by using various features extracted from images. In this paper, we focus on two high-level features, the object and the background, and assume that the semantic information in images is a good cue for predicting emotions. An object is one of the most important elements that define an image, and we discover through experiments that there is a high correlation between the objects and emotions in images in most cases. Even with the same object, there may be slight differences in emotion due to different backgrounds, and we use the semantic information of the background to improve the prediction performance. By combining the different levels of features, we build an emotion-based feedforward deep neural network that produces the emotion values of a given image. The output emotion values in our framework are continuous values in two-dimensional space (valence and arousal), which are more effective than using a small number of emotion categories to describe emotions. Experiments confirm the effectiveness of our network in predicting the emotions of images.
Published in: IEEE Transactions on Multimedia ( Volume: 20, Issue: 11, November 2018)
Funding Agency:

Department of Computer Science, Yonsei University, Seoul, South Korea
Hye-Rin Kim received the B.S. degree in computer engineering from
Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the
Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include
image processing, affective computing, and machine learning.
Hye-Rin Kim received the B.S. degree in computer engineering from
Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the
Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include
image processing, affective computing, and machine learning.View more

Department of Computer Science, Yonsei University, Seoul, South Korea
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science
in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D.
degree in computer science at the Department of Computer Science. His research interests include computer graphics,
image processing and machine learning.
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science
in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D.
degree in computer science at the Department of Computer Science. His research interests include computer graphics,
image processing and machine learning.View more

Department of Computer Science, Yonsei University, Seoul, South Korea
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from
Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor
with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include
computer vision, com...Show More
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from
Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor
with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include
computer vision, com...View more

Department of Computer Science, Yonsei University, Seoul, South Korea
In-Kwon Lee received the B.S. degree in computer science from Yonsei
University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from
Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a
Professor with the Department of Computer Science, Yonsei University. His research interests include computer
graphics, HCI and music...Show More
In-Kwon Lee received the B.S. degree in computer science from Yonsei
University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from
Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a
Professor with the Department of Computer Science, Yonsei University. His research interests include computer
graphics, HCI and music...View more

Department of Computer Science, Yonsei University, Seoul, South Korea
Hye-Rin Kim received the B.S. degree in computer engineering from
Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the
Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include
image processing, affective computing, and machine learning.
Hye-Rin Kim received the B.S. degree in computer engineering from
Korea Polytechnic University,Siheung, South Korea, in 2010 and the Ph.D. degree in computer science from the
Department of Computer Science, Yonsei University, Seoul, South Korea, in 2017. Her research interests include
image processing, affective computing, and machine learning.View more

Department of Computer Science, Yonsei University, Seoul, South Korea
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science
in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D.
degree in computer science at the Department of Computer Science. His research interests include computer graphics,
image processing and machine learning.
Yeong-Seok Kim received the B.S. and M.S. degrees in computer science
in 2011 and 2013, respectively, from Yonsei University, Seoul, South Korea, where he is currently working toward Ph.D.
degree in computer science at the Department of Computer Science. His research interests include computer graphics,
image processing and machine learning.View more

Department of Computer Science, Yonsei University, Seoul, South Korea
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from
Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor
with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include
computer vision, computer graphics/computational photography, and machine learning. He was an Area Chair for ICCV
2017, WACV 2015/2017. He serves regularly as the program committee members for CVPR, ECCV, and ICCV.
Seon Joo Kim (M’09) received the B.S. and M.S. degrees from
Yonsei University, Seoul, South Korea, in 1997 and 2001. He received the Ph.D. degree in computer science from the
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, in 2008. He is currently an Associate Professor
with the Department of Computer Science, Yonsei University, Seoul, South Korea. His research interests include
computer vision, computer graphics/computational photography, and machine learning. He was an Area Chair for ICCV
2017, WACV 2015/2017. He serves regularly as the program committee members for CVPR, ECCV, and ICCV.View more

Department of Computer Science, Yonsei University, Seoul, South Korea
In-Kwon Lee received the B.S. degree in computer science from Yonsei
University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from
Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a
Professor with the Department of Computer Science, Yonsei University. His research interests include computer
graphics, HCI and music technology.
In-Kwon Lee received the B.S. degree in computer science from Yonsei
University, Seoul, South Korea, in 1989, and the M.S. and Ph.D. degrees in computer science and engineering from
Pohang University of Science and Technology, Pohang, South Korea, in 1992 and 1997, respectively. He is a
Professor with the Department of Computer Science, Yonsei University. His research interests include computer
graphics, HCI and music technology.View more