Please use this identifier to cite or link to this item: https://scholarbank.nus.edu.sg/handle/10635/178622
Title: Who You Are Decides How You Tell
Authors: WU SHUANG 
FAN SHAOJING 
SHEN ZHIQI 
KANKANHALLI MOHAN S 
TUNG KUM HOE,ANTHONY 
Keywords: Image captioning
language and vision
multi-modal
human-centered
deep learning
Issue Date: 12-Oct-2020
Citation: WU SHUANG, FAN SHAOJING, SHEN ZHIQI, KANKANHALLI MOHAN S, TUNG KUM HOE,ANTHONY (2020-10-12). Who You Are Decides How You Tell. ScholarBank@NUS Repository.
Rights: CC0 1.0 Universal
Abstract: Image captioning is gaining significance in multiple applications such as content-based visual search and chat-bots. Much of the recent progress in this field embraces a data-driven approach without deep consideration of human behavioural characteristics. In this paper, we focus on human-centered automatic image captioning. Our study is based on the intuition that different people will generate a variety of image captions for the same scene, as their knowledge and opinion about the scene may differ. In particular, we first perform a series of human studies to investigate what influences human description of a visual scene. We identify three main factors: a person’s knowledge level of the scene, opinion on the scene, and gender. Based on our human study findings, we propose a novel human-centered algorithm that is able to generate human-like image captions. We evaluate the proposed model through traditional evaluation metrics, diversity metrics, and human-based evaluation. Experimental results demonstrate the superiority of our proposed model on generating diverse human-like image captions.
URI: https://scholarbank.nus.edu.sg/handle/10635/178622
Rights: CC0 1.0 Universal
Appears in Collections:Staff Publications
Elements

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
ACMMM2020_Who_You_Are_Decides_How_You_Tell.pdf2.55 MBAdobe PDF

OPEN

NoneView/Download

Page view(s)

213
checked on Aug 4, 2022

Download(s)

11
checked on Aug 4, 2022

Google ScholarTM

Check


This item is licensed under a Creative Commons License Creative Commons