Please use this identifier to cite or link to this item: https://doi.org/10.1007/978-3-030-58539-6_27
Title: Visual Relation Grounding in Videos
Authors: Junbin Xiao
Xindi Shang 
Xun Yang 
Sheng Tang
Tat-Seng Chua 
Issue Date: 7-Nov-2020
Publisher: Springer Science and Business Media Deutschland GmbH
Citation: Junbin Xiao, Xindi Shang, Xun Yang, Sheng Tang, Tat-Seng Chua (2020-11-07). Visual Relation Grounding in Videos. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12351 LNCS : 447-464. ScholarBank@NUS Repository. https://doi.org/10.1007/978-3-030-58539-6_27
Abstract: In this paper, we explore a novel task named visual Relation Grounding in Videos (vRGV). The task aims at spatio-temporally localizing the given relations in the form of subject-predicate-object in the videos, so as to provide supportive visual facts for other high-level video-language tasks (e.g., video-language grounding and video question answering). The challenges in this task include but not limited to: (1) both the subject and object are required to be spatio-temporally localized to ground a query relation; (2) the temporal dynamic nature of visual relations in videos is difficult to capture; and (3) the grounding should be achieved without any direct supervision in space and time. To ground the relations, we tackle the challenges by collaboratively optimizing two sequences of regions over a constructed hierarchical spatio-temporal region graph through relation attending and reconstruction, in which we further propose a message passing mechanism by spatial attention shifting between visual entities. Experimental results demonstrate that our model can not only outperform baseline approaches significantly, but also produces visually meaningful facts to support visual grounding. (Code is available at https://github.com/doc-doc/vRGV). © 2020, Springer Nature Switzerland AG.
Source Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
URI: https://scholarbank.nus.edu.sg/handle/10635/190947
ISBN: 9783030585389
ISSN: 0302-9743
DOI: 10.1007/978-3-030-58539-6_27
Appears in Collections:Staff Publications
Elements

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
Visual Relation Grounding in Videos.pdf3.35 MBAdobe PDF

CLOSED

None

Page view(s)

21
checked on Jun 17, 2021

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.