Bayesian Modeling of Temporal Coherence in Videos for Entity Discovery and Summarization

Mitra, Adway and Biswas, Soma and Bhattacharyya, Chiranjib (2017) Bayesian Modeling of Temporal Coherence in Videos for Entity Discovery and Summarization. In: IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 39 (3). pp. 430-443.

PDF IEE_Tra_Pat_Ana_Mac_Int_39-3_430_2017.pdf - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

A video is understood by users in terms of entities present in it. Entity Discovery is the task of building appearance model for each entity (e. g., a person), and finding all its occurrences in the video. We represent a video as a sequence of tracklets, each spanning 10-20 frames, and associated with one entity. We pose Entity Discovery as tracklet clustering, and approach it by leveraging Temporal Coherence (TC): the property that temporally neighboring tracklets are likely to be associated with the same entity. Our major contributions are the first Bayesian nonparametric models for TC at tracklet-level. We extend Chinese Restaurant Process (CRP) to TC-CRP, and further to Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to jointly model entities and temporal segments using mixture components and sparse distributions. For discovering persons in TV serial videos without meta-data like scripts, these methods show considerable improvement over state-of-the-art approaches to tracklet clustering in terms of clustering accuracy, cluster purity and entity coverage. The proposed methods can perform online tracklet clustering on streaming videos unlike existing approaches, and can automatically reject false tracklets. Finally we discuss entity-driven video summarization-where temporal segments of the video are selected based on the discovered entities, to create a semantically meaningful summary.

Item Type:	Journal Article
Publication:	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
Additional Information:	Copy right for this article belongs to the IEEE COMPUTER SOC, 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1314 USA
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation Division of Electrical Sciences > Electrical Engineering
Date Deposited:	03 Apr 2017 04:37
Last Modified:	03 Oct 2018 15:20
URI:	http://eprints.iisc.ac.in/id/eprint/56437

Actions (login required)

View Item