ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Understanding the perceived quality of video predictions

Somraj, N and Kashi, MS and Arun, SP and Soundararajan, R (2022) Understanding the perceived quality of video predictions. In: Signal Processing: Image Communication, 102 .

sig_pro_2022.pdf - Published Version

Download (2MB) | Preview
Official URL: https://doi.org/10.1016/j.image.2021.116626


The study of video prediction models is believed to be a fundamental approach to representation learning for videos. While a plethora of generative models for predicting the future frame pixel values given the past few frames exist, the quantitative evaluation of the predicted frames has been found to be extremely challenging. In this context, we study the problem of quality assessment of predicted videos. We create the Indian Institute of Science Predicted Videos Quality Assessment (IISc PVQA) Database consisting of 300 videos, obtained by applying different prediction models on different datasets, and accompanying human opinion scores. We collected subjective ratings of quality from 50 human participants for these videos. Our subjective study reveals that human observers were highly consistent in their judgments of quality of predicted videos. We benchmark several popularly used measures for evaluating video prediction and show that they do not adequately correlate with these subjective scores. We introduce two new features to effectively capture the quality of predicted videos, motion-compensated cosine similarities of deep features of predicted frames with past frames, and deep features extracted from rescaled frame differences. We show that our feature design leads to state-of-the-art quality prediction in accordance with human judgments on our IISc PVQA Database. The database and code are publicly available on our project website: https://nagabhushansn95.github.io/publications/2020/pvqa. © 2021 Elsevier B.V.

Item Type: Journal Article
Publication: Signal Processing: Image Communication
Publisher: Elsevier B.V.
Additional Information: The copyright for this article belongs to Elsevier B.V.
Keywords: Deep neural networks; Forecasting, Deep learning; Indian institute of science; Neural-networks; Perceived quality; Perceptual quality; Prediction modelling; Quality assessment; Video prediction; Video quality; Video quality assessment, Database systems
Department/Centre: Division of Biological Sciences > Centre for Neuroscience
Division of Electrical Sciences > Electrical Communication Engineering
Date Deposited: 10 Feb 2022 11:50
Last Modified: 10 Feb 2022 11:50
URI: http://eprints.iisc.ac.in/id/eprint/71151

Actions (login required)

View Item View Item