A single video super-resolution GAN for multiple downsampling operators based on pseudo-inverse image formation models

Santiago López-Tapia*, Alice Lucas, Rafael Molina, Aggelos K. Katsaggelos

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

The popularity of high and ultra-high definition displays has led to the need for methods to improve the quality of videos already obtained at much lower resolutions. A large amount of current CNN-based Video Super-Resolution methods are designed and trained to handle a specific degradation operator (e.g., bicubic downsampling) and are not robust to mismatch between training and testing degradation models. This causes their performance to deteriorate in real-life applications. Furthermore, many of them use the Mean-Squared-Error as the only loss during learning, causing the resulting images to be too smooth. In this work we propose a new Convolutional Neural Network for video super resolution which is robust to multiple degradation models. During training, which is performed on a large dataset of scenes with slow and fast motions, it uses the pseudo-inverse image formation model as part of the network architecture in conjunction with perceptual losses and a smoothness constraint that eliminates the artifacts originating from these perceptual losses. The experimental validation shows that our approach outperforms current state-of-the-art methods and is robust to multiple degradations.

Original languageEnglish (US)
Article number102801
JournalDigital Signal Processing: A Review Journal
Volume104
DOIs
StatePublished - Sep 2020

Keywords

  • Convolutional neuronal networks
  • Generative adversarial networks
  • Perceptual loss functions
  • Super-resolution
  • Video

ASJC Scopus subject areas

  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Statistics, Probability and Uncertainty
  • Computational Theory and Mathematics
  • Electrical and Electronic Engineering
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint Dive into the research topics of 'A single video super-resolution GAN for multiple downsampling operators based on pseudo-inverse image formation models'. Together they form a unique fingerprint.

Cite this