Semantic prior based generative adversarial network for video super-resolution

Xinyi Wu, Alice Lucas, Santiago Lopez-Tapia, Xijun Wang, Yul Hee Kim, Rafael Molina, Aggelos K. Katsaggelos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Semantic information is widely used in the deep learning literature to improve the performance of visual media processing. In this work, we propose a semantic prior based Generative Adversarial Network (GAN) model for video super-resolution. The model fully utilizes various texture styles from different semantic categories of video-frame patches, contributing to more accurate and efficient learning for the generator. Based on the GAN framework, we introduce the semantic prior by making use of the spatial feature transform during the learning process of the generator. The patch-wise semantic prior is extracted on the whole video frame by a semantic segmentation network. A hybrid loss function is designed to guide the learning performance. Experimental results show that our proposed model is advantageous in sharpening video frames, reducing noise and artifacts, and recovering realistic textures.

Original languageEnglish (US)
Title of host publicationEUSIPCO 2019 - 27th European Signal Processing Conference
PublisherEuropean Signal Processing Conference, EUSIPCO
ISBN (Electronic)9789082797039
DOIs
StatePublished - Sep 2019
Event27th European Signal Processing Conference, EUSIPCO 2019 - A Coruna, Spain
Duration: Sep 2 2019Sep 6 2019

Publication series

NameEuropean Signal Processing Conference
Volume2019-September
ISSN (Print)2219-5491

Conference

Conference27th European Signal Processing Conference, EUSIPCO 2019
CountrySpain
CityA Coruna
Period9/2/199/6/19

Keywords

  • Generative Adversarial Networks
  • Hybrid loss function
  • Semantic Segmentation
  • Spatial Feature Transform
  • Video Super-Resolution

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Semantic prior based generative adversarial network for video super-resolution'. Together they form a unique fingerprint.

  • Cite this

    Wu, X., Lucas, A., Lopez-Tapia, S., Wang, X., Kim, Y. H., Molina, R., & Katsaggelos, A. K. (2019). Semantic prior based generative adversarial network for video super-resolution. In EUSIPCO 2019 - 27th European Signal Processing Conference (European Signal Processing Conference; Vol. 2019-September). European Signal Processing Conference, EUSIPCO. https://doi.org/10.23919/EUSIPCO.2019.8902987