Deep generative cross-modal on-body accelerometer data synthesis from videos

Shibo Zhang, Nabil Alshurafa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

Human activity recognition (HAR) based on wearable sensors has brought tremendous benefit to several industries ranging from healthcare to entertainment. However, to build reliable machine-learned models from wearables, labeled on-body sensor datasets obtained from real-world settings are needed. It is often prohibitively expensive to obtain large-scale, labeled on-body sensor datasets from real-world deployments. The lack of labeled datasets is a major obstacle in the wearable sensor-based activity recognition community. To overcome this problem, I aim to develop two deep generative cross-modal architectures to synthesize accelerometer data streams from video data streams. In the proposed approach, a conditional generative adversarial network (cGAN) is first used to generate sensor data conditioned on video data. Then, a conditional variational autoencoder (cVAE)-cGAN is proposed to further improve representation of the data. The effectiveness and efficacy of the proposed methods will be evaluated through two popular applications in HAR: eating recognition and physical activity recognition. Extensive experiments will be conducted on public sensor-based activity recognition datasets by building models with synthetic data and comparing the models against those trained from real sensor data. This work aims to expand labeled on-body sensor data, by generating synthetic on-body sensor data from video, which will equip the community with methods to transfer labels from video to on-body sensors.

Original languageEnglish (US)
Title of host publicationUbiComp/ISWC 2020 Adjunct - Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers
PublisherAssociation for Computing Machinery
Pages223-227
Number of pages5
ISBN (Electronic)9781450380768
DOIs
StatePublished - Sep 10 2020
Event2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and 2020 ACM International Symposium on Wearable Computers, UbiComp/ISWC 2020 - Virtual, Online, Mexico
Duration: Sep 12 2020Sep 17 2020

Publication series

NameUbiComp/ISWC 2020 Adjunct - Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

Conference

Conference2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and 2020 ACM International Symposium on Wearable Computers, UbiComp/ISWC 2020
Country/TerritoryMexico
CityVirtual, Online
Period9/12/209/17/20

Keywords

  • accelerometer data synthesis
  • data augmentation
  • deep generative model
  • deep multi-modal learning
  • video-sensor data representation learning

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Deep generative cross-modal on-body accelerometer data synthesis from videos'. Together they form a unique fingerprint.

Cite this