A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication Channel

Srutarshi Banerjee, Henry H. Chopp, Jianping Zhang, Zihao W. Wang, Peng Kang, Oliver Cossairt, Aggelos Katsaggelos

Research output: Contribution to journalArticlepeer-review

Abstract

We present a novel adaptive multimodal intensity-event algorithm to optimize an overall objective of object tracking under bit rate constraints for a host-chip architecture. The chip is a computationally resource-constrained device acquiring high-resolution intensity frames and events, while the host is capable of performing computationally expensive tasks. We develop a joint intensity-neuromorphic event rate-distortion compression framework with a quadtree (QT)-based compression of intensity and events scheme. The goal of this compression framework is to optimally allocate bits to the intensity frames and neuromorphic events based on the minimum distortion at a given communication channel capacity. The data acquisition on the chip is driven by the presence of objects of interest in the scene as detected by an object detector. The most informative intensity and event data are communicated to the host under rate constraints so that the best possible tracking performance is obtained. The detection and tracking of objects in the scene are done on the distorted data at the host. Intensity and events are jointly used in a fusion framework to enhance the quality of the distorted images, in order to improve the object detection and tracking performance. The performance assessment of the overall system is done in terms of the multiple object tracking accuracy (MOTA) score. Compared with using intensity modality only, there is an improvement in MOTA using both these modalities in different scenarios.

Original languageEnglish (US)
Pages (from-to)1-15
Number of pages15
JournalIEEE Transactions on Neural Networks and Learning Systems
DOIs
StateAccepted/In press - 2022

Keywords

  • Compressed domain object detection and tracking
  • Computer architecture
  • dynamic programming (DP)
  • Image reconstruction
  • joint intensity-event imaging system
  • joint intensity-event rate-distortion optimization
  • Neuromorphics
  • Object detection
  • Optimization
  • quadtree (QT) segmentation
  • Rate-distortion
  • Task analysis

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication Channel'. Together they form a unique fingerprint.

Cite this