Perceiving Graphical and Pictorial Information via Hearing and Touch

Pubudu Madhawa Silva, Thrasyvoulos N. Pappas, Joshua Atkins, James E. West

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


We propose a dynamic interactive system for conveying visual information via hearing and touch to blind and visually impaired people. The system is implemented with a touch screen that allows the user to actively explore a two-dimensional layout consisting of one or more objects with the finger while listening to auditory feedback. Sound is used as the primary source of information for object localization, identification, and shape, while touch is used for pointing and kinesthetic feedback. A static overlay of raised-dot tactile patterns can also be added. The head-related transfer function is used for rendering sound directionality, and variations of sound intensity or other features are used for rendering proximity. The main focus is on conveying the shape of an object, but the rendering of a simple scene layout, that consists of objects in a linear arrangement, each with a distinct tapping sound, is also considered and compared to a 'virtual cane.' We consider a number of acoustic-tactile configurations and use empirical studies with visually blocked sighted participants to compare their effectiveness. Our findings demonstrate the advantages of spatial sound (directionality and proximity cues) for dynamic display of information (localization, identification, shape), while raised-dot patterns provide the best static shape rendition. We also show that the proposed configurations outperform existing techniques. The proposed approach is also expected to impact other applications where vision cannot be used.

Original languageEnglish (US)
Article number7544538
Pages (from-to)2432-2445
Number of pages14
JournalIEEE Transactions on Multimedia
Issue number12
StatePublished - Dec 2016


  • Acoustic-tactile representation of visual signals
  • semantic mapping
  • sensory substitution
  • user interface

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Perceiving Graphical and Pictorial Information via Hearing and Touch'. Together they form a unique fingerprint.

Cite this