Abstract
We present a System for Processing In-situ Bio-signal Data for Emotion Recog-nition and Sensing (SPIDERS+) – a low-cost, wireless, glasses-based platform for continuous in-situ monitoring of user's facial expressions (apparent emotions) and real emotions. We present algorithms to provide four core functions to detect eye shape and eyebrow movements, pupillometry, zygomaticus muscle movements, and head movements, using the bio-signals acquired from three non-contact sensors (IR camera, proximity sensor, and IMU). SPIDERS+ distinguishes between different classes of apparent and real emotion states based on the aforementioned four bio-signals. We prototype advanced functionalities including facial expression detection and real emotion classification with a facial expression detector based on landmarks and optical flow that leverages changes in a user's eyebrows and eye shapes to achieve the average true positive rate up to 83.87%, as well as a pupillometry-based real emotion classifier with higher accuracy than existing low-cost wearable platforms that use sensors requiring skin contact. In addition, SPIDERS+ also includes optional contact-based sensors such as photoplethysmography (PPG) and multi-channel electroencephalogram (EEG) to monitor other bio-signals. SPIDERS+ costs less than $20 to assemble and can continuously run for up to 9 h before recharging. We design, implement and evaluate accompanying desktop and smartphone applications to demonstrate that SPIDERS+ is a truly wireless and portable platform that can impact a wide range of applications, where the knowledge of the user's emotional state is critical.
Original language | English (US) |
---|---|
Article number | 101424 |
Journal | Pervasive and Mobile Computing |
Volume | 75 |
DOIs | |
State | Published - Aug 2021 |
Funding
This research was partially supported by the National Science Foundation under Grant Numbers CNS-1704899 , CNS-1815274 , CNS-11943396 , and CNS-1837022 . The views and conclusions contained here are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Columbia University, NSF, or the U.S. Government or any of its agencies.
Keywords
- Bio-signal acquisition
- Emotion monitoring
- Facial expression detection
- Pupillometry
- Wearables
ASJC Scopus subject areas
- Software
- Information Systems
- Hardware and Architecture
- Computer Science Applications
- Computer Networks and Communications