Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis

Nili E. Krausz; Denys Lamotte; Iason Batzianoulis; Levi J. Hargrove; Silvestro Micera; Aude Billard

doi:10.1109/TNSRE.2020.2992885

Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis

Nili E. Krausz^*, Denys Lamotte, Iason Batzianoulis, Levi J. Hargrove, Silvestro Micera, Aude Billard

^*Corresponding author for this work

Physical Medicine and Rehabilitation

Research output: Contribution to journal › Article › peer-review

31 Scopus citations

Abstract

We propose a novel controller for powered prosthetic arms, where fused EMG and gaze data predict the desired end-point for a full arm prosthesis, which could drive the forward motion of individual joints. We recorded EMG, gaze, and motion-tracking during pick-and-place trials with 7 able-bodied subjects. Subjects positioned an object above a random target on a virtual interface, each completing around 600 trials. On average across all trials and subjects gaze preceded EMG and followed a repeatable pattern that allowed for prediction. A computer vision algorithm was used to extract the initial and target fixations and estimate the target position in 2D space. Two SVRs were trained with EMG data to predict the x- and y- position of the hand; results showed that the y-estimate was significantly better than the x-estimate. The EMG and gaze predictions were fused using a Kalman Filter-based approach, and the positional error from using EMG-only was significantly higher than the fusion of EMG and gaze. The final target position Root Mean Squared Error (RMSE) decreased from 9.28 cm with an EMG-only prediction to 6.94 cm when using a gaze-EMG fusion. This error also increased significantly when removing some or all arm muscle signals. However, using fused EMG and gaze, there were no significant difference between predictors that included all muscles, or only a subset of muscles.

Original language	English (US)
Article number	9088146
Pages (from-to)	1471-1480
Number of pages	10
Journal	IEEE Transactions on Neural Systems and Rehabilitation Engineering
Volume	28
Issue number	6
DOIs	https://doi.org/10.1109/TNSRE.2020.2992885
State	Published - Jun 2020

Funding

Manuscript received December 11, 2019; revised April 13, 2020; accepted April 29, 2020. Date of publication May 6, 2020; date of current version June 5, 2020. This work was supported in part by the Swiss National Science Foundation through National Centre of Competence in Research in Robotics (NCCR Robotics) Wearable Scenario, NCCR Robotics Ph.D. Exchange Fellowship under Grant 51NF40–160592, and in part by the NIH T32 PRND Training Grant. (Nili E. Krausz and Denys Lamotte are co-first authors.) (Corresponding author: Nili E. Krausz.) Nili E. Krausz was with the Shirley Ryan AbilityLab (formerly RIC), Chicago, IL 60611 USA, and also with the LASA Laboratory, EPFL, 1015 Lausanne, Switzerland. She is now with the Rehabilitation Institute of Chicago, Northwestern University, Chicago, IL 60611 USA, and also with the Weizmann Institute of Science, Rehovot 7610001, Israel (e-mail: [email protected]).

Keywords

Kalman Filter
Prosthetics
computer vision
electromyography
end-point control
gaze tracking
sensory fusion
upper limb prosthesis

ASJC Scopus subject areas

Internal Medicine
General Neuroscience
Biomedical Engineering
Rehabilitation

Access to Document

10.1109/TNSRE.2020.2992885

Cite this

Krausz, N. E., Lamotte, D., Batzianoulis, I., Hargrove, L. J., Micera, S., & Billard, A. (2020). Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(6), 1471-1480. Article 9088146. https://doi.org/10.1109/TNSRE.2020.2992885

@article{26d46e4312894b7d98ba4ab985f798cf,

title = "Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis",

abstract = "We propose a novel controller for powered prosthetic arms, where fused EMG and gaze data predict the desired end-point for a full arm prosthesis, which could drive the forward motion of individual joints. We recorded EMG, gaze, and motion-tracking during pick-and-place trials with 7 able-bodied subjects. Subjects positioned an object above a random target on a virtual interface, each completing around 600 trials. On average across all trials and subjects gaze preceded EMG and followed a repeatable pattern that allowed for prediction. A computer vision algorithm was used to extract the initial and target fixations and estimate the target position in 2D space. Two SVRs were trained with EMG data to predict the x- and y- position of the hand; results showed that the y-estimate was significantly better than the x-estimate. The EMG and gaze predictions were fused using a Kalman Filter-based approach, and the positional error from using EMG-only was significantly higher than the fusion of EMG and gaze. The final target position Root Mean Squared Error (RMSE) decreased from 9.28 cm with an EMG-only prediction to 6.94 cm when using a gaze-EMG fusion. This error also increased significantly when removing some or all arm muscle signals. However, using fused EMG and gaze, there were no significant difference between predictors that included all muscles, or only a subset of muscles.",

keywords = "Kalman Filter, Prosthetics, computer vision, electromyography, end-point control, gaze tracking, sensory fusion, upper limb prosthesis",

author = "Krausz, {Nili E.} and Denys Lamotte and Iason Batzianoulis and Hargrove, {Levi J.} and Silvestro Micera and Aude Billard",

note = "Publisher Copyright: {\textcopyright} 2001-2011 IEEE.",

year = "2020",

month = jun,

doi = "10.1109/TNSRE.2020.2992885",

language = "English (US)",

volume = "28",

pages = "1471--1480",

journal = "IEEE Transactions on Neural Systems and Rehabilitation Engineering",

issn = "1534-4320",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "6",

}

Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis. / Krausz, Nili E.; Lamotte, Denys; Batzianoulis, Iason et al.
In: IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 28, No. 6, 9088146, 06.2020, p. 1471-1480.

Research output: Contribution to journal › Article › peer-review

TY - JOUR

T1 - Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis

AU - Krausz, Nili E.

AU - Lamotte, Denys

AU - Batzianoulis, Iason

AU - Hargrove, Levi J.

AU - Micera, Silvestro

AU - Billard, Aude

PY - 2020/6

Y1 - 2020/6

N2 - We propose a novel controller for powered prosthetic arms, where fused EMG and gaze data predict the desired end-point for a full arm prosthesis, which could drive the forward motion of individual joints. We recorded EMG, gaze, and motion-tracking during pick-and-place trials with 7 able-bodied subjects. Subjects positioned an object above a random target on a virtual interface, each completing around 600 trials. On average across all trials and subjects gaze preceded EMG and followed a repeatable pattern that allowed for prediction. A computer vision algorithm was used to extract the initial and target fixations and estimate the target position in 2D space. Two SVRs were trained with EMG data to predict the x- and y- position of the hand; results showed that the y-estimate was significantly better than the x-estimate. The EMG and gaze predictions were fused using a Kalman Filter-based approach, and the positional error from using EMG-only was significantly higher than the fusion of EMG and gaze. The final target position Root Mean Squared Error (RMSE) decreased from 9.28 cm with an EMG-only prediction to 6.94 cm when using a gaze-EMG fusion. This error also increased significantly when removing some or all arm muscle signals. However, using fused EMG and gaze, there were no significant difference between predictors that included all muscles, or only a subset of muscles.

AB - We propose a novel controller for powered prosthetic arms, where fused EMG and gaze data predict the desired end-point for a full arm prosthesis, which could drive the forward motion of individual joints. We recorded EMG, gaze, and motion-tracking during pick-and-place trials with 7 able-bodied subjects. Subjects positioned an object above a random target on a virtual interface, each completing around 600 trials. On average across all trials and subjects gaze preceded EMG and followed a repeatable pattern that allowed for prediction. A computer vision algorithm was used to extract the initial and target fixations and estimate the target position in 2D space. Two SVRs were trained with EMG data to predict the x- and y- position of the hand; results showed that the y-estimate was significantly better than the x-estimate. The EMG and gaze predictions were fused using a Kalman Filter-based approach, and the positional error from using EMG-only was significantly higher than the fusion of EMG and gaze. The final target position Root Mean Squared Error (RMSE) decreased from 9.28 cm with an EMG-only prediction to 6.94 cm when using a gaze-EMG fusion. This error also increased significantly when removing some or all arm muscle signals. However, using fused EMG and gaze, there were no significant difference between predictors that included all muscles, or only a subset of muscles.

KW - Kalman Filter

KW - Prosthetics

KW - computer vision

KW - electromyography

KW - end-point control

KW - gaze tracking

KW - sensory fusion

KW - upper limb prosthesis

UR - http://www.scopus.com/inward/record.url?scp=85086052268&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85086052268&partnerID=8YFLogxK

U2 - 10.1109/TNSRE.2020.2992885

DO - 10.1109/TNSRE.2020.2992885

M3 - Article

C2 - 32386160

AN - SCOPUS:85086052268

SN - 1534-4320

VL - 28

SP - 1471

EP - 1480

JO - IEEE Transactions on Neural Systems and Rehabilitation Engineering

JF - IEEE Transactions on Neural Systems and Rehabilitation Engineering

IS - 6

M1 - 9088146

ER -

Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this