From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking

Felix Aller; David Pinto-Fernandez; Diego Torricelli; Jose Luis Pons; Katja Mombaur

doi:10.1109/LRA.2019.2952291

From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking

Felix Aller^*, David Pinto-Fernandez, Diego Torricelli, Jose Luis Pons, Katja Mombaur

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.

Original language	English (US)
Article number	8894403
Pages (from-to)	914-920
Number of pages	7
Journal	IEEE Robotics and Automation Letters
Volume	5
Issue number	2
DOIs	https://doi.org/10.1109/LRA.2019.2952291
State	Published - Apr 2020
Externally published	Yes

Keywords

Humanoid robots
humanoid and bipedal locomotion
performance evaluation and benchmarking

ASJC Scopus subject areas

Control and Systems Engineering
Biomedical Engineering
Human-Computer Interaction
Mechanical Engineering
Computer Vision and Pattern Recognition
Computer Science Applications
Control and Optimization
Artificial Intelligence

Access to Document

10.1109/LRA.2019.2952291

Cite this

@article{44477e51805d476988d3bfa8d763ffa3,

title = "From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking",

abstract = "In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.",

keywords = "Humanoid robots, humanoid and bipedal locomotion, performance evaluation and benchmarking",

author = "Felix Aller and David Pinto-Fernandez and Diego Torricelli and Pons, {Jose Luis} and Katja Mombaur",

note = "Funding Information: Manuscript received June 7, 2019; accepted October 17, 2019. Date of publication November 8, 2019; date of current version January 28, 2020. This letter was recommended for publication by Associate Editor B. Ugurlu and Editor N. Tsagarakis upon evaluation of the reviewers{\textquoteright} comments. This work was supported by the project EUROBENCH (European Robotic Framework for Bipedal Locomotion Benchmarking) through H2020 Topic ICT 27-2017 under Grant 779963. (Corresponding author: Felix Aller.) F. Aller and K. Mombaur are with the Optimization, Robotics and Biomechanics Chair, Institute of Computer Engineering, Heidelberg University, 69117 Heidelberg, Germany (e-mail: felix.aller@ziti.uni-heidelberg; katja.mombaur@ziti.uni-heidelberg.de). Publisher Copyright: {\textcopyright} 2016 IEEE.",

year = "2020",

month = apr,

doi = "10.1109/LRA.2019.2952291",

language = "English (US)",

volume = "5",

pages = "914--920",

journal = "IEEE Robotics and Automation Letters",

issn = "2377-3766",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "2",

}

TY - JOUR

T1 - From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking

AU - Aller, Felix

AU - Pinto-Fernandez, David

AU - Torricelli, Diego

AU - Pons, Jose Luis

AU - Mombaur, Katja

N1 - Funding Information: Manuscript received June 7, 2019; accepted October 17, 2019. Date of publication November 8, 2019; date of current version January 28, 2020. This letter was recommended for publication by Associate Editor B. Ugurlu and Editor N. Tsagarakis upon evaluation of the reviewers’ comments. This work was supported by the project EUROBENCH (European Robotic Framework for Bipedal Locomotion Benchmarking) through H2020 Topic ICT 27-2017 under Grant 779963. (Corresponding author: Felix Aller.) F. Aller and K. Mombaur are with the Optimization, Robotics and Biomechanics Chair, Institute of Computer Engineering, Heidelberg University, 69117 Heidelberg, Germany (e-mail: felix.aller@ziti.uni-heidelberg; katja.mombaur@ziti.uni-heidelberg.de). Publisher Copyright: © 2016 IEEE.

PY - 2020/4

Y1 - 2020/4

N2 - In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.

AB - In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.

KW - Humanoid robots

KW - humanoid and bipedal locomotion

KW - performance evaluation and benchmarking

UR - http://www.scopus.com/inward/record.url?scp=85079803802&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85079803802&partnerID=8YFLogxK

U2 - 10.1109/LRA.2019.2952291

DO - 10.1109/LRA.2019.2952291

M3 - Article

AN - SCOPUS:85079803802

SN - 2377-3766

VL - 5

SP - 914

EP - 920

JO - IEEE Robotics and Automation Letters

JF - IEEE Robotics and Automation Letters

IS - 2

M1 - 8894403

ER -

From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this