TY - JOUR
T1 - From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking
AU - Aller, Felix
AU - Pinto-Fernandez, David
AU - Torricelli, Diego
AU - Pons, Jose Luis
AU - Mombaur, Katja
N1 - Funding Information:
Manuscript received June 7, 2019; accepted October 17, 2019. Date of publication November 8, 2019; date of current version January 28, 2020. This letter was recommended for publication by Associate Editor B. Ugurlu and Editor N. Tsagarakis upon evaluation of the reviewers’ comments. This work was supported by the project EUROBENCH (European Robotic Framework for Bipedal Locomotion Benchmarking) through H2020 Topic ICT 27-2017 under Grant 779963. (Corresponding author: Felix Aller.) F. Aller and K. Mombaur are with the Optimization, Robotics and Biomechanics Chair, Institute of Computer Engineering, Heidelberg University, 69117 Heidelberg, Germany (e-mail: [email protected]; [email protected]).
Publisher Copyright:
© 2016 IEEE.
PY - 2020/4
Y1 - 2020/4
N2 - In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.
AB - In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.
KW - Humanoid robots
KW - humanoid and bipedal locomotion
KW - performance evaluation and benchmarking
UR - http://www.scopus.com/inward/record.url?scp=85079803802&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85079803802&partnerID=8YFLogxK
U2 - 10.1109/LRA.2019.2952291
DO - 10.1109/LRA.2019.2952291
M3 - Article
AN - SCOPUS:85079803802
SN - 2377-3766
VL - 5
SP - 914
EP - 920
JO - IEEE Robotics and Automation Letters
JF - IEEE Robotics and Automation Letters
IS - 2
M1 - 8894403
ER -