From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking

Felix Aller*, David Pinto-Fernandez, Diego Torricelli, Jose Luis Pons, Katja Mombaur

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


In order to prepare humanoid robots for real-world applications, it is necessary to assess the robots' locomotion performance and to determine their readiness level. We have carried out an extensive statistical literature review of 204 publications evaluating what kind of benchmarking scenarios and performance indicators are used and if they are sufficient to take a robot from prototype to production. We report the results of our analysis and discuss the most relevant findings such as the overall increase in the number of publications and motion tasks considered. Previous benchmarking efforts have been devoted to the functional assessment of the robotic system itself. A trend towards goal level oriented performance indicators such as robustness to external disturbance and dependability can be observed. We also identify a deficiency in benchmarking of non-functional aspects like safety or human-robot interaction. Based on our results, we outline the creation of a benchmarking framework with respect to current benchmarking approaches also introducing testbeds considering currently neglected motion tasks with a focus on a high degree of standardization.

Original languageEnglish (US)
Article number8894403
Pages (from-to)914-920
Number of pages7
JournalIEEE Robotics and Automation Letters
Issue number2
StatePublished - Apr 2020
Externally publishedYes


  • Humanoid robots
  • humanoid and bipedal locomotion
  • performance evaluation and benchmarking

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'From the State of the Art of Assessment Metrics Toward Novel Concepts for Humanoid Robot Locomotion Benchmarking'. Together they form a unique fingerprint.

Cite this