Manipulability of comparative tests

Wojciech Olszewski*, Alvaro Sandroni

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Multiple self-proclaimed experts claim that they know the probabilities of future events. A tester does not know the odds of future events and she also does not know whether, among the multiple experts, there are some who do know the relevant probabilities. So the tester requires each expert to announce, before any data are observed, the probabilities of all future events. A test either rejects or does not reject each expert based on the observed data and the profile of the probabilities announced by the experts. We assume that the test controls for the type I error of rejecting the true probabilities. However, consider the case in which all experts are uninformed (i.e., they do not know anything about true probabilities). We show that they can still independently produce false forecasts that are likely to both pass the test, no matter how the data evolve in the future. Hence, the data may not suffice to effectively discredit uninformed, but strategic, experts.

Original languageEnglish (US)
Pages (from-to)5029-5034
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume106
Issue number13
DOIs
StatePublished - Mar 31 2009

Keywords

  • Comparative tests
  • Manipulability
  • Strategic expert

ASJC Scopus subject areas

  • General

Cite this