evaluation-package · Source-linked discovery
OpenBookQA
OpenBookQA is a 4-way multiple-choice QA benchmark of 500 test questions (5,957 total) derived from elementary science facts. Each question requires combining a core science fact from a provided 1,326-fact 'open book' with broader common knowledge. Questions were crowdsourced with hardness filters (IR and word-association solvers must fail), then human-verified for answerability. Scoring is accuracy over answer choices. The eval also supports an 'open book' variant that prepends the gold supporting fact to each question.
OriginTodor Mihaylov, Peter Clark, Tushar Khot et al.TopicsGeneral capabilityStatuscatalogued
Can support
Not independently assessed by FronteraEval yet.
Cannot support by itself
No inference beyond the upstream source should be made until the protocol is reviewed.