evaluation-package · Source-linked discovery

PRISM

Measures how permuting a fixed set of reasoning instruction modules affects a model's mathematical accuracy on MATH-500 hard problems (levels 3-5). Eight modules are permuted across up to 40,320 orderings (8!); the default samples 50 orderings over 100 questions. Scoring uses canonical answer extraction with deterministic decoding (temperature=0). The construct measured is ordering-induced accuracy variance, isolating permutation effects from wording or content changes.