evaluation-package · Source-linked discovery

PRISM GSM8K Exhaustive

Measures accuracy sensitivity to the ordering of six fixed reasoning instruction modules (RESTATE, IDENTIFY, PLAN, COMPUTE, CHECK, ANSWER) on a frozen 32-question GSM8K subset. All 720 permutations are evaluated with deterministic decoding (temperature=0). Scoring uses canonical numeric answer extraction. The primary metric is per-ordering accuracy, enabling a full fitness landscape map. The paper reports 6.3%-96.9% accuracy range across orderings.