evaluation-package · Source-linked discovery

PRISM Instruction Order

Measures how permuting a fixed set of six reasoning instruction modules (RESTATE, IDENTIFY, PLAN, COMPUTE, CHECK, ANSWER) affects LLM accuracy on a frozen 32-question GSM8K subset, with module wording held constant and deterministic decoding (temperature 0). By default the task samples 50 seeded random orderings (1,600 samples); `-T design=exhaustive` evaluates all 720 (6!) orderings. Answers are scored by canonical numeric extraction, and per-ordering metrics (worst, best, range, standard deviation) report ordering-induced accuracy variance. The eval implements the paper's ordering-landscape protocol; the paper's 6.3%-96.9% range was measured on a single model (Gemini 2.5 Flash-Lite) and is not a property of the eval itself.