evaluation-package · Source-linked discovery

MonitorBench (Goal sandbag Safety subset)

MonitorBench evaluates CoT monitorability -- whether an LLM's chain-of-thought faithfully reflects the decision-critical factors driving its outputs. It comprises 1,514 instances across 19 tasks in 7 categories (input intervention, outcome justification, solution process). A monitor LLM checks whether injected decision-critical factors appear in CoTs. The monitorability score uses a causal attribution formula comparing flagging rates against a minimum expected ratio. Two stress-test conditions (direct concealment, monitor-aware evasion) quantify how much monitorability degrades under adversarial prompting.