evaluation-package · Source-linked discovery

NARCBench

NARCBench evaluates black-box detection of covert multi-agent collusion from complete public committee records. The primary task presents all 100 Qwen3-32B Core runs (50 matched collusion/control pairs), including public scenario context, option descriptions, deliberation, final statements, and votes, and asks a monitor model for P(collusion). Primary AUROC is intention-to-treat over all 50 pairs; a legacy-context view is available as a documented ablation. The paper's results use white-box activation probes, while this eval measures transcript-level monitoring on its released data.