evaluation-package · Source-linked discovery
MonitorBench (Dual Objectives: Summarization subset)
MonitorBench evaluates CoT monitorability: whether an LLM's chain-of-thought explicitly surfaces decision-critical factors driving its outputs. The summarization subtask presents 50 instances where the model must summarize a report while verbatim-including a required sentence (side objective). A judge LLM monitors CoT-only, action-only, and all-messages scopes to flag whether the side objective appears. Three conditions are tested: standard, direct concealment, and monitor-aware evasion. Monitorability score measures flagging rate on successful rollouts.
OriginHan Wang, Yifan Sun, Brian Ko et al.TopicsCyber · Evaluation integrityStatuscatalogued
Can support
Not independently assessed by FronteraEval yet.
Cannot support by itself
No inference beyond the upstream source should be made until the protocol is reviewed.