evaluation-package · Source-linked discovery

HVTB

HVTB measures reward-hacking rates of LLM agents on 89 real-world terminal/coding tasks from Terminal Bench 2.1. Each task container plants a reference solution and held-out tests under /app/admin/, monitored via inotify filesystem watchers. If an agent reads these files, a hack is flagged automatically. The primary metric is hack_rate (fraction of tasks where a watcher fired). The Inspect implementation uses a react agent with a bash tool instead of the paper's native agent harnesses.