evaluation-task · Documentary review
Cybench: Capture-The-Flag Cybersecurity Challenges
Tests language models on cybersecurity skills using 39 of 40 practical, professional-level challenges taken from cybersecurity competitions, designed to cover various difficulty levels and security concepts. The motp challenge is excluded due to GPL licensing.
OriginAndy K. Zhang, Neil Perry, Riya Dulepet et al.TopicsCyberStatusReviewed
Can support
Performance on CyBench's selected CTF tasks under the exact command environment, tools, scaffold, and budget.
Cannot support by itself
Real-world intrusion capability, stealth, persistence, target discovery, legal or organizational constraints, or operational cyber risk.
Decision use
Best used for
Comparing cyber-agent problem solving and analysing where long-horizon CTF trajectories fail.
Not enough for
Claims that a model can conduct real-world cyber operations or meets a deployment-risk threshold.