evaluation-package · Source-linked discovery

CASTLE

CASTLE evaluates LLM vulnerability detection on 250 hand-crafted, compilable C programs covering 25 CWE types (6 vulnerable, 4 non-vulnerable per CWE). Models receive a system prompt requesting JSON output indicating vulnerability presence and CWE number. Scoring uses the CASTLE Score: +5 for correct vulnerability detection (minus 1 per extra false positive reported), +2 for correct true-negative identification, and -1 per false positive otherwise. TPR and FPR are also reported.