evaluation-package · Source-linked discovery
ContractBench
ContractBench evaluates LLM agents on 33 tasks probing two orthogonal failure modes: temporal validity (using API artifacts before expiry) and byte-level integrity (relaying artifacts without corruption). Tasks simulate real API patterns (presigned URLs, OAuth tokens, HMAC webhooks) using a virtual clock and SHA-256 hash verification. Each episode is scored deterministically via HTTP request logs against a 15-label failure taxonomy. Metric is success rate over k=3 runs per task (n=99 episodes per model). No LLM judge is used.
OriginJicheng Wang, Yifeng He, Zili Wang et al.TopicsGeneral capabilityStatuscatalogued
Can support
Not independently assessed by FronteraEval yet.
Cannot support by itself
No inference beyond the upstream source should be made until the protocol is reviewed.