evaluation-package · Source-linked discovery

WildClawBench

This is an Inspect AI wrapper of the original WildClawBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/internlm/WildClawBench. Users provide a pinned checkout of the original repository, tested here at commit 86d71447413d38f38740a021cb776f64eb396ee0, with the task data downloaded from the internlm/WildClawBench HuggingFace dataset (workspace folder, revision 75f945578aa00cbdb8f46e4d42e4f4e98f704b4f), and the wrapper reports native WildClawBench results through Inspect's scoring interface. WildClawBench contains 60 human-authored bilingual and multimodal long-horizon agent tasks across productivity flow, code intelligence, social interaction, search and retrieval, creative synthesis, and safety alignment; every native task is one Inspect sample, so --limit, --sample-id and --max-samples apply. The native harness runs OpenClaw-style agents in Docker and grades with deterministic rule checks, environment-state auditing, and LLM/VLM judges (42 of the 60 tasks call a judge, which the wrapper serves from the same OpenAI-compatible endpoint as the agent or a judge_model it exposes); each sample's score is the task's native overall_score, reported with Inspect mean/stderr and per-category means, and tasks whose native grading crashes raise an error rather than scoring zero (per-metric judge fallbacks are recorded in the score explanation). Task containers run with --network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.