evaluation-package · Source-linked discovery

PinchBench

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect's scoring interface. At that commit the upstream task manifest lists 147 real-world OpenClaw agent tasks in 11 categories (productivity, research, writing, coding, analysis, CSV analysis, log analysis, meeting analysis, memory, skills, and integrations); every native task is one Inspect sample, so --limit, --sample-id and --max-samples apply. The native harness grades 25 tasks automatically, 21 with an LLM judge and 101 with both; the wrapper runs that harness in Docker, serves the judge from the same OpenAI-compatible endpoint as the agent (or a judge_model it exposes), and scores each sample as native points earned over points available, reporting Inspect mean/stderr, the upstream aggregate (total points over total available) as native_score, and per-category means. Task containers run with --network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.