evaluation-package · Source-linked discovery

PinchBench

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect's scoring interface. PinchBench contains 53 real-world OpenClaw agent tasks across productivity, research, writing, coding, analysis, email, memory, and skill-discovery categories. The native harness grades tasks automatically, with an LLM judge, or both; this wrapper runs that harness in Docker and reports the native aggregate score, or the mean of per-task numeric scores when no aggregate is present, with Inspect mean/stderr metrics. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.