HN 표시: Mdarena – 자신의 PR과 비교하여 Claude.md를 벤치마킹하세요.

Benchmark your CLAUDE.md against your own PRs. Most CLAUDE.md files are written blindly. Research shows they often reduce agent success rates and cost 20%+ more tokens. mdarena lets you measure whether yours helps or hurts, on tasks from your actual codebase. pip install mdarena # Mine 50 merged PRs into a test set mdarena mine owner/repo --limit 50 --detect-tests # Benchmark multiple CLAUDE.md files + baseline (no context) mdarena run -c claude_v1.md -c claude_v2.md -c agents.md # See who wins mdarena report mdarena mine -> Fetch merged PRs, filter, build task set Auto-detect test commands from CI/package files mdarena run -> For each task x condition: - Checkout repo at pre-PR commit - Baseline: all CLAUDE.md files stripped - Context: inject CLAUDE.md, let Claude discover it - Run tests if available, capture git diff mdarena report -> Compare patches against gold (actual PR diff) - Test pass/fail (same as SWE-bench) - File/hunk overlap, cost, tokens - Statistical significance (paired t-test) mdarena can run your repo's actual tests to grade agent patches, the same way SWE-bench does it. # Auto-detect from CI/CD mdarena mine owner/repo --detect-tests # Or specify manually mdarena mine owner/repo --test-cmd "make test" --setup-cmd "npm install" Parses .github/workflows/*.yml , package.json , pyproject.toml , Cargo.toml , and go.mod . When tests aren't available, falls back to diff overlap scoring. Pass a directory to benchmark a full CLAUDE.md tree: mdarena run -c ./configs-v1/ -c ./configs-v2/ Each directory mirrors your repo structure. Baseline strips ALL CLAUDE.md and AGENTS.md files from the entire tree. We ran mdarena against a large production monorepo: 20 merged PRs, Claude Opus 4.6, three conditions (bare baseline, existing CLAUDE.md, hand-written alternative). Patches graded against real test suites. Not string matching, not LLM-as-judge. Key findings: - The existing CLAUDE.md improved test resolution by ~27% over bare baseline - A consolidated alternative that merged all per-directory guidance into one file performed no better than no CLAUDE.md at all - On hard tasks, per-directory instruction files gave the agent targeted context, while the consolidated version introduced noise that caused regressions The winning CLAUDE.md wasn't the longest or most detailed. It was the one that put the right context in front of the agent at the right time. # Import SWE-bench tasks pip install datasets mdarena load-swebench lite --limit 50 mdarena run -c my_claude.md # Or export your tasks as SWE-bench JSONL mdarena export-swebench Only benchmark repositories you trust. mdarena executes code from the repos it benchmarks (test commands run via shell=True , Claude Code runs with --dangerously-skip-permissions ). Sandboxes are isolated temp directories under /tmp but processes run as your user. Benchmark integrity: Because tasks come from historical PRs, the gold patch is in the repo's git history. Claude 4 Sonnet exploited this against SWE-bench by walking future commits via tags. mdarena prevents this with history-free checkouts: git archive exports a snapshot at base_commit into a fresh single-commit repo. Future commits don't exist in the object database at all. See tests/test_isolated_checkout.py for the integrity assertions. | Command | Description | |---|---| mdarena mine | Mine merged PRs into a task set | mdarena mine --detect-tests | Mine with auto-detected test extraction | mdarena run -c file.md | Benchmark a single CLAUDE.md | mdarena run -c a.md -c b.md | Compare multiple files head-to-head | mdarena run --no-run-tests | Skip test execution, diff overlap only | mdarena report | Analyze results, show comparison | mdarena load-swebench [dataset] | Import SWE-bench tasks | mdarena export-swebench | Export tasks as SWE-bench JSONL | git clone https://github.com/HudsonGri/mdarena.git cd mdarena uv sync uv run pytest uv run ruff check src/ See ROADMAP.md. MIT. See LICENSE.