근본 원인 분석 및 오류 마이닝 기능을 갖춘 오픈 소스 LLM 판사 평가 제품군

AI system reliability infrastructure. Evaluate any AI system's reliability across correctness, structure, and performance. pip install cane-eval Extensible reliability evaluation for AI systems — not just LLMs, but any AI agent, API, or pipeline. One tool, one score, one answer: would this break in production? Support Agent 28.4s Overall: [=========----------] 47 1 passed 1 warned 3 failed (5 total) Pass rate: 20% Latency: p50: 1.2s p95: 8.4s max: 12.1s Schema: 3/5 valid (60%) Reliability: [=======-----------] 52 (D) export ANTHROPIC_API_KEY=sk-ant-... cane-eval demo 1. Define tests (tests.yaml ): name: Support Agent criteria: - key: accuracy weight: 40 - key: completeness weight: 30 - key: hallucination weight: 30 # Optional: validate response structure schema: type: object required: [answer, sources] properties: answer: { type: string } sources: { type: array } # Optional: latency target for reliability scoring latency_target_ms: 5000 # Optional: configure reliability weights reliability: correctness_weight: 0.60 structural_weight: 0.20 performance_weight: 0.20 # Optional: parallel execution concurrency: 5 tests: - question: What is the return policy? expected_answer: 30-day return policy for unused items with receipt - question: How do I reset my password? expected_answer: Go to Settings > Security > Reset Password 2. Run: cane-eval run tests.yaml 3. Production checks: # Parallel execution cane-eval run tests.yaml -j 5 # Custom reliability weights (correctness:structural:performance) cane-eval run tests.yaml --reliability-weights 60:20:20 # Validate responses against JSON schema cane-eval run tests.yaml --schema schema.json --fail-on-schema # Fail if p95 latency exceeds 10 seconds cane-eval run tests.yaml --latency-p95 10000 # All together + mine failures into training data cane-eval run tests.yaml -j 5 --schema schema.json --latency-p95 10000 --mine --export dpo Every eval run produces an Agent Reliability Score (0-100) across three pillars: | Pillar | What it measures | How | |---|---|---| | Correctness | Does the answer look good? | LLM judge (accuracy, completeness, hallucination) | | Structural | Does the response match expected format? | JSON schema validation | | Performance | Is it fast enough for production? | p95 latency vs target | Grades: A (90+) production-ready, B (75+) mostly reliable, C (60+) needs work, D (40+) significant gaps, F ( float: if ctx.response_time_ms Agent --> LLM Judge -----> Reliability Score (A-F) | | | | v | | Schema Check | | Latency Stats | | | v v v Training Data Root Cause Failure (DPO/SFT/OpenAI) Analysis Mining MIT