
> judohq@0.1.0 eval:judo-helper
> NODE_PATH=./node_modules tsx ../research/experiment/eval/run.ts --gold eval/gold/full.json --pipeline standard --ragas-judge --json

◇ injected env (24) from .env.local // tip: ◈ secrets for agents [www.dotenvx.com]
=== Judo Helper Eval ===

Gold set:  eval/gold/full.json (150 items)
Pipeline:  standard
Chunking:  naive
Abstain:   threshold
Backend:   ollama
RAGAS judge: enabled (slow pass)
DB:        <PROJECT_ROOT>/judohq-supabase/data/judo-docs-union.sqlite

Running pipeline on gold set...

Running RAGAS faithfulness judge (claim extraction + NLI)...

[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"{\":\"The upper weight limit for Senior and Junior Women in the -48 kg individual competition category is up to and including 48.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"== Clarification Request ==\":\n\"The upper weight limit for Senior and Junior Women in the -48 kg individual competition category is up to and including 48.0 kg. This means a judoka can weigh exactly 48.0 kg or less to compete there.\"\n\n}"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"== Clarification Request ==\":\n\"The upper weight limit for Senior and Junior Women in the -48 kg individual competition category is up to and including 48.0 kg. This means a judoka can weigh exactly 48.0 kg or less to compete there.\"\n\n}"
Metrics:
  [FAIL] faithfulness: 0.494 (threshold 0.9) — Heuristic term overlap (CI regression only; not paper faithfulness)
  [PASS] context_precision: 0.923 (threshold 0.8) — Share of retrieved chunks containing reference key terms
  [PASS] context_recall: 0.811 (threshold 0.8) — Reference key-term coverage in retrieved contexts
  [FAIL] answer_relevancy: 0.718 (threshold 0.8) — Question key-term overlap in answer (or refusal when expected)
  [FAIL] faithfulness_judge_all: 0.714 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b; rubric: claims-v2-2026-07-10); all items
  [PASS] faithfulness_judge_answered: 0.967 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b; rubric: claims-v2-2026-07-10); answered items with extracted claims (n=86)
  [PASS] citation_precision: 1.000 (threshold 0.9) — No citations found in answers
  [PASS] citation_recall: 1.000 (threshold 0.8) — Alias of citation_precision for pilot harness
  [FAIL] hallucination_rate: 1.000 (threshold 0.98) — 124/124 answers with unverified or missing citations
  [FAIL] red_team_refusal: 0.923 (threshold 1) — 24/26 red-team items refused safely
  [FAIL] negative_rejection: 0.923 (threshold 1) — Alias for red_team_refusal on refusal-required items
  [PASS] id_recall: 0.992 (threshold 0.85) — 123/124 items retrieved at least one anchor

Stratified aggregates (SA-23):
  [FAIL] faithfulness_factual: 0.404 n=124 (threshold 0.9) — Heuristic faithfulness on factual gold (expect_refusal=false, non-meta-pointer)
  [FAIL] abstention_accuracy: 0.923 n=26 (threshold 1) — Correct refusal when expect_refusal=true
  [exploratory] faithfulness_meta_pointer: 1.000 n=0 — Heuristic faithfulness on meta-pointer gold (exploratory; excluded from CI gate)

Per-tier breakdown (W12-T two-tier policy):
  critical: n=31 answer_rate=0.645 refusal_rate=0.355
    faithfulness: 0.588 — Heuristic term overlap (CI regression only; not paper faithfulness)
    context_precision: 0.910 — Share of retrieved chunks containing reference key terms
    context_recall: 0.846 — Reference key-term coverage in retrieved contexts
    answer_relevancy: 0.786 — Question key-term overlap in answer (or refusal when expected)
    citation_precision: 1.000 — No citations found in answers
    citation_recall: 1.000 — Alias of citation_precision for pilot harness
    hallucination_rate: 1.000 — 20/20 answers with unverified or missing citations
    red_team_refusal: 1.000 — 11/11 red-team items refused safely
    negative_rejection: 1.000 — Alias for red_team_refusal on refusal-required items
    id_recall: 0.950 — 19/20 items retrieved at least one anchor
  standard: n=119 answer_rate=0.891 refusal_rate=0.109
    faithfulness: 0.470 — Heuristic term overlap (CI regression only; not paper faithfulness)
    context_precision: 0.925 — Share of retrieved chunks containing reference key terms
    context_recall: 0.804 — Reference key-term coverage in retrieved contexts
    answer_relevancy: 0.700 — Question key-term overlap in answer (or refusal when expected)
    citation_precision: 1.000 — No citations found in answers
    citation_recall: 1.000 — Alias of citation_precision for pilot harness
    hallucination_rate: 1.000 — 104/104 answers with unverified or missing citations
    red_team_refusal: 0.867 — 13/15 red-team items refused safely
    negative_rejection: 0.867 — Alias for red_team_refusal on refusal-required items
    id_recall: 1.000 — 104/104 items retrieved at least one anchor

Overall CI gate: FAIL

Experiment log: <PROJECT_ROOT>/research/experiment/eval/runs/2026-07-11T03-12-06-268Z_d5b22172-14b1-4617-8bdc-944e2dfc3f60
Report JSON: <PROJECT_ROOT>/research/experiment/eval/runs/2026-07-11T03-12-06-268Z_d5b22172-14b1-4617-8bdc-944e2dfc3f60/report.json
