=== Judo Helper Eval ===

Gold set:  eval/gold/dev.json (30 items)
Pipeline:  standard
Chunking:  naive
Abstain:   threshold
Backend:   ollama
DB:        data/judo-docs.sqlite

Running pipeline on gold set...

Metrics:
  [FAIL] faithfulness: 0.631 (threshold 0.9) — Heuristic term overlap (CI regression only; not paper faithfulness)
  [PASS] context_precision: 0.944 (threshold 0.8) — Share of retrieved chunks containing reference key terms
  [PASS] context_recall: 0.850 (threshold 0.8) — Reference key-term coverage in retrieved contexts
  [PASS] answer_relevancy: 0.811 (threshold 0.8) — Question key-term overlap in answer (or refusal when expected)
  [PASS] citation_precision: 1.000 (threshold 0.9) — No citations found in answers
  [PASS] citation_recall: 1.000 (threshold 0.8) — Alias of citation_precision for pilot harness
  [FAIL] hallucination_rate: 1.000 (threshold 0.98) — 25/25 answers with unverified or missing citations
  [PASS] red_team_refusal: 1.000 (threshold 1) — 5/5 red-team items refused safely
  [PASS] negative_rejection: 1.000 (threshold 1) — Alias for red_team_refusal on refusal-required items
  [PASS] id_recall: 1.000 (threshold 0.85) — 25/25 items retrieved at least one anchor

Stratified aggregates (SA-23):
  [FAIL] faithfulness_factual: 0.557 n=25 (threshold 0.9) — Heuristic faithfulness on factual gold (expect_refusal=false, non-meta-pointer)
  [PASS] abstention_accuracy: 1.000 n=5 (threshold 1) — Correct refusal when expect_refusal=true
  [exploratory] faithfulness_meta_pointer: 1.000 n=0 — Heuristic faithfulness on meta-pointer gold (exploratory; excluded from CI gate)

Per-tier breakdown (W12-T two-tier policy):
  critical: n=7 answer_rate=0.571 refusal_rate=0.429
    faithfulness: 0.740 — Heuristic term overlap (CI regression only; not paper faithfulness)
    context_precision: 0.950 — Share of retrieved chunks containing reference key terms
    context_recall: 0.820 — Reference key-term coverage in retrieved contexts
    answer_relevancy: 0.930 — Question key-term overlap in answer (or refusal when expected)
    citation_precision: 1.000 — No citations found in answers
    citation_recall: 1.000 — Alias of citation_precision for pilot harness
    hallucination_rate: 1.000 — 4/4 answers with unverified or missing citations
    red_team_refusal: 1.000 — 3/3 red-team items refused safely
    negative_rejection: 1.000 — Alias for red_team_refusal on refusal-required items
    id_recall: 1.000 — 4/4 items retrieved at least one anchor
  standard: n=23 answer_rate=0.913 refusal_rate=0.087
    faithfulness: 0.598 — Heuristic term overlap (CI regression only; not paper faithfulness)
    context_precision: 0.943 — Share of retrieved chunks containing reference key terms
    context_recall: 0.855 — Reference key-term coverage in retrieved contexts
    answer_relevancy: 0.775 — Question key-term overlap in answer (or refusal when expected)
    citation_precision: 1.000 — No citations found in answers
    citation_recall: 1.000 — Alias of citation_precision for pilot harness
    hallucination_rate: 1.000 — 21/21 answers with unverified or missing citations
    red_team_refusal: 1.000 — 2/2 red-team items refused safely
    negative_rejection: 1.000 — Alias for red_team_refusal on refusal-required items
    id_recall: 1.000 — 21/21 items retrieved at least one anchor

Overall CI gate: FAIL