=== Judo Helper Eval ===

Gold set:  eval/gold/full.json (150 items)
Pipeline:  constrained
Chunking:  naive
Abstain:   threshold
Backend:   ollama
RAGAS judge: enabled (slow pass)
DB:        data/judo-docs.sqlite

Running pipeline on gold set...

Running RAGAS faithfulness judge (claim extraction + NLI)...

[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw="{\""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw="```"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw="```"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw="```json\n{\n  \"claims\": [\n    {\n      \"text\": \"The minimum age to enter a Dan grading is 15.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"Junior judoka holding the grade of 1st Kyu under the age of 15 may collect promotion points but cannot register their promotion until they are 15.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"A 1st Kyu must have a record of contest activity on at"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw="```json\n{\n  \"claims\": [\n    {\n      \"text\": \"The minimum age to enter a Dan grading is 15.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"Junior judoka holding the grade of 1st Kyu under the age of 15 may collect promotion points but cannot register their promotion until they are 15.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"A 1st Kyu must have a record of contest activity on at"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw="```json\n{\n  \"claims\": [\n    {\n      \"text\": \"A child enrolls at their local club.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"A membership application is completed by the applicant or a coach and submitted online or in printed form.\",\n      \"supported\": true\n    },"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw="```json\n{\n  \"claims\": [\n    {\n      \"text\": \"A child enrolls at their local club.\",\n      \"supported\": true\n    },\n    {\n      \"text\": \"A membership application is completed by the applicant or a coach and submitted online or in printed form.\",\n      \"supported\": true\n    },"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw="The"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw="The"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw="The"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=gemma4:e4b raw=""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=gemma4:e4b raw=""
Metrics:
  [FAIL] faithfulness: 0.333 (threshold 0.9) — Heuristic term overlap (CI regression only; not paper faithfulness)
  [FAIL] context_precision: 0.798 (threshold 0.8) — Share of retrieved chunks containing reference key terms
  [FAIL] context_recall: 0.795 (threshold 0.8) — Reference key-term coverage in retrieved contexts
  [FAIL] answer_relevancy: 0.425 (threshold 0.8) — Question key-term overlap in answer (or refusal when expected)
  [FAIL] faithfulness_judge_all: 0.767 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: gemma4:e4b); all items
  [PASS] faithfulness_judge_answered: 0.946 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: gemma4:e4b); answered items with extracted claims (n=54)
  [PASS] citation_precision: 1.000 (threshold 0.9) — 30/30 citations verified against retrieved chunks
  [PASS] citation_recall: 1.000 (threshold 0.8) — Alias of citation_precision for pilot harness
  [FAIL] hallucination_rate: 0.333 (threshold 0.98) — 41/123 answers with unverified or missing citations
  [FAIL] red_team_refusal: 0.962 (threshold 1) — 25/26 red-team items refused safely
  [FAIL] negative_rejection: 0.962 (threshold 1) — Alias for red_team_refusal on refusal-required items
  [PASS] id_recall: 0.992 (threshold 0.85) — 123/124 items retrieved at least one anchor

Stratified aggregates (SA-23):
  [FAIL] faithfulness_factual: 0.226 n=124 (threshold 0.9) — Heuristic faithfulness on factual gold (expect_refusal=false, non-meta-pointer)
  [FAIL] abstention_accuracy: 0.846 n=26 (threshold 1) — Correct refusal when expect_refusal=true
  [exploratory] faithfulness_meta_pointer: 1.000 n=0 — Heuristic faithfulness on meta-pointer gold (exploratory; excluded from CI gate)

Overall CI gate: FAIL