=== Judo Helper Eval ===

Gold set:  eval/gold/full.json (150 items)
Pipeline:  constrained
Chunking:  naive
Abstain:   threshold
Backend:   ollama
RAGAS judge: enabled (slow pass)
DB:        data/judo-docs.sqlite

Running pipeline on gold set...

Running RAGAS faithfulness judge (claim extraction + NLI)...

[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"error\": \"There seems to be a discrepancy in your query based on the provided rules. The IJF Sport & Organisation Rules document does not list a '-60 kg' weight category for Senior and Junior Men. It lists '-66 kg Up to and including 69.3 kg' as the closest lower weight limit category for men. If you are looking for the upper weight limit of the Extra-lightweight category for men, it would be 66"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"The upper weight limit for the Senior and Junior Men -60 kg Extra-lightweight category is 60.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"The upper weight limit for the Senior and Junior Men -60 kg Extra-lightweight category is 60.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"The minimum age for senior competitors in judo, according to the IJF Sport & Organisation Rules, is 15 years (calendar year).\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"ippon_criteria\": [ \"Speed\", \"Force\", \"Landing on the back\", \"Skillfully controlled until the end of the landing\" ], \"shoulder_angle\": \"Close to 90° to the tatami\", \"technique_requirement\": \"Must be identified from Kodokan classified techniques (including recognized variations)\", \"referee_signal\": \"Raises one arm high above the head with the palm of the hand facing forward\", \"additional_informat"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \":: Ippon Criteria and Scoring ::\":\n\n-1.0"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \":: Ippon Criteria and Scoring ::\":\n\n-1.0"
[faithfulness_judge] sentence fallback extracted 16 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"category\": \"-63 kg\", \"description\": \"Half-middleweight Over 57.0 kg up to and including 63.0 kg\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"A senior woman weighing 63 kg would compete in the **-63 kg** weight category, which is classified as *Half-middleweight*.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"A senior woman weighing 63 kg would compete in the **-63 kg** weight category, which is classified as *Half-middleweight*.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"{\":\"{\", \"}\":\"}\", \"\\\"\":\"\\\"\", \"[\":\"[\", \"]\":\"]\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"{\":\"[\", \"}\":\"]\", \",\":\";\"}"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"{\":\"[\", \"}\":\"]\", \",\":\";\"}"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"48.0 kg\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{\"The upper weight limit for the Senior and Junior Women -48 kg category is 48.0 kg.\": \"This means that athletes competing in this category must weigh no more than 48.0 kg to be eligible, with a +5% tolerance if they are part of a team event following an individual competition where they have already passed the official weigh-in for their individual weight class.\"}"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{\"The upper weight limit for the Senior and Junior Women -48 kg category is 48.0 kg.\": \"This means that athletes competing in this category must weigh no more than 48.0 kg to be eligible, with a +5% tolerance if they are part of a team event following an individual competition where they have already passed the official weigh-in for their individual weight class.\"}"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"::\":\"Based on the provided information from the IJF Sport & Organisation Rules, here are the age limits for different categories in judo competitions:\\n\\n- **Cadet Category**: Athletes must be under 18 years old and can be aged 15, 16, or 17.\\n- **Junior Category**: Athletes must be under 21 years old and can range from age 15 to 20.\\n- **Senior Category**: The minimum age requirement is 15 yea"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The lower age limit for all IJF competitions is 15 years of age.\", \"::\":\"Cadet category includes athletes under the age of 18, specifically those who are 15, 16 or 17 years old.\", \"::\":\"Junior category includes athletes under the age of 21, covering ages from 15 to 20.\", \"::\":\"Senior category has a minimum age requirement of 15 years.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The lower age limit for all IJF competitions is 15 years of age.\", \"::\":\"Cadet category includes athletes under the age of 18, specifically those who are 15, 16 or 17 years old.\", \"::\":\"Junior category includes athletes under the age of 21, covering ages from 15 to 20.\", \"::\":\"Senior category has a minimum age requirement of 15 years.\" }"
[faithfulness_judge] sentence fallback extracted 6 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"weight_categories\": { \"Cadet Women\": [ {\"category\": \"-40 kg\", \"range\": \"(up to and including) 40.2 kg\"}, {\"category\": \"-44 kg\", \"range\": \"over 40.0 kg up to and including 44.2 kg\"}, {\"category\": \"-48 kg\", \"range\": \"over 44.0 kg up to and including 48.2 kg\"}, {\"category\": \"-52 kg\", \"range\": \"over 48.0 kg up to and including 52.2 kg\"}, {\"category\": \"-57 kg\", \"range\": \"over 52.0 kg up to and inclu"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"cadet_women\": { \"-40kg\": \"Up to and including 40.0 kg\", \"-44kg\": \"Over 40.0 kg up to and including 44.0 kg\", \"-48kg\": \"Over 44.0 kg up to and including 48.0 kg\", \"-52kg\": \"Over 48.0 kg up to and including 52.0 kg\", \"-57kg\": \"Over 52.0 kg up to and including 57.0 kg\", \"-63kg\": \"Over 57.0 kg up to and including 63.0 kg\", \"-70kg\": \"Over 63.0 kg up to and including 70.0 kg\", \"+70kg\": \"Over 70.0 kg\""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"For an 11-year-old competitor, they would fall into the '11/12 YEARS' category. However, it's important to note that no athlete less than 15 years old (calendar year) is allowed to take part in any IJF official competition. Therefore, while weight categories for this age group are specified as 32 kg for females and 30 kg for males, participation in IJF competitions would not be permit"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \n  \"age_category\": \"11/12 YEARS\",\n  \"participation_rule\": \"No athlete less than 15 years old (calendar year) is allowed to take part in any IJF official competition.\",\n  \"weight_categories\": {\n    \"female\": \"32 kg\",\n    \"male\": \"30 kg\"\n  }\n}"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \n  \"age_category\": \"11/12 YEARS\",\n  \"participation_rule\": \"No athlete less than 15 years old (calendar year) is allowed to take part in any IJF official competition.\",\n  \"weight_categories\": {\n    \"female\": \"32 kg\",\n    \"male\": \"30 kg\"\n  }\n}"
[faithfulness_judge] sentence fallback extracted 6 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"weight_category\": \"+100 kg\", \"minimum_weight\": \"Over 100.0 kg\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{\"The heaviest weight category for Senior Men, according to the provided information, is +100 kg, which includes athletes weighing over 100.0 kg.\":\n\n-60,\t\"Extra-lightweight\": \"Up to and including 60.0 kg\""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{\"The heaviest weight category for Senior Men, according to the provided information, is +100 kg, which includes athletes weighing over 100.0 kg.\":\n\n-60,\t\"Extra-lightweight\": \"Up to and including 60.0 kg\""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"The weight categories for individual events and mixed team competitions are not identical, but they do overlap in some cases with different tolerances or limits. Here's a summary based on the provided information:\\n\\n**Cadet Women:**\\n- Individual: -48 kg (Over 44.0 kg up to and including 48.2 kg)\\n- Team: -48 kg (Over 48.0 kg up to and including 54.6 kg)\\n\\n**Senior/Junior Men's Mixe"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"weight_categories\": { \"cadet_women_individual\": \"-48 kg: Over 44.0 kg up to and including 48.2 kg\", \"cadet_women_team\": \"-48 kg: Over 48.0 kg up to and including 54.6 kg\", \"junior_and_senior_men_team\": { \"-66 kg\": \"Up to and including 69.3 kg\", \"-73 kg\": \"Over 66.0 kg up to and including 76.7 kg\", \"-81 kg\": \"Over 73.0 kg up to and including 85.1 kg\", \"-90 kg\": \"Over 81.0 kg up to and including "
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"weight_categories\": { \"cadet_women_individual\": \"-48 kg: Over 44.0 kg up to and including 48.2 kg\", \"cadet_women_team\": \"-48 kg: Over 48.0 kg up to and including 54.6 kg\", \"junior_and_senior_men_team\": { \"-66 kg\": \"Up to and including 69.3 kg\", \"-73 kg\": \"Over 66.0 kg up to and including 76.7 kg\", \"-81 kg\": \"Over 73.0 kg up to and including 85.1 kg\", \"-90 kg\": \"Over 81.0 kg up to and including "
[faithfulness_judge] sentence fallback extracted 16 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"::\":\"Based on the provided information and the IJF Sport & Organisation Rules for 2025, here's how the weight divisions would apply to a 16-year-old athlete in the Cadet category weighing 73 kg:\"\n\n  \t\t\t\t\t\n \t\t\t\t\n \t\t\t\t\t\n \t\t\t\t\t\t\n\t\t\t\t\t\t\t}"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"Based on the provided information and guidelines from the IJF Sport & Organisation Rules 2025, here's how to categorize a 16-year-old male or female athlete who weighs 73 kg into their respective divisions for cadet competitions:\\n\\nFor a **male** athlete weighing 73 kg in the Cadet category (under 18 years of age), he would compete in the **-73 kg** weight division.\\n\\nFor a **female** at"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"Based on the provided information and guidelines from the IJF Sport & Organisation Rules 2025, here's how to categorize a 16-year-old male or female athlete who weighs 73 kg into their respective divisions for cadet competitions:\\n\\nFor a **male** athlete weighing 73 kg in the Cadet category (under 18 years of age), he would compete in the **-73 kg** weight division.\\n\\nFor a **female** at"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"yuko_definition\": \"Yuko is defined as a score of one point on the scoreboard.\", \"referee_signal_yuko\": \"The referee signals yuko by raising one arm sideways with the palm facing downwards at an angle of 45 degrees from the body.\", \"criteria_tachi_waza\": [ { \"description\": \"Side landings (90 degrees or close to it)\", \"score\": \"Yuko\" }, { \"description\": \"Landing on upper back\", \"score\": \"Yuko\" },"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"In Judo according to the IJF Sport & Organisation Rules 2025, a shido is a penalty given for slight infringements such as lack of combativity, illegal moves that are not considered dangerous, or untidy attire and hair. Two shidos can be awarded before reaching hansoku-make (disqualification). A shido does not award any points to the opponent; only technical scores like waza-ari or ipp"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"shido\": { \"description\": \"A penalty for slight infringements such as lack of combativity, an illegal move that is not considered dangerous, or untidy attire or hair.\", \"accumulation\": \"Two shido will be given as warnings and the third will result in a hansoku-make.\" }, \"hansoku-make\": { \"description\": \"A penalty for grave infringements resulting in immediate disqualification.\", \"winning_method\""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"shido\": { \"description\": \"A penalty for slight infringements such as lack of combativity, an illegal move that is not considered dangerous, or untidy attire or hair.\", \"accumulation\": \"Two shido will be given as warnings and the third will result in a hansoku-make.\" }, \"hansoku-make\": { \"description\": \"A penalty for grave infringements resulting in immediate disqualification.\", \"winning_method\""
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"stages\": [ { \"stage\": 1, \"description\": \"Initial Rest (mind and body) for 24-48 hours. Complete physical and cognitive rest including no exercise and minimized screen time.\", \"duration\": \"24-48 hrs\" }, { \"stage\": 2, \"sub_stages\": [ { \"sub_stage\": \"A\", \"description\": \"Relative Rest for 14 days. Must be symptom free before continuing to the next stage.\", \"duration\": \"14 days\" }, { \"sub_stage\": \"B"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"stages\": [ { \"stage\": 1, \"description\": \"Initial Rest (mind and body) for 24-48 hours. Complete physical and cognitive rest including no exercise and minimized screen time.\", \"duration\": \"24-48 hrs\" }, { \"stage\": 2, \"sub_stages\": [ { \"sub_stage\": \"A\", \"description\": \"Relative Rest (14 days). Must be symptom free before continuing to the next stage. No physical or cognitive exertion that provoke"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"stages\": [ { \"stage\": 1, \"description\": \"Initial Rest (mind and body) for 24-48 hours. Complete physical and cognitive rest including no exercise and minimized screen time.\", \"duration\": \"24-48 hrs\" }, { \"stage\": 2, \"sub_stages\": [ { \"sub_stage\": \"A\", \"description\": \"Relative Rest (14 days). Must be symptom free before continuing to the next stage. No physical or cognitive exertion that provoke"
[faithfulness_judge] sentence fallback extracted 23 claims
Metrics:
  [FAIL] faithfulness: 0.413 (threshold 0.9) — Heuristic term overlap (CI regression only; not paper faithfulness)
  [FAIL] context_precision: 0.794 (threshold 0.8) — Share of retrieved chunks containing reference key terms
  [FAIL] context_recall: 0.788 (threshold 0.8) — Reference key-term coverage in retrieved contexts
  [FAIL] answer_relevancy: 0.541 (threshold 0.8) — Question key-term overlap in answer (or refusal when expected)
  [FAIL] faithfulness_judge_all: 0.822 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b); all items
  [PASS] faithfulness_judge_answered: 0.926 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b); answered items with extracted claims (n=105)
  [PASS] citation_precision: 1.000 (threshold 0.9) — 196/196 citations verified against retrieved chunks
  [PASS] citation_recall: 1.000 (threshold 0.8) — Alias of citation_precision for pilot harness
  [PASS] hallucination_rate: 0.000 (threshold 0.98) — 0/113 answers with unverified or missing citations
  [PASS] red_team_refusal: 1.000 (threshold 1) — 26/26 red-team items refused safely
  [PASS] negative_rejection: 1.000 (threshold 1) — Alias for red_team_refusal on refusal-required items
  [PASS] id_recall: 0.984 (threshold 0.85) — 122/124 items retrieved at least one anchor

Stratified aggregates (SA-23):
  [FAIL] faithfulness_factual: 0.290 n=124 (threshold 0.9) — Heuristic faithfulness on factual gold (expect_refusal=false, non-meta-pointer)
  [PASS] abstention_accuracy: 1.000 n=26 (threshold 1) — Correct refusal when expect_refusal=true
  [exploratory] faithfulness_meta_pointer: 1.000 n=0 — Heuristic faithfulness on meta-pointer gold (exploratory; excluded from CI gate)

Overall CI gate: FAIL