=== Judo Helper Eval ===

Gold set:  eval/gold/full.json (150 items)
Pipeline:  constrained
Chunking:  naive
Abstain:   threshold
Backend:   ollama
RAGAS judge: enabled (slow pass)
DB:        data/judo-docs.sqlite

Running pipeline on gold set...

Running RAGAS faithfulness judge (claim extraction + NLI)...

[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"::\":\"The upper weight limit for the senior men's -60 kg category in Judo, according to the provided rules, does not explicitly appear as a standalone category. However, based on the pattern of other categories and the information given, it can be inferred that if there were a -60 kg category for seniors (which is more commonly seen in cadet or junior classifications), its upper limit would foll"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The upper weight limit for the senior men's -60 kg category as per the provided information isn't directly stated, but typically in judo classifications, each weight class has a specific range. The categories listed show that the -60 kg category for cadet men goes up to and including 63.0 kg. For seniors, the next higher category is -66 kg which starts at over 60.0 kg, indicating that the "
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The upper weight limit for the senior men's -60 kg category as per the provided information isn't directly stated, but based on the pattern of other categories, it can be inferred that:\", \"answer\":\"The senior men's category listed explicitly in the given text starts from -66 kg. However, if we follow the structure and logic presented for other weight classes, the upper limit for a -60 kg c"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"::\":\"The minimum age requirement for senior competitors, both male and female, according to the IJF Sport & Organisation Rules 2025, is 15 years old (calendar year).\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The minimum age requirement for participating in senior judo competitions, according to the IJF Sport & Organisation Rules, is 15 years old (calendar year). This means that athletes must be at least 15 years old by January 1st of the competition year to compete in the senior category.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"::\":\"The minimum age requirement for participating in senior judo competitions, according to the International Judo Federation (IJF) rules, is 15 years old in the calendar year of the competition.\" }"
[faithfulness_judge] sentence fallback extracted 1 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"ippon_criteria\": [ { \"criteria_name\": \"Speed\", \"description\": \"The technique must be executed with speed.\" }, { \"criteria_name\": \"Force\", \"description\": \"The throw must demonstrate significant force and control.\" }, { \"criteria_name\": \"Back Landing\", \"description\": \"The opponent must land primarily on their back, specifically the shoulders at an angle close to 90° relative to the tatami. Contac"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"ippon_criteria\": [ { \"criteria_name\": \"Speed\", \"description\": \"The technique must be executed with speed.\" }, { \"criteria_name\": \"Force\", \"description\": \"The throw must demonstrate significant force and control.\" }, { \"criteria_name\": \"Back Landing\", \"description\": \"The opponent must land primarily on their back, with the shoulders at an angle close to 90° relative to the tatami. Shoulder and u"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"ippon_criteria\": [ { \"description\": \"The action must be a continuation of Kodokan classified judo techniques or recognized variations without stopping.\", \"details\": \"If there is any stop during the execution, no score is given.\" }, { \"description\": \"Continuity in technique execution is essential; interruptions disqualify an ippon from being awarded.\", \"details\": \"\" }, { \"description\": \"The four"
[faithfulness_judge] sentence fallback extracted 16 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"error\": \"The athlete weighing exactly 63.0 kg does not fit into any of the specified senior women's categories according to the provided IJF rules.\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"error\": \"The athlete weighing exactly 63.0 kg does not fit into any of the specified senior women's categories according to the provided IJF rules.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"error\": \"Invalid weight category\", \"details\": \"A weight of exactly 63.0 kg does not fit within any senior women's weight categories as defined in the provided IJF rules excerpts. The -57 kg category ends at 57.0 kg and the -63 kg category starts above 63.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"error\": \"The provided text does not specify unique weight categories for veterans (masters) beyond referencing that they use the same weight categories as senior men and women. The exact weight divisions such as -60kg or +100kg are not listed under the veteran section, but rather it is implied to follow the senior weight classes.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"{\":\"{\", \"}\":\"}\", \"\\\"\":\"\\\"\", \"(\":\"(\", \")\":\")\", \"[\":\"[\", \"]\":\"]\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"{\":\"ERROR\", \"}\":\"IN THE GIVEN DOCUMENTS, THERE IS NO DIRECT MENTION OF HOW IJF BELT COLORS CORRESPOND TO SPECIFIC BJA KYU GRADES. HOWEVER, BASED ON THE INFORMATION PROVIDED IN THE 'BRITISH JUDO ASSOCIATION MON GRADE PROMOTION SYLLABUS' AND 'IJF BELT GRADING GUIDE', WE CAN MAKE SOME REASONABLE INFERENCES:\"}"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"{\":\"ERROR\", \"}\":\"IN THE GIVEN DOCUMENTS, THERE IS NO DIRECT MENTION OF HOW IJF BELT COLORS ARE CONVERTED TO SPECIFIC BJA KYU GRADES. HOWEVER, BASED ON THE INFORMATION PROVIDED IN THE 'BRITISH JUDO ASSOCIATION MON GRADE PROMOTION SYLLABUS' AND 'IJF BELT GRADES GUIDE', WE CAN MAKE SOME REASONABLE INFERENCES:\"}"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"relevant_excerpts\": [ { \"text\": \"Players must convert from Mon to Kyu grades upon reaching age 18.\", \"page\": \"not specified\" }, { \"text\": \"The syllabus is almost identical between the Mon and Kyu grades to allow easy transfer from Mon to Kyu\", \"page\": \"2 of 30\" } ], \"answer\": \"Based on the provided information, there is no explicit minimum time-in-grade requirement for an 18th Mon judoka before"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"@/components/Answer\":{ \"answer\":\"The provided official excerpts do not specify a mandatory 'minimum time-in-grade' requirement for an 18th Mon judoka before converting to the Kyu grade pathway. The documents indicate that players holding a Mon grade must convert upon reaching age 18, and they may choose to convert at any point after turning 14 years of age. While there are detailed time-in-grad"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"@/components/Answer\":{ \"answer\":\"The provided official excerpts do not specify a mandatory 'minimum time-in-grade' requirement for an 18th Mon judoka before converting to the Kyu grade pathway. The documents indicate that players holding a Mon grade must convert upon reaching age 18, and they may choose to convert at any point after turning 14 years of age. While there are detailed time-in-grad"
[faithfulness_judge] sentence fallback extracted 5 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"Under IJF rules, the upper weight limit for senior women's -48 kg category is up to and including 48.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"Under IJF rules, the upper weight limit for senior women's -48 kg category is up to and including 48.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"Under IJF rules, the upper weight limit for senior women's -48 kg category is up to and including 48.0 kg.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"relevant_factoids\": [ { \"page\": \"[4]\", \"factoid\": \"BJA Kyu Grade Promotion Syllabus (January 2025), p.3\" }, { \"page\": \"[5]\", \"factoid\": \"IJF Sport & Organisation Rules 2025, p.93\" }, { \"page\": \"[6]\", \"factoid\": \"BJA Dan Grade Promotion Syllabus (July 2025), p.15\" }, { \"page\": \"[7]\", \"factoid\": \"IJF Sport & Organisation Rules 2025, p.246\" } ], \"answer\": \"There is no specific age at which an athl"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"kg\": \"\", \"2\": \"\", \"3\": \"\", \"Age Groups\": { \"Athletes under 15 years of age\": \"not allowed to compete in any official IJF event\", \"Cadet\": \"men and women under 18, ages 15–17\", \"Junior\": \"men and women under 21, ages 15–20\", \"Senior\": \"men and women with only a lower age limit of 15 years\" }, \"transition from U18 to Senior\": \"Athletes transition from the U18 category to Senior status when they r"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"kg\": \"\", \"2\": \"\", \"3\": \"\", \"Age Groups\": [ { \"Name\": \"Cadet\", \"Description\": \"Men and women under 18, ages 15 to 17 years (calendar year).\" }, { \"Name\": \"Junior\", \"Description\": \"Men and women under 21, ages 15 to 20 years (calendar year).\" }, { \"Name\": \"Senior\", \"Description\": \"Men and women with only a lower age limit of 15 years (calendar year).\" } ] }"
[faithfulness_judge] sentence fallback extracted 4 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"=\": \"There seems to be a discrepancy in your request. Based on the provided IJF Sport & Organisation Rules 2025 document excerpts, here are the correct weight categories for cadet men's and women's individual events:\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"=\": \"The provided information does not explicitly list all cadet men's weight categories. However, based on the given data for other age groups and events, we can infer the missing details for cadet men's individual events as follows:\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"=\": \"The provided information does not explicitly list all cadet men's weight categories. However, based on the given data for other age groups and events, we can infer the missing details for cadet men's individual events as follows:\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"An 11-year-old athlete would fall under the 'recommendations only' category as per IJF guidelines, which means national federations can design their own concepts for this age group. Specifically, an 11-year-old falls into the 9/10 years or potentially extended categories depending on the national federation's rules since official IJF competitions do not include athletes younger than 1"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"An 11-year-old athlete would fall under the recommendations provided by national federations to design their own concepts, as per IJF guidelines since official IJF competitions do not include athletes younger than 15 years old (calendar year). The specific age category for an 11-year-old is typically part of the 9/10 years group or potentially extended based on national federation rul"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"An 11-year-old athlete would fall under the age category recommendations provided by national federations for designing their own concepts, as per IJF rules which do not specify official competition categories below the age of 15 years old (calendar year). The specific age group recommendation is typically 9/10 years or potentially an extension depending on the national federation's r"
[faithfulness_judge] sentence fallback extracted 6 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"weightCategory\": \"+100 kg\", \"categoryType\": \"Heavyweight\", \"description\": \"Over 100.0 kg, no upper weight limit\" }"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"The heaviest weight category for senior men under IJF rules, as per the provided information, is the +100 kg (Heavyweight) category.\" }"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"@/\": \"The heaviest weight category for senior men under IJF rules, as per the provided information, is the +100 kg (Heavyweight) category.\" }"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"According to the IJF Sport & Organisation Rules 2025, the weight categories and their specific tolerances vary between individual competitions and mixed teams competitions. For Cadet Women's individual events, athletes in the -48 kg category must weigh over 44.0 kg up to and including 48.2 kg due to a +200 g tolerance for underclothing. However, for Senior and Junior Mixed Teams, the "
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"The IJF Sport & Organisation Rules 2025 specify distinct weight categories and tolerance rules for both individual and mixed team competitions. For Cadet Women's individual events, athletes in the -48 kg category are allowed a maximum of 48.2 kg (including an additional 200g allowance). In contrast, for Senior and Junior Mixed Team competitions, the equivalent category is listed as -5"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"The IJF Sport & Organisation Rules 2025 specify distinct weight categories and tolerance rules for both individual and mixed team competitions. For Cadet Women's individual events, athletes in the -48 kg category are allowed a maximum of 48.2 kg (including an additional 200g allowance). In contrast, for Senior and Junior Mixed Team competitions, the equivalent category is listed as -5"
[faithfulness_judge] sentence fallback extracted 9 claims
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"answer\": \"Based on the information provided, a 16-year-old male athlete weighing exactly 73 kg would compete in the Cadet Men -73 kg category. This is because the specified weight range for this division is 'Over 66.0 kg up to and including 73.0 kg'. For female athletes of the same age, there isn't a specific category listed that matches exactly 73 kg; however, they would likely fall into the +"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"Based on the information provided, a 16-year-old male athlete weighing exactly 73 kg would compete in the Cadet Men -73 kg category. This is because the specified weight range for this division is 'Over 66.0 kg up to and including 73.0 kg'. For female athletes of the same age, there isn't a specific category listed that matches exactly 73 kg; however, they would likely fall into the +"
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"answer\": \"Based on the information provided, a 16-year-old male athlete weighing exactly 73 kg would compete in the Cadet Men -73 kg category. This is because the specified weight range for this division is 'Over 66.0 kg up to and including 73.0 kg'. For female athletes of the same age, there isn't a specific category listed that matches exactly 73 kg; however, they would likely fall into the +"
[faithfulness_judge] empty claims (attempt 1/3, simple=false) model=qwen2.5:14b raw="{ \"shido\": { \"description\": \"Slight infringements that serve as warnings and do not give a score to the opponent.\", \"accumulation\": \"Two shidos can be awarded before reaching hansoku-make (third penalty).\", \"outcome\": \"The third shido results in hansoku-make, disqualifying the athlete.\" }, \"hansoku-make\": { \"description\": \"Grave infringements that result in immediate disqualification without prior"
[faithfulness_judge] empty claims (attempt 2/3, simple=true) model=qwen2.5:14b raw="{ \"shido\": { \"description\": \"Slight infringements that do not give a score to the opponent but can accumulate toward disqualification.\", \"accumulation\": \"Two shidos serve as warnings, and the third results in hansoku-make.\" }, \"hansoku-make\": { \"direct_penalty\": \"Grave infringements awarded without prior accumulation of shido\", \"outcome\": \"Immediate disqualification with opponent winning by ippon\""
[faithfulness_judge] empty claims (attempt 3/3, simple=true) model=qwen2.5:14b raw="{ \"shido\": { \"description\": \"Slight infringements that serve as warnings and do not give a score to the opponent.\", \"accumulation\": \"Two shidos can be awarded before reaching hansoku-make (third penalty).\", \"outcome\": \"The third shido results in hansoku-make, disqualifying the athlete.\" }, \"hansoku-make\": { \"description\": \"Grave infringements that result in immediate disqualification without prior"
Metrics:
  [FAIL] faithfulness: 0.500 (threshold 0.9) — Heuristic term overlap (CI regression only; not paper faithfulness)
  [FAIL] context_precision: 0.794 (threshold 0.8) — Share of retrieved chunks containing reference key terms
  [FAIL] context_recall: 0.788 (threshold 0.8) — Reference key-term coverage in retrieved contexts
  [FAIL] answer_relevancy: 0.736 (threshold 0.8) — Question key-term overlap in answer (or refusal when expected)
  [FAIL] faithfulness_judge_all: 0.847 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b); all items
  [FAIL] faithfulness_judge_answered: 0.886 (threshold 0.9) — Claim-level NLI vs retrieved context (judge: qwen2.5:14b); answered items with extracted claims (n=114)
  [PASS] citation_precision: 1.000 (threshold 0.9) — 236/236 citations verified against retrieved chunks
  [PASS] citation_recall: 1.000 (threshold 0.8) — Alias of citation_precision for pilot harness
  [PASS] hallucination_rate: 0.000 (threshold 0.98) — 0/122 answers with unverified or missing citations
  [PASS] red_team_refusal: 1.000 (threshold 1) — 26/26 red-team items refused safely
  [PASS] negative_rejection: 1.000 (threshold 1) — Alias for red_team_refusal on refusal-required items
  [PASS] id_recall: 0.984 (threshold 0.85) — 122/124 items retrieved at least one anchor

Stratified aggregates (SA-23):
  [FAIL] faithfulness_factual: 0.395 n=124 (threshold 0.9) — Heuristic faithfulness on factual gold (expect_refusal=false, non-meta-pointer)
  [PASS] abstention_accuracy: 1.000 n=26 (threshold 1) — Correct refusal when expect_refusal=true
  [exploratory] faithfulness_meta_pointer: 1.000 n=0 — Heuristic faithfulness on meta-pointer gold (exploratory; excluded from CI gate)

Overall CI gate: FAIL