================================================================================
OSF REGISTRATION FORM — COPY-PASTE BLOCKS
Study: Judo Helper RAG Dissertation
Prepared: 2026-06-23 (SA-60)
Source: research/osf-preregistration/PREREGISTRATION.md + ANALYSIS-PLAN.md
================================================================================

--------------------------------------------------------------------------------
FIELD: Study title
--------------------------------------------------------------------------------
Constrained versus Agentic RAG in a Safety-Critical, Low-Resource Domain

--------------------------------------------------------------------------------
FIELD: Authors / contributors
--------------------------------------------------------------------------------
Daniel [surname] — Principal investigator / dissertation author
[Add co-authors or supervisors if required by your institution]

--------------------------------------------------------------------------------
FIELD: Study design / description
--------------------------------------------------------------------------------
Confirmatory comparison of constrained single-shot RAG versus agentic RAG (ReAct and self-reflective variants) on a closed BJA/IJF/safeguarding corpus embedded in the JudoHQ Judo Helper (/api/ai/judo-helper).

Gold evaluation set: 150 items in full.json (120 held-out test items test-001–test-120; 30 dev items for engineering only). Nested ablations: chunking (naive, table-aware, late) and abstention control (threshold, evaluator-halt, unbounded).

Platform: local Ollama inference with hybrid retrieval (HYBRID_SEARCH=true, cross-encoder reranking).

--------------------------------------------------------------------------------
FIELD: Research questions
--------------------------------------------------------------------------------
RQ1 (primary): To what extent does agentic RAG improve or degrade faithfulness, citation accuracy, latency, and answer relevancy versus constrained RAG?

RQ2 (ablation): What is the impact of table-aware and late chunking versus naive chunking on retrieval metrics, stratified by GRADE difficulty?

RQ3 (ablation): How do safe abstention strategies compare on correct Soft Refusal Rate versus Attack Success Rate under adversarial prompting?

--------------------------------------------------------------------------------
FIELD: Hypotheses
--------------------------------------------------------------------------------
RQ1 — Architectural efficacy
H1.1: Agentic RAG shows no improvement or a decrease in faithfulness versus constrained RAG (reasoning drift). Null: no difference in faithfulness. One-tailed.
H1.2: Constrained RAG achieves higher citation precision (ALiiCE). Null: no difference. One-tailed.
H1.3: Agentic loops incur disproportionately higher latency and token cost per query. Null: no difference. One-tailed.

RQ2 — Chunking ablation
H2.1: Naive chunking disproportionately degrades context recall for numerical/tabular queries versus textual queries. One-tailed.
H2.2: Table-aware chunking narrows or eliminates any agentic advantage on multi-evidence queries. Exploratory / two-tailed.
H2.3: Late chunking improves multi-hop context recall over naive chunking independently of table-aware segmentation. Exploratory / two-tailed.

RQ3 — Safe abstention
H3.1: Agentic RAG exhibits higher Attack Success Rate under adversarial prompting. Null: no difference. One-tailed.
H3.2: Thresholded context-relevance gating achieves higher correct Soft Refusal Rate than agentic evaluator-halt. Null: no difference. One-tailed.

--------------------------------------------------------------------------------
FIELD: Primary outcome / endpoint
--------------------------------------------------------------------------------
Primary metric: faithfulness_judge_answered

Definition: Claim-level NLI entailment of extracted claims against retrieved context (not reference-answer vocabulary overlap).

Direction: One-tailed test for H1.1 (agentic ≤ constrained).

Production gate threshold: ≥ 0.85 minimum; 0.90 strong target.

Judge model: gemma4:e4b (OLLAMA_JUDGE_MODEL); generator/chat model gemma4:12b (OLLAMA_CHAT_MODEL).

Exclusions from denominator: expect_refusal items; fixed-abstention strings; items with empty claim extraction.

Secondary headline (regression visibility only, not primary claim): faithfulness_judge_all.

--------------------------------------------------------------------------------
FIELD: Secondary outcomes
--------------------------------------------------------------------------------
hallucination_rate
citation_precision (ALiiCE)
context_recall
answer_relevancy
latency and token cost per query
red_team_refusal / Attack Success Rate
correct Soft Refusal Rate (abstention accuracy)

Multiplicity control: Holm–Bonferroni correction at α = 0.05 across secondary metrics (primary endpoint not corrected against secondaries).

--------------------------------------------------------------------------------
FIELD: Sample size / evaluation set
--------------------------------------------------------------------------------
Full gold set: n = 150 items (full.json).
Confirmatory claims: n = 120 held-out test items (test-001 through test-120).
Dev split: n = 30 (dev-001 through dev-030) — engineering tuning only; excluded from confirmatory claims.

Human anchor subsample: n = 30 items for prediction-powered inference (PPI), stratified across GRADE difficulty and topic; two independent annotators; Cohen's κ reported.

--------------------------------------------------------------------------------
FIELD: Analysis plan — primary (H1.1)
--------------------------------------------------------------------------------
Unit of analysis: paired item-level scores across architectures on the same gold item. Replicate-level aggregation: mean per item across R = 5 seeds before architecture comparison.

Metric: faithfulness_judge_answered on test split.

Methods:
- Point estimate: mean across test items per architecture
- Uncertainty: bootstrap 95% CI, 10,000 resamples, paired by item
- Hypothesis test: Wilcoxon signed-rank (paired), one-tailed (agentic ≤ constrained)
- Effect size: rank-biserial correlation; Cohen's d on item-level paired differences
- α = 0.05
- PPI overlay: rectified estimator anchoring machine judge scores to n = 30 human-labelled anchor items (Angelopoulos et al.)

Primary contrast: constrained (naive chunking, threshold abstention) versus agentic ReAct (naive chunking, evaluator-halt abstention).

--------------------------------------------------------------------------------
FIELD: Analysis plan — secondary and refusal outcomes
--------------------------------------------------------------------------------
Secondary continuous metrics: same paired bootstrap CI + Wilcoxon framework; Holm–Bonferroni step-down across secondary family at α = 0.05.

Binary/refusal outcomes (H3.1, H3.2): McNemar test on paired binary outcomes; bootstrap CI on proportions for descriptive reporting.

Stratified reporting by GRADE stratum (single-hop-text, single-hop-tabular, multi-hop-text, multi-hop-tabular, out-of-scope). Under-powered strata flagged exploratory.

--------------------------------------------------------------------------------
FIELD: Replicate design / randomisation
--------------------------------------------------------------------------------
Replicates: R = 5 passes per architecture × condition.
Seeds: fixed set {42, 43, 44, 45, 46} — one seed per replicate.
Decoding: temperature = 0.1 (held constant).
Item order: randomised per run (seeded).

Confirmatory run order (fixed):
1. Constrained + naive + threshold abstention (baseline)
2. Agentic ReAct + naive + evaluator halt
3. Agentic self-reflective + naive + evaluator halt
4. Constrained + table-aware + threshold abstention (Ablation A)
5. Constrained + naive + no abstention (Ablation B)

Locked environment (pre-registration):
OLLAMA_CHAT_MODEL=gemma4:12b
OLLAMA_JUDGE_MODEL=gemma4:e4b
HYBRID_SEARCH=true
RERANK_BACKEND=cross-encoder

--------------------------------------------------------------------------------
FIELD: Exclusions and reporting rules
--------------------------------------------------------------------------------
- Heuristic faithfulness (term-overlap vs reference answer): CI regression only; never reported as paper headline.
- faithfulness_meta_pointer stratum: exploratory diagnostic; excluded from CI gate.
- Dev split optimisations disclosed; confirmatory claims only on frozen test split.
- Gold repairs (SA-31): meta-pointer reference_answer fields replaced with corpus-exact facts before registration freeze; see GOLD-REPAIR-LOG.md.
- Test split test-001–test-120 immutable after registration (SHA256: 45df8643d736026472f883741213fb5cef9e2f299a983287b8c08f4a167c2e3c).

--------------------------------------------------------------------------------
FIELD: Deviations policy
--------------------------------------------------------------------------------
Any post-registration deviation from this plan (metric definition change, unplanned exclusion, additional comparison) will be reported as exploratory with rationale in the dissertation limitations section. Corpus errors after freeze follow GOLD-FREEZE.md exception process (protocol deviation on OSF, gold-full-v2 if needed).

--------------------------------------------------------------------------------
FIELD: Data and materials availability
--------------------------------------------------------------------------------
Gold set: research/experiment/eval/gold/full.json (uploaded with this registration).
Eval harness script checksums: scripts-hash.txt.
Analysis scripts: committed in repository at hashes listed in scripts-hash.txt.
Raw run outputs: research/experiment/eval/runs/<run_id>/report.json.

--------------------------------------------------------------------------------
FIELD: Supporting document references
--------------------------------------------------------------------------------
PREREGISTRATION.md — full study design
ANALYSIS-PLAN.md — statistical methods detail
GOLD-FREEZE.md — freeze policy and confirmatory run parameters
GOLD-REPAIR-LOG.md — pre-freeze gold repair audit trail
manifest.json — package file checksums

================================================================================
END OF REGISTRATION-FIELDS.txt
================================================================================
