The cleanest parser was the worst retriever
Document RAG does not retrieve directly from a PDF; it retrieves from an index built from parsed, chunked text. Intrinsic parsing scores inspect output in isolation and cannot show whether answer-bearing evidence survived in a form the retriever can find.
In the audited deployment grid, table-enabled MinerU-on has the highest measured Boundary Clarity among complete outputs, 0.713 versus Prod's 0.610. Retrieval reverses the result: MinerU-on reaches 0.123 Hit@1 and 0.137 RCPS, while Prod reaches 0.549 and 0.583. The intrinsic winner retrieves worst in this pool.
Across the four complete 294-page outputs with defined Boundary Clarity, its correlation with RCPS is r = −0.74. Adding Marker's partial 38-page output gives r = −0.83 at n = 5. These small-pool results are descriptive, not a general law. In a source-aligned OHR-Bench check, semantic corruption lowers retrieval in every parser family while Boundary Clarity stays stable, changes non-monotonically or rises.
Select the pipeline, locate the loss, then decide whether to act
RCPS is a deployment workflow around an existing text-RAG stack, not another parser or similarity model.
The fixed frame contains 294 pages — 229 Korean government and 65 arXiv — with 663 held-out question–answer pairs. Each parser produces page-level Markdown, and each chunker turns it into a candidate index. The same queries, retrievers, depths and relevance rule score every pair, producing an RCPS matrix rather than one production-specific score.
The top-ranked pair is the provisional deployment choice. Coverage then inspects its parser output and chunks without a retriever. A covered span points downstream to the index or retriever; a split span calls for rechunking or overlap; an absent span calls for parser inspection. Switching or training is reserved for genuine evidence loss, and every changed configuration returns through RCPS before deployment.
One retrieval score for every parser–chunker pair
RCPS uses standard mean reciprocal rank in a controlled comparison; the contribution is fixing the probe and decision rule.
Let C(P) be the corpus from parser P and chunker C, and D the fixed probe. RCPS(P,C) averages MRR@k(r, C(P), D) over every declared retriever r and depth k. A query contributes 1/j when its first relevant chunk appears at rank j ≤ k, and zero otherwise. MRR@k averages over D; RCPS then averages across retrievers and depths, so higher is better.
Fixing C ranks parsers; fixing P ranks chunkers. The ranking belongs to the candidate pool and probe, not to the parser intrinsically.
Each page is parsed once per parser before every candidate corpus is chunked, indexed and searched. With m parsers, c chunkers and |R| retrievers, this requires m parsing runs and mc|R| retrieval evaluations. No training or manually labelled chunks are required because the source page and answer span define relevance.
Held-out question–answer pairs test whether the retriever can find evidence, not whether Markdown resembles a reference transcription.
We average three retrievers at k ∈ {1, 5, 10}: BGE-M3, multilingual-e5-large and Qwen3-Embedding-8B. A fixed deployment retriever can be used alone.
A hit must come from the reference page and contain the answer after shared Unicode, whitespace and Markdown normalisation. The rule is identical for every candidate.
Coverage identifies the layer that lost the answer
A retrieval score selects a pipeline but cannot locate a miss. Coverage separates parsing loss from chunking loss before retrieval.
For Prod, 134 of 663 spans (20.2%) have no normalised exact match before chunking. None of eight chunkers splits more than 15 spans (2.3%). The absent rate cannot change after chunking: in one audited case, MinerU-on writes A = 180 m² as A = 180m, and no chunker can restore the exponent.
MinerU-on's exact-match absent rate is 66.1%, 45.9 percentage points above Prod. A tolerant L4 matcher changes the rates, not the gap: 62.1% versus 16.9%, a 45.2-percentage-point difference. In the retained MinerU-off full-set check, GPT-5.4 marks 56% of Prod's exact-match-absent cases as recoverable surface artifacts, yet the MinerU-off–Prod retrieval-unusable gap remains 50.4 percentage points.
Two authors labelled 100 parser-masked absent cases and agreed on 81 before adjudication (κ = 0.615). Final labels mark 42 of 50 MinerU-on, 12 of 30 Prod and 19 of 20 PaddleOCR cases as retrieval-unusable. This stratified sample verifies direction, not population rates.
Table evidence sharpens the diagnosis: exact-match absence is 87.9% for MinerU-off, 41.7% for MinerU-on and 13.9% for Prod. Causes include dropped cells, text confined to captions, stamps, seals or figure labels, and corrupted numerals or units.
The normalised answer span has no exact match in the source-page transcription. Rechunking cannot restore it; inspect whether the cause is surface form or genuine evidence loss.
The span appears intact in a chunk, clearing the parser and chunker under the operational matcher. If retrieval fails, inspect the index or retriever.
The span exists in the page transcription but crosses chunk boundaries. Rechunking or overlap can repair this class directly.
The main comparison keeps one 294-page frame fixed
RCPS and coverage hold the corpus and probe fixed while candidate pipelines change.
Qwen3-VL-30B produced the manually de-noised pseudo-reference Markdown; GPT-5.4 generated the probe. An LLM-assisted check accepted 94/100 sampled pairs as clear, correct and supported. Neither complete set was human-verified.
MinerU-off is the submitted output with table recognition disabled; MinerU-on is a later audited, table-enabled 294-page run. Software and retrieval environments also changed, so their difference is not a controlled table-recognition ablation.
- 01Corpus: 294 pages, split into 229 Korean government pages and 65 arXiv pages.
- 02Probe: 663 verbatim-answerable Q–A, split into 527 government and 136 arXiv questions.
- 03Evidence frame: answers occur on 242 pages; 52 Q–A-free pages remain as distractors in every selection index. Coverage inspects all 294 outputs.
- 04External check: 1,043 source-aligned Law–Manual Q–A pairs from OHR-Bench test semantic perturbations. These variants share source outputs and are not independent parsers.
- 05Training frames: pooled KoGovDoc-RAG uses the same 663 Q–A but only 242 evidence pages; the pre-specified pilot uses 202 Q–A on 73 pages; OHR compatibility uses 2,036 Q–A across six domains. These denominators are not interchangeable with the 294-page selection frame.
The 30B teacher and 2B production parser form the top tier
The grid compares five complete 294-page outputs under the same parser-native chunker and RCPS protocol. Marker remains a labelled 38-page partial run.
The 30B teacher leads at 0.584 RCPS, only 0.001 above Prod. Prod has slightly higher Hit@1, 0.549 versus 0.545. The teacher ranks above Prod in only 62.5% of 1,000 fixed-seed probe subsets, making latency and compute cost better tie-breaks than the point estimate.
MinerU-on has the highest measured Boundary Clarity but the lowest RCPS and Hit@1 among complete outputs. PaddleOCR retrieves similarly but has no adjacent parser-native boundaries, so Boundary Clarity is undefined rather than accidentally missing.
The mixed score also hides domain sensitivity: MinerU-on reaches 0.046 RCPS on 527 government questions but 0.486 on 136 arXiv questions. RCPS must be rerun on the deployment probe, not used as a portable leaderboard.
Audited parser comparison · 294 pages and 663 Q–A · BC = Boundary Clarity (higher is better); CS = Chunk Stickiness (lower is better)
| Parser | BC ↑ | CS ↓ | RCPS ↑ | Hit@1 ↑ |
|---|---|---|---|---|
| Qwen3-VL-30B teacher | 0.623 | 3.38 | 0.584 | 0.545 |
| Prod, ours (2B) | 0.610 | 3.07 | 0.583 | 0.549 |
| Qwen3-VL-2B base | 0.520 | 3.74 | 0.532 | 0.500 |
| PaddleOCR | Undefined | 3.46 | 0.140 | 0.125 |
| MinerU-on | 0.713 | Not recomputed | 0.137 | 0.123 |
| Marker (38 pages) | 0.717 | 3.41 | 0.073 | 0.068 |
Changing the parser moves more than changing the chunker in this pool
Four candidates implement distinct boundaries: md-h3 uses Markdown headings through level 3; parser-native uses blank-line paragraphs; fixed-500 uses 500-character windows; LumberChunker uses a local instruction model to detect topic shifts.
With Prod fixed, md-h3 leads at 0.593 RCPS, followed by parser-native at 0.583, LumberChunker at 0.557 and fixed-500 at 0.535. This 0.058 range is far below the heterogeneous parser range of 0.447 and close to the 0.052 range among three vision–language parsers.
Across 1,000 random 500-of-663 Q–A subsets, the chunker order is unchanged in 96.1% of draws; md-h3 beats parser-native in 96.5%. The six-configuration parser ranking has mean Kendall τa = 0.902, with changes limited to the near-tied teacher–Prod and PaddleOCR–MinerU-on pairs. Prod beats Base, MinerU-off, PaddleOCR and MinerU-on in every draw.
Chunker comparison with Prod fixed · 294 pages and 663 Q–A
| Chunker | RCPS ↑ | Rank |
|---|---|---|
| md-h3 | 0.593 | 01 |
| Parser-native | 0.583 | 02 |
| LumberChunker | 0.557 | 03 |
| Fixed-500 | 0.535 | 04 |
- 01A stored MinerU-off grid preserves both full orders when the three-retriever average uses only MRR@10: Kendall τa = 1.0.
- 02Raw substring relevance lowers every RCPS by 0.024–0.041 without changing either full ranking.
- 03A pre-audit BGE-M3-only ablation changes only the near-tied 30B–Prod order; it does not test the later MinerU-on–PaddleOCR pair.
The top RCPS choice also wins the end-to-end check
BGE-M3 retrieves five chunks for each of 663 questions; GPT-5.4 generates and judges the answer. Prod reaches 72.5% accuracy, versus 23.8% for MinerU-on and 20.5% for PaddleOCR, so the RCPS winner remains the winner with a reader.
The lower pair reverses, so this does not validate the full ranking. It uses one retriever rather than the three-retriever average, and one GPT-5.4 checkpoint both generates and judges. We treat it only as a check of the winner.
Three-parser answer-generation check · BGE-M3 top five · 663 Q–A
| Parser | Answer accuracy | Exact match | Answered | RCPS |
|---|---|---|---|---|
| Prod | 72.5% | 49.8% | 87.9% | 0.583 |
| MinerU-on | 23.8% | 14.5% | 39.1% | 0.137 |
| PaddleOCR | 20.5% | 11.5% | 45.9% | 0.140 |
Training stays behind the selection and diagnosis gate
RADP asks whether retrieval-oriented training helps after RCPS selection and confirmed parser-side loss. It is neither the main contribution nor the first action.
The pre-specified pilot requires a five-percentage-point RCPS gain with a 95% confidence-interval lower bound above zero; it misses that gate. On the audited 2,036-Q–A OHR compatibility subset, two DPO checkpoints improve Hit@5 by 0.95 and 1.15 percentage points, while a matched edit-distance control gains 1.36. Direct control-versus-DPO intervals cross zero, and SimPO point estimates are negative. The study does not isolate a retrieval-reward benefit.
In the pooled 242-page KoGovDoc-RAG analysis, three DPO checkpoints gain 1.96–2.11 Hit@5 points over Prod, but every two-sided interval crosses zero. Because the reported configuration followed multiple trials, the estimates are exploratory.
On the 294-page selection frame, fine-tuning raises Prod Hit@1 by 4.9 points over Base; selecting Prod over MinerU-on changes it by 42.6 points. This descriptive comparison reinforces selection before optimisation without bounding training causally.
The result is a stop rule: select candidates first, use coverage to locate the layer, and train only for unresolved parser-side loss.
What can be audited, and what a clean checkout still cannot reproduce
The release separates tracked evidence from experiments that still depend on external or author-held material.
- 01Tracked: the frozen 663-Q–A probe, portable 294-page source map, RCPS and coverage code, both MinerU outputs, per-Q–A arrays, stability results and deterministic compatibility audits.
- 02Released separately: nine LoRA adapters with portable configurations, hashes and available structured trainer states, checked from a clean checkout with CPU-only gates.
- 03Not public in full: source documents, every third-party parser output, embedding caches, raw preference-pair text and original training logs. A fresh-clone end-to-end rerun remains unavailable.
- 04Adjudicated human-study inputs remain author-only; the public repository reports aggregates. The mixed-version seven-domain OHR artifact is excluded rather than repaired into a v2 claim.
Where the claim stops
- L01The parser comparison has five complete 294-page outputs; the Boundary Clarity correlation uses only four. One partial 38-page run does not make the result general.
- L02KoGovDoc-RAG uses de-noised Qwen3-VL-30B pseudo-references and GPT-5.4-generated Q–A. A sample check accepted 94/100, but neither full set was human-verified and shared model lineage may bias the probe.
- L03RCPS measures verbatim answer-span retrieval for one corpus, probe, candidate pool, retriever set and relevance rule. It does not predict unscored corpora or implicit and paraphrased answers.
- L04The 20.2% absent rate is matcher-defined, not semantic loss. GPT-5.4 classifies 56% of Prod's exact-match-absent cases as surface artifacts. Tolerant matchers preserve the gap; the stratified human sample confirms only its direction, not population rates.
- L05The end-to-end check covers three parsers and uses one GPT-5.4 checkpoint to generate and judge. It supports the winner, not the full ordering.
- L06RADP evidence is not confirmatory: OHR is a compatibility subset rather than a full v2 rerun, the fidelity control performs similarly to DPO, and the measurements do not identify a causal mechanism.
- L07A clean checkout audits released results and nine adapters but cannot reproduce the full chain without source documents, missing parser outputs and embedding caches.