my parser returned 39 chapters. every one was a person's name.
cutting a book into chapters looks like a solved problem until you try it on 26 real ones. the bug was never the model — it was the six layers i built to correct the model.
a tutor that answers from your textbook has to know where the chapters are. that sounds like a solved problem — PDFs have outlines, books have tables of contents, and a competent model can read a contents page. i believed that for about a month.
then i ran my production parser against the 26 books actually in the library and read every result by hand. the Santhali edition of the Indian Constitution came back with 39 chapters, and every one of them was a person's name — ISHWAR MARANDI, KRISHNA CHANDRA TUDU. a contributors page, set in all-caps, matched a heuristic that thought all-caps meant heading.
that was the funny one. the rest were worse, because they looked fine.
the audit
nine trade books, nine different failures. not nine bugs — one architecture, failing nine ways:
| book | what came out |
|---|---|
| The Mom Test | every chapter duplicated |
| The Hard Thing About Hard Things | starts at chapter 2 |
| The Accidental CTO | chapters 11, 15, 19, 16, 20 — eight missing |
| Thinking, Fast and Slow | appendices placed first |
| The Power of Habit | "chapter 6" stamped on four different blobs |
| Diary of a CEO | "Cover", "Copyright", "Dedication" as chapters |
| 7 Habits | part one and all seven habits missing |
zero of nine were correct. one was right, and when i traced why, it was right by luck: the verifier had collapsed a good 16-chapter extraction down to 1, and an unverified fallback path — whose page numbers happened to line up — saved it.
that is the detail that reframed the whole thing. i had been reading these as model failures. they weren't. in every broken case i checked, the model's raw extraction was good: 16 chapters for Zero to One, 50 for Thinking Fast and Slow, 40 for Diary of a CEO. the corruption happened entirely downstream, in the machinery i had written to make the model's output trustworthy.
the root cause nobody warns you about
here it is, and it is embarrassingly simple.
a book has two coordinate systems, and they disagree.
the model reads the contents page and reports printed page numbers — the ones inked on the paper. your code slices the file by PDF page index. those differ by however many unnumbered front-matter pages the publisher put in front of page 1. roman numerals, a blank leaf, a half-title, praise pages. it's different for every book, and nothing in the file tells you the offset.
so every chapter lands slightly wrong. and the natural response — the one i took — is to build a verifier that re-anchors each chapter by searching the body for its title.
that verifier became the single most destructive component in the pipeline.
it drops what it can't match. and it cannot match a lot: a body banner reading CHAPTER ONE — The Mom Test doesn't substring-match the contents entry The Mom Test. a chapter opening with a stylised drop-cap doesn't match its own first word. short titles get refused as too ambiguous. every one of those is a chapter that silently vanishes — no error, no warning, just a book that starts at chapter 2.
then i added a fallback for when the verifier came up empty. then a fallback for the fallback. six layers, plus plausibility gates that corrected each other into new failures. the winning layer was different for every book. output quality had become a function of which fallback happened to fire — which is another way of saying it was random.
what i'd tell someone starting this
four things i paid for and would not have guessed.
a repair layer for a bad primitive will do more damage than the bug. the printed-page/PDF-page mismatch is a real problem. the verifier existed only to paper over it. papering over it cost me good extractions in a way the original bug never did. delete the mismatch instead — pick one coordinate system and never leave it.
anchor on content, not on coordinates. the fix that mattered: stop asking the model where a chapter is, and ask it what the chapter says — the first several words of its actual body prose, verbatim. that's a string you can search for. it doesn't need an offset table, and it can't drift.
the search has to be forward-only. locate each chapter after the previous one's position, never globally. that one constraint structurally eliminates duplicates and scrambles — the two failures i'd been fighting with dedup logic and sort passes. you don't validate them away, you make them unrepresentable. (this matters more than it sounds: opening phrases genuinely recur earlier in a book, because chapters quote questions posed before them. earliest-match finds the quote.)
your granularity problem is not a prompting problem. this is the one that cost me the most time. ask a model to segment a heading-dense book twice and you get two different answers — i measured 14 chapters on one run and 117 on the next, same book, same prompt. no amount of prompt tightening fixes that, because "is this a chapter or a subsection?" is genuinely ambiguous from the text alone. it needs an external authority. the printed contents page is one, and it's sitting right there in the file.
though only when it's complete. a statute whose front matter lists the first four of its eleven chapters will happily cap your extraction at four. so the contents page gets to be the authority conditionally — and working out that condition, and which of two strategies to run once you trust it, was most of the remaining work. that part i'm keeping.
the honest fallback
the last piece is a product decision disguised as an engineering one.
the new pipeline ends with an invariant gate: positions strictly increasing, most of the spine located, no single chapter swallowing the book, body well covered. if the gate fails, the parser is not allowed to guess. it emits one clearly-labelled coarse split, flags the book as low quality, and logs it for re-ingest.
that was hard to accept. a coarse split is visibly worse than chapters. but the alternative isn't chapters — the alternative is 39 people's names, presented with exactly the same confidence as a correct result. i deleted the regex heuristic entirely. it never failed, which is precisely what made it dangerous.
results, same 26 books: seven of nine trade books parse to correct real chapters, two take the honest fallback, zero produce garbage. the government corpus went from 25 fabricated chapters on an NCERT prelims file to one honest document. Thinking, Fast and Slow went from appendices-first fragments to 39 real chapters. the 740-page Economic Survey, once i got the contents-page condition right, went from 8 chapters to 17.
the remaining imperfections are all under-segmentation — occasionally a middle chapter folds into its neighbour. i'll take that trade every time. an under-segmented book is quietly worse; a fabricated one is confidently wrong, and confidently wrong is the failure mode that loses a user.
the lesson
i spent weeks improving prompts for a component that was already returning the right answer. the extraction was never the weak link. the weak link was six layers of correction, each one added in good faith to fix the layer before it, collectively converting a good signal into a random one.
when a pipeline fails differently on every input, stop debugging the inputs. the variance is the bug. something in there has no invariant, and every fallback you add is another way to be wrong that you won't be able to tell apart from being right.