The Boat License That Made Me Hand-Roll a RAG Chatbot

I live in Nice. When you live in Nice, the Med is right there, and at some point “I should get the permis bateau and just rent a boat” stops being a fantasy and becomes a todo item. So in June I decided to go from zero to the permis bateau côtier — the French coastal boating license — in about two months. I’m a software engineer, not a sailor. I knew nothing about feux, balisage, or VHF etiquette.

Nothing about the exam itself is hard. 40 multiple-choice questions, 35 correct answers needed, mostly common sense once you’ve seen the material. The hard part is the material: it’s dry, scattered across official PDFs and tutorial sites, and full of details that all look the same after an hour — “is the red buoy on your left or your right when entering the port? wait, which direction is ‘entering’ again?”

Where it started: /teach

My first instinct was to lean on AI. A friend pointed me at Matt Pocock’s /teach skill — you give it a topic, it turns into a patient tutor that quizzes you and explains your mistakes. Great concept. But the actual experience kept annoying me: content came as slides with embedded videos, and the videos had no timestamp access. Miss a point, and you’re scrubbing through a video by eye because the slide deck and the video aren’t in sync. For a memory-based exam, that’s the wrong ergonomics entirely.

I’m a developer. I have opinions about ergonomics. So I closed the tab and built my own study site — pure static HTML/CSS/JS, no framework, no build step. Fifteen lessons covering the tronc commun, the coastal track, and the fluvial track. Nineteen printable reference sheets. Interactive quizzes with immediate feedback. Two full 40-question mock exams that grade you in exam conditions (5 errors max, no feedback until you hit “Corriger”). It’s deployable to any static host in one command. That part was fast and fun, and it genuinely works — spaced repetition, interleaving, all the good stuff, without a single video.

The plot twist: LLMs are bad at this

Here’s where the story gets interesting. I had cloned Matt’s idea in spirit — a tutor you can ask questions — so I started asking the smartest models I had access to: “two motorboats are converging, who gives way?” “what does a single short sound signal mean?” “which side is the red buoy on when entering a port?”

GPT-5 and Claude Opus. Flunk. Not occasionally — consistently. They’ll tell you the right answer with total confidence, then contradict the official rulebook on the very next question. The French maritime regulations aren’t obscure: they’re public, well-documented, and haven’t changed much in decades. The models just don’t have them weighted highly enough in their training data to be reliable at the level of detail the exam requires — and the exam requires exact detail. “5 nœuds in the 300-meter band” is not “you should go slow near the beach.” One wrong digit, and it’s a wrong answer.

It’s the perfect failure case for a hallucination: a narrow, factual, low-stakes-but-exact domain. Criticize the model and it apologizes and picks the other wrong answer. Ask five times and you get five confident, contradictory answers. I needed the answers to come from my corpus — the lessons and sheets I’d already written and audited — not from the model’s memory.

So I built a RAG chatbot over my own study material. That’s the whole origin story of permis-bateau-rag, and honestly, it turned into the most satisfying side project I’ve done this year.

The pipeline

The design constraint I set for myself: I wanted it to feel like an actual product, not a weekend glue script. Five stages, each a standalone Python script, all feeding one DuckDB database (rag/permis.duckdb).

One of those five stages is gone now. I’ll get to why — it’s the most useful thing I learned from this project.

• T1 — extract. A zero-dependency parser using only the stdlib html.parser turns my lessons, reference sheets, and study notes into corpus.jsonl plus qcm.json. No BeautifulSoup, no puppeteer, no JS runtime. Just Python’s boring, reliable html module.
• T2 — embed. Voyage AI’s voyage-3-lite produces 512-dim embeddings. Chunking started as a 400-word sliding window with 50-word overlap, which I later replaced with section-based chunking: one h2/h3 section per chunk, prefixed with its heading path, so the embedder sees “Balisage Maritime / Cardinal marks” instead of an anonymous slab of words. The sliding window kept slicing through the middle of tables and buoy cards — I had a whole helper function whose only job was to compensate for a badly placed chunk boundary. Why Voyage? Great quality-per-dollar on embeddings, and the pricing is basically free for a corpus this size.
• T3 — graph. (Removed. See below.) A Mistral model extracted entities and relationships from each chunk — 10 entity types (règles de route, feux, balisage, VHF, écluses…) and 8 relation types — deduplicated by normalized name. This was the only step that cost real money: one API call per chunk, checkpointed so you couldn’t lose progress. It was also the step that made this project feel clever, which turns out to be a warning sign.
• T4 — retrieve. Cosine similarity over embeddings, top-k. For a while there was a second path: seed entities from your question, walk one hop through the knowledge graph, pull the linked chunks, rerank with a bonus for entity sharing. Being able to compare them head-to-head was the point — and it’s what let me kill it.
• T5 — rag. The CLI chatbot. Ask a question about rules of the road, feux, balisage, VHF, écluses — it retrieves context, streams a Mistral answer, and cites its sources. Off-topic? It refuses politely instead of hallucinating. There’s also a quiz mode that pulls random QCMs with explained corrections.

Why DuckDB? Because I like the tech and it earns its keep here: one file, zero servers, real SQL, and it makes the whole pipeline reproducible from scratch. Rebuilding the base is now two commands — extract.py, embed.py — and costs nothing beyond the embeddings, which is a direct consequence of deleting the graph.

The honest benchmark

I ran a real benchmark: 60 stratified QCMs sampled from my 165 audited questions, plus 10 out-of-domain traps. Fixed seed, reproducible runs.

That was my first read, and I wrote a paragraph here claiming the graph earned its keep on source diversity — that it surfaced reference sheets the classic path missed. I’ve since deleted both the paragraph and the graph. It wasn’t true.

And the spot-checks on factual accuracy — the nautical mile is 1852m, latitude scale on the vertical edges, underlined soundings mean they dry out — all correct. Scoring is conservative on purpose; the keyword-proxy metrics understate how good the actual answers are.

I’ve been using it daily since. It’s genuinely better than any general-purpose model at exam questions, because it’s not trying to know the answer — it’s retrieving it from somewhere I’ve already verified, with a source I can click.

The knowledge graph: a clean negative result

Two points on a table with a bolded 92% is not evidence. It’s a rounding artifact I wanted to believe, because I’d spent the most interesting afternoon of the project building that graph. So I went back and did it properly.

First, the graph was broken in a way that flattered it. The rerank bonus counted entities shared with the seed chunks — but a seed chunk shares 100% of its entities with itself, so every seed saturated the bonus. And the cap was 0.30, while the entire spread of cosine similarity across my corpus is 0.338, with a median gap of 0.0045 between adjacent ranks. The bonus was worth 66 ranks. It wasn’t nuancing the vector score; it was obliterating it. “Graph retrieval” was really “take the vector top-8, then reorder it by how many entities the extractor happened to find in each chunk” — which has nothing to do with the question you asked.

Fix both bugs, and the graph becomes bit-identical to plain cosine similarity on the QCM benchmark. 48/48, 53/53, 55/55 at every tier. The miscalibration was the only thing that had ever made the two differ.

Second, the QCM benchmark can’t see a graph’s value anyway. A multiple-choice question is answerable from one section. That’s the anti-pattern for GraphRAG. So I wrote a second benchmark: 24 composite questions of the kind you’d actually type into a chatbot — “Mistral forecast at force 7 tomorrow, I’ve got a category C boat and I wanted to go out in zone 2 — do I go?” Each needs 2 to 4 different sources. I defined the required sources and expected facts by reading the corpus before running any retrieval, and validated that every expected fact genuinely exists in the corpus. Then I added a third arm designed to give the graph its best shot: instead of a rerank bonus, use the graph for entity coverage — greedily pick chunks that contribute query-related entities not yet covered, which is exactly what should help on a multi-source question.

That +3.1 points on facts looks like something. It isn’t. Paired across 24 questions: t = 0.78, 95% CI [−4.7, +11.0], 6 wins / 4 losses / 14 ties. Then I ran the control that settles it — classic against itself, same retrieval, same prompt, two generations at temperature 0.2. Two identical runs differ by ±6.3 points. The same configuration scored 73.3%, then 77.2%, then 77.4%. The run-to-run variance of one config is larger than the advantage I was crediting to the graph.

And it isn’t saturation. At top-3 (69%), top-4 (78%) and top-6 (84%) — where there’s plenty of headroom — the numbers are identical to the decimal.

Three measurements explain why:

• 71% of the extracted entities appear in exactly one chunk (1053 of 1481). A singleton entity connects nothing. It can’t bridge anything.
• The 1-hop expansion selected 67–83% of the corpus as candidates. The graph wasn’t filtering; it was reselecting the corpus with extra steps.
• The entities that do span chunks are generic — vent, mouillage, écluse, VHF. They relate everything to everything, which is the same as relating nothing.

So I deleted it: 381 lines, three DuckDB tables (6,131 rows), one Mistral call per chunk on every rebuild, and a use_graph flag threaded through the API, the CLI, and the front-end. The database went from 3.94 MB to 2.90 MB and a full rebuild now costs nothing.

When would it have been worth it? A graph pays for itself on a large corpus with dense, recurring entities, where answers are spread across many documents and questions need implicit multi-hop jumps — and where you need to show why an answer was reached. My corpus is 206 chunks, flat, deliberately well-structured, with questions that are nearly always answerable from one well-titled section. Section-based chunking had already fixed the only real problem: the top-9 now spreads across 5 distinct sources on its own, because each chunk carries its own heading path. Dense retrieval over a small, clean, well-organized corpus is already near-optimal. There was no gap left for a graph to fill.

The uncomfortable version of this lesson: the graph didn’t fail because it was badly built. It failed because the corpus was good. I’d spent effort making the study material clean, hierarchical, and non-redundant — and that effort is precisely what made the clever retrieval layer redundant. Better data made the sophisticated technique worthless.

One caveat I owe the reader, because I’d want it myself: my 24 composite questions are multi-source but not truly multi-hop. They name both domains explicitly (“at sea and on a river”), so the embedder finds both without needing an implicit jump. A question set with genuinely implicit hops would be the stronger test, and I haven’t built it. That said — on the one question closest to an implicit jump (“night, 4 miles offshore, engine dead, wind force 7”), the graph scored worse than plain vector search: 50% versus 75%.

The meta-lesson

This project sits at a weird intersection: it’s a learning project (I was studying for a boat license), an AI engineering project (RAG, embeddings, knowledge graphs), and a dogfooding project (I used the tool to study for the exam I built it for). My Hermes agent already nags me every evening at 7 PM with three random concepts, and now the chatbot answers the “wait, what was the rule again?” questions with sources.

The part that would’ve saved me time: I assumed a good LLM would be enough for a narrow factual exam. It isn’t. The moment I treated my own corpus as the source of truth and the model as a reader of that corpus instead of an oracle, the quality jumped immediately. That’s the RAG lesson nobody believes until they feel it.

The second lesson cost me more, and I only learned it because I went back and measured properly: build the control before you believe the result. I had two points on a table and a story I liked, and that was enough to keep 381 lines of knowledge graph alive for weeks. The thing that killed it wasn’t a better idea — it was running the same configuration twice and discovering that the noise floor was twice the size of my finding. If you’re evaluating anything with an LLM in the loop, run it against itself first. Whatever gap you can’t beat, you can’t claim.

The exam is at the end of August. I feel ready — not because I memorized anything, but because I can ask a tool I trust “who has priority here, again?” and get an answer from a source I wrote and audited, in seconds, on the boat.

Or on the couch. Let’s be honest, mostly on the couch.

You can browse the study site at permis-bateau.legrand.sh and the RAG pipeline on GitHub — MIT-licensed code, personal corpus.