The Boat License That Made Me Hand-Roll a RAG Chatbot

I live in Nice. When you live in Nice, the Med is right there, and at some point “I should get the permis bateau and just rent a boat” stops being a fantasy and becomes a todo item. So in June I decided to go from zero to the permis bateau côtier — the French coastal boating license — in about two months. I’m a software engineer, not a sailor. I knew nothing about feux, balisage, or VHF etiquette.

Nothing about the exam itself is hard. 40 multiple-choice questions, 35 correct answers needed, mostly common sense once you’ve seen the material. The hard part is the material: it’s dry, scattered across official PDFs and tutorial sites, and full of details that all look the same after an hour — “is the red buoy on your left or your right when entering the port? wait, which direction is ‘entering’ again?”

Where it started: /teach

My first instinct was to lean on AI. A friend pointed me at Matt Pocock’s /teach skill — you give it a topic, it turns into a patient tutor that quizzes you and explains your mistakes. Great concept. But the actual experience kept annoying me: content came as slides with embedded videos, and the videos had no timestamp access. Miss a point, and you’re scrubbing through a video by eye because the slide deck and the video aren’t in sync. For a memory-based exam, that’s the wrong ergonomics entirely.

I’m a developer. I have opinions about ergonomics. So I closed the tab and built my own study site — pure static HTML/CSS/JS, no framework, no build step. Fifteen lessons covering the tronc commun, the coastal track, and the fluvial track. Nineteen printable reference sheets. Interactive quizzes with immediate feedback. Two full 40-question mock exams that grade you in exam conditions (5 errors max, no feedback until you hit “Corriger”). It’s deployable to any static host in one command. That part was fast and fun, and it genuinely works — spaced repetition, interleaving, all the good stuff, without a single video.

The plot twist: LLMs are bad at this

Here’s where the story gets interesting. I had cloned Matt’s idea in spirit — a tutor you can ask questions — so I started asking the smartest models I had access to: “two motorboats are converging, who gives way?” “what does a single short sound signal mean?” “which side is the red buoy on when entering a port?”

GPT-5 and Claude Opus. Flunk. Not occasionally — consistently. They’ll tell you the right answer with total confidence, then contradict the official rulebook on the very next question. The French maritime regulations aren’t obscure: they’re public, well-documented, and haven’t changed much in decades. The models just don’t have them weighted highly enough in their training data to be reliable at the level of detail the exam requires — and the exam requires exact detail. “5 nœuds in the 300-meter band” is not “you should go slow near the beach.” One wrong digit, and it’s a wrong answer.

It’s the perfect failure case for a hallucination: a narrow, factual, low-stakes-but-exact domain. Criticize the model and it apologizes and picks the other wrong answer. Ask five times and you get five confident, contradictory answers. I needed the answers to come from my corpus — the lessons and sheets I’d already written and audited — not from the model’s memory.

So I built a RAG chatbot over my own study material. That’s the whole origin story of permis-bateau-rag, and honestly, it turned into the most satisfying side project I’ve done this year.

The pipeline

The design constraint I set for myself: I wanted it to feel like an actual product, not a weekend glue script. Five stages, each a standalone Python script, all feeding one DuckDB database (rag/permis.duckdb) with five tables: documents, qcm, entities, relationships, document_entities.

• T1 — extract. A zero-dependency parser using only the stdlib html.parser turns my lessons, reference sheets, and study notes into corpus.jsonl plus qcm.json. No BeautifulSoup, no puppeteer, no JS runtime. Just Python’s boring, reliable html module.
• T2 — embed. Voyage AI’s voyage-3-lite produces 512-dim embeddings, chunked at 400 words with a 50-word overlap. Why Voyage? Great quality-per-dollar on embeddings, and the pricing is basically free for a corpus this size.
• T3 — graph. A Mistral model extracts entities and relationships from each chunk — 10 entity types (règles de route, feux, balisage, VHF, écluses…) and 8 relation types — deduplicated by normalized name. This is the step that costs real money (about 80 API calls, checkpointed so you can’t lose progress), and the step that made this project genuinely interesting.
• T4 — retrieve. Two retrieval paths: classic cosine similarity over embeddings, and a graph path that starts from your question’s seed entities, walks one hop through the knowledge graph, pulls the linked chunks, and reranks them with a bonus for entity sharing. You can compare them head-to-head — that was the point.
• T5 — rag. The CLI chatbot. Ask a question about rules of the road, feux, balisage, VHF, écluses — it retrieves context, streams a Mistral answer, and cites its sources. Off-topic? It refuses politely instead of hallucinating. There’s also a quiz mode that pulls random QCMs with explained corrections.

Why DuckDB? Because I like the tech and it earns its keep here: one file, zero servers, real SQL, and it makes the whole pipeline reproducible from scratch — extract.py, embed.py, graph.py rebuilds the entire base from the corpus. (And yes, the “classic pipeline” — old-fashioned cosine-similarity retrieval — is still there as the fallback, because it turns out it’s genuinely good at this task.)

The honest benchmark

I ran a real benchmark: 60 stratified QCMs sampled from my 165 audited questions, plus 10 out-of-domain traps. Fixed seed, reproducible runs.

The honest finding: the knowledge graph does not beat classic retrieval on ranking. On a small, well-structured corpus, cosine similarity over good embeddings is hard to beat. Where the graph earns its keep is source diversity — it surfaces reference sheets (feux, signals, fluvial signage) that the classic path misses entirely, and it makes the citations feel human. Also the refusal behavior: the graph path gives the model the “this is outside my domain” signal more reliably.

And the spot-checks on factual accuracy — the nautical mile is 1852m, latitude scale on the vertical edges, underlined soundings mean they dry out — all correct. Scoring is conservative on purpose; the keyword-proxy metrics understate how good the actual answers are.

I’ve been using it daily since. It’s genuinely better than any general-purpose model at exam questions, because it’s not trying to know the answer — it’s retrieving it from somewhere I’ve already verified, with a source I can click.

The meta-lesson

This project sits at a weird intersection: it’s a learning project (I was studying for a boat license), an AI engineering project (RAG, embeddings, knowledge graphs), and a dogfooding project (I used the tool to study for the exam I built it for). My Hermes agent already nags me every evening at 7 PM with three random concepts, and now the chatbot answers the “wait, what was the rule again?” questions with sources.

The part that would’ve saved me time: I assumed a good LLM would be enough for a narrow factual exam. It isn’t. The moment I treated my own corpus as the source of truth and the model as a reader of that corpus instead of an oracle, the quality jumped immediately. That’s the RAG lesson nobody believes until they feel it.

The exam is at the end of August. I feel ready — not because I memorized anything, but because I can ask a tool I trust “who has priority here, again?” and get an answer from a source I wrote and audited, in seconds, on the boat.

Or on the couch. Let’s be honest, mostly on the couch.

You can browse the study site at permis-bateau.legrand.sh and the RAG pipeline on GitHub — MIT-licensed code, personal corpus.