I Can Generate Fifty Thousand Questions. My Kid Asked How Many There Are.

My kid finished a set of problems, looked up, and asked: how many questions are there in this thing? I couldn't answer.

Not because I'd forgotten. Because in the design I had at that moment, the question had no answer. What I'd built wasn't a question bank—it was 74 random generators. Every click rolled up a fresh question; once you answered it, it was gone. Asking how many questions it holds is like asking how many point sequences a die has.

That's when it hit me that I'd fused two different things into one: "can keep producing new questions" and "has a bank you can look things up in" are completely different animals. The first is my instinct as an engineer—if labor repeats, automate it. The second is the instinct of a parent and a kid—I need to know what I'm practicing, how much I've done, and how much is left.

The site started on August 8th. In five days I tore it down four times. All four teardowns share one root cause: I kept treating "technically possible" as "this is what the product should be." This post covers the whole build, and how to use it.

1. It starts with an exam paper, not an idea

First, the thing I consider most important: every question on this site traces back to a real exam paper.

The starting point is the third-grade Chinese final from the second semester of the 2025 school year. Reading question 17 says it plainly: "find three idioms in the text." The answers are printed right there in the last paragraph of Shrimp Delight—plus one more in paragraph 2. You could copy them off the page without thinking.

My kid wrote down two idioms that do not appear anywhere in the text. He made them up from memory.

That isn't a vocabulary problem. It's a behavior problem: the question said "in the text," and he didn't go back to the text. Question 16 on the same paper is the same story—paragraph 5 clearly describes the shrimp moving forward and backward, and he summarized only "forward," dropping half of it.

So the first station on the Chinese track is called "Go back to the text for evidence," and the third is "Get the sequence straight." Every one of them grew out of a real point where marks were actually lost on that paper. That's the first hard rule I set for myself:

Topics invented out of thin air won't move the score. Get a paper into the archive first, extract where marks were actually lost, then open the course.

The homepage shows all ten primary-school subjects, but only Chinese and Math are open. The other eight are marked "in preparation," each with a written reason for why it isn't open yet. That isn't laziness—the eight empty slots are themselves the next batch of work, and far more honest than a curriculum table pretending to be complete. The validator is strict: a subject either has courses or is marked soon; once marked soon it may not carry real courses; a placeholder subject must state its reason. Any structure that fails those three is refused at render time.

2. First teardown: an infinite stream is a bad design

Version one was an "infinite stream": one question on screen, answer it, get judged, next one appears, never ends, and the algorithm always feeds you whatever you're weakest at.

Technically that's optimal—maximum practice on your weakest topic per unit of time. I was rather pleased with myself.

My kid used it for ten minutes and said: I don't know what I just did.

He was right, and more precisely right than I'd realized. An infinite stream optimizes throughput, at the cost of no boundary and no trace. The person answering never gets a single signal that says "I finished something." The TOEFL, a final exam, any real test—none of them are organized that way: a set is a set. There's a question 1 and a question 12, and when you're on 8 you know where you are.

So I rebuilt it as sets. Twelve questions by default, with a beginning and an end:

Weak topics and previously-missed types get priority at assembly time, but once drawn, the set is fixed—it doesn't shift while you're working through it. Ordered easy to hard.

The key sentence is this: "infinite" and "unstructured" are two different things. Questions can still be produced without limit; they just have to come out one set at a time. I'd bound the two words together purely because my head was full of the Anki flashcard model, not because my kid needed it.

3. Second teardown: a mistake log can't store the original question

For version two of the mistake log I did the obvious thing: you get one wrong, save that question, replay it during review.

Two rounds in, he'd memorized the answers. Third round, all correct. Looked like mastery. Not one of them was.

So the mistake log now records question types, not questions. An entry holds only {type id, type name, one-line summary, date, times wrong, times right in a row}—it physically does not store the answers or the options, so replaying the original isn't possible even if I wanted it. At review time the same type calls make() for a fresh question: new numbers, new options, new text.

That change also removed something dirtier. On the replay path, the question object was deserialized out of storage with no closure, so it couldn't run its own verification function—and at the time I'd stuffed in a verify: () => true.

A verification that always returns true is worse than no verification, because it makes the self-check permanently green. Once that path was gone, that line had no reason to exist. There is now a gate that scans the whole repo for always-true verifications; write one back in and it goes red.

Two more tuning decisions on correction:

• Getting it right once counts as one correction; two in a row clears it from the log. Treat one lucky guess as mastery and you'll miss it again next time.
• Getting it wrong zeroes the streak and increments the wrong count. Types you miss repeatedly carry more weight at the next assembly.
• Ordinary sets also up-weight missed types (+6)—you don't have to go tap "practice mistakes"; they come find you.

4. Third teardown: a generator is not a bank

Which brings us back to the opening question.

When he asked "how many questions are there," I had 74 generators and not one table that could answer him. Worse, I couldn't tell whether the questions were any good either—I had no way to lay out the output of 74 generators and look at it. I could only click around and spot-check.

So I built bank.db, a single SQLite file, one row per question:

items    stem / prompt / answer / explanation / hint / type / difficulty /
         topic / source paper + question number / what my kid actually wrote / date added
options  every option of a multiple choice (which is right, why the others are wrong)
topics   topics      papers  source exams      texts  source passages
gens     type registry: generated live (open) or materialized (enumerated)
sources  fingerprint of each page's generator source—the drift gate runs off this

The site itself is still zero-backend, zero-CDN, a single HTML file you can open by double-clicking. The bank isn't a runtime dependency; it's a build-time source of truth: build the bank → inline it into each practice page → export a separate overview page.

The overview page filters (subject / topic / type / difficulty / source), does full-text search, lets you tick questions and export a printable set, shows weak topics by area, and lets you click a question to see which paper and which question number it came from. Adding questions goes through the terminal—no admin UI. That was my own call, and the reasoning is simple: an admin UI has to be written, maintained, and verified, and I add questions a few times a week. bank.py add --json is enough. Every extra line of UI code is a cost paid for nothing.

5. Fourth teardown: full enumeration explodes

The day I built the bank I did something stupid: I enumerated all 74 generators into it.

The reasoning sounds perfectly sound—a generator's output space is finite, so call make() until nothing new comes out, store it all, and the bank is complete. Technically it holds up. I even wrote a convergence test: 8,000 consecutive calls with nothing new counts as exhausted.

Result: 1,787 questions. One reading passage blew up into 380. Another set produced 1,226. The first four types alone accounted for 79%. One page went from a bit over 100KB to 912KB.

That isn't a bank. It's the Cartesian product of a word list and its distractors. And it grows combinatorially: three more exam papers and the pages would be unusable.

The real problem is that I'd gotten "complete" wrong. What my kid needs complete is "not one of the 13 original questions from that paper is missing"—not "this generator's output space has been exhausted."

Rebuilt, with the rule reduced to one sentence: the bank is built around original exam questions; generators are not pre-expanded; similar questions are made on demand.

The bank now holds 227: 13 original exam questions plus 214 samples. Pages are back to 134KB. Size grows linearly with the number of exam papers, not with combinations—ten more papers just adds ten papers' worth.

Questions are still produced live by the generators. The gens table records how many variants each type can actually produce: 50,725 in total. That's the real measure of variety. How many samples sit in the bank isn't. I initially used the sample count as the metric and nearly misjudged every type as "too repetitive."

Each question header now carries three buttons:

Original exam questions have no generation rule, so "build a similar question" tells you to go make one in the terminal. On that path I use a model to write a new question against the same skill, but the adjudication isn't delegated: every line of textual evidence the new question cites must be found verbatim in the source passage, or it's thrown out. Retrieval and execution can be outsourced. Whether the answer is right cannot.

6. Verification has to take a different road

This is the one thing in the whole build I care most about, and the only one I never once compromised on.

Every question's make() must return a verify(), and verify has to reach the answer by a completely different road than the one that produced it.

Why two roads? Because reusing the same formula to "verify" is testing a body double. When it's wrong, both sides are wrong together and the self-check is green forever. It's non-investigation disguised as investigation—which makes it worse than not investigating, because it looks so much like the real thing.

At engine startup every type runs verify 120 times; one failure writes the reason into the page and the render gate goes red. This has actually caught things: I mistyped one character of an idiom and the self-check went red on the spot.

I should also be clear about what it can't catch. The go-back-to-the-source part is as hard as the math. Pure word discrimination (misspellings, near-synonyms) can only be cross-checked against an independent word list—it catches "option marked backwards / more than one correct answer / duplicate distractors," but not "the word list itself is wrong." A true/false mark flipped the wrong way is caught now (via consistency with the stated trap—that one was added after it slipped through once), but "marked correctly, trap written convincingly, and yet the claim doesn't actually match the source" is not. That's semantics. You have to read it against the source yourself.

So after check_all.py runs every gate, the last screen lists what no gate covers: whether the generated Chinese reads naturally, whether the explanations suit a third grader, whether the layout is cramped on a phone, whether the archive is faithful to the original paper. All green is not the same as correct, and that sentence has to be printed in the output, not just written in the docs.

7. I once wrote a gate that could never go red

While building the bank I added a gate: at render time, compare the number of questions inlined into the page against the number in the bank; mismatch goes red.

It looks like a gate. In fact it compares at the moment of rendering, with both numbers produced by the same run of the same function, comparing against itself—always equal. I only found out while writing the reverse-check probe for it: I couldn't make it go red no matter what I did.

The fix was to genuinely separate the two sides: stamp the page with a fingerprint covering the entire inlined payload (not just question IDs—changing a stem without changing the count has to be detectable), and have a separate, independent gate compare bank against page. At the same time, the payload construction moved into one function shared by the renderer and the staleness gate—write it twice and the two copies will eventually compute different numbers.

After that I set a rule and made it machine-enforced:

Every time you add a gate, register its reverse check—put the bug it's supposed to catch back in and confirm it actually stops it. Before writing a gate, answer one question: under what circumstances does this go red? If you can't answer, it isn't a gate.

reverse_checks.yaml now registers 41 probes, and gate_audit.py reconciles both directions: every gate in the run must be registered here, and every gate registered here must still exist there. Debt is allowed, but it must carry a written waiver: explaining why—a waiver is visible debt, not a silent skip.

gate_audit.py also handles three other things, all of them wrecks from this round:

1. Always-true verifications—any verify: () => true is red on sight
2. Artifact size caps—300KB for a practice page, 800KB for the bank page. Forced out of me by that 912KB
3. Commands promised in copy must actually exist—I once wrote "run bank.py similar in the terminal" into a dialog on the page while that subcommand wasn't implemented yet

That third one deserves emphasis. Write a nonexistent command into your docs and whoever reads them—including you, six months from now—will type it and get stuck. It's part of the machine checks now.

8. The engagement layer lives in the engine, not on a page

Whether a kid clicks a second time has almost nothing to do with question quality. It has to do with "almost done" being permanently in view.

All of it went into the engine, so it takes effect across every practice page on re-render. No per-page patches:

Storage splits in two: per-course mastery and mistake log under the course's own key; XP, level, badges, streak, and daily quest under a global key. So progression is one line across subjects, not one for Chinese and another for Math.

Three hard constraints learned the hard way:

• Every effects layer is pointer-events:none. Confetti and popup cards sit on top of the question; let them swallow a single click and the whole page becomes unclickable. There's a smoke test for exactly this.
• What a badge looks like lives in one JSON file, read by both the practice pages and the homepage, with the predicate functions keyed to match. A mismatch aborts rendering.
• The daily quota has one source of truth in the curriculum file. A page can override it, but the default may not be written per-page—otherwise changing it once means changing it ten times.

9. Turning a spreadsheet into an account that adds up

My kid's points system started as a spreadsheet: 81 rules, partitioned by school subject, rewards and deductions side by side. I structured it into the system rule by rule; it's now 83 rules and 12 redeemable items.

Five rule shapes, and the score is always computed server-side—whatever the request claims doesn't count:

A few judgment calls worth pulling out:

Tier expressions are evaluated against a whitelist, never eval. The expressions come from a config file, and eval hands over arbitrary code execution. The gate has 6 injection cases (__import__ / __subclasses__ / open / lambda / exec / list comprehension) reverse-verified.

The page preview and the actual ledger entry run the same compute function. The parent sees the preview; the kid gets the entry. Compute them separately and they will eventually disagree—and by then you can't explain it.

The tier table is ordered and the first match returns. That's how "does not stack" from the original spreadsheet is implemented. 190 skips in 3 sets has to match before 180 in 3 sets; get the order wrong and you underpay.

Deductions can go negative; redemptions can't overdraw. The deduction clauses were always "you owe it, earn it back." But buying something can't run on credit. The two paths have different flags.

The admin password issues no cookie and must be typed every time. A kid holding an already-logged-in tablet still can't add points. A points system where you can award yourself points becomes a point-farming game by day two, and the incentive dies on the spot.

Practice points are capped server-side at 30 a day, and an empty submission earns nothing. That's the only channel that doesn't involve a parent, so the quota has to be computed server-side. The empty-submission check has a trap in it: you must count questions answered correctly, not rows in the summary—counting rows scores "submit without answering anything" as a perfect set, which is an open farming route.

The ledger is append-only and the balance is always accumulated from the flow. Store a separate balance field and it will drift from the flow, and a drifting ledger is worse than no ledger. Reversal is done by contra-entry: an offsetting record plus a void stamp on the original. Nothing is deleted.

One more I hesitated over and then wrote in anyway: the first 30 days of history all carry a seed flag, with "(opening simulation)" appended to each description. You can have the "climbs from 0 to 1000 with realistic wobble" look, but the ledger doesn't get to lie—six months from now it has to be obvious at a glance which entries actually happened. The seeding command refuses to run if the ledger already contains real activity.

The login gate follows the same thinking: it's server-side, not front-end. A password box in the front end is nothing—right-click, view source, or just type a deep URL and you're past it. The server serves the static files itself and checks the session before it does. That split isn't only about security; it's about being able to exercise the same path locally: put the gate in the nginx layer and you can't test it locally at all, which means you don't have a gate.

10. How to use it

Add a course

Three steps, each with a gate:

# 1. Write primary-<subject>/<slug>.practice.md
#    frontmatter requires title / slug / date / out; source / unit recommended
#    body = short explanation + a gens fence (the generator code)
python3 engine/practice_render.py primary-<subject>/<slug>.practice.md

# 2. Add a node to the curriculum, then re-render the homepage
python3 engine/path_render.py

# 3. All gates (including real-browser end-to-end smoke)
python3 engine/check_all.py

date is required; the renderer refuses to render without it. The reason: six months from now, to answer "why is this question here," you need to be able to follow the date back to the exam paper it came from. The date shows in three places on the page, and on every course card on the homepage—read live from the source file, never duplicated into the curriculum, so the two can't drift apart.

Add a question

cat > /tmp/q.json <<'EOF'
{ "type":"choice", "q":"stem", "ask":"prompt", "level":2,
  "options":[{"t":"the right one","ok":true},{"t":"distractor","ok":false,"why":"why it's wrong"}],
  "sol":"step-by-step explanation (required)", "hint":"point the way, don't give the answer" }
EOF
python3 engine/bank.py add --topic chinese-text-evidence --json /tmp/q.json
python3 engine/practice_render.py primary-chinese/text-evidence.practice.md

The moment it lands it runs the gates; failure rolls back automatically. After adding you must re-render the page—the staleness gate will stop you, but don't wait for it to.

Archive an exam paper

One directory per paper: source text / questions / official answers / what my kid wrote / error analysis, plus scans. Only two rules, both hard:

• If you can't read it, mark confidence: low, leave it null, and don't guess. Photographed papers with red and blue pen layered on top of each other are the norm, and a guessed answer poisons the error statistics—worse than missing data. Every low entry must appear in the "needs your review" list; the machine reconciles it, and one missing entry goes red.
• Transcribe compositions character for character, misspellings and pinyin substitutions preserved—those are the evidence of what's being lost in character recognition and writing. Cleaning them up is falsifying evidence.

Precisely because originals must be transcribed verbatim, the site's "no half-width double quotes in Chinese sentences" rule is waived for original exam questions. Rules serve the facts, not the other way around.

Deploy

The archive directory never goes online. The deploy directory is assembled from a whitelist, with a fail-closed gate on top: any name registered in the archive, any reference to an archive path, any non-HTML file that sneaks in—abort, don't ship. The name blacklist is read live from the archive files, never duplicated into the deploy script.

That gate has caught real things, both of them mine: a description in the curriculum and one JS comment carrying a local path.

The site itself has no email-code authentication—make a kid clear a verification code every time he practices and the site simply won't get used. Instead: username and password, no search engine indexing, and the archive never uploaded at all. That's a deliberate trade, not an oversight.

11. Five days, four teardowns, one root cause

Looking back, the four teardowns are four shapes of the same mistake:

The common thread: every time, I verified that something could be done, and then treated that as evidence that it should be. Technical feasibility is a seductive answer, because it can be proven. "Should it" can't—"should it" can only be settled by putting it in front of a real person and seeing whether it helps.

The site now has 7,240 lines of engine code, 17 machine gates, 41 reverse-check probes, 227 questions you can look up, and 50,725 questions it can produce.

But the most valuable change I made was giving "how many questions are there" an answer. That isn't a line of code. It's a question I didn't think needed answering.

The person who asked it is in third grade.