From 07fab4c094702de1c3ffd55e4a8c5ea3aeb00638 Mon Sep 17 00:00:00 2001 From: Zareian Date: Thu, 17 Sep 2026 05:59:19 +0330 Subject: [PATCH] write down what the last two days changed about content and answer quality MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md تنها جایی است که هر جلسه‌ی تازه اول می‌خواندش، و چیزهایی که این دوره یاد گرفتیم جایی ثبت نشده بود. چهار بخش اضافه شد و «وضعیتِ فعلی» به بالا آمد: • محتوا حالا در content/seed/ کامیت می‌شود و deploy خودش بالا می‌آوردش — بدونِ Qdrant، بدونِ اعتبارِ AI، بدونِ SSH. و اینکه سوال با مسیرِ عنوان به درس وصل می‌شود نه با id، چون idها بینِ دو دیتابیس یکی نیستند. • لایه‌ی راستی‌آزمایی: کتاب از روی تصویر خوانده می‌شود، پس خطا همان‌جا وارد می‌شود. برچسب سه‌حالته است و سوال فقط از agreesها ساخته می‌شود. • دو درسی که گران تمام شدند: نتیجه‌ی دوحالته نرخ را سه برابر نشان می‌دهد، و «۳/۱۰۴» در این کتاب‌ها عددِ اعشاری است نه کسر. • شکافِ شکل‌ها (۴۲٪ محتوا) و اینکه اصلاحش در پرامپت هست ولی اجرا نشده. • تفاوتِ ۴۲۹ سقفِ حساب با ۴۰۳ اتمامِ اعتبار — و اینکه متنِ دومی فاصله دارد نه زیرخط، که یک بار ۷۷۰ فراخوانیِ بی‌فایده به بار آورد. «وضعیت تا ۲۰۲۶-۰۹-۱۷» هم اضافه شد: بک‌اند آماده و تست‌شده است، و دو مانعِ باقی‌مانده دستِ ما نیست — سرور روی کدِ قدیمی، و فرانتی که به هیچ APIی وصل نیست. هر دو قبل از اینکه کسی دنبالِ باگ بگردد باید چک شوند. هیچ فایلِ تازه‌ای اضافه نشد؛ همان فایلِ موجود به‌روز شد. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 95 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7155f30..6d00d24 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,12 +137,104 @@ those defaults, not replaced by them. counts, security warnings, and recent traffic. It is how you check whether anyone has actually connected without asking a human. +## Content: where it lives and how it gets to a server + +`content/seed/` holds the whole curriculum and question bank as committed JSON — +`curriculum.json` (6 grades, 48 units, 230 nodes, 1,928 lessons) and `questions.json` +(1,441 multiple-choice questions). `deploy.sh` runs `seed_content` on every release, so a +fresh server fills itself with one command: **no Qdrant, no AI credit, no SSH session.** + +`seed_content` is idempotent and never deletes. Questions bind to lessons **by title path** +(grade › unit › node › lesson), never by primary key — ids differ between any two +databases, and binding by id silently attaches questions to the wrong lesson. + +`seed_content --export` writes the files back from the local DB. It refuses to overwrite a +non-empty `questions.json` from a DB that has no questions (that used to wipe the file +silently); use `--export --only curriculum` when you only mean the curriculum. + +## Answer quality: the verification layer + +Books are read by a **vision model from page images**, so extraction errors enter there — +a misread digit, a dropped decimal separator, a solution attached to the neighbouring +problem. These are invisible in retrieval and only surface when something trusts the text. + +`rag_verify_solutions` re-solves every extracted problem independently and compares. +`LessonChunk.solution_verdict` is three-state — `agrees` / `disagrees` / `unjudgeable` — +and the verdict also rides in the Qdrant payload under a key `retrieval._payload_text` +does not read, so the tutor's context is unchanged. Current state: 1,112 agree, 123 +disagree, 881 unjudgeable. **Question generation only ever reads `agrees` chunks.** + +Two things that were learned the expensive way and are already encoded in the prompts: + +- A boolean verdict is wrong. With two states every "I can't tell" becomes "wrong", which + reported ~3× the real rate. Read samples before trusting any bulk run. +- `۳/۱۰۴` in these books is the decimal 3.104, not the fraction 3÷104. + +`solution` itself is never rewritten — the extracted text stays as the record, and the +independent answer is stored separately in `checked_answer`. + +## The figure gap (largest open data issue) + +881 chunks (~42%) reference a figure or table that is not in the text, so they cannot be +answered or turned into questions. **The vision model saw the figure; it was simply never +asked to write it down.** `rag/services/pdf.py` now carries a self-contained rule telling +it to inline measurements and labels into the problem text — books ingested before that +rule predate it. Recovering them means re-ingesting those pages (~$2.7 of API credit). + +## Quiz assembly + +The app plays at **node** level (the island on the map) but questions hang off **lessons**, +and a lesson here is the finest grain — one problem type from the book, not a class +session. `GET /api/nodes/{id}/quiz` assembles a quiz from a node's lessons, at most one +question per lesson. Shuffle is seeded on `nodeId:userId`, so a refresh returns the same +quiz; `questionCount` is reported separately from `requested` so a thin node cannot look +like an error. 172 of 230 nodes currently hold a full four-question quiz. + +## AI provider notes + +`OPENAI_BASE_URL` points at GapGPT (an OpenAI proxy reachable from Iran) and it exposes +~127 models, including reasoning ones. Two failures look alike and must not be treated +alike — `is_account_limit()` in `rag_verify_solutions` is the shared guard: + +- `429 initial_usage_limit_exceeded` — account cap, hours long. Retrying **extends** it. +- `403 insufficient user quota` — credit exhausted. Note the spaces: an earlier guard + matched `insufficient_quota` with an underscore, missed it, and burned 770 doomed calls. + +`/dashboard/billing/usage` returns cents and is the only reliable balance signal; the +`hard_limit_usd` in `/dashboard/billing/subscription` is a placeholder, not a balance. + +**The key is currently out of credit.** Bulk work since then runs on Claude Code subagents +instead. Sonnet handles question authoring well (~88% yield); Haiku does not — it cannot +reliably pull the final answer out of a solution and produced ~40% broken output. + ## Known gaps (per README pre-deploy checklist) - Still on SQLite, not Postgres. - Throttle rates need periodic review against real traffic. - `DEV_MASTER_OTP` (`0000`) must stay gated behind `DEBUG` — never enabled in prod. -- The server runs with `DJANGO_DEBUG=1` on a public domain, so OTP `0000` currently logs - anyone in as anyone. Harmless until real users exist; `/api/health` warns about it. - No video content exists yet, though `Lesson.kind`/`video_url` can express it. -- Question generation and textbook ingest are blocked on AI API credit. +- 58 of 230 nodes still hold fewer than four questions; their verified chunks ran out. +- Kavenegar is unconfigured, so `POST /api/auth/otp/request` returns 503. With `DEBUG=0` + there is no way to log into the API without creating an `OTPCode` row in the admin + (`/admin/accounts/otpcode/add/`) — this is how to hand a tester a working login. + +## State as of 2026-09-17 — read this first + +**Backend is done and verified; nothing is blocked on us.** The full game loop was driven +end to end against a fresh database with `DEBUG=0`: grades → login → set grade → home → +node quiz → submit answer → CORS header for the frontend origin. `main` is pushed and +clean at `16d3d9f`; 435 tests green. + +Two things are blocked on other people, and both are worth checking before assuming a bug: + +1. **The server still runs pre-16d3d9f code with an empty database.** `git pull && + ./deploy.sh` brings both code and content. Verify with `curl /api/health` — it should + report `questions: 1441`. A 404 there means the pull never landed. +2. **The frontend is not wired to any API.** `src/api/` was merged (PR #1) but is + imported nowhere; `QuestionScreen` in `App.tsx` still renders four hardcoded + multiplication questions. **PR #3** (`zareian/api-client-2`, 3 commits) is still open on + upstream. Until someone connects the screens, none of the 1,441 questions are visible. + +The frontend's `Question` type carries a `type` field that nothing reads, and +`count_objects` in `Question.TYPE_CHOICES` is an empty shell — no rows, no serializer +branch, no rendering anywhere. Do not build content for it without asking first.