write down what the last two days changed about content and answer quality

CLAUDE.md تنها جایی است که هر جلسه‌ی تازه اول می‌خواندش، و چیزهایی که این دوره
یاد گرفتیم جایی ثبت نشده بود. چهار بخش اضافه شد و «وضعیتِ فعلی» به بالا آمد:

• محتوا حالا در content/seed/ کامیت می‌شود و deploy خودش بالا می‌آوردش — بدونِ
  Qdrant، بدونِ اعتبارِ AI، بدونِ SSH. و اینکه سوال با مسیرِ عنوان به درس وصل
  می‌شود نه با id، چون idها بینِ دو دیتابیس یکی نیستند.
• لایه‌ی راستی‌آزمایی: کتاب از روی تصویر خوانده می‌شود، پس خطا همان‌جا وارد
  می‌شود. برچسب سه‌حالته است و سوال فقط از agreesها ساخته می‌شود.
• دو درسی که گران تمام شدند: نتیجه‌ی دوحالته نرخ را سه برابر نشان می‌دهد، و
  «۳/۱۰۴» در این کتاب‌ها عددِ اعشاری است نه کسر.
• شکافِ شکل‌ها (۴۲٪ محتوا) و اینکه اصلاحش در پرامپت هست ولی اجرا نشده.
• تفاوتِ ۴۲۹ سقفِ حساب با ۴۰۳ اتمامِ اعتبار — و اینکه متنِ دومی فاصله دارد نه
  زیرخط، که یک بار ۷۷۰ فراخوانیِ بی‌فایده به بار آورد.

«وضعیت تا ۲۰۲۶-۰۹-۱۷» هم اضافه شد: بک‌اند آماده و تست‌شده است، و دو مانعِ
باقی‌مانده دستِ ما نیست — سرور روی کدِ قدیمی، و فرانتی که به هیچ APIی وصل نیست.
هر دو قبل از اینکه کسی دنبالِ باگ بگردد باید چک شوند.

هیچ فایلِ تازه‌ای اضافه نشد؛ همان فایلِ موجود به‌روز شد.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 05:59:19 +03:30
parent 16d3d9f149
commit 07fab4c094

View File

@@ -137,12 +137,104 @@ those defaults, not replaced by them.
counts, security warnings, and recent traffic. It is how you check whether anyone has counts, security warnings, and recent traffic. It is how you check whether anyone has
actually connected without asking a human. actually connected without asking a human.
## Content: where it lives and how it gets to a server
`content/seed/` holds the whole curriculum and question bank as committed JSON —
`curriculum.json` (6 grades, 48 units, 230 nodes, 1,928 lessons) and `questions.json`
(1,441 multiple-choice questions). `deploy.sh` runs `seed_content` on every release, so a
fresh server fills itself with one command: **no Qdrant, no AI credit, no SSH session.**
`seed_content` is idempotent and never deletes. Questions bind to lessons **by title path**
(grade › unit › node › lesson), never by primary key — ids differ between any two
databases, and binding by id silently attaches questions to the wrong lesson.
`seed_content --export` writes the files back from the local DB. It refuses to overwrite a
non-empty `questions.json` from a DB that has no questions (that used to wipe the file
silently); use `--export --only curriculum` when you only mean the curriculum.
## Answer quality: the verification layer
Books are read by a **vision model from page images**, so extraction errors enter there —
a misread digit, a dropped decimal separator, a solution attached to the neighbouring
problem. These are invisible in retrieval and only surface when something trusts the text.
`rag_verify_solutions` re-solves every extracted problem independently and compares.
`LessonChunk.solution_verdict` is three-state — `agrees` / `disagrees` / `unjudgeable` —
and the verdict also rides in the Qdrant payload under a key `retrieval._payload_text`
does not read, so the tutor's context is unchanged. Current state: 1,112 agree, 123
disagree, 881 unjudgeable. **Question generation only ever reads `agrees` chunks.**
Two things that were learned the expensive way and are already encoded in the prompts:
- A boolean verdict is wrong. With two states every "I can't tell" becomes "wrong", which
reported ~3× the real rate. Read samples before trusting any bulk run.
- `۳/۱۰۴` in these books is the decimal 3.104, not the fraction 3÷104.
`solution` itself is never rewritten — the extracted text stays as the record, and the
independent answer is stored separately in `checked_answer`.
## The figure gap (largest open data issue)
881 chunks (~42%) reference a figure or table that is not in the text, so they cannot be
answered or turned into questions. **The vision model saw the figure; it was simply never
asked to write it down.** `rag/services/pdf.py` now carries a self-contained rule telling
it to inline measurements and labels into the problem text — books ingested before that
rule predate it. Recovering them means re-ingesting those pages (~$2.7 of API credit).
## Quiz assembly
The app plays at **node** level (the island on the map) but questions hang off **lessons**,
and a lesson here is the finest grain — one problem type from the book, not a class
session. `GET /api/nodes/{id}/quiz` assembles a quiz from a node's lessons, at most one
question per lesson. Shuffle is seeded on `nodeId:userId`, so a refresh returns the same
quiz; `questionCount` is reported separately from `requested` so a thin node cannot look
like an error. 172 of 230 nodes currently hold a full four-question quiz.
## AI provider notes
`OPENAI_BASE_URL` points at GapGPT (an OpenAI proxy reachable from Iran) and it exposes
~127 models, including reasoning ones. Two failures look alike and must not be treated
alike — `is_account_limit()` in `rag_verify_solutions` is the shared guard:
- `429 initial_usage_limit_exceeded` — account cap, hours long. Retrying **extends** it.
- `403 insufficient user quota` — credit exhausted. Note the spaces: an earlier guard
matched `insufficient_quota` with an underscore, missed it, and burned 770 doomed calls.
`/dashboard/billing/usage` returns cents and is the only reliable balance signal; the
`hard_limit_usd` in `/dashboard/billing/subscription` is a placeholder, not a balance.
**The key is currently out of credit.** Bulk work since then runs on Claude Code subagents
instead. Sonnet handles question authoring well (~88% yield); Haiku does not — it cannot
reliably pull the final answer out of a solution and produced ~40% broken output.
## Known gaps (per README pre-deploy checklist) ## Known gaps (per README pre-deploy checklist)
- Still on SQLite, not Postgres. - Still on SQLite, not Postgres.
- Throttle rates need periodic review against real traffic. - Throttle rates need periodic review against real traffic.
- `DEV_MASTER_OTP` (`0000`) must stay gated behind `DEBUG` — never enabled in prod. - `DEV_MASTER_OTP` (`0000`) must stay gated behind `DEBUG` — never enabled in prod.
- The server runs with `DJANGO_DEBUG=1` on a public domain, so OTP `0000` currently logs
anyone in as anyone. Harmless until real users exist; `/api/health` warns about it.
- No video content exists yet, though `Lesson.kind`/`video_url` can express it. - No video content exists yet, though `Lesson.kind`/`video_url` can express it.
- Question generation and textbook ingest are blocked on AI API credit. - 58 of 230 nodes still hold fewer than four questions; their verified chunks ran out.
- Kavenegar is unconfigured, so `POST /api/auth/otp/request` returns 503. With `DEBUG=0`
there is no way to log into the API without creating an `OTPCode` row in the admin
(`/admin/accounts/otpcode/add/`) — this is how to hand a tester a working login.
## State as of 2026-09-17 — read this first
**Backend is done and verified; nothing is blocked on us.** The full game loop was driven
end to end against a fresh database with `DEBUG=0`: grades → login → set grade → home →
node quiz → submit answer → CORS header for the frontend origin. `main` is pushed and
clean at `16d3d9f`; 435 tests green.
Two things are blocked on other people, and both are worth checking before assuming a bug:
1. **The server still runs pre-16d3d9f code with an empty database.** `git pull &&
./deploy.sh` brings both code and content. Verify with `curl /api/health` — it should
report `questions: 1441`. A 404 there means the pull never landed.
2. **The frontend is not wired to any API.** `src/api/` was merged (PR #1) but is
imported nowhere; `QuestionScreen` in `App.tsx` still renders four hardcoded
multiplication questions. **PR #3** (`zareian/api-client-2`, 3 commits) is still open on
upstream. Until someone connects the screens, none of the 1,441 questions are visible.
The frontend's `Question` type carries a `type` field that nothing reads, and
`count_objects` in `Question.TYPE_CHOICES` is an empty shell — no rows, no serializer
branch, no rendering anywhere. Do not build content for it without asking first.