write down what the last two days changed about content and answer quality

CLAUDE.md تنها جایی است که هر جلسه‌ی تازه اول می‌خواندش، و چیزهایی که این دوره
یاد گرفتیم جایی ثبت نشده بود. چهار بخش اضافه شد و «وضعیتِ فعلی» به بالا آمد:

• محتوا حالا در content/seed/ کامیت می‌شود و deploy خودش بالا می‌آوردش — بدونِ
  Qdrant، بدونِ اعتبارِ AI، بدونِ SSH. و اینکه سوال با مسیرِ عنوان به درس وصل
  می‌شود نه با id، چون idها بینِ دو دیتابیس یکی نیستند.
• لایه‌ی راستی‌آزمایی: کتاب از روی تصویر خوانده می‌شود، پس خطا همان‌جا وارد
  می‌شود. برچسب سه‌حالته است و سوال فقط از agreesها ساخته می‌شود.
• دو درسی که گران تمام شدند: نتیجه‌ی دوحالته نرخ را سه برابر نشان می‌دهد، و
  «۳/۱۰۴» در این کتاب‌ها عددِ اعشاری است نه کسر.
• شکافِ شکل‌ها (۴۲٪ محتوا) و اینکه اصلاحش در پرامپت هست ولی اجرا نشده.
• تفاوتِ ۴۲۹ سقفِ حساب با ۴۰۳ اتمامِ اعتبار — و اینکه متنِ دومی فاصله دارد نه
  زیرخط، که یک بار ۷۷۰ فراخوانیِ بی‌فایده به بار آورد.

«وضعیت تا ۲۰۲۶-۰۹-۱۷» هم اضافه شد: بک‌اند آماده و تست‌شده است، و دو مانعِ
باقی‌مانده دستِ ما نیست — سرور روی کدِ قدیمی، و فرانتی که به هیچ APIی وصل نیست.
هر دو قبل از اینکه کسی دنبالِ باگ بگردد باید چک شوند.

هیچ فایلِ تازه‌ای اضافه نشد؛ همان فایلِ موجود به‌روز شد.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 05:59:19 +03:30
parent 16d3d9f149
commit 07fab4c094

View File

@@ -137,12 +137,104 @@ those defaults, not replaced by them.
counts, security warnings, and recent traffic. It is how you check whether anyone has
actually connected without asking a human.
## Content: where it lives and how it gets to a server
`content/seed/` holds the whole curriculum and question bank as committed JSON —
`curriculum.json` (6 grades, 48 units, 230 nodes, 1,928 lessons) and `questions.json`
(1,441 multiple-choice questions). `deploy.sh` runs `seed_content` on every release, so a
fresh server fills itself with one command: **no Qdrant, no AI credit, no SSH session.**
`seed_content` is idempotent and never deletes. Questions bind to lessons **by title path**
(grade › unit › node › lesson), never by primary key — ids differ between any two
databases, and binding by id silently attaches questions to the wrong lesson.
`seed_content --export` writes the files back from the local DB. It refuses to overwrite a
non-empty `questions.json` from a DB that has no questions (that used to wipe the file
silently); use `--export --only curriculum` when you only mean the curriculum.
## Answer quality: the verification layer
Books are read by a **vision model from page images**, so extraction errors enter there —
a misread digit, a dropped decimal separator, a solution attached to the neighbouring
problem. These are invisible in retrieval and only surface when something trusts the text.
`rag_verify_solutions` re-solves every extracted problem independently and compares.
`LessonChunk.solution_verdict` is three-state — `agrees` / `disagrees` / `unjudgeable` —
and the verdict also rides in the Qdrant payload under a key `retrieval._payload_text`
does not read, so the tutor's context is unchanged. Current state: 1,112 agree, 123
disagree, 881 unjudgeable. **Question generation only ever reads `agrees` chunks.**
Two things that were learned the expensive way and are already encoded in the prompts:
- A boolean verdict is wrong. With two states every "I can't tell" becomes "wrong", which
reported ~3× the real rate. Read samples before trusting any bulk run.
- `۳/۱۰۴` in these books is the decimal 3.104, not the fraction 3÷104.
`solution` itself is never rewritten — the extracted text stays as the record, and the
independent answer is stored separately in `checked_answer`.
## The figure gap (largest open data issue)
881 chunks (~42%) reference a figure or table that is not in the text, so they cannot be
answered or turned into questions. **The vision model saw the figure; it was simply never
asked to write it down.** `rag/services/pdf.py` now carries a self-contained rule telling
it to inline measurements and labels into the problem text — books ingested before that
rule predate it. Recovering them means re-ingesting those pages (~$2.7 of API credit).
## Quiz assembly
The app plays at **node** level (the island on the map) but questions hang off **lessons**,
and a lesson here is the finest grain — one problem type from the book, not a class
session. `GET /api/nodes/{id}/quiz` assembles a quiz from a node's lessons, at most one
question per lesson. Shuffle is seeded on `nodeId:userId`, so a refresh returns the same
quiz; `questionCount` is reported separately from `requested` so a thin node cannot look
like an error. 172 of 230 nodes currently hold a full four-question quiz.
## AI provider notes
`OPENAI_BASE_URL` points at GapGPT (an OpenAI proxy reachable from Iran) and it exposes
~127 models, including reasoning ones. Two failures look alike and must not be treated
alike — `is_account_limit()` in `rag_verify_solutions` is the shared guard:
- `429 initial_usage_limit_exceeded` — account cap, hours long. Retrying **extends** it.
- `403 insufficient user quota` — credit exhausted. Note the spaces: an earlier guard
matched `insufficient_quota` with an underscore, missed it, and burned 770 doomed calls.
`/dashboard/billing/usage` returns cents and is the only reliable balance signal; the
`hard_limit_usd` in `/dashboard/billing/subscription` is a placeholder, not a balance.
**The key is currently out of credit.** Bulk work since then runs on Claude Code subagents
instead. Sonnet handles question authoring well (~88% yield); Haiku does not — it cannot
reliably pull the final answer out of a solution and produced ~40% broken output.
## Known gaps (per README pre-deploy checklist)
- Still on SQLite, not Postgres.
- Throttle rates need periodic review against real traffic.
- `DEV_MASTER_OTP` (`0000`) must stay gated behind `DEBUG` — never enabled in prod.
- The server runs with `DJANGO_DEBUG=1` on a public domain, so OTP `0000` currently logs
anyone in as anyone. Harmless until real users exist; `/api/health` warns about it.
- No video content exists yet, though `Lesson.kind`/`video_url` can express it.
- Question generation and textbook ingest are blocked on AI API credit.
- 58 of 230 nodes still hold fewer than four questions; their verified chunks ran out.
- Kavenegar is unconfigured, so `POST /api/auth/otp/request` returns 503. With `DEBUG=0`
there is no way to log into the API without creating an `OTPCode` row in the admin
(`/admin/accounts/otpcode/add/`) — this is how to hand a tester a working login.
## State as of 2026-09-17 — read this first
**Backend is done and verified; nothing is blocked on us.** The full game loop was driven
end to end against a fresh database with `DEBUG=0`: grades → login → set grade → home →
node quiz → submit answer → CORS header for the frontend origin. `main` is pushed and
clean at `16d3d9f`; 435 tests green.
Two things are blocked on other people, and both are worth checking before assuming a bug:
1. **The server still runs pre-16d3d9f code with an empty database.** `git pull &&
./deploy.sh` brings both code and content. Verify with `curl /api/health` — it should
report `questions: 1441`. A 404 there means the pull never landed.
2. **The frontend is not wired to any API.** `src/api/` was merged (PR #1) but is
imported nowhere; `QuestionScreen` in `App.tsx` still renders four hardcoded
multiplication questions. **PR #3** (`zareian/api-client-2`, 3 commits) is still open on
upstream. Until someone connects the screens, none of the 1,441 questions are visible.
The frontend's `Question` type carries a `type` field that nothing reads, and
`count_objects` in `Question.TYPE_CHOICES` is an empty shell — no rows, no serializer
branch, no rendering anywhere. Do not build content for it without asking first.