write down what the last two days changed about content and answer quality
CLAUDE.md تنها جایی است که هر جلسهی تازه اول میخواندش، و چیزهایی که این دوره یاد گرفتیم جایی ثبت نشده بود. چهار بخش اضافه شد و «وضعیتِ فعلی» به بالا آمد: • محتوا حالا در content/seed/ کامیت میشود و deploy خودش بالا میآوردش — بدونِ Qdrant، بدونِ اعتبارِ AI، بدونِ SSH. و اینکه سوال با مسیرِ عنوان به درس وصل میشود نه با id، چون idها بینِ دو دیتابیس یکی نیستند. • لایهی راستیآزمایی: کتاب از روی تصویر خوانده میشود، پس خطا همانجا وارد میشود. برچسب سهحالته است و سوال فقط از agreesها ساخته میشود. • دو درسی که گران تمام شدند: نتیجهی دوحالته نرخ را سه برابر نشان میدهد، و «۳/۱۰۴» در این کتابها عددِ اعشاری است نه کسر. • شکافِ شکلها (۴۲٪ محتوا) و اینکه اصلاحش در پرامپت هست ولی اجرا نشده. • تفاوتِ ۴۲۹ سقفِ حساب با ۴۰۳ اتمامِ اعتبار — و اینکه متنِ دومی فاصله دارد نه زیرخط، که یک بار ۷۷۰ فراخوانیِ بیفایده به بار آورد. «وضعیت تا ۲۰۲۶-۰۹-۱۷» هم اضافه شد: بکاند آماده و تستشده است، و دو مانعِ باقیمانده دستِ ما نیست — سرور روی کدِ قدیمی، و فرانتی که به هیچ APIی وصل نیست. هر دو قبل از اینکه کسی دنبالِ باگ بگردد باید چک شوند. هیچ فایلِ تازهای اضافه نشد؛ همان فایلِ موجود بهروز شد. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
98
CLAUDE.md
98
CLAUDE.md
@@ -137,12 +137,104 @@ those defaults, not replaced by them.
|
||||
counts, security warnings, and recent traffic. It is how you check whether anyone has
|
||||
actually connected without asking a human.
|
||||
|
||||
## Content: where it lives and how it gets to a server
|
||||
|
||||
`content/seed/` holds the whole curriculum and question bank as committed JSON —
|
||||
`curriculum.json` (6 grades, 48 units, 230 nodes, 1,928 lessons) and `questions.json`
|
||||
(1,441 multiple-choice questions). `deploy.sh` runs `seed_content` on every release, so a
|
||||
fresh server fills itself with one command: **no Qdrant, no AI credit, no SSH session.**
|
||||
|
||||
`seed_content` is idempotent and never deletes. Questions bind to lessons **by title path**
|
||||
(grade › unit › node › lesson), never by primary key — ids differ between any two
|
||||
databases, and binding by id silently attaches questions to the wrong lesson.
|
||||
|
||||
`seed_content --export` writes the files back from the local DB. It refuses to overwrite a
|
||||
non-empty `questions.json` from a DB that has no questions (that used to wipe the file
|
||||
silently); use `--export --only curriculum` when you only mean the curriculum.
|
||||
|
||||
## Answer quality: the verification layer
|
||||
|
||||
Books are read by a **vision model from page images**, so extraction errors enter there —
|
||||
a misread digit, a dropped decimal separator, a solution attached to the neighbouring
|
||||
problem. These are invisible in retrieval and only surface when something trusts the text.
|
||||
|
||||
`rag_verify_solutions` re-solves every extracted problem independently and compares.
|
||||
`LessonChunk.solution_verdict` is three-state — `agrees` / `disagrees` / `unjudgeable` —
|
||||
and the verdict also rides in the Qdrant payload under a key `retrieval._payload_text`
|
||||
does not read, so the tutor's context is unchanged. Current state: 1,112 agree, 123
|
||||
disagree, 881 unjudgeable. **Question generation only ever reads `agrees` chunks.**
|
||||
|
||||
Two things that were learned the expensive way and are already encoded in the prompts:
|
||||
|
||||
- A boolean verdict is wrong. With two states every "I can't tell" becomes "wrong", which
|
||||
reported ~3× the real rate. Read samples before trusting any bulk run.
|
||||
- `۳/۱۰۴` in these books is the decimal 3.104, not the fraction 3÷104.
|
||||
|
||||
`solution` itself is never rewritten — the extracted text stays as the record, and the
|
||||
independent answer is stored separately in `checked_answer`.
|
||||
|
||||
## The figure gap (largest open data issue)
|
||||
|
||||
881 chunks (~42%) reference a figure or table that is not in the text, so they cannot be
|
||||
answered or turned into questions. **The vision model saw the figure; it was simply never
|
||||
asked to write it down.** `rag/services/pdf.py` now carries a self-contained rule telling
|
||||
it to inline measurements and labels into the problem text — books ingested before that
|
||||
rule predate it. Recovering them means re-ingesting those pages (~$2.7 of API credit).
|
||||
|
||||
## Quiz assembly
|
||||
|
||||
The app plays at **node** level (the island on the map) but questions hang off **lessons**,
|
||||
and a lesson here is the finest grain — one problem type from the book, not a class
|
||||
session. `GET /api/nodes/{id}/quiz` assembles a quiz from a node's lessons, at most one
|
||||
question per lesson. Shuffle is seeded on `nodeId:userId`, so a refresh returns the same
|
||||
quiz; `questionCount` is reported separately from `requested` so a thin node cannot look
|
||||
like an error. 172 of 230 nodes currently hold a full four-question quiz.
|
||||
|
||||
## AI provider notes
|
||||
|
||||
`OPENAI_BASE_URL` points at GapGPT (an OpenAI proxy reachable from Iran) and it exposes
|
||||
~127 models, including reasoning ones. Two failures look alike and must not be treated
|
||||
alike — `is_account_limit()` in `rag_verify_solutions` is the shared guard:
|
||||
|
||||
- `429 initial_usage_limit_exceeded` — account cap, hours long. Retrying **extends** it.
|
||||
- `403 insufficient user quota` — credit exhausted. Note the spaces: an earlier guard
|
||||
matched `insufficient_quota` with an underscore, missed it, and burned 770 doomed calls.
|
||||
|
||||
`/dashboard/billing/usage` returns cents and is the only reliable balance signal; the
|
||||
`hard_limit_usd` in `/dashboard/billing/subscription` is a placeholder, not a balance.
|
||||
|
||||
**The key is currently out of credit.** Bulk work since then runs on Claude Code subagents
|
||||
instead. Sonnet handles question authoring well (~88% yield); Haiku does not — it cannot
|
||||
reliably pull the final answer out of a solution and produced ~40% broken output.
|
||||
|
||||
## Known gaps (per README pre-deploy checklist)
|
||||
|
||||
- Still on SQLite, not Postgres.
|
||||
- Throttle rates need periodic review against real traffic.
|
||||
- `DEV_MASTER_OTP` (`0000`) must stay gated behind `DEBUG` — never enabled in prod.
|
||||
- The server runs with `DJANGO_DEBUG=1` on a public domain, so OTP `0000` currently logs
|
||||
anyone in as anyone. Harmless until real users exist; `/api/health` warns about it.
|
||||
- No video content exists yet, though `Lesson.kind`/`video_url` can express it.
|
||||
- Question generation and textbook ingest are blocked on AI API credit.
|
||||
- 58 of 230 nodes still hold fewer than four questions; their verified chunks ran out.
|
||||
- Kavenegar is unconfigured, so `POST /api/auth/otp/request` returns 503. With `DEBUG=0`
|
||||
there is no way to log into the API without creating an `OTPCode` row in the admin
|
||||
(`/admin/accounts/otpcode/add/`) — this is how to hand a tester a working login.
|
||||
|
||||
## State as of 2026-09-17 — read this first
|
||||
|
||||
**Backend is done and verified; nothing is blocked on us.** The full game loop was driven
|
||||
end to end against a fresh database with `DEBUG=0`: grades → login → set grade → home →
|
||||
node quiz → submit answer → CORS header for the frontend origin. `main` is pushed and
|
||||
clean at `16d3d9f`; 435 tests green.
|
||||
|
||||
Two things are blocked on other people, and both are worth checking before assuming a bug:
|
||||
|
||||
1. **The server still runs pre-16d3d9f code with an empty database.** `git pull &&
|
||||
./deploy.sh` brings both code and content. Verify with `curl /api/health` — it should
|
||||
report `questions: 1441`. A 404 there means the pull never landed.
|
||||
2. **The frontend is not wired to any API.** `src/api/` was merged (PR #1) but is
|
||||
imported nowhere; `QuestionScreen` in `App.tsx` still renders four hardcoded
|
||||
multiplication questions. **PR #3** (`zareian/api-client-2`, 3 commits) is still open on
|
||||
upstream. Until someone connects the screens, none of the 1,441 questions are visible.
|
||||
|
||||
The frontend's `Question` type carries a `type` field that nothing reads, and
|
||||
`count_objects` in `Question.TYPE_CHOICES` is an empty shell — no rows, no serializer
|
||||
branch, no rendering anywhere. Do not build content for it without asking first.
|
||||
|
||||
Reference in New Issue
Block a user