new app — cleaned-up RAG pipeline (Qdrant retrieval + GPT), ported from Rag_backend

Self-contained app; wired into settings/urls in the next commit. Data
stays in the parallel team's live Qdrant — nothing is copied into our DB.

apps.py
  - RagConfig.
models.py
  - LessonChunk: DB-of-record for one summarized lesson chunk
    (original_text + summary + embedding_vector) mirrored as a single
    Qdrant point; loose lesson_id/grade_id ints, no cross-app FK.
    LessonChunkImage for future multimodal lessons.
migrations/0001_initial.py
  - initial tables for the two models.
services/clients.py
  - lazy, lru_cache'd OpenAI + Qdrant clients (Rag_backend builds them at
    import time and crashes without keys). RagNotConfigured raised when a
    key/lib is missing; Qdrant client uses check_compatibility=False for
    the 1.19-client / 1.17-server minor version gap.
services/config.py
  - rag_is_configured(), collection-name template, grade-title resolution
    (RAG_GRADE_TITLE_OVERRIDES -> content.Grade.title), retrieval top-k.
services/known_collections.py
  - exact live Qdrant catalog: 6 collection names, math chapter-number ->
    exact "فصل N: ..." string tables per (subject, grade), and the
    علوم/اجتماعی lesson-number -> title maps. filter_key_for_subject()
    picks `chapter` for math, `lesson_number` otherwise;
    resolve_math_chapter() maps a plain chapter number (Persian or Latin
    digits) to the stored string.
services/embedding.py
  - embed_text() pinned to OPENAI_EMBEDDING_MODEL (text-embedding-3-small,
    1536d) so vectors match the other team's existing collections.
services/chunking.py
  - split_text_into_chunks(), summarize_chunk() (gpt-4o-mini, JSON out);
    plain exceptions instead of ninja HttpError.
services/qdrant_store.py
  - ensure_collection(), upsert_points(), search(): the lesson filter
    matches the value both as str and native, since Qdrant stores
    lesson_number as int and chapter as string.
services/retrieval.py
  - resolve_target() -> (collection, filter_key, filter_value), reused by
    the cache key and debug block.
  - retrieve_context() -> (chunks, meta): embed -> Qdrant search scoped to
    one lesson/chapter -> top-k. Math payloads (no `summary`) fall back to
    generalSolution/problem/solution; a 404 collection returns empty.
services/generation.py
  - generate_answer() and stream_answer() (token iterator for SSE);
    context-grounded Persian prompt that explains from retrieved text
    rather than refusing.
services/pdf.py
  - textbook PDF -> page images -> gpt-4o vision problem-type extraction
    (adapted from Rag_backend mathapp); returns dicts, no on-disk JSON.
    pymupdf/pdf2image optional — missing libs raise RagNotConfigured.
services/ingest.py
  - ingest_text() / ingest_pdf(): chunk -> summarize -> embed -> persist
    LessonChunk + upsert Qdrant, in one transaction.
services/answer_cache.py
  - short-TTL cache of the final answer keyed by
    sha1(collection | filter_value | normalized-question).
services/tutor.py
  - thin answer_question() (retrieve -> generate) for direct callers.
serializers.py, urls.py, views.py
  - POST /api/rag/ingest/text, POST /api/rag/ingest/pdf (HasInternalServiceKey,
    throttle scope rag_ingest), GET /api/rag/chunks review list.
    RagProviderUnavailable -> clean 503.
admin.py
  - LessonChunk admin with an inline for its images.
management/commands/rag_catalog.py
  - read-only live Qdrant listing (--values, --json); never writes.
tests.py
  - mocked-client tests: ask-with-retrieval, ingest, SSE stream, answer
    cache, and rag_is_configured() gating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-29 02:29:54 +03:30
parent 38999f3541
commit ac93e17305
26 changed files with 1661 additions and 0 deletions

44
rag/services/config.py Normal file
View File

@@ -0,0 +1,44 @@
"""
هم‌خوانیِ نامِ collection و کلیدِ فیلترِ Qdrant با دادهٔ موجودِ تیمِ دیگر.
در Rag_backend نامِ collection این‌طور ساخته می‌شه:
f"{subject.name} {subject.grade.title}".lower()
و فیلترِ درس با کلیدِ payloadِ "lesson_number" (برای درس‌های متنی) یا
"chapter" (برای ریاضی) انجام می‌شه. اینجا هر دو از settings قابلِ تنظیم‌ان
چون اسمِ دقیقِ گرید حساسه (کاربر قبلاً با اسمِ نادرست به مشکل خورده بود).
"""
from django.conf import settings
def rag_is_configured() -> bool:
return bool(
settings.OPENAI_API_KEY
and settings.QDRANT_API_KEY
and not settings.RAG_FORCE_MOCK
)
def resolve_grade_title(grade_id, fallback: str = '') -> str:
"""gradeId → عنوانِ گرید همان‌طور که در نامِ collectionِ تیمِ دیگه اومده."""
overrides = getattr(settings, 'RAG_GRADE_TITLE_OVERRIDES', {}) or {}
if grade_id in overrides:
return str(overrides[grade_id])
if grade_id is not None:
try:
from content.models import Grade
grade = Grade.objects.filter(pk=grade_id).only('title').first()
if grade is not None:
return grade.title
except Exception:
pass
return fallback
def build_collection_name(subject: str, grade_title: str) -> str:
template = getattr(settings, 'RAG_COLLECTION_NAME_TEMPLATE', '{subject} {grade}')
return template.format(subject=(subject or '').strip(), grade=(grade_title or '').strip()).lower()
def retrieval_top_k() -> int:
return int(getattr(settings, 'RAG_RETRIEVAL_TOP_K', 4))