new app — cleaned-up RAG pipeline (Qdrant retrieval + GPT), ported from Rag_backend

Self-contained app; wired into settings/urls in the next commit. Data
stays in the parallel team's live Qdrant — nothing is copied into our DB.

apps.py
  - RagConfig.
models.py
  - LessonChunk: DB-of-record for one summarized lesson chunk
    (original_text + summary + embedding_vector) mirrored as a single
    Qdrant point; loose lesson_id/grade_id ints, no cross-app FK.
    LessonChunkImage for future multimodal lessons.
migrations/0001_initial.py
  - initial tables for the two models.
services/clients.py
  - lazy, lru_cache'd OpenAI + Qdrant clients (Rag_backend builds them at
    import time and crashes without keys). RagNotConfigured raised when a
    key/lib is missing; Qdrant client uses check_compatibility=False for
    the 1.19-client / 1.17-server minor version gap.
services/config.py
  - rag_is_configured(), collection-name template, grade-title resolution
    (RAG_GRADE_TITLE_OVERRIDES -> content.Grade.title), retrieval top-k.
services/known_collections.py
  - exact live Qdrant catalog: 6 collection names, math chapter-number ->
    exact "فصل N: ..." string tables per (subject, grade), and the
    علوم/اجتماعی lesson-number -> title maps. filter_key_for_subject()
    picks `chapter` for math, `lesson_number` otherwise;
    resolve_math_chapter() maps a plain chapter number (Persian or Latin
    digits) to the stored string.
services/embedding.py
  - embed_text() pinned to OPENAI_EMBEDDING_MODEL (text-embedding-3-small,
    1536d) so vectors match the other team's existing collections.
services/chunking.py
  - split_text_into_chunks(), summarize_chunk() (gpt-4o-mini, JSON out);
    plain exceptions instead of ninja HttpError.
services/qdrant_store.py
  - ensure_collection(), upsert_points(), search(): the lesson filter
    matches the value both as str and native, since Qdrant stores
    lesson_number as int and chapter as string.
services/retrieval.py
  - resolve_target() -> (collection, filter_key, filter_value), reused by
    the cache key and debug block.
  - retrieve_context() -> (chunks, meta): embed -> Qdrant search scoped to
    one lesson/chapter -> top-k. Math payloads (no `summary`) fall back to
    generalSolution/problem/solution; a 404 collection returns empty.
services/generation.py
  - generate_answer() and stream_answer() (token iterator for SSE);
    context-grounded Persian prompt that explains from retrieved text
    rather than refusing.
services/pdf.py
  - textbook PDF -> page images -> gpt-4o vision problem-type extraction
    (adapted from Rag_backend mathapp); returns dicts, no on-disk JSON.
    pymupdf/pdf2image optional — missing libs raise RagNotConfigured.
services/ingest.py
  - ingest_text() / ingest_pdf(): chunk -> summarize -> embed -> persist
    LessonChunk + upsert Qdrant, in one transaction.
services/answer_cache.py
  - short-TTL cache of the final answer keyed by
    sha1(collection | filter_value | normalized-question).
services/tutor.py
  - thin answer_question() (retrieve -> generate) for direct callers.
serializers.py, urls.py, views.py
  - POST /api/rag/ingest/text, POST /api/rag/ingest/pdf (HasInternalServiceKey,
    throttle scope rag_ingest), GET /api/rag/chunks review list.
    RagProviderUnavailable -> clean 503.
admin.py
  - LessonChunk admin with an inline for its images.
management/commands/rag_catalog.py
  - read-only live Qdrant listing (--values, --json); never writes.
tests.py
  - mocked-client tests: ask-with-retrieval, ingest, SSE stream, answer
    cache, and rag_is_configured() gating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

rag/services/ingest.py

@@ -0,0 +1,138 @@
+"""
+هماهنگ‌کننده‌ی ingestion: متن/PDF درس → chunk/خلاصه/embedding → ذخیره در
+DB (LessonChunk، منبعِ حقیقت) + upsert به Qdrant (index).
+
+نامِ collection و کلیدهای payload عمداً همون قراردادِ Rag_backend را دنبال
+می‌کنن تا با دادهٔ موجودِ تیمِ دیگه سازگار بمونه.
+"""
+import logging
+import os
+import tempfile
+
+from django.db import transaction
+
+from ..models import LessonChunk
+from .chunking import chunk_and_summarize
+from .config import build_collection_name, resolve_grade_title
+from .embedding import embed_text
+from . import pdf as pdf_service
+from .qdrant_store import ensure_collection, upsert_points
+
+logger = logging.getLogger(__name__)
+
+
+def _payload(chunk: LessonChunk) -> dict:
+    return {
+        'lesson_id': chunk.lesson_id,
+        'lesson_number': chunk.lesson_number,
+        'chapter_number': chunk.chapter,
+        'chunk_order': chunk.order,
+        'lesson_title': chunk.title,
+        'summary': chunk.summary,
+    }
+
+
+def _persist_and_upsert(rows: list[dict], *, lesson_id, grade_id, subject,
+                        lesson_number, chapter, title, collection) -> list[LessonChunk]:
+    ensure_collection(collection)
+    created: list[LessonChunk] = []
+    points: list[dict] = []
+
+    with transaction.atomic():
+        for row in rows:
+            vector = embed_text(row['summary'])
+            chunk = LessonChunk.objects.create(
+                lesson_id=lesson_id,
+                grade_id=grade_id,
+                subject=subject,
+                lesson_number=lesson_number,
+                chapter=chapter,
+                order=row['order'],
+                title=title,
+                original_text=row['original_text'],
+                summary=row['summary'],
+                embedding_vector=vector,
+                collection=collection,
+            )
+            created.append(chunk)
+            points.append({'id': chunk.qdrant_point_id, 'vector': vector, 'payload': _payload(chunk)})
+
+    upsert_points(collection, points)
+    return created
+
+
+def ingest_text(*, lesson_id: int, grade_id, subject: str, lesson_number: int,
+                text: str, chapter: int | None = None, title: str = '') -> dict:
+    grade_title = resolve_grade_title(grade_id)
+    collection = build_collection_name(subject, grade_title)
+    title = title or f'lesson_{lesson_number}'
+
+    rows = chunk_and_summarize(text)
+    if not rows:
+        return {'status': 'empty', 'chunks_created': 0, 'collection': collection}
+
+    created = _persist_and_upsert(
+        rows, lesson_id=lesson_id, grade_id=grade_id, subject=subject,
+        lesson_number=lesson_number, chapter=chapter, title=title, collection=collection,
+    )
+    return {
+        'status': 'success',
+        'collection': collection,
+        'chunks_created': len(created),
+        'chunk_ids': [c.id for c in created],
+    }
+
+
+def ingest_pdf(*, lesson_id: int, grade_id, subject: str, lesson_number: int,
+               pdf_bytes: bytes, chapter: int | None = None, title: str = '',
+               chapter_name: str = '') -> dict:
+    """
+    هر صفحه‌ی PDF → عکس → استخراجِ انواعِ مسئله با vision → هر problemType یک
+    chunk. برای درس‌های متنی به‌جای این از ingest_text استفاده کن.
+    """
+    grade_title = resolve_grade_title(grade_id)
+    collection = build_collection_name(subject, grade_title)
+    title = title or f'lesson_{lesson_number}'
+    chapter_name = chapter_name or title
+
+    fd, tmp_path = tempfile.mkstemp(suffix='.pdf')
+    try:
+        with os.fdopen(fd, 'wb') as fh:
+            fh.write(pdf_bytes)
+
+        start_page, detected_grade = pdf_service.detect_grade_and_start_page(tmp_path)
+        images = pdf_service.extract_page_images(tmp_path, first_page=start_page)
+
+        rows: list[dict] = []
+        order = 0
+        for image in images:
+            try:
+                parsed = pdf_service.extract_problem_types(image, detected_grade or grade_title, chapter_name)
+            except Exception:
+                logger.exception('rag.ingest_pdf: تحلیلِ یک صفحه شکست خورد؛ رد شد')
+                continue
+            for problem_type in parsed.get('problemTypes', []) or []:
+                text = pdf_service.problem_type_to_text(problem_type)
+                if not text:
+                    continue
+                order += 1
+                rows.append({'order': order, 'original_text': text, 'summary': text})
+    finally:
+        try:
+            os.remove(tmp_path)
+        except OSError:
+            pass
+
+    if not rows:
+        return {'status': 'empty', 'chunks_created': 0, 'collection': collection}
+
+    created = _persist_and_upsert(
+        rows, lesson_id=lesson_id, grade_id=grade_id, subject=subject,
+        lesson_number=lesson_number, chapter=chapter, title=title, collection=collection,
+    )
+    return {
+        'status': 'success',
+        'collection': collection,
+        'chunks_created': len(created),
+        'chunk_ids': [c.id for c in created],
+    }