new app — cleaned-up RAG pipeline (Qdrant retrieval + GPT), ported from Rag_backend

Self-contained app; wired into settings/urls in the next commit. Data
stays in the parallel team's live Qdrant — nothing is copied into our DB.

apps.py
  - RagConfig.
models.py
  - LessonChunk: DB-of-record for one summarized lesson chunk
    (original_text + summary + embedding_vector) mirrored as a single
    Qdrant point; loose lesson_id/grade_id ints, no cross-app FK.
    LessonChunkImage for future multimodal lessons.
migrations/0001_initial.py
  - initial tables for the two models.
services/clients.py
  - lazy, lru_cache'd OpenAI + Qdrant clients (Rag_backend builds them at
    import time and crashes without keys). RagNotConfigured raised when a
    key/lib is missing; Qdrant client uses check_compatibility=False for
    the 1.19-client / 1.17-server minor version gap.
services/config.py
  - rag_is_configured(), collection-name template, grade-title resolution
    (RAG_GRADE_TITLE_OVERRIDES -> content.Grade.title), retrieval top-k.
services/known_collections.py
  - exact live Qdrant catalog: 6 collection names, math chapter-number ->
    exact "فصل N: ..." string tables per (subject, grade), and the
    علوم/اجتماعی lesson-number -> title maps. filter_key_for_subject()
    picks `chapter` for math, `lesson_number` otherwise;
    resolve_math_chapter() maps a plain chapter number (Persian or Latin
    digits) to the stored string.
services/embedding.py
  - embed_text() pinned to OPENAI_EMBEDDING_MODEL (text-embedding-3-small,
    1536d) so vectors match the other team's existing collections.
services/chunking.py
  - split_text_into_chunks(), summarize_chunk() (gpt-4o-mini, JSON out);
    plain exceptions instead of ninja HttpError.
services/qdrant_store.py
  - ensure_collection(), upsert_points(), search(): the lesson filter
    matches the value both as str and native, since Qdrant stores
    lesson_number as int and chapter as string.
services/retrieval.py
  - resolve_target() -> (collection, filter_key, filter_value), reused by
    the cache key and debug block.
  - retrieve_context() -> (chunks, meta): embed -> Qdrant search scoped to
    one lesson/chapter -> top-k. Math payloads (no `summary`) fall back to
    generalSolution/problem/solution; a 404 collection returns empty.
services/generation.py
  - generate_answer() and stream_answer() (token iterator for SSE);
    context-grounded Persian prompt that explains from retrieved text
    rather than refusing.
services/pdf.py
  - textbook PDF -> page images -> gpt-4o vision problem-type extraction
    (adapted from Rag_backend mathapp); returns dicts, no on-disk JSON.
    pymupdf/pdf2image optional — missing libs raise RagNotConfigured.
services/ingest.py
  - ingest_text() / ingest_pdf(): chunk -> summarize -> embed -> persist
    LessonChunk + upsert Qdrant, in one transaction.
services/answer_cache.py
  - short-TTL cache of the final answer keyed by
    sha1(collection | filter_value | normalized-question).
services/tutor.py
  - thin answer_question() (retrieve -> generate) for direct callers.
serializers.py, urls.py, views.py
  - POST /api/rag/ingest/text, POST /api/rag/ingest/pdf (HasInternalServiceKey,
    throttle scope rag_ingest), GET /api/rag/chunks review list.
    RagProviderUnavailable -> clean 503.
admin.py
  - LessonChunk admin with an inline for its images.
management/commands/rag_catalog.py
  - read-only live Qdrant listing (--values, --json); never writes.
tests.py
  - mocked-client tests: ask-with-retrieval, ingest, SSE stream, answer
    cache, and rag_is_configured() gating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

rag/services/pdf.py

@@ -0,0 +1,126 @@
+"""
+استخراج از PDF کتابِ درسی + تحلیلِ عکسِ صفحه با مدلِ vision.
+
+برداشتِ پاک‌سازی‌شده از mathapp/services/read_pdf.py و info_question.py مخزنِ
+Rag_backend. به‌جای نوشتنِ نتیجه در یک فایلِ JSON روی دیسک، دیکشنری
+برمی‌گردونه تا لایه‌ی ingest تصمیم بگیره.
+
+pymupdf (fitz) و pdf2image اختیاری‌ان؛ نبودشون فقط این ماژول رو از کار
+می‌ندازه (RagNotConfigured)، نه بقیه‌ی پروژه رو.
+"""
+import base64
+import json
+from io import BytesIO
+
+from django.conf import settings
+
+from . import clients
+from .clients import RagNotConfigured
+
+
+def extract_page_images(pdf_path: str, first_page: int = 1, dpi: int = 200):
+    """لیستی از تصاویرِ PIL برای صفحاتِ PDF از first_page به بعد."""
+    try:
+        from pdf2image import convert_from_path
+    except ModuleNotFoundError as exc:
+        raise RagNotConfigured('pdf2image نصب نیست یا poppler موجود نیست.') from exc
+    return convert_from_path(pdf_path, dpi=dpi, first_page=first_page)
+
+
+def detect_grade_and_start_page(pdf_path: str) -> tuple[int, str | None]:
+    """
+    از متنِ صفحه‌ی اولِ PDF، پایه‌ی تحصیلی و صفحه‌ای که محتوای واقعی از اونجا
+    شروع می‌شه رو حدس می‌زنه (مثلِ نسخه‌ی Rag_backend).
+    """
+    try:
+        import fitz  # PyMuPDF
+    except ModuleNotFoundError as exc:
+        raise RagNotConfigured('pymupdf (fitz) نصب نیست.') from exc
+
+    start_pages = {'پنجم دبستان': 11, 'ششم دبستان': 9}
+    grades = ['هفتم', 'اول دبستان', 'دوم دبستان', 'سوم دبستان',
+              'چهارم دبستان', 'پنجم دبستان', 'ششم دبستان']
+
+    doc = fitz.open(pdf_path)
+    try:
+        text = doc[0].get_text()
+    finally:
+        doc.close()
+
+    grade = next((g for g in grades if g in text), None)
+    return start_pages.get(grade, 7), grade
+
+
+_VISION_JSON_TEMPLATE = """
+{
+  "grade": "",
+  "chapter": "",
+  "problemTypes": [
+    {"type": "", "keywords": [], "generalSolution": "",
+     "exampleFromImage": {"problem": "", "solution": ""}}
+  ]
+}
+"""
+
+
+def extract_problem_types(image, grade: str, chapter_name: str) -> dict:
+    """
+    یک عکسِ صفحه‌ی کتاب → ساختارِ JSONِ انواعِ مسئله (فارسی)، با مدلِ vision.
+    خروجی همون شکلی‌ست که uploadِ ریاضیِ تیمِ دیگه هم تولید می‌کرد.
+    """
+    prompt = f"""
+You are a specialist elementary math teacher for children aged 8 to 12.
+Analyze the uploaded image and extract every unique math problem type for the given chapter.
+
+For each type produce: type, keywords (3-7, Persian), generalSolution (step-by-step),
+exampleFromImage (one real problem from the image + full solution).
+
+Output must follow EXACTLY this JSON structure:
+{_VISION_JSON_TEMPLATE}
+
+Rules:
+- Everything in Persian (Farsi).
+- grade = "{grade}"
+- chapter = "{chapter_name}"
+- exampleFromImage must be a real problem from the image (not invented).
+- Output ONLY the JSON object. No markdown, no backticks.
+"""
+
+    buffered = BytesIO()
+    image.save(buffered, format='JPEG')
+    img_b64 = base64.b64encode(buffered.getvalue()).decode()
+
+    client = clients.get_openai_client()
+    response = client.chat.completions.create(
+        model=settings.OPENAI_VISION_MODEL,
+        messages=[
+            {'role': 'system', 'content': 'You process educational visual content and respond ONLY with valid JSON.'},
+            {'role': 'user', 'content': [
+                {'type': 'text', 'text': prompt},
+                {'type': 'image_url', 'image_url': {'url': f'data:image/jpeg;base64,{img_b64}', 'detail': 'high'}},
+            ]},
+        ],
+        max_tokens=3000,
+        temperature=0.1,
+    )
+    raw = (response.choices[0].message.content or '').strip()
+    if raw.startswith('```json'):
+        raw = raw[len('```json'):].strip()
+    if raw.startswith('```'):
+        raw = raw[3:].strip()
+    if raw.endswith('```'):
+        raw = raw[:-3].strip()
+    return json.loads(raw)
+
+
+def problem_type_to_text(problem_type: dict) -> str:
+    """یک problemType → متنِ قابلِ embedding (مثلِ doc_text در Rag_backend)."""
+    example = problem_type.get('exampleFromImage', {}) or {}
+    keywords = ' '.join(problem_type.get('keywords', []) or [])
+    return (
+        f"Type: {problem_type.get('type', '')}\n"
+        f"Keywords: {keywords}\n"
+        f"General solution: {problem_type.get('generalSolution', '')}\n"
+        f"Example problem: {example.get('problem', '')}\n"
+        f"Example solution: {example.get('solution', '')}"
+    ).strip()