new app — cleaned-up RAG pipeline (Qdrant retrieval + GPT), ported from Rag_backend

Self-contained app; wired into settings/urls in the next commit. Data
stays in the parallel team's live Qdrant — nothing is copied into our DB.

apps.py
  - RagConfig.
models.py
  - LessonChunk: DB-of-record for one summarized lesson chunk
    (original_text + summary + embedding_vector) mirrored as a single
    Qdrant point; loose lesson_id/grade_id ints, no cross-app FK.
    LessonChunkImage for future multimodal lessons.
migrations/0001_initial.py
  - initial tables for the two models.
services/clients.py
  - lazy, lru_cache'd OpenAI + Qdrant clients (Rag_backend builds them at
    import time and crashes without keys). RagNotConfigured raised when a
    key/lib is missing; Qdrant client uses check_compatibility=False for
    the 1.19-client / 1.17-server minor version gap.
services/config.py
  - rag_is_configured(), collection-name template, grade-title resolution
    (RAG_GRADE_TITLE_OVERRIDES -> content.Grade.title), retrieval top-k.
services/known_collections.py
  - exact live Qdrant catalog: 6 collection names, math chapter-number ->
    exact "فصل N: ..." string tables per (subject, grade), and the
    علوم/اجتماعی lesson-number -> title maps. filter_key_for_subject()
    picks `chapter` for math, `lesson_number` otherwise;
    resolve_math_chapter() maps a plain chapter number (Persian or Latin
    digits) to the stored string.
services/embedding.py
  - embed_text() pinned to OPENAI_EMBEDDING_MODEL (text-embedding-3-small,
    1536d) so vectors match the other team's existing collections.
services/chunking.py
  - split_text_into_chunks(), summarize_chunk() (gpt-4o-mini, JSON out);
    plain exceptions instead of ninja HttpError.
services/qdrant_store.py
  - ensure_collection(), upsert_points(), search(): the lesson filter
    matches the value both as str and native, since Qdrant stores
    lesson_number as int and chapter as string.
services/retrieval.py
  - resolve_target() -> (collection, filter_key, filter_value), reused by
    the cache key and debug block.
  - retrieve_context() -> (chunks, meta): embed -> Qdrant search scoped to
    one lesson/chapter -> top-k. Math payloads (no `summary`) fall back to
    generalSolution/problem/solution; a 404 collection returns empty.
services/generation.py
  - generate_answer() and stream_answer() (token iterator for SSE);
    context-grounded Persian prompt that explains from retrieved text
    rather than refusing.
services/pdf.py
  - textbook PDF -> page images -> gpt-4o vision problem-type extraction
    (adapted from Rag_backend mathapp); returns dicts, no on-disk JSON.
    pymupdf/pdf2image optional — missing libs raise RagNotConfigured.
services/ingest.py
  - ingest_text() / ingest_pdf(): chunk -> summarize -> embed -> persist
    LessonChunk + upsert Qdrant, in one transaction.
services/answer_cache.py
  - short-TTL cache of the final answer keyed by
    sha1(collection | filter_value | normalized-question).
services/tutor.py
  - thin answer_question() (retrieve -> generate) for direct callers.
serializers.py, urls.py, views.py
  - POST /api/rag/ingest/text, POST /api/rag/ingest/pdf (HasInternalServiceKey,
    throttle scope rag_ingest), GET /api/rag/chunks review list.
    RagProviderUnavailable -> clean 503.
admin.py
  - LessonChunk admin with an inline for its images.
management/commands/rag_catalog.py
  - read-only live Qdrant listing (--values, --json); never writes.
tests.py
  - mocked-client tests: ask-with-retrieval, ingest, SSE stream, answer
    cache, and rag_is_configured() gating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

rag/views.py

@@ -0,0 +1,157 @@
+import base64
+import binascii
+import logging
+
+from drf_spectacular.types import OpenApiTypes
+from drf_spectacular.utils import OpenApiParameter, extend_schema
+from rest_framework.exceptions import APIException
+from rest_framework.parsers import FormParser, JSONParser, MultiPartParser
+from rest_framework.response import Response
+from rest_framework.throttling import ScopedRateThrottle
+from rest_framework.views import APIView
+
+from payment.permissions import HasInternalServiceKey
+
+from .models import LessonChunk
+from .serializers import (
+    ChunkListResponseSerializer,
+    ChunkSerializer,
+    IngestPdfRequestSerializer,
+    IngestResponseSerializer,
+    IngestTextRequestSerializer,
+)
+from .services import ingest as ingest_service
+from .services.clients import RagNotConfigured
+
+logger = logging.getLogger(__name__)
+
+_INTERNAL_KEY_PARAM = OpenApiParameter(
+    'X-Internal-Key', OpenApiTypes.STR, OpenApiParameter.HEADER, required=True,
+    description='کلید مخفیِ فراخوانی داخلی/سرور-به-سرور (settings.INTERNAL_SERVICE_KEY).',
+)
+
+
+class RagProviderUnavailable(APIException):
+    status_code = 503
+    default_detail = 'سرویس هوش مصنوعی/بردار موقتاً در دسترس نیست. لطفاً بعداً تلاش کنید.'
+    default_code = 'rag_provider_unavailable'
+
+
+def _ingest_response(result: dict) -> Response:
+    return Response(IngestResponseSerializer({
+        'status': result['status'],
+        'collection': result['collection'],
+        'chunksCreated': result['chunks_created'],
+        'chunkIds': result.get('chunk_ids', []),
+    }).data)
+
+
+class IngestTextView(APIView):
+    """POST /api/rag/ingest/text — آپلودِ متنِ یک درس، chunk/خلاصه/embedding و ذخیره در DB + Qdrant."""
+
+    permission_classes = [HasInternalServiceKey]
+    throttle_scope = 'rag_ingest'
+    throttle_classes = [ScopedRateThrottle]
+
+    @extend_schema(
+        request=IngestTextRequestSerializer,
+        responses=IngestResponseSerializer,
+        parameters=[_INTERNAL_KEY_PARAM],
+        tags=['RAG (rag)'],
+    )
+    def post(self, request):
+        serializer = IngestTextRequestSerializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        v = serializer.validated_data
+        try:
+            result = ingest_service.ingest_text(
+                lesson_id=v['lessonId'],
+                grade_id=v.get('gradeId'),
+                subject=v['subject'],
+                lesson_number=v['lessonNumber'],
+                text=v['text'],
+                chapter=v.get('chapter'),
+                title=v.get('title') or '',
+            )
+        except RagNotConfigured as exc:
+            logger.warning('rag ingest_text: not configured: %s', exc)
+            raise RagProviderUnavailable(detail=str(exc))
+        except Exception:
+            logger.exception('rag ingest_text failed')
+            raise RagProviderUnavailable()
+        return _ingest_response(result)
+
+
+class IngestPdfView(APIView):
+    """POST /api/rag/ingest/pdf — آپلودِ PDF کتاب، استخراجِ صفحات با vision و ذخیره در DB + Qdrant."""
+
+    permission_classes = [HasInternalServiceKey]
+    throttle_scope = 'rag_ingest'
+    throttle_classes = [ScopedRateThrottle]
+    parser_classes = [MultiPartParser, FormParser, JSONParser]
+
+    @extend_schema(
+        request=IngestPdfRequestSerializer,
+        responses=IngestResponseSerializer,
+        parameters=[_INTERNAL_KEY_PARAM],
+        tags=['RAG (rag)'],
+    )
+    def post(self, request):
+        serializer = IngestPdfRequestSerializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        v = serializer.validated_data
+
+        upload = v.get('pdf')
+        if upload is not None:
+            pdf_bytes = upload.read()
+        else:
+            try:
+                pdf_bytes = base64.b64decode(v['pdfBase64'], validate=True)
+            except (binascii.Error, ValueError):
+                raise RagProviderUnavailable(detail='pdfBase64 معتبر نیست.')
+
+        try:
+            result = ingest_service.ingest_pdf(
+                lesson_id=v['lessonId'],
+                grade_id=v.get('gradeId'),
+                subject=v['subject'],
+                lesson_number=v['lessonNumber'],
+                pdf_bytes=pdf_bytes,
+                chapter=v.get('chapter'),
+                title=v.get('title') or '',
+                chapter_name=v.get('chapterName') or '',
+            )
+        except RagNotConfigured as exc:
+            logger.warning('rag ingest_pdf: not available: %s', exc)
+            raise RagProviderUnavailable(detail=str(exc))
+        except Exception:
+            logger.exception('rag ingest_pdf failed')
+            raise RagProviderUnavailable()
+        return _ingest_response(result)
+
+
+class ChunkListView(APIView):
+    """GET /api/rag/chunks — فهرستِ chunkهای ذخیره‌شده (بازبینی)، فیلترِ lessonId/subject."""
+
+    permission_classes = [HasInternalServiceKey]
+
+    @extend_schema(
+        responses=ChunkListResponseSerializer,
+        parameters=[
+            _INTERNAL_KEY_PARAM,
+            OpenApiParameter('lessonId', OpenApiTypes.INT, OpenApiParameter.QUERY, required=False),
+            OpenApiParameter('subject', OpenApiTypes.STR, OpenApiParameter.QUERY, required=False),
+        ],
+        tags=['RAG (rag)'],
+    )
+    def get(self, request):
+        qs = LessonChunk.objects.all()
+        lesson_id = request.query_params.get('lessonId')
+        subject = request.query_params.get('subject')
+        if lesson_id:
+            qs = qs.filter(lesson_id=lesson_id)
+        if subject:
+            qs = qs.filter(subject=subject)
+        rows = list(qs[:200])
+        data = {'count': len(rows), 'results': ChunkSerializer(rows, many=True).data}
+        return Response(ChunkListResponseSerializer(data).data)