Zareian
ac93e17305
Self-contained app; wired into settings/urls in the next commit. Data
stays in the parallel team's live Qdrant — nothing is copied into our DB.
apps.py
- RagConfig.
models.py
- LessonChunk: DB-of-record for one summarized lesson chunk
(original_text + summary + embedding_vector) mirrored as a single
Qdrant point; loose lesson_id/grade_id ints, no cross-app FK.
LessonChunkImage for future multimodal lessons.
migrations/0001_initial.py
- initial tables for the two models.
services/clients.py
- lazy, lru_cache'd OpenAI + Qdrant clients (Rag_backend builds them at
import time and crashes without keys). RagNotConfigured raised when a
key/lib is missing; Qdrant client uses check_compatibility=False for
the 1.19-client / 1.17-server minor version gap.
services/config.py
- rag_is_configured(), collection-name template, grade-title resolution
(RAG_GRADE_TITLE_OVERRIDES -> content.Grade.title), retrieval top-k.
services/known_collections.py
- exact live Qdrant catalog: 6 collection names, math chapter-number ->
exact "فصل N: ..." string tables per (subject, grade), and the
علوم/اجتماعی lesson-number -> title maps. filter_key_for_subject()
picks `chapter` for math, `lesson_number` otherwise;
resolve_math_chapter() maps a plain chapter number (Persian or Latin
digits) to the stored string.
services/embedding.py
- embed_text() pinned to OPENAI_EMBEDDING_MODEL (text-embedding-3-small,
1536d) so vectors match the other team's existing collections.
services/chunking.py
- split_text_into_chunks(), summarize_chunk() (gpt-4o-mini, JSON out);
plain exceptions instead of ninja HttpError.
services/qdrant_store.py
- ensure_collection(), upsert_points(), search(): the lesson filter
matches the value both as str and native, since Qdrant stores
lesson_number as int and chapter as string.
services/retrieval.py
- resolve_target() -> (collection, filter_key, filter_value), reused by
the cache key and debug block.
- retrieve_context() -> (chunks, meta): embed -> Qdrant search scoped to
one lesson/chapter -> top-k. Math payloads (no `summary`) fall back to
generalSolution/problem/solution; a 404 collection returns empty.
services/generation.py
- generate_answer() and stream_answer() (token iterator for SSE);
context-grounded Persian prompt that explains from retrieved text
rather than refusing.
services/pdf.py
- textbook PDF -> page images -> gpt-4o vision problem-type extraction
(adapted from Rag_backend mathapp); returns dicts, no on-disk JSON.
pymupdf/pdf2image optional — missing libs raise RagNotConfigured.
services/ingest.py
- ingest_text() / ingest_pdf(): chunk -> summarize -> embed -> persist
LessonChunk + upsert Qdrant, in one transaction.
services/answer_cache.py
- short-TTL cache of the final answer keyed by
sha1(collection | filter_value | normalized-question).
services/tutor.py
- thin answer_question() (retrieve -> generate) for direct callers.
serializers.py, urls.py, views.py
- POST /api/rag/ingest/text, POST /api/rag/ingest/pdf (HasInternalServiceKey,
throttle scope rag_ingest), GET /api/rag/chunks review list.
RagProviderUnavailable -> clean 503.
admin.py
- LessonChunk admin with an inline for its images.
management/commands/rag_catalog.py
- read-only live Qdrant listing (--values, --json); never writes.
tests.py
- mocked-client tests: ask-with-retrieval, ingest, SSE stream, answer
cache, and rag_is_configured() gating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
99 lines
3.9 KiB
Python
99 lines
3.9 KiB
Python
"""
|
||
مرحله‌ی generation: پرسش + context بازیابی‌شده → پاسخِ فارسی + چند پیشنهادِ
|
||
ادامه.
|
||
|
||
اگه context خالی باشه، مدل به‌عنوانِ معلمِ عمومی جواب می‌ده (بدونِ ادعای
|
||
استناد به کتاب) — این حالت وقتی پیش می‌آد که subject/lessonNumber ارسال
|
||
نشده یا اون درس هنوز در Qdrant نیست.
|
||
"""
|
||
import logging
|
||
from collections.abc import Iterator
|
||
|
||
from django.conf import settings
|
||
|
||
from . import clients
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
_DEFAULT_FOLLOW_UPS = ['یک مثال دیگه بزن', 'بیشتر توضیح بده']
|
||
|
||
_SYSTEM = 'تو یک معلمِ فارسی‌زبانِ دبستانی هستی و پاسخ‌ها را روشن، ساده و کوتاه می‌دهی.'
|
||
|
||
|
||
def _prompt_with_context(question: str, contexts: list[str]) -> str:
|
||
joined = '\n\n---\n\n'.join(contexts)
|
||
return f"""
|
||
تو معلمِ دبستانی هستی. با تکیه بر «متنِ درس» زیر و با زبانِ خودت، به سوالِ
|
||
دانش‌آموز پاسخِ روشن و ساده بده. لازم نیست پاسخ عیناً در متن باشد؛ می‌توانی
|
||
از اطلاعاتِ متن نتیجه‌گیری و توضیح بدهی.
|
||
فقط اگر متنِ درس آشکارا درباره‌ی موضوعِ دیگری‌ست و هیچ ربطی به سوال ندارد،
|
||
بنویس: «این موضوع در این درس پوشش داده نشده است.»
|
||
|
||
متنِ درس:
|
||
{joined}
|
||
|
||
سوالِ دانش‌آموز: {question}
|
||
"""
|
||
|
||
|
||
def _prompt_without_context(question: str) -> str:
|
||
return f"""
|
||
به این سوالِ یک دانش‌آموزِ دبستانی به فارسی، ساده و کوتاه پاسخ بده.
|
||
|
||
سوال: {question}
|
||
"""
|
||
|
||
|
||
def generate_answer(question: str, context_chunks: list[dict] | None) -> tuple[str, list[str]]:
|
||
"""
|
||
context_chunks: خروجیِ retrieve_context ([{title, summary, score}, ...]) یا None/[].
|
||
خروجی: (answer, follow_ups)
|
||
"""
|
||
contexts = [c['summary'] for c in (context_chunks or []) if c.get('summary')]
|
||
prompt = _prompt_with_context(question, contexts) if contexts else _prompt_without_context(question)
|
||
|
||
client = clients.get_openai_client()
|
||
completion = client.chat.completions.create(
|
||
model=settings.OPENAI_CHAT_MODEL,
|
||
messages=[
|
||
{'role': 'system', 'content': _SYSTEM},
|
||
{'role': 'user', 'content': prompt},
|
||
],
|
||
temperature=0.3,
|
||
)
|
||
answer = (completion.choices[0].message.content or '').strip()
|
||
if not answer:
|
||
answer = 'متأسفم، الان نتونستم پاسخِ مناسبی بسازم. لطفاً دوباره بپرس.'
|
||
return answer, list(_DEFAULT_FOLLOW_UPS)
|
||
|
||
|
||
def stream_answer(question: str, context_chunks: list[dict] | None) -> Iterator[str]:
|
||
"""
|
||
همون generate_answer ولی توکن‌به‌توکن (برای SSE). فقط قطعه‌های متنِ پاسخ
|
||
را yield می‌کند؛ follow_upها ثابت‌اند (_DEFAULT_FOLLOW_UPS) و caller خودش
|
||
اضافه‌شان می‌کند.
|
||
"""
|
||
contexts = [c['summary'] for c in (context_chunks or []) if c.get('summary')]
|
||
prompt = _prompt_with_context(question, contexts) if contexts else _prompt_without_context(question)
|
||
|
||
client = clients.get_openai_client()
|
||
stream = client.chat.completions.create(
|
||
model=settings.OPENAI_CHAT_MODEL,
|
||
messages=[
|
||
{'role': 'system', 'content': _SYSTEM},
|
||
{'role': 'user', 'content': prompt},
|
||
],
|
||
temperature=0.3,
|
||
stream=True,
|
||
)
|
||
for event in stream:
|
||
if not event.choices:
|
||
continue
|
||
delta = event.choices[0].delta.content or ''
|
||
if delta:
|
||
yield delta
|
||
|
||
|
||
def default_follow_ups() -> list[str]:
|
||
return list(_DEFAULT_FOLLOW_UPS)
|