API output without Persian diacritics or written ezafe ی (company decision); tutor answers too
plain_persian() strips kasra/fatha/damma/shadda/sukun/hamza-above and the written «هی» ezafe, keeping tanwin. Applied at output, not in data: PlainPersianJSONRenderer runs one byte-level pass over every JSON response (~5 ms per MB locally), and to_plain() cleans tutor answers including the stream. Rewriting titles in the DB/seed would have broken question binding by title path and SeedRecord keys. Tests updated to expect the cleaned text; config/tests_plain_persian.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -256,8 +256,9 @@ def _to_plain(text: str) -> str:
|
||||
text = re.sub(r'[ \t]+\n', '\n', text)
|
||||
text = re.sub(r'\n{3,}', '\n\n', text)
|
||||
# S_n، a_1، x^2 و توانِ پیچیده که بالا رد شد: همان تبدیلِ seed/دیتابیس (یک رفتار در همهجا).
|
||||
from content.plain_math import convert as _plain
|
||||
return _plain(text).strip()
|
||||
from content.plain_math import convert as _plain, plain_persian
|
||||
# بیاعراب و بی «ی»ِ اضافه، مثلِ همهی پاسخهای API (config/renderers.py)؛ پاسخِ استریمی از آن renderer رد نمیشود.
|
||||
return plain_persian(_plain(text)).strip()
|
||||
|
||||
|
||||
class StreamSanitizer:
|
||||
|
||||
Reference in New Issue
Block a user