Postgres connection pool, clean 503 under overload, time indexes, 4 workers

Postgres here allows 60 connections, but each of the 96 gunicorn threads held its
own persistent one: at 100 concurrent users (local Postgres with the same limit)
21% of requests were 500 "remaining connection slots are reserved". Now each worker
has a psycopg pool (8), so the same load gives 0 errors and 714 req/s (was 460).

- A full pool answers 503 + Retry-After (DRF handler and middleware), not 500.
- The AI tutor and the OTP SMS call return their connection before the network wait.
- statement_timeout 30 s, idle-in-transaction 60 s; bad pool settings and the old
  DB_CONN_MAX_AGE stop startup instead of failing every request.
- /api/health dbPool: per-worker pool stats and workers x pool + 6 vs max_connections.
- Indexes on the columns global counters filter by time (weekly leaderboard,
  global SMS cap, global AI cap); the leaderboard recomputes once per expiry.
- Worker count moves into gunicorn.conf.py (4 on a shared 8-core box).
- security doc: Redis item closed (server uses our own Redis since 2026-10-08).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -125,7 +125,7 @@ API docs: Swagger `/api/docs/`, Redoc `/api/redoc/`, admin `/admin/`.
 
 **Structured logging**: `config.middleware.APIRequestLoggingMiddleware` (last in `MIDDLEWARE`, closest to the view) logs every `/api/` request as JSON (method/path/status/duration/user, no body/headers/secrets) via `config/log_formatters.py`.
 
-**Postgres** is the server's DB since 2026-10-08 (own cluster `14/zareian`, port 5440, localhost only; DB `zareian_mathapp`, role `zareian_app`; selected by `DB_NAME`/`DB_USER`/`DB_PASSWORD`/`DB_HOST`/`DB_PORT` in `.env`; passwords in `~/mathapp-v2/.pg_password`, `.redis_password`). It is separate from the company's `main` cluster on 5432 — never touch that. Without `DB_NAME` the app falls back to SQLite (local dev, tests); both run the full suite green. Data was moved with `dumpdata`/`loaddata`; model-count comparison matched. Guards that relied on SQLite's `IMMEDIATE` now rely on real row locks; concurrency tests pass on Postgres, not load-tested. `deploy.sh` backs up with `pg_dump -Fc` before `migrate`; the off-site live copy is in «بکاپ: کپیِ زنده» below. Operator guide: `~/Desktop/گزارشات Tadrisino/گزارش-۲۹/راهنمای-Postgres-و-Redis.html`. The pre-move `db.sqlite3` and migration dumps were deleted from the server; no SQLite file is left there.
+**Postgres** is the server's DB since 2026-10-08 (own cluster `14/zareian`, port 5440, localhost only; DB `zareian_mathapp`, role `zareian_app`; selected by `DB_NAME`/`DB_USER`/`DB_PASSWORD`/`DB_HOST`/`DB_PORT` in `.env`; passwords in `~/mathapp-v2/.pg_password`, `.redis_password`). It is separate from the company's `main` cluster on 5432 — never touch that. Without `DB_NAME` the app falls back to SQLite (local dev, tests); both run the full suite green. Data was moved with `dumpdata`/`loaddata`; model-count comparison matched. Guards that relied on SQLite's `IMMEDIATE` now rely on real row locks; concurrency tests pass on Postgres. **Connections come from a per-worker pool** (`OPTIONS.pool`, `DB_POOL_MAX_SIZE` 8; `CONN_MAX_AGE` is gone and `DB_CONN_MAX_AGE` in `.env` stops startup): Postgres here has `max_connections=60`, and the old one-connection-per-thread setup (3 workers × 32 threads) turned 21% of requests into 500s at 100 concurrent users in a local load test against a 60-connection Postgres; with the pool 0 errors. A full pool gives 503 + `Retry-After` (`config/db.py`), not 500; `/api/health` → `dbPool` checks `workers × pool + 6 ≤ max_connections`. The AI tutor and the OTP SMS call hand their connection back before the network wait (`release_db_connection`). Worker count lives in `gunicorn.conf.py` (4; the systemd unit no longer passes `--workers`). Not load-tested on the server itself. `deploy.sh` backs up with `pg_dump -Fc` before `migrate`; the off-site live copy is in «بکاپ: کپیِ زنده» below. Operator guide: `~/Desktop/گزارشات Tadrisino/گزارش-۲۹/راهنمای-Postgres-و-Redis.html`. The pre-move `db.sqlite3` and migration dumps were deleted from the server; no SQLite file is left there.
 
 **SMS** goes through Kavenegar (`KAVENEGAR_API_KEY`/`KAVENEGAR_SENDER`); failures return explicit 503, not a fake success.