make a deploy fail loudly when the service never restarted

سرور دو روز کدِ قدیمی را سرو کرد و هیچ‌کس نفهمید. علتش این بود که
`systemctl restart` شکست خورده بود ولی پروسه‌ی قبلی هنوز بالا بود و همان
۲۰۰ را می‌داد — پس اسکریپت «✓ سرویس سالم بالا آمد» چاپ کرد و با کدِ صفر
تمام شد. محتوا و فایل‌های استاتیک هم روی دیسک بودند و سرو نمی‌شدند.

حالا زمانِ بالا آمدنِ سرویس قبل و بعد مقایسه می‌شود. اگر عوض نشده باشد،
۲۰۰ گرفتن دیگر موفقیت حساب نمی‌شود و اسکریپت با خطا تمام می‌کند.

نامِ عامِ `gunicorn` هم از فهرستِ حدسِ سرویس برداشته شد: روی سرورِ واقعی یک
`gunicorn.service` هست که مالِ تیمِ دیگری است، و حدس زدنش یعنی ری‌استارت
کردنِ سرویسِ یک نفرِ دیگر. نامِ درستِ ما `gunicorn-mathapp` است.

`api/health/` هم کنارِ `api/health` ثبت شد. پایشِ سرور شکلِ اسلش‌دار را صدا
می‌زد و چون ثبت نبود، کسی روی سرور `config/urls.py` را دستی ویرایش کرده
بود — و از آن به بعد هر `git pull` روی همان فایل گیر می‌کرد.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 22:56:51 +03:30
parent 07fab4c094
commit de118db58f
3 changed files with 94 additions and 4 deletions

View File

@@ -218,6 +218,63 @@ reliably pull the final answer out of a solution and produced ~40% broken output
there is no way to log into the API without creating an `OTPCode` row in the admin
(`/admin/accounts/otpcode/add/`) — this is how to hand a tester a working login.
## The server: read this before running anything on it
`api.math.tadrisino2.ir` runs on a **shared company machine** (`administrator@78.38.35.249`,
hostname `haghzad`), not a box that belongs to this project. Our app is one tenant among
many. Also on it, all in production use by other people:
- **Gitea itself** (`gitea.service`) — the company's git server, including this repo's
origin. Breaking it means nobody can push or pull anything, anywhere.
- Mail (`postfix`, `dovecot`), DNS (`named`), `mariadb`, `postgresql@14`, `rabbitmq`,
`metabase`, `redis-server`, `nginx`.
- Other apps: `apirag`, `artinafront`, `metaverse`, `celerybeat`, and a **generic
`gunicorn.service` that is not ours**.
- Other people's home directories: `aria`, `artina`, `behdad`, `zahra`, `git`, `gitea`.
They are mode `0750` — unreadable as `administrator`, trivially readable under `sudo`.
Do not read them. Nothing in this project needs them.
**Our account is `administrator`, and `sudo -l` reports `(ALL : ALL) ALL`.** It is also in
the `docker` and `lxd` groups, each of which is root by another route. There is no
technical guard rail between a careless command and the whole company's data. The only
guard rail is the operator.
Ours is exactly one directory: **`/home/administrator/mathapp-v2`**, served by
**`gunicorn-mathapp.service`** (unix socket, proxied by nginx). Note `~/mathapp` also
exists, is not a git repo, and is not what runs — do not touch it.
Rules that follow from this, and are not negotiable:
1. **Never run a destructive command outside `~/mathapp-v2`.** No `rm -rf`, no
`git clean`, no `chown -R`, no dropping databases, anywhere else. When a command
takes a path, write the path out in full rather than relying on the current directory.
2. **Restart only `gunicorn-mathapp`.** `systemctl restart gunicorn` is a different
team's app. `deploy.sh` no longer guesses that name, and neither should you.
3. **Never delete the project directory or "re-clone fresh".** `db.sqlite3`, `.env` and
`.django_secret_key` live inside it and are all gitignored. A fresh clone silently
destroys the database, the config and every logged-in session. Use
`git reset --hard origin/main` when the tree needs to go back to a known state.
4. **Never `git clean`** there for the same reason.
5. **Read `.env` only through Django.** `python -c` printing `settings.CACHES[...]` and
booleans is fine; `cat .env` is not, and neither is pasting its values into a chat.
6. **Take a timestamped backup before editing anything under `/etc`,** and say where you
put it.
7. **Disk is the standing risk.** It has been sitting at ~95% full (about 6 GB free on a
113 GB volume). Check `df -h /` before anything that writes a lot — a full disk on
this machine takes the mail server and Gitea down with us, not just our app.
`DEBUG` must never be `1` on this server: it turns on `DEV_MASTER_OTP` (`0000`), which
accepts any phone number as a valid login, and renders tracebacks with settings in them.
`DJANGO_DEBUG` belongs in `.env` only — not in the systemd unit, where it once sat as
`Environment="DJANGO_DEBUG=1"` and was saved only by `.env` happening to override it.
`DJANGO_SECRET_KEY` is what signs our bearer tokens. It must be a real random value that
exists nowhere in git. If the env var is simply absent and `DEBUG=0`, `config/settings.py`
generates a strong one and stores it in `.django_secret_key` (mode `0600`) — that is the
preferred setup, because there is then no key to leak. **Never let the server run on the
`django-insecure-…` fallback from `config/settings.py`: that string is committed here, so
anyone who can read this repo can forge a token for any user.**
## State as of 2026-09-17 — read this first
**Backend is done and verified; nothing is blocked on us.** The full game loop was driven