mirror of
https://github.com/Such-Software/wownerogue.git
synced 2026-10-24 09:38:32 +02:00
jwinterm
ed24d43b56
Removes operator-private material from the shipped tree, corrects documentation
that had drifted from the code, deletes dead code, and applies a consistent
prose style.
Private material moved out of the public tree (docs/private/, git-ignored):
- the wallet keyfile recovery runbook and its helper scripts, which together
published the mainnet wallet file path, the service account, backup snapshot
filenames, and the wallet RPC listener state. The helper duplicated the same
constants, so relocating the document alone would not have removed them.
- the stagenet parallel-candidate runbook and its fixtures.
- the dated production-state record, the internal remediation plan, and the
TODO list. None are reference documentation for an outside reader.
Redacted or genericised:
- operator home-directory paths in shipped scripts and units. The asset build
script now takes KENNEY_ASSETS_DIR instead of a hard-coded path.
- private LAN addresses in tests, env examples and deploy docs, replaced with
RFC 5737 documentation addresses.
- the default alert sender and the notification hub example.
- the Monero wallet unit now matches the Wownero one: a dedicated service
account and a config file, instead of the operator's login account with the
wallet password in argv where any local user can read it from /proc.
Ignore rules now cover every .env variant rather than the bare name, plus local
agent state. The deployment docs instruct operators to create .env.mainnet and
.env.stagenet, which the exact-name rule would have allowed to be committed.
Documentation corrected against the code: per-chain dungeon level counts, the
premium unlock rule, Tavern chat scope, backup retention, the chain-profile
fallback, the environment template location, and several sections describing
shipped work as pending.
Dead code removed: configPersistence, rpccalls, a duplicate vendored ROT.js
bundle, the unused tile-picker overlay, the unreachable legacy game-size path
and its vestigial parameter, spectator stubs that shadowed the real
SpectatorManager, and commented-out debugging. nip98 now imports the shared
nostr loader rather than duplicating it.
Prose style: em-dashes removed throughout in favour of colons, semicolons and
commas; comments restated as present-tense descriptions of the code rather than
narration of past changes.
127 suites / 1015 tests pass.
260 lines
9.8 KiB
Markdown
260 lines
9.8 KiB
Markdown
# Logs and Backup
|
|
|
|
Operational reference for application logs and PostgreSQL backups in a production deployment.
|
|
|
|
---
|
|
|
|
## Application Logs
|
|
|
|
The server has no logging library and writes no log files. All diagnostics go to stdout and stderr,
|
|
so under systemd they land in the journal of the unit that started the process. Log retention is
|
|
therefore a journald configuration question, not an application one.
|
|
|
|
The repository ships two application units in `scripts/deploy/`: `wownerogue.service` (Wownero
|
|
mainnet instance) and `monerogue.service` (Monero stagenet instance).
|
|
|
|
### Viewing Logs
|
|
|
|
```bash
|
|
# All logs for a unit
|
|
journalctl -u wownerogue
|
|
|
|
# Follow live
|
|
journalctl -u wownerogue -f
|
|
|
|
# Last 100 lines
|
|
journalctl -u wownerogue -n 100
|
|
|
|
# Since today
|
|
journalctl -u wownerogue --since today
|
|
|
|
# A specific time range
|
|
journalctl -u wownerogue --since "2026-01-01" --until "2026-01-15"
|
|
|
|
# Errors only
|
|
journalctl -u wownerogue -p err
|
|
|
|
# Export for offline analysis
|
|
journalctl -u wownerogue --since "2026-01-01" > wownerogue-jan.log
|
|
```
|
|
|
|
The same commands work for `monerogue`, the wallet-RPC units, and the backup unit.
|
|
|
|
### Journal Retention
|
|
|
|
Journald limits apply to the whole journal, not to an individual unit, so these settings bound
|
|
every service on the host. Create a drop-in such as `/etc/systemd/journald.conf.d/limits.conf`:
|
|
|
|
```ini
|
|
[Journal]
|
|
# Total disk budget for all journals
|
|
SystemMaxUse=500M
|
|
# Size cap per journal file
|
|
SystemMaxFileSize=50M
|
|
# Discard entries older than this
|
|
MaxRetentionSec=30day
|
|
Compress=yes
|
|
```
|
|
|
|
Apply it:
|
|
|
|
```bash
|
|
sudo mkdir -p /etc/systemd/journald.conf.d/
|
|
sudo install -m 0644 limits.conf /etc/systemd/journald.conf.d/limits.conf
|
|
sudo systemctl restart systemd-journald
|
|
```
|
|
|
|
Journald is a rolling operational log, not an audit archive. Records that must survive log
|
|
rotation belong in the database. Accounting events are exported through the durable outbox
|
|
described in [FINANCIAL_EVENT_EXPORT.md](./FINANCIAL_EVENT_EXPORT.md).
|
|
|
|
---
|
|
|
|
## PostgreSQL Backups
|
|
|
|
### The Validated Backup Job
|
|
|
|
`scripts/deploy/wowngeon-db-backup.sh`, with its `wowngeon-db-backup.service` and
|
|
`wowngeon-db-backup.timer` units, is the production backup path. Each run:
|
|
|
|
- Dumps every database named in the script's loop. The shipped list is `wownerogue` and
|
|
`monerogue`; a single-instance operator edits that loop to match the databases actually present.
|
|
- Writes custom-format dumps (`pg_dump --format=custom --no-owner`) to a temporary file, verifies
|
|
the catalog with `pg_restore --list`, sets mode `0600`, then renames into place, so a reader never
|
|
observes a partial dump under its final name.
|
|
- Records a SHA-256 sidecar next to each dump, also mode `0600`.
|
|
- Deletes dumps and sidecars older than `BACKUP_RETENTION_DAYS`, and stale `.tmp` files older than
|
|
one day.
|
|
|
|
Configuration comes from two environment variables:
|
|
|
|
| Variable | Default | Behaviour |
|
|
| --- | --- | --- |
|
|
| `BACKUP_ROOT` | `/var/backups/wowngeon/daily` | The script refuses to run against any other value. Changing the destination requires a reviewed change to the script itself. |
|
|
| `BACKUP_RETENTION_DAYS` | `14` | Must be a positive integer. The service unit sets it explicitly to `14`. |
|
|
|
|
The script also exits if `BACKUP_ROOT` does not exist or is a symlink, so a misconfigured or
|
|
tampered path fails loudly instead of writing dumps somewhere unexpected.
|
|
|
|
Retention is deliberately short because this directory sits on the same disk as the database and
|
|
exists for fast local recovery. Long-horizon history is the job of an independent, separately
|
|
encrypted off-host archive, which the operator configures according to their recovery policy.
|
|
|
|
The service unit runs as `postgres` with `UMask=0077` and a hardened sandbox: `ProtectSystem=strict`
|
|
with `ReadWritePaths=/var/backups/wowngeon/daily` as the only writable location, an empty capability
|
|
bounding set, and `RestrictAddressFamilies=AF_UNIX` so the job can reach the local PostgreSQL socket
|
|
and nothing on the network. Because it authenticates as the `postgres` OS user over that socket, it
|
|
needs no database password and reads no application `.env` file.
|
|
|
|
The timer fires daily at 03:20 local time with a randomized delay of up to 20 minutes, and
|
|
`Persistent=true` so a run missed while the host was off happens at the next boot.
|
|
|
|
### Installation
|
|
|
|
Install these artifacts through the operator's reviewed change process rather than from the runtime
|
|
release directory, matching the boundary described in [DEPLOY.md](./DEPLOY.md). The service unit
|
|
expects the script at `/usr/local/sbin/wowngeon-db-backup`.
|
|
|
|
```bash
|
|
# Backup destination, owned by the database user only
|
|
sudo install -d -o postgres -g postgres -m 0700 /var/backups/wowngeon/daily
|
|
|
|
# Script and units
|
|
sudo install -o root -g root -m 0755 \
|
|
scripts/deploy/wowngeon-db-backup.sh /usr/local/sbin/wowngeon-db-backup
|
|
sudo install -o root -g root -m 0644 \
|
|
scripts/deploy/wowngeon-db-backup.service /etc/systemd/system/
|
|
sudo install -o root -g root -m 0644 \
|
|
scripts/deploy/wowngeon-db-backup.timer /etc/systemd/system/
|
|
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now wowngeon-db-backup.timer
|
|
|
|
# Confirm the schedule
|
|
systemctl list-timers wowngeon-db-backup.timer
|
|
```
|
|
|
|
### Running a Backup on Demand
|
|
|
|
```bash
|
|
sudo systemctl start wowngeon-db-backup.service
|
|
journalctl -u wowngeon-db-backup.service -n 50
|
|
ls -lh /var/backups/wowngeon/daily/
|
|
```
|
|
|
|
Dumps are named `<database>-<UTC timestamp>.dump`, for example
|
|
`wownerogue-20260726T032013Z.dump`.
|
|
|
|
### Verifying a Backup
|
|
|
|
Check integrity before trusting a dump:
|
|
|
|
```bash
|
|
cd /var/backups/wowngeon/daily
|
|
sudo -u postgres sha256sum -c wownerogue-20260726T032013Z.dump.sha256
|
|
sudo -u postgres pg_restore --list wownerogue-20260726T032013Z.dump | head
|
|
```
|
|
|
|
### Restoring
|
|
|
|
Custom-format dumps restore with `pg_restore`, not `psql`. Restore into a scratch database first
|
|
and promote only after inspection.
|
|
|
|
```bash
|
|
# Restore into a fresh database
|
|
sudo -u postgres createdb -O wownerogue wownerogue_restore
|
|
sudo -u postgres pg_restore --no-owner --exit-on-error \
|
|
--dbname=wownerogue_restore /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump
|
|
|
|
# Sanity-check the restored copy
|
|
sudo -u postgres psql -d wownerogue_restore -c "SELECT COUNT(*) FROM users;"
|
|
sudo -u postgres psql -d wownerogue_restore -c \
|
|
"SELECT filename FROM schema_migrations ORDER BY filename DESC LIMIT 1;"
|
|
|
|
# Discard the scratch copy when finished
|
|
sudo -u postgres dropdb wownerogue_restore
|
|
```
|
|
|
|
Restoring over a live database requires stopping the application unit first, because the server
|
|
runs migrations at startup and holds pooled connections:
|
|
|
|
```bash
|
|
sudo systemctl stop wownerogue
|
|
sudo -u postgres pg_restore --no-owner --clean --if-exists --exit-on-error \
|
|
--dbname=wownerogue /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump
|
|
sudo systemctl start wownerogue
|
|
```
|
|
|
|
Rehearse this path on a restored copy periodically. A backup that has never been restored is an
|
|
untested assumption. The disposable clone migration gate in
|
|
[CLONE_MIGRATIONS.md](./CLONE_MIGRATIONS.md) exercises restore plus the full migration set and is
|
|
the natural place to combine the two rehearsals.
|
|
|
|
---
|
|
|
|
## Simple Single-Database Alternative
|
|
|
|
`scripts/backup_db.sh` is a smaller portable script for operators who do not want the hardened
|
|
job: one database, gzipped plain SQL, credentials from the environment. Its companion units are
|
|
`scripts/wownerogue-backup.service` and `scripts/wownerogue-backup.timer` (daily at 03:00 with up
|
|
to 15 minutes of jitter).
|
|
|
|
| Variable | Default |
|
|
| --- | --- |
|
|
| `DB_NAME` | `wownerogue` |
|
|
| `DB_USER` | `postgres` |
|
|
| `DB_HOST` | `localhost` |
|
|
| `DB_PORT` | `5432` |
|
|
| `BACKUP_DIR` | `/var/backups/wownerogue` |
|
|
| `RETENTION_DAYS` | `30` |
|
|
|
|
The first five match the application's own database settings in `src/.env.example`, so the service
|
|
unit loads that file directly with `EnvironmentFile=`. Each run writes
|
|
`<DB_NAME>_<YYYYmmdd_HHMMSS>.sql.gz`, refreshes a `<DB_NAME>_LATEST.sql.gz` symlink, deletes a
|
|
partial file if `pg_dump` fails, and prunes dumps older than `RETENTION_DAYS`.
|
|
|
|
Because it connects over TCP as a database role, it needs working credentials, typically a
|
|
`~/.pgpass` entry or `PGPASSWORD` in the unit environment. Plain-SQL dumps restore with `psql`:
|
|
|
|
```bash
|
|
gunzip -c /var/backups/wownerogue/wownerogue_LATEST.sql.gz \
|
|
| psql -h "$DB_HOST" -U "$DB_USER" -d wownerogue_restore
|
|
```
|
|
|
|
A cron equivalent, for hosts without systemd timers:
|
|
|
|
```cron
|
|
0 3 * * * cd /var/www/wownerogue/src && set -a && . ./.env && set +a && /var/www/wownerogue/scripts/backup_db.sh >> /var/log/wownerogue-backup.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## What Database Backups Do Not Cover
|
|
|
|
A database dump alone does not restore a working instance. Back up these separately, with access
|
|
controls at least as strict as the dumps:
|
|
|
|
- **Wallet keyfiles and their passwords.** The wallet-RPC units keep wallet state under
|
|
`/var/lib/<instance>-wallet` (see `scripts/deploy/wownero-wallet-rpc.conf.example`). Losing these
|
|
loses the funds; the database records balances, never keys.
|
|
- **Instance secrets.** `/etc/wownerogue/app.env`, `/etc/wownerogue/wallet-rpc.env`, and the
|
|
equivalents for other instances. `src/.env.example` is the full annotated template for what these
|
|
contain.
|
|
- **Reverse proxy and TLS configuration**, which lives outside the application tree.
|
|
|
|
Keep at least one copy of everything above off the host that runs the database. Same-disk backups
|
|
protect against operator error, not against losing the machine.
|
|
|
|
---
|
|
|
|
## Monitoring Checklist
|
|
|
|
- Timer is active and scheduled: `systemctl list-timers wowngeon-db-backup.timer`
|
|
- Last run succeeded: `systemctl status wowngeon-db-backup.service`
|
|
- Recent dumps exist and are non-empty: `ls -lh /var/backups/wowngeon/daily/ | tail`
|
|
- Checksums verify: `sha256sum -c` on the newest sidecar
|
|
- Dump sizes track database growth rather than dropping sharply
|
|
- Backup volume has headroom: `df -h /var/backups`
|
|
- Off-host copies are current
|
|
- A restore rehearsal has happened recently enough to be meaningful
|