# Logs and Backup Operational reference for application logs and PostgreSQL backups in a production deployment. --- ## Application Logs The server has no logging library and writes no log files. All diagnostics go to stdout and stderr, so under systemd they land in the journal of the unit that started the process. Log retention is therefore a journald configuration question, not an application one. The repository ships two application units in `scripts/deploy/`: `wownerogue.service` (Wownero mainnet instance) and `monerogue.service` (Monero stagenet instance). ### Viewing Logs ```bash # All logs for a unit journalctl -u wownerogue # Follow live journalctl -u wownerogue -f # Last 100 lines journalctl -u wownerogue -n 100 # Since today journalctl -u wownerogue --since today # A specific time range journalctl -u wownerogue --since "2026-01-01" --until "2026-01-15" # Errors only journalctl -u wownerogue -p err # Export for offline analysis journalctl -u wownerogue --since "2026-01-01" > wownerogue-jan.log ``` The same commands work for `monerogue`, the wallet-RPC units, and the backup unit. ### Journal Retention Journald limits apply to the whole journal, not to an individual unit, so these settings bound every service on the host. Create a drop-in such as `/etc/systemd/journald.conf.d/limits.conf`: ```ini [Journal] # Total disk budget for all journals SystemMaxUse=500M # Size cap per journal file SystemMaxFileSize=50M # Discard entries older than this MaxRetentionSec=30day Compress=yes ``` Apply it: ```bash sudo mkdir -p /etc/systemd/journald.conf.d/ sudo install -m 0644 limits.conf /etc/systemd/journald.conf.d/limits.conf sudo systemctl restart systemd-journald ``` Journald is a rolling operational log, not an audit archive. Records that must survive log rotation belong in the database. Accounting events are exported through the durable outbox described in [FINANCIAL_EVENT_EXPORT.md](./FINANCIAL_EVENT_EXPORT.md). --- ## PostgreSQL Backups ### The Validated Backup Job `scripts/deploy/wowngeon-db-backup.sh`, with its `wowngeon-db-backup.service` and `wowngeon-db-backup.timer` units, is the production backup path. Each run: - Dumps every database named in the script's loop. The shipped list is `wownerogue` and `monerogue`; a single-instance operator edits that loop to match the databases actually present. - Writes custom-format dumps (`pg_dump --format=custom --no-owner`) to a temporary file, verifies the catalog with `pg_restore --list`, sets mode `0600`, then renames into place, so a reader never observes a partial dump under its final name. - Records a SHA-256 sidecar next to each dump, also mode `0600`. - Deletes dumps and sidecars older than `BACKUP_RETENTION_DAYS`, and stale `.tmp` files older than one day. Configuration comes from two environment variables: | Variable | Default | Behaviour | | --- | --- | --- | | `BACKUP_ROOT` | `/var/backups/wowngeon/daily` | The script refuses to run against any other value. Changing the destination requires a reviewed change to the script itself. | | `BACKUP_RETENTION_DAYS` | `14` | Must be a positive integer. The service unit sets it explicitly to `14`. | The script also exits if `BACKUP_ROOT` does not exist or is a symlink, so a misconfigured or tampered path fails loudly instead of writing dumps somewhere unexpected. Retention is deliberately short because this directory sits on the same disk as the database and exists for fast local recovery. Long-horizon history is the job of an independent, separately encrypted off-host archive, which the operator configures according to their recovery policy. The service unit runs as `postgres` with `UMask=0077` and a hardened sandbox: `ProtectSystem=strict` with `ReadWritePaths=/var/backups/wowngeon/daily` as the only writable location, an empty capability bounding set, and `RestrictAddressFamilies=AF_UNIX` so the job can reach the local PostgreSQL socket and nothing on the network. Because it authenticates as the `postgres` OS user over that socket, it needs no database password and reads no application `.env` file. The timer fires daily at 03:20 local time with a randomized delay of up to 20 minutes, and `Persistent=true` so a run missed while the host was off happens at the next boot. ### Installation Install these artifacts through the operator's reviewed change process rather than from the runtime release directory, matching the boundary described in [DEPLOY.md](./DEPLOY.md). The service unit expects the script at `/usr/local/sbin/wowngeon-db-backup`. ```bash # Backup destination, owned by the database user only sudo install -d -o postgres -g postgres -m 0700 /var/backups/wowngeon/daily # Script and units sudo install -o root -g root -m 0755 \ scripts/deploy/wowngeon-db-backup.sh /usr/local/sbin/wowngeon-db-backup sudo install -o root -g root -m 0644 \ scripts/deploy/wowngeon-db-backup.service /etc/systemd/system/ sudo install -o root -g root -m 0644 \ scripts/deploy/wowngeon-db-backup.timer /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now wowngeon-db-backup.timer # Confirm the schedule systemctl list-timers wowngeon-db-backup.timer ``` ### Running a Backup on Demand ```bash sudo systemctl start wowngeon-db-backup.service journalctl -u wowngeon-db-backup.service -n 50 ls -lh /var/backups/wowngeon/daily/ ``` Dumps are named `-.dump`, for example `wownerogue-20260726T032013Z.dump`. ### Verifying a Backup Check integrity before trusting a dump: ```bash cd /var/backups/wowngeon/daily sudo -u postgres sha256sum -c wownerogue-20260726T032013Z.dump.sha256 sudo -u postgres pg_restore --list wownerogue-20260726T032013Z.dump | head ``` ### Restoring Custom-format dumps restore with `pg_restore`, not `psql`. Restore into a scratch database first and promote only after inspection. ```bash # Restore into a fresh database sudo -u postgres createdb -O wownerogue wownerogue_restore sudo -u postgres pg_restore --no-owner --exit-on-error \ --dbname=wownerogue_restore /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump # Sanity-check the restored copy sudo -u postgres psql -d wownerogue_restore -c "SELECT COUNT(*) FROM users;" sudo -u postgres psql -d wownerogue_restore -c \ "SELECT filename FROM schema_migrations ORDER BY filename DESC LIMIT 1;" # Discard the scratch copy when finished sudo -u postgres dropdb wownerogue_restore ``` Restoring over a live database requires stopping the application unit first, because the server runs migrations at startup and holds pooled connections: ```bash sudo systemctl stop wownerogue sudo -u postgres pg_restore --no-owner --clean --if-exists --exit-on-error \ --dbname=wownerogue /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump sudo systemctl start wownerogue ``` Rehearse this path on a restored copy periodically. A backup that has never been restored is an untested assumption. The disposable clone migration gate in [CLONE_MIGRATIONS.md](./CLONE_MIGRATIONS.md) exercises restore plus the full migration set and is the natural place to combine the two rehearsals. --- ## Simple Single-Database Alternative `scripts/backup_db.sh` is a smaller portable script for operators who do not want the hardened job: one database, gzipped plain SQL, credentials from the environment. Its companion units are `scripts/wownerogue-backup.service` and `scripts/wownerogue-backup.timer` (daily at 03:00 with up to 15 minutes of jitter). | Variable | Default | | --- | --- | | `DB_NAME` | `wownerogue` | | `DB_USER` | `postgres` | | `DB_HOST` | `localhost` | | `DB_PORT` | `5432` | | `BACKUP_DIR` | `/var/backups/wownerogue` | | `RETENTION_DAYS` | `30` | The first five match the application's own database settings in `src/.env.example`, so the service unit loads that file directly with `EnvironmentFile=`. Each run writes `_.sql.gz`, refreshes a `_LATEST.sql.gz` symlink, deletes a partial file if `pg_dump` fails, and prunes dumps older than `RETENTION_DAYS`. Because it connects over TCP as a database role, it needs working credentials, typically a `~/.pgpass` entry or `PGPASSWORD` in the unit environment. Plain-SQL dumps restore with `psql`: ```bash gunzip -c /var/backups/wownerogue/wownerogue_LATEST.sql.gz \ | psql -h "$DB_HOST" -U "$DB_USER" -d wownerogue_restore ``` A cron equivalent, for hosts without systemd timers: ```cron 0 3 * * * cd /var/www/wownerogue/src && set -a && . ./.env && set +a && /var/www/wownerogue/scripts/backup_db.sh >> /var/log/wownerogue-backup.log 2>&1 ``` --- ## What Database Backups Do Not Cover A database dump alone does not restore a working instance. Back up these separately, with access controls at least as strict as the dumps: - **Wallet keyfiles and their passwords.** The wallet-RPC units keep wallet state under `/var/lib/-wallet` (see `scripts/deploy/wownero-wallet-rpc.conf.example`). Losing these loses the funds; the database records balances, never keys. - **Instance secrets.** `/etc/wownerogue/app.env`, `/etc/wownerogue/wallet-rpc.env`, and the equivalents for other instances. `src/.env.example` is the full annotated template for what these contain. - **Reverse proxy and TLS configuration**, which lives outside the application tree. Keep at least one copy of everything above off the host that runs the database. Same-disk backups protect against operator error, not against losing the machine. --- ## Monitoring Checklist - Timer is active and scheduled: `systemctl list-timers wowngeon-db-backup.timer` - Last run succeeded: `systemctl status wowngeon-db-backup.service` - Recent dumps exist and are non-empty: `ls -lh /var/backups/wowngeon/daily/ | tail` - Checksums verify: `sha256sum -c` on the newest sidecar - Dump sizes track database growth rather than dropping sharply - Backup volume has headroom: `df -h /var/backups` - Off-host copies are current - A restore rehearsal has happened recently enough to be meaningful