wownerogue/docs/LOGS_AND_BACKUP.md

Logs and Backup

Operational reference for application logs and PostgreSQL backups in a production deployment.

---

Application Logs

The server has no logging library and writes no log files. All diagnostics go to stdout and stderr, so under systemd they land in the journal of the unit that started the process. Log retention is therefore a journald configuration question, not an application one.

The repository ships two application units in scripts/deploy/: wownerogue.service (Wownero mainnet instance) and monerogue.service (Monero stagenet instance).

Viewing Logs

# All logs for a unit
journalctl -u wownerogue

# Follow live
journalctl -u wownerogue -f

# Last 100 lines
journalctl -u wownerogue -n 100

# Since today
journalctl -u wownerogue --since today

# A specific time range
journalctl -u wownerogue --since "2026-01-01" --until "2026-01-15"

# Errors only
journalctl -u wownerogue -p err

# Export for offline analysis
journalctl -u wownerogue --since "2026-01-01" > wownerogue-jan.log

The same commands work for monerogue, the wallet-RPC units, and the backup unit.

Journal Retention

Journald limits apply to the whole journal, not to an individual unit, so these settings bound every service on the host. Create a drop-in such as /etc/systemd/journald.conf.d/limits.conf:

[Journal]
# Total disk budget for all journals
SystemMaxUse=500M
# Size cap per journal file
SystemMaxFileSize=50M
# Discard entries older than this
MaxRetentionSec=30day
Compress=yes

Apply it:

sudo mkdir -p /etc/systemd/journald.conf.d/
sudo install -m 0644 limits.conf /etc/systemd/journald.conf.d/limits.conf
sudo systemctl restart systemd-journald

Journald is a rolling operational log, not an audit archive. Records that must survive log rotation belong in the database. Accounting events are exported through the durable outbox described in FINANCIAL_EVENT_EXPORT.md.

---

PostgreSQL Backups

The Validated Backup Job

scripts/deploy/wowngeon-db-backup.sh, with its wowngeon-db-backup.service and wowngeon-db-backup.timer units, is the production backup path. Each run:

• Dumps every database named in the script's loop. The shipped list is wownerogue and monerogue; a single-instance operator edits that loop to match the databases actually present.
• Writes custom-format dumps (pg_dump --format=custom --no-owner) to a temporary file, verifies the catalog with pg_restore --list, sets mode 0600, then renames into place, so a reader never observes a partial dump under its final name.
• Records a SHA-256 sidecar next to each dump, also mode 0600.
• Deletes dumps and sidecars older than BACKUP_RETENTION_DAYS, and stale .tmp files older than one day.

Configuration comes from two environment variables:

Variable	Default	Behaviour
BACKUP_ROOT	/var/backups/wowngeon/daily	The script refuses to run against any other value. Changing the destination requires a reviewed change to the script itself.
BACKUP_RETENTION_DAYS	14	Must be a positive integer. The service unit sets it explicitly to 14.

The script also exits if BACKUP_ROOT does not exist or is a symlink, so a misconfigured or tampered path fails loudly instead of writing dumps somewhere unexpected.

Retention is deliberately short because this directory sits on the same disk as the database and exists for fast local recovery. Long-horizon history is the job of an independent, separately encrypted off-host archive, which the operator configures according to their recovery policy.

The service unit runs as postgres with UMask=0077 and a hardened sandbox: ProtectSystem=strict with ReadWritePaths=/var/backups/wowngeon/daily as the only writable location, an empty capability bounding set, and RestrictAddressFamilies=AF_UNIX so the job can reach the local PostgreSQL socket and nothing on the network. Because it authenticates as the postgres OS user over that socket, it needs no database password and reads no application .env file.

The timer fires daily at 03:20 local time with a randomized delay of up to 20 minutes, and Persistent=true so a run missed while the host was off happens at the next boot.

Installation

Install these artifacts through the operator's reviewed change process rather than from the runtime release directory, matching the boundary described in DEPLOY.md. The service unit expects the script at /usr/local/sbin/wowngeon-db-backup.

# Backup destination, owned by the database user only
sudo install -d -o postgres -g postgres -m 0700 /var/backups/wowngeon/daily

# Script and units
sudo install -o root -g root -m 0755 \
    scripts/deploy/wowngeon-db-backup.sh /usr/local/sbin/wowngeon-db-backup
sudo install -o root -g root -m 0644 \
    scripts/deploy/wowngeon-db-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 \
    scripts/deploy/wowngeon-db-backup.timer /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl enable --now wowngeon-db-backup.timer

# Confirm the schedule
systemctl list-timers wowngeon-db-backup.timer

Running a Backup on Demand

sudo systemctl start wowngeon-db-backup.service
journalctl -u wowngeon-db-backup.service -n 50
ls -lh /var/backups/wowngeon/daily/

Dumps are named <database>-<UTC timestamp>.dump, for example wownerogue-20260726T032013Z.dump.

Verifying a Backup

Check integrity before trusting a dump:

cd /var/backups/wowngeon/daily
sudo -u postgres sha256sum -c wownerogue-20260726T032013Z.dump.sha256
sudo -u postgres pg_restore --list wownerogue-20260726T032013Z.dump | head

Restoring

Custom-format dumps restore with pg_restore, not psql. Restore into a scratch database first and promote only after inspection.

# Restore into a fresh database
sudo -u postgres createdb -O wownerogue wownerogue_restore
sudo -u postgres pg_restore --no-owner --exit-on-error \
    --dbname=wownerogue_restore /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump

# Sanity-check the restored copy
sudo -u postgres psql -d wownerogue_restore -c "SELECT COUNT(*) FROM users;"
sudo -u postgres psql -d wownerogue_restore -c \
    "SELECT filename FROM schema_migrations ORDER BY filename DESC LIMIT 1;"

# Discard the scratch copy when finished
sudo -u postgres dropdb wownerogue_restore

Restoring over a live database requires stopping the application unit first, because the server runs migrations at startup and holds pooled connections:

sudo systemctl stop wownerogue
sudo -u postgres pg_restore --no-owner --clean --if-exists --exit-on-error \
    --dbname=wownerogue /var/backups/wowngeon/daily/wownerogue-20260726T032013Z.dump
sudo systemctl start wownerogue

Rehearse this path on a restored copy periodically. A backup that has never been restored is an untested assumption. The disposable clone migration gate in CLONE_MIGRATIONS.md exercises restore plus the full migration set and is the natural place to combine the two rehearsals.

---

Simple Single-Database Alternative

scripts/backup_db.sh is a smaller portable script for operators who do not want the hardened job: one database, gzipped plain SQL, credentials from the environment. Its companion units are scripts/wownerogue-backup.service and scripts/wownerogue-backup.timer (daily at 03:00 with up to 15 minutes of jitter).

Variable	Default
DB_NAME	wownerogue
DB_USER	postgres
DB_HOST	localhost
DB_PORT	5432
BACKUP_DIR	/var/backups/wownerogue
RETENTION_DAYS	30

The first five match the application's own database settings in src/.env.example, so the service unit loads that file directly with EnvironmentFile=. Each run writes <DB_NAME>_<YYYYmmdd_HHMMSS>.sql.gz, refreshes a <DB_NAME>_LATEST.sql.gz symlink, deletes a partial file if pg_dump fails, and prunes dumps older than RETENTION_DAYS.

Because it connects over TCP as a database role, it needs working credentials, typically a ~/.pgpass entry or PGPASSWORD in the unit environment. Plain-SQL dumps restore with psql:

gunzip -c /var/backups/wownerogue/wownerogue_LATEST.sql.gz \
    | psql -h "$DB_HOST" -U "$DB_USER" -d wownerogue_restore

A cron equivalent, for hosts without systemd timers:

0 3 * * * cd /var/www/wownerogue/src && set -a && . ./.env && set +a && /var/www/wownerogue/scripts/backup_db.sh >> /var/log/wownerogue-backup.log 2>&1

---

What Database Backups Do Not Cover

A database dump alone does not restore a working instance. Back up these separately, with access controls at least as strict as the dumps:

• Wallet keyfiles and their passwords. The wallet-RPC units keep wallet state under /var/lib/<instance>-wallet (see scripts/deploy/wownero-wallet-rpc.conf.example). Losing these loses the funds; the database records balances, never keys.
• Instance secrets. /etc/wownerogue/app.env, /etc/wownerogue/wallet-rpc.env, and the equivalents for other instances. src/.env.example is the full annotated template for what these contain.
• Reverse proxy and TLS configuration, which lives outside the application tree.

Keep at least one copy of everything above off the host that runs the database. Same-disk backups protect against operator error, not against losing the machine.

---

Monitoring Checklist

• Timer is active and scheduled: systemctl list-timers wowngeon-db-backup.timer
• Last run succeeded: systemctl status wowngeon-db-backup.service
• Recent dumps exist and are non-empty: ls -lh /var/backups/wowngeon/daily/ | tail
• Checksums verify: sha256sum -c on the newest sidecar
• Dump sizes track database growth rather than dropping sharply
• Backup volume has headroom: df -h /var/backups
• Off-host copies are current
• A restore rehearsal has happened recently enough to be meaningful