Files
wownerogue/docs/DEPLOY.md
T
jwinterm 2ff8d0be67 feat: multi-node daemon failover and graceful degradation when nothing answers
Closes the two findings left open from the pre-launch review.

Node redundancy (it was node reachability, not chain health)
  RPC_ENDPOINTS accepts any number of daemons in preference order; PRIMARY /
  FALLBACK still work and are appended. The list is deduplicated, which exposes
  that the stock config pointed both at the SAME host — what looked like
  redundancy was one node.

  One logical call now walks every endpoint before failing, instead of needing
  three consecutive failures to try exactly one alternate. Each node still has
  to pass the chain-identity check before it may answer, so failover cannot
  silently serve another chain's data. `failoverActive` is no longer a one-way
  latch: the preferred node is re-tested every RPC_PREFERRED_RETRY_MS, so
  recovery no longer requires a restart. healthCheck probes every node.

Graceful failure
  Entries are refused while no node is reachable. A run is measured in blocks
  and its entry block comes from the last successful poll, so a game admitted
  during an outage was anchored to an already-past block and killed as a timeout
  on the first recovered poll — the player lost a run, and in paid mode an
  entry, to our infrastructure. The gate sits in _runAdmission, ahead of any
  credit spend, and says so to the player.

Wallet: refused, not failed over
  Two wallet-rpc processes over one wallet file can build transactions from the
  same outputs and corrupt the cache. A standby wallet is a custody decision for
  the operator, not something this code may take on its own — daemons are
  read-only and interchangeable, wallets are neither. So a wallet outage
  declines paid intake cleanly and leaves credits and free play working.

Auth limiter evicts instead of refusing
  The bounded key map 429'd any NEW ip once full, turning a memory cap into a
  DoS primitive: a distributed source could hold it at capacity and lock every
  first-time caller out of Smirk login. It now reclaims expired entries and, if
  still full, evicts the oldest live windows. Same memory ceiling, still a
  working limiter.

129 suites / 1026 tests pass.
2026-07-26 07:49:18 -04:00

17 KiB
Raw Blame History

Production Deployment Guide

This guide covers deploying Wownerogue to a production environment.

For evidence about the release currently serving play.wowne.ro, see the dated production-state record. That record is not a substitute for this runbook or authorization for another activation.

Operated profiles versus independent deployments

Such Software (apps@such.software) operates only play.wowne.ro (Wownero mainnet pay-for-credits leaderboard/prestige, no prize/payout/cash-out and not marketed as gambling) and monerogue.app (Monero stagenet direct-entry-only NO REAL VALUE 2×/3× solo test gambling mechanics, with purchased-credit entry and crypto-match payouts off). Classification of any product under applicable law requires jurisdiction-specific advice. Their reviewed environment templates set OPERATED_PRODUCT_PROFILE, so preflight/startup rejects scope drift. Independent MIT deployments must leave that variable unset and identify their actual operator.

MIT rights are subject to retaining the copyright and permission notice. The software is provided “AS IS”, without warranty, and the code/docs are not legal advice or compliance approval. Each third-party operator is solely responsible for its deployment, legal obligations, funds, players, claims, and support; Such Software neither endorses nor accepts responsibility for it.


Prerequisites

  • Node.js 22.x LTS or later
  • PostgreSQL 12+
  • Wownero or Monero wallet-rpc running and synced
  • A domain with SSL certificate (via reverse proxy)

Deployment Checklist

  • Set NODE_ENV=production
  • Configure TLS via reverse proxy (Nginx/Caddy)
  • Set secure database credentials
  • Configure wallet-rpc with authentication
  • Set ADMIN_API_KEY for admin endpoints
  • Run npm run preflight with the final environment
  • Keep PAYOUTS_ENABLED=false on real-money prestige instances
  • Complete a stagenet payment and payout before enabling a payout instance
  • Firewall: allow Node 3000/3001 only from the reverse proxy; keep wallet/daemon RPC loopback-only
  • Protect /admin.html with basic auth
  • Set up database backups (see LOGS_AND_BACKUP.md)
  • Configure log rotation
  • Run database migrations
  • If 3D is offered, provision the gitignored runtime GLBs and verify library/model requests stay same-origin

Create Dedicated User

Create an isolated, non-login service identity. Release directories and the current link remain root-owned; the service identity receives read/traverse access only and must never own its code:

sudo useradd --system --shell /usr/sbin/nologin --home-dir /var/www/wownerogue --no-create-home wownerogue
sudo install -d -m 0750 -o root -g wownerogue /var/www/wownerogue
sudo install -d -m 0750 -o root -g wownerogue /var/www/wownerogue/releases
sudo install -d -m 0750 -o root -g wownerogue /etc/wownerogue

# Create log directory
sudo install -d -m 0750 -o wownerogue -g wownerogue /var/log/wownerogue

Stage reviewed artifacts, not a repository clone

Build the runtime artifact from a clean, reviewed commit as described in RELEASE_ARTIFACT.md. The production host needs no Git checkout, deploy key, or write-capable source identity. The reviewed fleet playbooks/wowngeon-stage-candidate.yml role transfers exactly one tarball to a newly selected root-owned mode-0700 candidate inbox and compares its remote SHA-256 with the independently recorded literal digest. Follow the fleet Wowngeon runbook; the role is a one-shot custody boundary, not deployment.

The executable extraction/activation boundary lives only in the separately reviewed fleet repository as the default-closed wowngeon_release_activate role. This application checkout cannot attest which fleet revision is installed or authorize its execution. Follow the hash-pinned fleet runbook and literal receipt contract; do not manually extract beneath releases/, install dependencies there, change current, restart an application, or run migrations merely because a blob has arrived.


Install Node.js

Install Node.js LTS using NodeSource (run as root):

# Install Node.js 22.x LTS (or check https://nodejs.org for current LTS)
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -y nodejs

# Verify installation
node --version
npm --version

The reviewed fleet extraction/activation role installs exactly the locked production graph in its private, writable staging directory with npm ci --omit=dev --ignore-scripts --no-audit --no-fund, verifies it with npm ls --omit=dev, and seals the finished release root-owned and non-writable before it can be selected. Never make a release tree writable by wownerogue. Online advisory lookup is a separate explicitly authorized source-review operation; never run npm audit fix in staging or on a live release.


Set File Permissions

# Releases and selector are immutable to the service identity.
sudo chown root:wownerogue /var/www/wownerogue /var/www/wownerogue/releases
sudo chmod 0750 /var/www/wownerogue /var/www/wownerogue/releases
sudo chmod 0640 /etc/wownerogue/app.env

Do not use recursive ownership changes on /var/www/wownerogue; they can hand the service process write access to immutable code or the rollback selector. The reviewed fleet role must verify every individual release's ownership and modes before activation.

Database ownership - the application currently runs its own migrations at startup, so its database role must own the database/schema (or otherwise retain DDL rights). Create it this way:

-- Create role with minimal privileges
CREATE USER wownerogue WITH PASSWORD 'secure-password-here';
CREATE DATABASE wownerogue OWNER wownerogue;

Do not revoke schema CREATE from this role while application startup owns migration execution; that makes a fresh release fail partway through startup. A separate one-shot migrator role is the follow-up path if runtime DDL privileges need to be removed.

Database Setup

Run Migrations

Database migrations run automatically, in filename order, inside one transaction per file and are recorded in schema_migrations. Do not apply migration files directly with psql: bypassing the ledger makes startup attempt them again. Before a release, take a restricted PostgreSQL backup and test the complete migration set against a restored copy of each production database using the fail-closed disposable clone migration gate.

The migration ledger does not prove that rows predating a NOT VALID constraint are clean. Run the read-only historical audit and the rollback-only native validation proof described in FINANCIAL_CONSTRAINT_VALIDATION.md for each restored database. The native VALIDATE CONSTRAINT gate deliberately refuses live database names.

If the Wownero mainnet service will export accounting events, review and configure the durable financial-event outbox before cutover. Monero stagenet export must remain unset: those no-value test events are marked ignored locally.

Install the reviewed wowngeon-db-backup.sh, .service, and .timer through the hash-pinned fleet operations change—not from the runtime artifact—and create /var/backups/wowngeon/daily as postgres:postgres mode 0700, then enable the timer. Each run writes custom-format dumps atomically, verifies their catalogs with pg_restore --list, and records a SHA-256 sidecar. Configure retention/remote replication according to the operator's recovery policy; the supplied job deliberately does not delete backups.

Database Reset (Development Only)

npm run db:drop    # Drop all tables
npm run db:create  # Recreate schema

systemd Service

Install reviewed application, wallet, backup, and firewall artifacts only through the hash-pinned fleet operations change; the runtime artifact intentionally has no service units. Application units read secrets from /etc/wownerogue/app.env or /etc/monerogue/app.env plus the narrow wallet-RPC environment, never from an immutable release directory. They run npm run preflight before every start and use /var/www/<instance>/current/src so rollback is an atomic symlink switch.

For the two Such Software services, do not replace this boundary with manual install, systemctl, or symlink commands. Follow the default-closed wallet candidate/promotion, candidate validation, and one-instance activation procedures in DEPLOY_INSTANCES.md and the hash-pinned ~/src/such-fleet/RUNBOOK-wowngeon.md. Independent MIT operators must design and review their own equivalent control plane.

The systemd hardening options:

  • NoNewPrivileges - prevents privilege escalation
  • ProtectSystem=strict - mounts filesystem read-only except allowed paths
  • ProtectHome=yes - hides /home, /root, /run/user
  • PrivateTmp=yes - isolates /tmp
  • ReadWritePaths - whitelists the wallet state directory in wallet-RPC units

Reverse Proxy (Nginx Proxy Manager)

If using Nginx Proxy Manager:

  1. Add Proxy Host
    • Domain Names: yourdomain.com
    • Scheme: http
    • Forward Hostname/IP: Your server's LAN IP (e.g., 192.168.1.100)
    • Forward Port: 3000
    • Enable "Websockets Support" toggle

Record the proxy's fixed source address in /etc/wowngeon/firewall.env as NPM_SOURCE_IPV4=<address>. Direct access to 3000/3001 must be rejected; this also makes the single-hop TRUST_PROXY_HOPS=1 setting safe.

  1. SSL Tab

    • Request a new SSL certificate
    • Enable "Force SSL"
  2. Advanced Tab - Paste this config for WebSocket support:

location /socket.io/ {
    proxy_pass http://192.168.1.100:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}

Updating the Deployment

Run the full suite from the clean source commit, then build and verify the runtime artifact using RELEASE_ARTIFACT.md. Use the fleet blob-staging playbook, then proceed only through the reviewed, hash-pinned wowngeon_release_activate role after its clone, wallet, accounting, drain, predecessor, and receipt gates pass. That role—not an application service identity or a manual shell session—builds a new immutable /var/www/<instance>/releases/<release-id>, installs the locked graph, verifies the clone/preflight evidence, seals ownership, and atomically selects current. Never run git pull, npm install, or npm audit fix inside the active release. Keep the previous release and database backup until the new version has passed public health, WebSocket, payment-intake, and (stagenet only) payout smoke tests.


Multi-Instance Deployment

To run multiple instances (e.g., Wownero on port 3000 and Monero stagenet on port 3001):

Directory Structure

/var/www/
├── wownerogue/        # releases/<id> + current symlink (port 3000)
└── monerogue/         # releases/<id> + current symlink (port 3001)
/etc/
├── wownerogue/app.env # root:wownerogue, 0640
└── monerogue/app.env  # root:monerogue, 0640

Configuration Differences

Setting Wownero Instance Monero Instance
PORT 3000 3001
CRYPTO_TYPE WOW XMR
MONERO_NETWORK mainnet stagenet
DB_NAME wownerogue monerogue
PRIMARY_WALLET_ENDPOINT http://127.0.0.1:34570 http://127.0.0.1:38083
PRIMARY_RPC_ENDPOINT http://127.0.0.1:34568 http://127.0.0.1:38081

Daemon redundancy and graceful degradation

RPC_ENDPOINTS takes any number of daemons, comma or whitespace separated, in preference order:

RPC_ENDPOINTS=http://127.0.0.1:34568,http://10.42.1.40:34568,http://10.42.1.41:34568

PRIMARY_RPC_ENDPOINT / FALLBACK_RPC_ENDPOINT still work and are appended to the list. The list is deduplicated — note that the stock configuration sets primary and fallback to the same host, so by default there is exactly one node and no redundancy at all. Add at least one genuinely separate daemon before relying on failover.

Behaviour:

  • one logical RPC call tries every configured node before it fails;
  • each node must pass the same chain-identity check before it may answer, so failover can never silently serve a different chain's data;
  • after failing over, the preferred node is re-tested every RPC_PREFERRED_RETRY_MS (default 60s, minimum 1s), so recovery does not need a restart;
  • when no node answers, the service reports unhealthy and throws. getBlockCountStrict() — used by fairness and match seeding — never substitutes a cached height.

Entries are refused while no node is reachable. A run's lifetime is counted in blocks and its entry block comes from the last successful poll; admitting a game during an outage anchors it to an already-past block and it is killed as a timeout on the first recovered poll. The player is told Can't reach a <CHAIN> node right now… and nothing is charged. Credits are not consumed, because the gate runs before any spend.

The wallet is deliberately NOT failed over. Two wallet-rpc processes serving one wallet file can build transactions from the same outputs and corrupt the wallet cache — a standby wallet is a custody decision for the operator, not something the app may do on its own. Daemons are read-only and interchangeable; wallets are neither. When the wallet is unreachable the server declines paid intake with Payments are temporarily unavailable…, and credits and free play keep working.

Separate systemd Service

Install the hash-pinned fleet copy of monerogue.service; it is intentionally absent from the runtime artifact. Do not hand-maintain a second, weaker unit. The reviewed template contains the same preflight, graceful-stop, immutable-release path, and systemd sandbox as the mainnet service.


Wallet Output Management

Wownero (and Monero) lock change outputs for several blocks after spending. If the wallet has only one large output and multiple payouts fire in quick succession, the second payout will fail because the change output from the first is still locked.

Solution: Pre-split wallet outputs into many smaller ones. The game server includes a CLI tool for this:

# Preview what would happen (no transaction sent)
node scripts/splitOutputs.js --amount 10 --count 30 --dry-run

# Split into 30 outputs of 10 WOW each (requires ~300 WOW + fees)
node scripts/splitOutputs.js --amount 10 --count 30

This creates 30 independently-spendable outputs, allowing up to 30 concurrent payouts before any output locking becomes an issue.

When to re-split: As payouts go out, outputs get consumed and consolidated via change. Periodically check the wallet and re-split when the number of spendable outputs drops low. The server also batches payouts that happen within 5 seconds of each other into a single transaction to conserve outputs.

Recommended setup: After initial wallet funding, run the split script. For a game with typical traffic, 20-30 outputs of 10 WOW each provides good concurrency headroom.


Monitoring

Check Service Status

systemctl status wownerogue
journalctl -u wownerogue -f  # Follow logs

Health Endpoint

curl http://localhost:3000/health/live
curl --fail http://localhost:3000/health/ready

/health/live checks the process. /health/ready returns 503 until PostgreSQL, the chain daemon, and (for paid instances) wallet RPC are ready. Public probes do not expose wallet balances or RPC endpoints; those remain in the authenticated admin dashboard.

Admin Dashboard

Access /admin.html with your ADMIN_API_KEY for:

  • Wallet balance and connection status
  • Pending/failed payouts
  • Game statistics
  • User search

Troubleshooting

Server Won't Start

# Check logs for errors
journalctl -u wownerogue -n 50

# Validate the protected environment without printing it
cd /var/www/wownerogue/current/src
sudo -u wownerogue bash -c 'set -a; . /etc/wownerogue/app.env; set +a; npm run preflight'

# Test database connection
psql -h localhost -U wownerogue -d wownerogue -c "SELECT 1;"

Wallet RPC Issues

# Test wallet RPC directly
curl --digest -u user:password -X POST http://127.0.0.1:34570/json_rpc \
  -d '{"jsonrpc":"2.0","id":"0","method":"get_balance"}' \
  -H 'Content-Type: application/json'

Database Migrations

If migrations fail, check the schema_migrations ledger:

SELECT filename, applied_at FROM schema_migrations ORDER BY filename;

Migrations are tracked by name to prevent duplicate execution.