# Production Deployment Guide

This guide covers deploying Wownerogue to a production environment.

For evidence about the release currently serving `play.wowne.ro`, see the dated
[production-state record](PRODUCTION_STATE_2026-07-25.md). That record is not a substitute for this
runbook or authorization for another activation.

## Operated profiles versus independent deployments

Such Software (`apps@such.software`) operates only `play.wowne.ro` (Wownero mainnet
pay-for-credits leaderboard/prestige, no prize/payout/cash-out and not marketed as gambling) and
`monerogue.app` (Monero stagenet direct-entry-only **NO REAL VALUE** 2×/3× solo test gambling
mechanics, with purchased-credit entry and crypto-match payouts off).
Classification of any product under applicable law requires jurisdiction-specific advice.
Their reviewed environment templates set `OPERATED_PRODUCT_PROFILE`, so preflight/startup rejects
scope drift. Independent MIT deployments must leave that variable unset and identify their actual
operator.

MIT rights are subject to retaining the copyright and permission notice. The software is provided
“AS IS”, without warranty, and the code/docs are not legal advice or compliance approval. Each
third-party operator is solely responsible for its deployment, legal obligations, funds, players,
claims, and support; Such Software neither endorses nor accepts responsibility for it.

---

## Prerequisites

- Node.js 22.x LTS or later
- PostgreSQL 12+
- Wownero or Monero wallet-rpc running and synced
- A domain with SSL certificate (via reverse proxy)

---

## Deployment Checklist

- [ ] Set `NODE_ENV=production`
- [ ] Configure TLS via reverse proxy (Nginx/Caddy)
- [ ] Set secure database credentials
- [ ] Configure wallet-rpc with authentication
- [ ] Set `ADMIN_API_KEY` for admin endpoints
- [ ] Run `npm run preflight` with the final environment
- [ ] Keep `PAYOUTS_ENABLED=false` on real-money prestige instances
- [ ] Complete a stagenet payment and payout before enabling a payout instance
- [ ] Firewall: allow Node 3000/3001 only from the reverse proxy; keep wallet/daemon RPC loopback-only
- [ ] Protect `/admin.html` with basic auth
- [ ] Set up database backups (see [LOGS_AND_BACKUP.md](./LOGS_AND_BACKUP.md))
- [ ] Configure log rotation
- [ ] Run database migrations
- [ ] If 3D is offered, provision the gitignored runtime GLBs and verify library/model requests
      stay same-origin

---

## Create Dedicated User

Create an isolated, non-login service identity. Release directories and the `current` link remain
root-owned; the service identity receives read/traverse access only and must never own its code:

```bash
sudo useradd --system --shell /usr/sbin/nologin --home-dir /var/www/wownerogue --no-create-home wownerogue
sudo install -d -m 0750 -o root -g wownerogue /var/www/wownerogue
sudo install -d -m 0750 -o root -g wownerogue /var/www/wownerogue/releases
sudo install -d -m 0750 -o root -g wownerogue /etc/wownerogue

# Create log directory
sudo install -d -m 0750 -o wownerogue -g wownerogue /var/log/wownerogue
```

---

## Stage reviewed artifacts, not a repository clone

Build the runtime artifact from a clean, reviewed commit as described in
[RELEASE_ARTIFACT.md](./RELEASE_ARTIFACT.md). The production host needs no Git checkout, deploy key,
or write-capable source identity. The reviewed fleet
`playbooks/wowngeon-stage-candidate.yml` role transfers exactly one tarball to a newly selected
root-owned mode-0700 candidate inbox and compares its remote SHA-256 with the independently
recorded literal digest. Follow the fleet Wowngeon runbook; the role is a one-shot custody boundary,
not deployment.

The executable extraction/activation boundary lives only in the separately reviewed fleet
repository as the default-closed `wowngeon_release_activate` role. This application checkout cannot
attest which fleet revision is installed or authorize its execution. Follow the hash-pinned fleet
runbook and literal receipt contract; do not manually extract beneath `releases/`, install
dependencies there, change `current`, restart an application, or run migrations merely because a
blob has arrived.

---

## Install Node.js

Install Node.js LTS using NodeSource (run as root):

```bash
# Install Node.js 22.x LTS (or check https://nodejs.org for current LTS)
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -y nodejs

# Verify installation
node --version
npm --version
```

The reviewed fleet extraction/activation role installs exactly the locked production graph in its
private, writable staging directory with
`npm ci --omit=dev --ignore-scripts --no-audit --no-fund`, verifies it with
`npm ls --omit=dev`, and seals the finished release root-owned and non-writable before it can be
selected. Never make a release tree writable by `wownerogue`. Online advisory lookup is a separate
explicitly authorized source-review operation; never run `npm audit fix` in staging or on a live
release.

---

## Set File Permissions

```bash
# Releases and selector are immutable to the service identity.
sudo chown root:wownerogue /var/www/wownerogue /var/www/wownerogue/releases
sudo chmod 0750 /var/www/wownerogue /var/www/wownerogue/releases
sudo chmod 0640 /etc/wownerogue/app.env
```

Do not use recursive ownership changes on `/var/www/wownerogue`; they can hand the service process
write access to immutable code or the rollback selector. The reviewed fleet role must verify
every individual release's ownership and modes before activation.

**Database ownership** - the application currently runs its own migrations at startup, so its
database role must own the database/schema (or otherwise retain DDL rights). Create it this way:

```sql
-- Create role with minimal privileges
CREATE USER wownerogue WITH PASSWORD 'secure-password-here';
CREATE DATABASE wownerogue OWNER wownerogue;
```

Do not revoke schema `CREATE` from this role while application startup owns migration execution;
that makes a fresh release fail partway through startup. A separate one-shot migrator role is the
follow-up path if runtime DDL privileges need to be removed.

## Database Setup

### Run Migrations

Database migrations run automatically, in filename order, inside one transaction per file and are
recorded in `schema_migrations`. Do not apply migration files directly with `psql`: bypassing the
ledger makes startup attempt them again. Before a release, take a restricted PostgreSQL backup and
test the complete migration set against a restored copy of each production database using the
fail-closed [disposable clone migration gate](CLONE_MIGRATIONS.md).

The migration ledger does not prove that rows predating a `NOT VALID` constraint are clean. Run
the read-only historical audit and the rollback-only native validation proof described in
[`FINANCIAL_CONSTRAINT_VALIDATION.md`](FINANCIAL_CONSTRAINT_VALIDATION.md) for each restored
database. The native `VALIDATE CONSTRAINT` gate deliberately refuses live database names.

If the Wownero mainnet service will export accounting events, review and configure the durable
[financial-event outbox](FINANCIAL_EVENT_EXPORT.md) before cutover. Monero stagenet export must
remain unset: those no-value test events are marked ignored locally.

Install the reviewed `wowngeon-db-backup.sh`, `.service`, and `.timer` through the hash-pinned fleet
operations change—not from the runtime artifact—and create
`/var/backups/wowngeon/daily` as `postgres:postgres` mode `0700`, then enable the timer. Each run
writes custom-format dumps atomically, verifies their catalogs with `pg_restore --list`, and records
a SHA-256 sidecar. Configure retention/remote replication according to the operator's recovery
policy; the supplied job deliberately does not delete backups.

### Database Reset (Development Only)

```bash
npm run db:drop    # Drop all tables
npm run db:create  # Recreate schema
```

---

## systemd Service

Install reviewed application, wallet, backup, and firewall artifacts only through the hash-pinned
fleet operations change; the runtime artifact intentionally has no service units. Application
units read secrets from `/etc/wownerogue/app.env` or `/etc/monerogue/app.env` plus the narrow
wallet-RPC environment, never from an immutable release directory. They run `npm run preflight`
before every start and use `/var/www/<instance>/current/src` so rollback is an atomic symlink
switch.

For the two Such Software services, do not replace this boundary with manual `install`, `systemctl`,
or symlink commands. Follow the default-closed wallet candidate/promotion, candidate validation,
and one-instance activation procedures in [DEPLOY_INSTANCES.md](DEPLOY_INSTANCES.md) and the
hash-pinned `~/src/such-fleet/RUNBOOK-wowngeon.md`. Independent MIT operators must design and review
their own equivalent control plane.

The systemd hardening options:
- `NoNewPrivileges` - prevents privilege escalation
- `ProtectSystem=strict` - mounts filesystem read-only except allowed paths
- `ProtectHome=yes` - hides /home, /root, /run/user
- `PrivateTmp=yes` - isolates /tmp
- `ReadWritePaths` - whitelists the wallet state directory in wallet-RPC units

---

## Reverse Proxy (Nginx Proxy Manager)

If using Nginx Proxy Manager:

1. **Add Proxy Host**
   - Domain Names: `yourdomain.com`
   - Scheme: `http`
   - Forward Hostname/IP: Your server's LAN IP (e.g., `192.168.1.100`)
   - Forward Port: `3000`
   - Enable "Websockets Support" toggle

Record the proxy's fixed source address in `/etc/wowngeon/firewall.env` as
`NPM_SOURCE_IPV4=<address>`. Direct access to 3000/3001 must be rejected; this also makes the
single-hop `TRUST_PROXY_HOPS=1` setting safe.

2. **SSL Tab**
   - Request a new SSL certificate
   - Enable "Force SSL"

3. **Advanced Tab** - Paste this config for WebSocket support:

```nginx
location /socket.io/ {
    proxy_pass http://192.168.1.100:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}
```

---

## Updating the Deployment

Run the full suite from the clean source commit, then build and verify the runtime artifact using
[RELEASE_ARTIFACT.md](./RELEASE_ARTIFACT.md). Use the fleet blob-staging playbook, then proceed only
through the reviewed, hash-pinned `wowngeon_release_activate` role after its clone, wallet,
accounting, drain, predecessor, and receipt gates pass. That role—not an application service
identity or a manual shell session—builds a new immutable
`/var/www/<instance>/releases/<release-id>`, installs the locked graph, verifies the clone/preflight
evidence, seals ownership, and atomically selects `current`.
Never run `git pull`, `npm install`, or `npm audit fix` inside the active release. Keep the previous
release and database backup until the new version has passed public health, WebSocket,
payment-intake, and (stagenet only) payout smoke tests.

---

## Multi-Instance Deployment

To run multiple instances (e.g., Wownero on port 3000 and Monero stagenet on port 3001):

### Directory Structure

```
/var/www/
├── wownerogue/        # releases/<id> + current symlink (port 3000)
└── monerogue/         # releases/<id> + current symlink (port 3001)
/etc/
├── wownerogue/app.env # root:wownerogue, 0640
└── monerogue/app.env  # root:monerogue, 0640
```

### Configuration Differences

| Setting | Wownero Instance | Monero Instance |
|---------|------------------|-----------------|
| `PORT` | 3000 | 3001 |
| `CRYPTO_TYPE` | WOW | XMR |
| `MONERO_NETWORK` | mainnet | stagenet |
| `DB_NAME` | wownerogue | monerogue |
| `PRIMARY_WALLET_ENDPOINT` | http://127.0.0.1:34570 | http://127.0.0.1:38083 |
| `PRIMARY_RPC_ENDPOINT` | http://127.0.0.1:34568 | http://127.0.0.1:38081 |

### Daemon redundancy and graceful degradation

`RPC_ENDPOINTS` takes any number of daemons, comma or whitespace separated, in **preference order**:

```
RPC_ENDPOINTS=http://127.0.0.1:34568,http://10.42.1.40:34568,http://10.42.1.41:34568
```

`PRIMARY_RPC_ENDPOINT` / `FALLBACK_RPC_ENDPOINT` still work and are appended to the list. The list
is deduplicated — note that the stock configuration sets primary and fallback to the *same* host, so
by default there is exactly one node and no redundancy at all. Add at least one genuinely separate
daemon before relying on failover.

Behaviour:

- one logical RPC call tries **every** configured node before it fails;
- each node must pass the same chain-identity check before it may answer, so failover can never
  silently serve a different chain's data;
- after failing over, the preferred node is re-tested every `RPC_PREFERRED_RETRY_MS` (default 60s,
  minimum 1s), so recovery does not need a restart;
- when **no** node answers, the service reports unhealthy and throws. `getBlockCountStrict()` —
  used by fairness and match seeding — never substitutes a cached height.

**Entries are refused while no node is reachable.** A run's lifetime is counted in blocks and its
entry block comes from the last successful poll; admitting a game during an outage anchors it to an
already-past block and it is killed as a timeout on the first recovered poll. The player is told
`Can't reach a <CHAIN> node right now…` and nothing is charged. Credits are not consumed, because
the gate runs before any spend.

**The wallet is deliberately NOT failed over.** Two `wallet-rpc` processes serving one wallet file
can build transactions from the same outputs and corrupt the wallet cache — a standby wallet is a
custody decision for the operator, not something the app may do on its own. Daemons are read-only
and interchangeable; wallets are neither. When the wallet is unreachable the server declines paid
intake with `Payments are temporarily unavailable…`, and credits and free play keep working.

### Separate systemd Service

Install the hash-pinned fleet copy of `monerogue.service`; it is intentionally absent from the
runtime artifact. Do not hand-maintain a second, weaker unit. The reviewed template contains the
same preflight, graceful-stop, immutable-release path, and systemd sandbox as the mainnet service.

---

## Wallet Output Management

Wownero (and Monero) lock change outputs for several blocks after spending. If the wallet has only one large output and multiple payouts fire in quick succession, the second payout will fail because the change output from the first is still locked.

**Solution**: Pre-split wallet outputs into many smaller ones. The game server includes a CLI tool for this:

```bash
# Preview what would happen (no transaction sent)
node scripts/splitOutputs.js --amount 10 --count 30 --dry-run

# Split into 30 outputs of 10 WOW each (requires ~300 WOW + fees)
node scripts/splitOutputs.js --amount 10 --count 30
```

This creates 30 independently-spendable outputs, allowing up to 30 concurrent payouts before any output locking becomes an issue.

**When to re-split**: As payouts go out, outputs get consumed and consolidated via change. Periodically check the wallet and re-split when the number of spendable outputs drops low. The server also batches payouts that happen within 5 seconds of each other into a single transaction to conserve outputs.

**Recommended setup**: After initial wallet funding, run the split script. For a game with typical traffic, 20-30 outputs of 10 WOW each provides good concurrency headroom.

---

## Monitoring

### Check Service Status

```bash
systemctl status wownerogue
journalctl -u wownerogue -f  # Follow logs
```

### Health Endpoint

```bash
curl http://localhost:3000/health/live
curl --fail http://localhost:3000/health/ready
```

`/health/live` checks the process. `/health/ready` returns 503 until PostgreSQL, the chain daemon,
and (for paid instances) wallet RPC are ready. Public probes do not expose wallet balances or RPC
endpoints; those remain in the authenticated admin dashboard.

### Admin Dashboard

Access `/admin.html` with your `ADMIN_API_KEY` for:
- Wallet balance and connection status
- Pending/failed payouts
- Game statistics
- User search

---

## Troubleshooting

### Server Won't Start

```bash
# Check logs for errors
journalctl -u wownerogue -n 50

# Validate the protected environment without printing it
cd /var/www/wownerogue/current/src
sudo -u wownerogue bash -c 'set -a; . /etc/wownerogue/app.env; set +a; npm run preflight'

# Test database connection
psql -h localhost -U wownerogue -d wownerogue -c "SELECT 1;"
```

### Wallet RPC Issues

```bash
# Test wallet RPC directly
curl --digest -u user:password -X POST http://127.0.0.1:34570/json_rpc \
  -d '{"jsonrpc":"2.0","id":"0","method":"get_balance"}' \
  -H 'Content-Type: application/json'
```

### Database Migrations

If migrations fail, check the `schema_migrations` ledger:

```sql
SELECT filename, applied_at FROM schema_migrations ORDER BY filename;
```

Migrations are tracked by name to prevent duplicate execution.