jwinterm
ed24d43b56
Removes operator-private material from the shipped tree, corrects documentation
that had drifted from the code, deletes dead code, and applies a consistent
prose style.
Private material moved out of the public tree (docs/private/, git-ignored):
- the wallet keyfile recovery runbook and its helper scripts, which together
published the mainnet wallet file path, the service account, backup snapshot
filenames, and the wallet RPC listener state. The helper duplicated the same
constants, so relocating the document alone would not have removed them.
- the stagenet parallel-candidate runbook and its fixtures.
- the dated production-state record, the internal remediation plan, and the
TODO list. None are reference documentation for an outside reader.
Redacted or genericised:
- operator home-directory paths in shipped scripts and units. The asset build
script now takes KENNEY_ASSETS_DIR instead of a hard-coded path.
- private LAN addresses in tests, env examples and deploy docs, replaced with
RFC 5737 documentation addresses.
- the default alert sender and the notification hub example.
- the Monero wallet unit now matches the Wownero one: a dedicated service
account and a config file, instead of the operator's login account with the
wallet password in argv where any local user can read it from /proc.
Ignore rules now cover every .env variant rather than the bare name, plus local
agent state. The deployment docs instruct operators to create .env.mainnet and
.env.stagenet, which the exact-name rule would have allowed to be committed.
Documentation corrected against the code: per-chain dungeon level counts, the
premium unlock rule, Tavern chat scope, backup retention, the chain-profile
fallback, the environment template location, and several sections describing
shipped work as pending.
Dead code removed: configPersistence, rpccalls, a duplicate vendored ROT.js
bundle, the unused tile-picker overlay, the unreachable legacy game-size path
and its vestigial parameter, spectator stubs that shadowed the real
SpectatorManager, and commented-out debugging. nip98 now imports the shared
nostr loader rather than duplicating it.
Prose style: em-dashes removed throughout in favour of colons, semicolons and
commas; comments restated as present-tense descriptions of the code rather than
narration of past changes.
127 suites / 1015 tests pass.
5.2 KiB
Financial reconciliation
Payouts and refunds move real funds, so any row that is not in a terminal state is treated as a liability with an unknown on-chain outcome rather than a failed transfer that can be resent. This document describes the states, the retry rules that follow from them, and the manual reconciliation procedure.
Where to look
The admin dashboard (html/admin.html) has separate Payouts and Refunds tabs. Each
has a status filter whose Needs Attention option selects exactly the non-terminal states:
- Payouts:
pending,processing,failed,needs_review,permanently_failed. - Refunds:
requested,processing,needs_review.
The overview panel and each list also report per-status counts. Compare the row against the service logs and the wallet's outgoing-transfer history before taking any action.
Payout states
| State | Meaning |
|---|---|
pending |
Queued for the batch payout worker in gameModeManager. That worker owns these rows exclusively. |
processing |
Claimed by a worker. The wallet call may already have broadcast. |
completed |
Settled, with a transaction hash recorded. |
failed |
The wallet call reported an error and the row is below the retry limit. |
permanently_failed |
Retries reached PAYOUT_MAX_RETRIES (default 3). |
needs_review |
Quarantined for manual reconciliation. Never retried automatically. |
batched |
Legacy status from an earlier batching scheme. Current code does not write it; it remains accepted by the status constraint and is counted in dashboard totals. |
PayoutRetryService retries only failed rows below the retry limit, claiming each with
FOR UPDATE SKIP LOCKED so the batcher and the retry worker cannot both send the same row.
It also sweeps processing rows that have not advanced within its stale window and moves
them to needs_review with the reason recorded in last_error. The stale window defaults to
max(PAYOUT_RETRY_INTERVAL_MS * 3, 15 minutes).
Refund states
Refunds have their own durable outbox table, payment_refunds. Its state shape is enforced
by a database CHECK constraint, so a state and its timestamps or evidence cannot disagree.
| State | Meaning |
|---|---|
recorded |
Entitlements and payment state were reversed with no on-chain transfer requested. Terminal. |
requested |
Queued for a transfer. Not yet claimed. |
processing |
Claimed. The wallet call may already have broadcast. |
completed |
Settled. The constraint requires a non-empty tx_hash. |
needs_review |
Quarantined. The constraint requires a non-empty error_message. |
Only requested rows may be claimed. No code path moves processing back to requested,
so a crash or a lost wallet response can never produce an automatic second transfer.
Retry rules
- A payout in
processing,needs_review, or any state carrying a transaction hash must never be retried. Absence of a matching transfer in wallet history is not proof that no broadcast occurred. POST /api/admin/payouts/:id/retryaccepts only a hashlessfailedorpermanently_failedrow and requires an explicitconfirm(query parameter or JSON body); anything else returns 409 or 400. The dashboard exposes the Retry button only for those rows and sends the confirmation after a local prompt. The update predicate re-checks every safety condition, so a concurrent status change cannot be overwritten.- Use the retry action only after proving the recorded error happened before broadcast and confirming that wallet history contains no matching transfer.
- There is no resend action for any refund state, in the dashboard or the API.
Reconciliation checklist
- Preserve the database row, service logs, wallet files, and current backups before changing anything.
- Match currency and network, destination, exact atomic amount, creation and processing times, transaction hash if present, and wallet outgoing-transfer history.
- If a matching wallet transfer exists, record its hash and confirmation evidence in an audited remediation. Do not create a replacement transfer.
- If no transfer appears, treat that as inconclusive until the wallet is synchronized and the relevant time window and mempool have been checked. Escalate ambiguous cases instead of retrying.
- Re-run the read-only audit,
scripts/deploy/financial-audit.sql, after any reviewed database remediation and retain the before and after evidence with the incident record.
Alerts
AlertService runs periodic checks over the same aggregate states and sends two financial
alerts: one for any payout or refund in needs_review, and one for payout processing rows
or refund requested/processing rows that have not moved within FINANCIAL_REVIEW_STALE_MS
(default 900000, that is 15 minutes). Both resolve automatically once their count returns to
zero. Alerts are read-only: they never move, retry, or cancel a transfer.
Environment variables referenced here are documented in src/.env.example.
No generic resolve endpoint
There is deliberately no "mark resolved" endpoint. A reviewed state transition needs a case-specific, audited remediation because the correct result depends on wallet evidence and on whether value was actually transferred.