wownerogue/docs/FINANCIAL_RECONCILIATION.md

# Financial reconciliation

Payouts and refunds move real funds, so any row that is not in a terminal state is treated
as a liability with an unknown on-chain outcome rather than a failed transfer that can be
resent. This document describes the states, the retry rules that follow from them, and the
manual reconciliation procedure.

## Where to look

The admin dashboard (`html/admin.html`) has separate **Payouts** and **Refunds** tabs. Each
has a status filter whose **Needs Attention** option selects exactly the non-terminal states:

- Payouts: `pending`, `processing`, `failed`, `needs_review`, `permanently_failed`.
- Refunds: `requested`, `processing`, `needs_review`.

The overview panel and each list also report per-status counts. Compare the row against the
service logs and the wallet's outgoing-transfer history before taking any action.

## Payout states

| State | Meaning |
| --- | --- |
| `pending` | Queued for the batch payout worker in `gameModeManager`. That worker owns these rows exclusively. |
| `processing` | Claimed by a worker. The wallet call may already have broadcast. |
| `completed` | Settled, with a transaction hash recorded. |
| `failed` | The wallet call reported an error and the row is below the retry limit. |
| `permanently_failed` | Retries reached `PAYOUT_MAX_RETRIES` (default 3). |
| `needs_review` | Quarantined for manual reconciliation. Never retried automatically. |
| `batched` | Legacy status from an earlier batching scheme. Current code does not write it; it remains accepted by the status constraint and is counted in dashboard totals. |

`PayoutRetryService` retries only `failed` rows below the retry limit, claiming each with
`FOR UPDATE SKIP LOCKED` so the batcher and the retry worker cannot both send the same row.
It also sweeps `processing` rows that have not advanced within its stale window and moves
them to `needs_review` with the reason recorded in `last_error`. The stale window defaults to
`max(PAYOUT_RETRY_INTERVAL_MS * 3, 15 minutes)`.

## Refund states

Refunds have their own durable outbox table, `payment_refunds`. Its state shape is enforced
by a database CHECK constraint, so a state and its timestamps or evidence cannot disagree.

| State | Meaning |
| --- | --- |
| `recorded` | Entitlements and payment state were reversed with no on-chain transfer requested. Terminal. |
| `requested` | Queued for a transfer. Not yet claimed. |
| `processing` | Claimed. The wallet call may already have broadcast. |
| `completed` | Settled. The constraint requires a non-empty `tx_hash`. |
| `needs_review` | Quarantined. The constraint requires a non-empty `error_message`. |

Only `requested` rows may be claimed. No code path moves `processing` back to `requested`,
so a crash or a lost wallet response can never produce an automatic second transfer.

## Retry rules

- A payout in `processing`, `needs_review`, or any state carrying a transaction hash must
  never be retried. Absence of a matching transfer in wallet history is not proof that no
  broadcast occurred.
- `POST /api/admin/payouts/:id/retry` accepts only a hashless `failed` or
  `permanently_failed` row and requires an explicit `confirm` (query parameter or JSON body);
  anything else returns 409 or 400. The dashboard exposes the Retry button only for those
  rows and sends the confirmation after a local prompt. The update predicate re-checks every
  safety condition, so a concurrent status change cannot be overwritten.
- Use the retry action only after proving the recorded error happened before broadcast and
  confirming that wallet history contains no matching transfer.
- There is no resend action for any refund state, in the dashboard or the API.

## Reconciliation checklist

1. Preserve the database row, service logs, wallet files, and current backups before
   changing anything.
2. Match currency and network, destination, exact atomic amount, creation and processing
   times, transaction hash if present, and wallet outgoing-transfer history.
3. If a matching wallet transfer exists, record its hash and confirmation evidence in an
   audited remediation. Do not create a replacement transfer.
4. If no transfer appears, treat that as inconclusive until the wallet is synchronized and
   the relevant time window and mempool have been checked. Escalate ambiguous cases instead
   of retrying.
5. Re-run the read-only audit, `scripts/deploy/financial-audit.sql`, after any reviewed
   database remediation and retain the before and after evidence with the incident record.

## Alerts

`AlertService` runs periodic checks over the same aggregate states and sends two financial
alerts: one for any payout or refund in `needs_review`, and one for payout `processing` rows
or refund `requested`/`processing` rows that have not moved within `FINANCIAL_REVIEW_STALE_MS`
(default 900000, that is 15 minutes). Both resolve automatically once their count returns to
zero. Alerts are read-only: they never move, retry, or cancel a transfer.

Environment variables referenced here are documented in `src/.env.example`.

## No generic resolve endpoint

There is deliberately no "mark resolved" endpoint. A reviewed state transition needs a
case-specific, audited remediation because the correct result depends on wallet evidence and
on whether value was actually transferred.