feat: multi-node daemon failover and graceful degradation when nothing answers

Closes the two findings left open from the pre-launch review.

Node redundancy (it was node reachability, not chain health)
  RPC_ENDPOINTS accepts any number of daemons in preference order; PRIMARY /
  FALLBACK still work and are appended. The list is deduplicated, which exposes
  that the stock config pointed both at the SAME host — what looked like
  redundancy was one node.

  One logical call now walks every endpoint before failing, instead of needing
  three consecutive failures to try exactly one alternate. Each node still has
  to pass the chain-identity check before it may answer, so failover cannot
  silently serve another chain's data. `failoverActive` is no longer a one-way
  latch: the preferred node is re-tested every RPC_PREFERRED_RETRY_MS, so
  recovery no longer requires a restart. healthCheck probes every node.

Graceful failure
  Entries are refused while no node is reachable. A run is measured in blocks
  and its entry block comes from the last successful poll, so a game admitted
  during an outage was anchored to an already-past block and killed as a timeout
  on the first recovered poll — the player lost a run, and in paid mode an
  entry, to our infrastructure. The gate sits in _runAdmission, ahead of any
  credit spend, and says so to the player.

Wallet: refused, not failed over
  Two wallet-rpc processes over one wallet file can build transactions from the
  same outputs and corrupt the cache. A standby wallet is a custody decision for
  the operator, not something this code may take on its own — daemons are
  read-only and interchangeable, wallets are neither. So a wallet outage
  declines paid intake cleanly and leaves credits and free play working.

Auth limiter evicts instead of refusing
  The bounded key map 429'd any NEW ip once full, turning a memory cap into a
  DoS primitive: a distributed source could hold it at capacity and lock every
  first-time caller out of Smirk login. It now reclaims expired entries and, if
  still full, evicts the oldest live windows. Same memory ceiling, still a
  working limiter.

129 suites / 1026 tests pass.
This commit is contained in:
jwinterm
2026-07-26 07:49:18 -04:00
parent dacbf356ed
commit 2ff8d0be67
7 changed files with 630 additions and 105 deletions
+33 -8
View File
@@ -30,25 +30,50 @@ function expectedAuthHost(req) {
return (req && (req.get('host') || req.headers?.host)) || null;
}
/**
* Per-IP auth rate limiter with a bounded key map.
*
* The bound must be enforced by EVICTION, never by refusing new keys. Refusing turned the memory
* cap into a denial-of-service primitive: an attacker sourcing from many addresses could hold the
* map at capacity and every legitimate first-time caller of /challenge or /verify got a 429, locking
* the whole population out of Smirk login. Eviction keeps the same memory ceiling while leaving the
* limiter functional — the worst case is that a very old window's count is forgotten, which grants
* one extra window to an attacker who was already being counted.
*/
function createIpRateLimiter({ max }) {
const hits = new Map();
let lastSweep = 0;
const sweepExpired = (now) => {
for (const [key, value] of hits) {
if (now - value.windowStart >= AUTH_WINDOW_MS) hits.delete(key);
}
lastSweep = now;
};
// Discard the oldest live windows in one batch, so the O(n) pass is amortised rather than paid
// on every request once the map is full.
const evictOldest = (count) => {
const oldest = Array.from(hits.entries())
.sort((left, right) => left[1].windowStart - right[1].windowStart)
.slice(0, count);
for (const [key] of oldest) hits.delete(key);
};
return (req, res, next) => {
const now = Date.now();
if (now - lastSweep >= AUTH_WINDOW_MS) {
for (const [key, value] of hits) {
if (now - value.windowStart >= AUTH_WINDOW_MS) hits.delete(key);
}
lastSweep = now;
}
if (now - lastSweep >= AUTH_WINDOW_MS) sweepExpired(now);
const ip = req.ip || req.socket?.remoteAddress || 'unknown';
let entry = hits.get(ip);
if (!entry || now - entry.windowStart >= AUTH_WINDOW_MS) {
if (!entry && hits.size >= AUTH_RATE_MAX_KEYS) {
res.setHeader('Retry-After', '60');
return res.status(429).json({ error: 'Too many authentication attempts' });
// Sweeping is otherwise lazy (once per window), so a map full of expired entries would
// stay full. Reclaim first, and only evict live windows if that was not enough.
sweepExpired(now);
if (hits.size >= AUTH_RATE_MAX_KEYS) {
evictOldest(Math.max(1, Math.ceil(AUTH_RATE_MAX_KEYS * 0.1)));
}
}
entry = { count: 0, windowStart: now };
hits.set(ip, entry);