Fail-Open vs Fail-Closed Middleware: Hono + Redis (2026)

Fail-open lets a request through when a dependency is down; fail-closed rejects it. Rate limits and feature flags should fail open so a Redis blip doesn't 503 the API. Authentication MUST fail closed — allowing a request you couldn't authorize is a security bypass.

TL;DR

You'll build a Hono 4.12.23 + ioredis 5.11.0 service with three middleware: a fail-closed API-key check (rejects when Redis is down — preventing an auth bypass), a fail-open feature-flag check with an explicit safe default (lets traffic through), and a fail-open sliding-window rate limiter (allows requests rather than 503-ing the API). All three share a typed CircuitBreaker class (~40 lines) so a sustained outage stops hammering the dead dependency. End-to-end verified in the sandbox with Redis unreachable: auth returns 503 auth_unavailable while flag + rate-limit silently fall back, exactly as designed.

What you'll learn

• The fail-open vs fail-closed decision per middleware type
• How to write a small typed circuit breaker in TypeScript (no library needed)
• How to add per-call deadlines on top of ioredis's commandTimeout
• Why enableOfflineQueue: false is the right ioredis default for resilient apps
• How to compose three middleware in Hono so the policies don't fight each other
• Verification with curl against an unreachable Redis so you can see each policy fire

Prerequisites

• Node.js 24 (Active LTS as of May 2026, supported through April 2028) — or Node 22 if you're still on Maintenance LTS. Verified in the sandbox on Node 22.22.0.
• Docker (any recent version) — used to run Redis locally for the happy-path test
• npm 10+ (Node 22 ships npm 10; Node 24 ships npm 11)
• A scratch directory; we'll call it failover-demo/

No managed Redis account is needed — a local redis:8-alpine container is fine for the verification step. If you already have Upstash or Redis Cloud, set REDIS_URL accordingly.

Step 1: scaffold the project

mkdir failover-demo && cd failover-demo
cat > package.json <<'EOF'
{ "name": "failover-demo", "version": "0.0.0", "private": true, "type": "module" }
EOF
npm install --save-exact hono@4.12.23 @hono/node-server@2.0.4 ioredis@5.11.0
npm install --save-exact --save-dev typescript@6.0.3 tsx@4.22.3 @types/node@25.9.1
mkdir -p src/middleware src/lib

The --save-exact flag pins every dependency to the verified patch — without it npm writes ^X.Y.Z and the next npm install may resolve a higher patch (lesson from a prior tutorial that bit the same trap).¹

Create tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "lib": ["ES2023"],
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "verbatimModuleSyntax": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "resolveJsonModule": true,
    "noEmit": true,
    "isolatedModules": true,
    "allowImportingTsExtensions": true
  },
  "include": ["src/**/*.ts"]
}

The allowImportingTsExtensions: true is required because we'll write import paths with explicit .ts extensions — tsx resolves them at runtime and the compiler needs permission to see them.

Step 2: build a typed circuit breaker

A circuit breaker has three states: CLOSED (calls flow through), OPEN (every call fast-fails without touching the dependency), and HALF_OPEN (a single probe is allowed to test recovery). When the dependency is healthy the breaker stays CLOSED. When failures cross a threshold it OPENS, giving the downstream a chance to recover. After a reset window it goes HALF_OPEN and the next call decides which way to go.²

You can pull in opossum for a battle-tested implementation,³ but for a tutorial that wants the failure model visible in code review, ~40 lines of TypeScript is clearer. Write src/lib/circuit-breaker.ts:

export type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

export class CircuitOpenError extends Error {
  readonly code = 'CIRCUIT_OPEN' as const;
  constructor(name: string) {
    super(`${name}: circuit is OPEN`);
    this.name = 'CircuitOpenError';
  }
}

export interface CircuitBreakerOptions {
  /** Open the circuit after this many consecutive failures. */
  failureThreshold: number;
  /** Keep the circuit OPEN for this long before allowing a probe. */
  resetMs: number;
  /** Human-readable name for logs/errors. */
  name: string;
}

export class CircuitBreaker {
  private state: CircuitState = 'CLOSED';
  private failures = 0;
  private openedAt = 0;

  constructor(private readonly opts: CircuitBreakerOptions) {}

  getState(): CircuitState {
    if (this.state === 'OPEN' && Date.now() - this.openedAt >= this.opts.resetMs) {
      this.state = 'HALF_OPEN';
    }
    return this.state;
  }

  async fire<T>(fn: () => Promise<T>): Promise<T> {
    const state = this.getState();
    if (state === 'OPEN') {
      throw new CircuitOpenError(this.opts.name);
    }
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  private onSuccess(): void {
    this.failures = 0;
    this.state = 'CLOSED';
  }

  private onFailure(): void {
    this.failures += 1;
    if (this.state === 'HALF_OPEN' || this.failures >= this.opts.failureThreshold) {
      this.state = 'OPEN';
      this.openedAt = Date.now();
    }
  }
}

Two design choices worth flagging. First, this counts consecutive failures rather than a percentage of failures in a rolling window — that's simpler to reason about and good enough until you start seeing flapping. If you start seeing noisy near-threshold behavior in production, that's the signal to switch to opossum's rolling-bucket model. Second, the state check happens inside fire() rather than a separate scheduler — there's no timer to clean up, and the half-open probe just IS the next call after the reset window expires.

Step 3: add per-call deadlines

ioredis ships a commandTimeout option that rejects a single Redis request on the client side after N ms (the server may still execute the command — the timeout is purely a client-side promise rejection). That's useful but it's a per-command bound, and between an EventLoop hiccup and a chain of multi().incr().expire().exec() round-trips a single middleware can still spend more wall-clock time inside the breaker than you intended.

A Promise.race against an explicit deadline gives the middleware a wall-clock budget regardless of what the client is doing.⁴ Write src/lib/with-deadline.ts:

export class DeadlineExceededError extends Error {
  readonly code = 'DEADLINE_EXCEEDED' as const;
  constructor(ms: number, label: string) {
    super(`${label}: deadline of ${ms}ms exceeded`);
    this.name = 'DeadlineExceededError';
  }
}

export function withDeadline<T>(
  p: Promise<T>,
  ms: number,
  label: string,
): Promise<T> {
  let timer: NodeJS.Timeout;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(() => reject(new DeadlineExceededError(ms, label)), ms);
  });
  return Promise.race([p, timeout]).finally(() => clearTimeout(timer));
}

Node has no Promise cancellation primitive,⁵ so the original Redis call continues in the background after the deadline rejects. That's fine as long as you also pin commandTimeout on the client so each in-flight command has its own client-side rejection too — both layers compose. commandTimeout is the per-command budget on the Redis client; withDeadline is your middleware's wall-clock budget for the entire operation, which may pipeline several commands.

Step 4: configure the Redis client for fast failure

The single most important ioredis option for resilience is enableOfflineQueue: false. The default is true, which means commands issued while the client is disconnected get queued in memory and replayed on reconnect.⁶ That sounds friendly but it has a hidden hazard: under a sustained outage the queue grows until something OOMs, and when Redis comes back the API processes a thundering herd of stale commands all at once. With the offline queue off, commands fail immediately during a disconnect — exactly the signal the circuit breaker needs.

Write src/redis.ts:

import { Redis } from 'ioredis';

/**
 * One shared Redis client per process. `lazyConnect: true` means no TCP socket
 * is opened until the first command (or an explicit .connect() call). With
 * `enableOfflineQueue: false`, commands issued while disconnected reject
 * immediately rather than queueing — that's what lets the circuit breaker
 * see failures fast.
 */
export function makeRedis(url: string) {
  return new Redis(url, {
    lazyConnect: true,
    enableOfflineQueue: false,
    maxRetriesPerRequest: 1,
    commandTimeout: 200,
    retryStrategy: (attempt) => Math.min(1000 * 2 ** attempt, 30_000),
    reconnectOnError: () => 1,
  });
}

The defaults you're overriding here: enableOfflineQueue defaults to true and maxRetriesPerRequest defaults to 20 — both are surprises if you wanted fast-fail.⁷ The retryStrategy returns the backoff delay in ms for the Nth reconnect attempt, capped at 30s so a long outage doesn't pin the event loop. reconnectOnError: () => 1 retries a failing command on the reconnect attempt instead of dropping it.

Step 5: fail-closed auth middleware

API-key verification is the canonical fail-closed case. If you can't reach Redis to confirm the key maps to a real user, the only safe response is to refuse the request — returning 200 would be an authentication bypass.⁸ Write src/middleware/auth.ts:

import { createMiddleware } from 'hono/factory';
import type { Redis } from 'ioredis';
import { CircuitBreaker } from '../lib/circuit-breaker.ts';
import { withDeadline } from '../lib/with-deadline.ts';

export type AuthVars = { userId: string };

export function authMiddleware(redis: Redis, breaker: CircuitBreaker) {
  return createMiddleware<{ Variables: AuthVars }>(async (c, next) => {
    const key = c.req.header('x-api-key');
    if (!key) {
      return c.json({ error: 'missing_api_key' }, 401);
    }
    let userId: string | null;
    try {
      userId = await breaker.fire(() =>
        withDeadline(redis.get(`apikey:${key}`), 150, 'auth.lookup'),
      );
    } catch {
      // Circuit open, deadline exceeded, Redis disconnected, command error —
      // any failure means we cannot verify, so refuse.
      return c.json({ error: 'auth_unavailable' }, 503, {
        'Retry-After': '30',
      });
    }
    if (userId === null) {
      return c.json({ error: 'invalid_api_key' }, 401);
    }
    c.set('userId', userId);
    await next();
  });
}

Three details worth flagging:

1. The bare catch {} is deliberate — for security middleware, the failure mode is "cannot verify," not "this specific error class." A CircuitOpenError and a Redis pipeline error and a deadline timeout all mean the same thing: refuse the request.
2. The 503 carries Retry-After: 30 so clients (and Cloudflare/ALB retry policies) know how long to back off. Without that header, well-behaved clients may retry immediately and amplify the load on a degraded backend.⁹
3. The 150ms deadline is the request-side budget for auth. If your p99 healthy GET is 10ms, 150ms leaves 15x headroom — small enough that even a partial-failure Redis (slow but reachable) trips the breaker quickly.

Step 6: fail-open feature flag with a safe default

The fail-open case looks similar but inverts the policy: if the kill switch lookup fails, fall back to a hardcoded default and let the request through. The judgment call is in picking that default. For a "checkout disabled" kill switch the safe default is false (don't disable — a Redis outage shouldn't also take down checkout). For an "experimental UI enabled" flag the safe default is also false (don't show unfinished UI when you can't read the config). The pattern is: choose the default that minimises blast radius if you guess wrong.¹⁰

Write src/middleware/feature-flag.ts:

import { createMiddleware } from 'hono/factory';
import type { Redis } from 'ioredis';
import { CircuitBreaker } from '../lib/circuit-breaker.ts';
import { withDeadline } from '../lib/with-deadline.ts';

export function killSwitchMiddleware(
  redis: Redis,
  breaker: CircuitBreaker,
  flag: string,
  safeDefault: boolean,
) {
  return createMiddleware(async (c, next) => {
    let killed = safeDefault;
    try {
      const raw = await breaker.fire(() =>
        withDeadline(redis.get(`flag:${flag}`), 100, 'flag.lookup'),
      );
      killed = raw === '1';
    } catch {
      // Fall through with safeDefault; tag the response so observability
      // tooling can count fallback events.
      c.header('x-flag-fallback', flag);
    }
    if (killed) {
      return c.json({ error: 'feature_disabled' }, 503);
    }
    await next();
  });
}

The x-flag-fallback header is the small observability investment that pays for itself the first time you wonder "did anyone actually hit the fallback path during the outage?" Grep your access logs for the header name and you have a precise count. Same pattern works for OpenFeature integrations — emit a counter on every fallback evaluation.

Step 7: fail-open sliding-window rate limiter

Rate limits exist to protect the API from abuse — they are not what the API does for a living. When Redis is unreachable the right degraded behaviour is "no limits enforced," not "503 every request." The trade-off is explicit: a brief outage may let a request burst through unbounded. That's almost always preferable to taking the whole API offline because the limiter is down.¹¹

Write src/middleware/rate-limit.ts:

import { createMiddleware } from 'hono/factory';
import { getConnInfo } from '@hono/node-server/conninfo';
import type { Redis } from 'ioredis';
import { CircuitBreaker } from '../lib/circuit-breaker.ts';
import { withDeadline } from '../lib/with-deadline.ts';

export function rateLimitMiddleware(
  redis: Redis,
  breaker: CircuitBreaker,
  opts: { limit: number; windowSec: number },
) {
  return createMiddleware(async (c, next) => {
    const ip = getConnInfo(c).remote.address ?? 'unknown';
    const key = `rl:${ip}:${Math.floor(Date.now() / 1000 / opts.windowSec)}`;
    let count: number;
    try {
      const reply = await breaker.fire(() =>
        withDeadline(
          redis.multi().incr(key).expire(key, opts.windowSec).exec(),
          80,
          'rl.incr',
        ),
      );
      const incrResult = reply?.[0];
      count = (incrResult?.[1] as number | undefined) ?? 0;
    } catch {
      c.header('x-ratelimit-fallback', 'open');
      await next();
      return;
    }
    c.header('x-ratelimit-limit', String(opts.limit));
    c.header('x-ratelimit-remaining', String(Math.max(0, opts.limit - count)));
    if (count > opts.limit) {
      return c.json({ error: 'rate_limited' }, 429, {
        'Retry-After': String(opts.windowSec),
      });
    }
    await next();
  });
}

The MULTI keeps INCR and EXPIRE in a single round-trip and guarantees the TTL lands even on the first request of a new window. exec() returns Array<[Error | null, unknown]> per queued command, so the count is at reply[0][1]. Note that for proper IP extraction behind a proxy you'd combine this with getConnInfo + a trusted X-Forwarded-For hop selector — the sibling rate-limit tutorial covers that in detail.¹²

Step 8: compose into a Hono app

Now wire all three middleware onto a single route. Write src/app.ts:

import { Hono } from 'hono';
import { logger } from 'hono/logger';
import { makeRedis } from './redis.ts';
import { CircuitBreaker } from './lib/circuit-breaker.ts';
import { authMiddleware, type AuthVars } from './middleware/auth.ts';
import { killSwitchMiddleware } from './middleware/feature-flag.ts';
import { rateLimitMiddleware } from './middleware/rate-limit.ts';

export function createApp(redisUrl: string) {
  const redis = makeRedis(redisUrl);

  // One breaker per dependency surface. A single shared breaker across all
  // middleware works too — but a slow `incr` would then open the auth
  // circuit, which masks which call is actually broken. Start with one
  // breaker per Redis-surface and split if you need finer-grained isolation.
  const breaker = new CircuitBreaker({
    name: 'redis',
    failureThreshold: 5,
    resetMs: 5_000,
  });

  const app = new Hono<{ Variables: AuthVars }>();
  app.use(logger());

  app.get('/healthz', (c) => c.json({ ok: true }));

  app.get(
    '/api/checkout',
    rateLimitMiddleware(redis, breaker, { limit: 30, windowSec: 60 }),
    killSwitchMiddleware(redis, breaker, 'checkout', /*safeDefault=*/ false),
    authMiddleware(redis, breaker),
    (c) => c.json({ ok: true, user: c.var.userId }),
  );

  return { app, redis };
}

Middleware order matters and the order here is deliberate. Rate-limit runs first so an unauthenticated flood doesn't hammer Redis for auth lookups; kill-switch runs next so a disabled feature returns fast without an auth round-trip; auth runs last because it's both the most expensive (a key lookup) and the most sensitive (its result decides whether the user is identified at all). Whatever order you pick, write it down — middleware composition is one of the most common sources of subtle production bugs.

Write src/index.ts:

import { serve } from '@hono/node-server';
import { createApp } from './app.ts';

const url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379';
const { app, redis } = createApp(url);

// Required: ioredis emits 'error' on every reconnect failure. Without a
// listener the process crashes with `unhandledRejection` on the first
// disconnect. The middleware applies the actual fail policy per-request;
// this listener exists only so the process survives the event.
redis.on('error', (err) => {
  console.warn('[redis] error:', err.message);
});

// Connect on boot so a typo in REDIS_URL surfaces during deploy, not at
// first request. We swallow the error — the middleware degrades correctly
// if the boot connect fails.
await redis.connect().catch((err) => {
  console.warn('[boot] redis connect failed; serving in degraded mode:', err.message);
});

const port = Number(process.env.PORT ?? 3000);
serve({ fetch: app.fetch, port });
console.log(`listening on http://127.0.0.1:${port}`);

The redis.on('error', ...) is not optional. Without it, the first reconnect failure becomes an unhandledRejection and crashes the process. The middleware can't help you if the runtime is dead.

Step 9: verify the happy path with Redis up

Start Redis and the server in two terminals:

# Terminal 1
docker run --rm -p 6379:6379 redis:8-alpine

# Terminal 2 — same directory as the project
SECRET_KEY=letmein
docker exec -i $(docker ps -qf ancestor=redis:8-alpine) \
  redis-cli SET apikey:$SECRET_KEY user-42
node --import tsx src/index.ts

node --import tsx loads tsx as a module-register hook so the .ts imports resolve without a separate build step. It needs Node 20.6+ for the module.register() API. If you prefer the standalone CLI (with watch mode and other flags), npx tsx src/index.ts works equally well — tsx's own FAQ recommends the standalone form for development, since the loader form adds a small per-request transpile overhead.

Now exercise the API:

# Healthcheck — no Redis path
curl -s -w '\n%{http_code}\n' http://127.0.0.1:3000/healthz
# {"ok":true}
# 200

# Missing key — auth middleware short-circuits with 401
curl -s -w '\n%{http_code}\n' http://127.0.0.1:3000/api/checkout
# {"error":"missing_api_key"}
# 401

# Valid key — happy path
curl -s -i -H 'x-api-key: letmein' http://127.0.0.1:3000/api/checkout
# HTTP/1.1 200 OK
# x-ratelimit-limit: 30
# x-ratelimit-remaining: 29
# {"ok":true,"user":"user-42"}

The x-ratelimit-limit and x-ratelimit-remaining headers are how clients learn their budget without a separate API call. The IETF rate-limit headers RFC describes a richer schema (RateLimit + RateLimit-Policy structured fields) if you want full spec compliance.¹³

Step 10: verify the failure path with Redis down

Stop the Redis container with docker stop (or just kill the terminal). The server keeps running. Now hammer the API:

# 1. Auth fails closed — 503 with Retry-After
curl -s -i -H 'x-api-key: letmein' http://127.0.0.1:3000/api/checkout
# HTTP/1.1 503 Service Unavailable
# retry-after: 30
# x-flag-fallback: checkout
# x-ratelimit-fallback: open
# {"error":"auth_unavailable"}

# 2. The breaker is shared across all three middleware, so each failing
#    request bumps its counter by up to 3 (one per middleware that calls
#    fire()). With failureThreshold: 5 the circuit opens within the first
#    2 requests, and every subsequent call short-circuits in milliseconds
#    instead of waiting for the per-middleware deadline budget:
for i in {1..8}; do
  time curl -s -o /dev/null -w '%{http_code} ' \
    -H 'x-api-key: letmein' http://127.0.0.1:3000/api/checkout
done
# 503 503 503 503 503 503 503 503
# (every call settles in single-digit ms once the breaker is OPEN —
#  if you want a "5 consecutive failures per operation" model instead,
#  pass a separate CircuitBreaker into each middleware factory)

This is the exact behaviour the sandbox build of the tutorial reproduces end-to-end. The fail-open middleware emit their fallback headers (x-flag-fallback, x-ratelimit-fallback) which makes the degraded state machine-grep-able — Datadog or a sidecar log shipper can alert on a sudden spike of fallback responses without you having to instrument every middleware.

Bring Redis back up:

docker run --rm -p 6379:6379 redis:8-alpine

After the resetMs: 5_000 window the breaker goes HALF_OPEN. The very next call is a probe — if it succeeds the breaker closes, if it fails it goes straight back to OPEN. Your API recovers without a restart and without manual intervention.

Verification checklist

You can confirm the wiring is correct without running the load test by typechecking:

npx tsc --noEmit
# (no output = clean)

Then a 10-second smoke test against an unreachable port simulates Redis being down without needing Docker at all:

REDIS_URL=redis://127.0.0.1:9999 node --import tsx src/index.ts &
sleep 1
curl -s -i -H 'x-api-key: t' http://127.0.0.1:3000/api/checkout
# Expect: 503 with x-flag-fallback + x-ratelimit-fallback headers
# (this is the exact response shape verified in the sandbox)
kill %1

Troubleshooting

Stream isn't writeable and enableOfflineQueue options is false. This is the message ioredis returns when you send a command while the socket is disconnected and offline queueing is off. That's the intended behaviour — your catch {} block should swallow it. If you're seeing it bubble up as a 500, your try/catch is wrapped at the wrong level. Make sure the await breaker.fire(...) is inside the try, not outside it.

Process exits with unhandledRejection: connect ECONNREFUSED. You forgot the redis.on('error', ...) listener in src/index.ts. ioredis emits 'error' on every reconnect failure (which can happen every few seconds during a long outage). The listener doesn't need to do anything — it just has to exist so Node doesn't treat the event as unhandled.

The breaker opens too quickly during normal traffic. Two likely causes. (1) Your commandTimeout and withDeadline budgets are tighter than your p99 healthy latency, so legitimate slow calls look like failures. Run redis-cli --latency against your Redis to baseline the round-trip, then set the deadline to roughly 10x the p99. (2) You're sharing one breaker across heterogeneous workloads — a slow KEYS scan can open the same breaker an INCR uses. Split into per-operation breakers if the workloads differ meaningfully.

The breaker never opens. Either the failure threshold is too high for your traffic pattern, or your enableOfflineQueue is still true, in which case commands silently queue during the outage and never throw. Verify the client's actual options with console.log(redis.options.enableOfflineQueue) at boot.

Auth returns 503 even after Redis recovers. The breaker's resetMs hasn't elapsed yet, so it's still OPEN. Either reduce resetMs (5s is reasonable for an API; 60s for a batch worker) or call the probe sooner — but don't reduce it below your typical recovery time, or HALF_OPEN will flap.

Next steps

The pattern generalises. Anywhere your middleware reads from a dependency you don't control, the same three questions apply: what does this middleware protect (security or capacity)? what happens to the user if the dependency is gone? what's the safe default? Auth-style middleware (session lookups, JWT validation against a JWKS endpoint) almost always fail closed. Telemetry middleware (logging, tracing) almost always fail open — never let your observability layer take down the API. Business middleware (feature flags, A/B test assignment, geo lookups) live in the middle and need an explicit safe default per flag.

A few directions for follow-up reading on this site:

• The Upstash + Hono sliding-window rate-limit tutorial goes deeper on the rate-limiter side, including IP extraction behind a proxy and the IETF RateLimit-Policy header.
• The Bun + Hono on Fly.io production guide covers how to deploy the resulting service — including how to wire SIGTERM handling so the breaker's in-memory state is flushed cleanly on rolling restarts.
• For backend-architecture background on the pattern itself, backend architecture patterns from monoliths to microservices places circuit breakers in the broader context of service-to-service resilience.

If you want a production-grade circuit breaker rather than the 40-line teaching version, reach for opossum³. The shape differs slightly — opossum wraps a function at construction time (new CircuitBreaker(fn, opts).fire(...args)) rather than at call time, so you instantiate one breaker per Redis operation (or per logical resource) rather than passing a generic breaker around. In exchange you get percentage-based thresholds, rolling buckets, and event hooks for metrics emission. The fail-open vs fail-closed policies in your middleware stay exactly the same.

Footnotes

1. npm CLI documentation, npm install command (-E / --save-exact): writes the resolved version verbatim into package.json instead of with the default caret range. https://docs.npmjs.com/cli/v11/commands/npm-install/. ↩

2. Michael Nygard, Release It! (Pragmatic Bookshelf), introduces the three-state circuit breaker; Martin Fowler's summary: https://martinfowler.com/bliki/CircuitBreaker.html. ↩

3. opossum on npm: https://www.npmjs.com/package/opossum — v9.0.0 published 2025-06-05, Apache-2.0, engines ^20 || ^22 || ^24. The package ships JavaScript only; TypeScript users currently rely on @types/opossum (which lags the runtime version) or write thin local type shims. ↩ ↩²

4. Google SRE book (the first volume, "Site Reliability Engineering: How Google Runs Production Systems"), chapter 22 "Addressing Cascading Failures": deadlines and load shedding are first-class techniques for preventing one slow dependency from collapsing the whole call graph. https://sre.google/sre-book/addressing-cascading-failures/. ↩

5. Node.js Globals documentation, AbortController class: a Web-API-compatible utility used to signal cancellation in selected Promise-based APIs that opt in (such as fetch, fs, setTimeout). There is no general Promise-cancellation primitive in the language. https://nodejs.org/api/globals.html#class-abortcontroller. ↩

6. ioredis README, "Offline Queue" section (covers enableOfflineQueue) and "Auto-reconnect" section (covers retryStrategy and reconnectOnError): https://github.com/redis/ioredis/blob/main/README.md#offline-queue. ↩

7. ioredis defaults (verified at runtime in the sandbox build of this tutorial): enableOfflineQueue: true, maxRetriesPerRequest: 20. Override both for fail-fast behaviour. ↩

8. OWASP Application Security Verification Standard, V2 (Authentication): a system that "fails open" on authentication failure is explicitly a vulnerability. https://owasp.org/www-project-application-security-verification-standard/. ↩

9. RFC 9110 §10.2.3 — Retry-After may be either an HTTP-date or a non-negative number of seconds; this tutorial uses the seconds form. https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3. ↩

10. OpenFeature specification, Flag Evaluation API (Requirement 1.4.10): "Flag evaluation calls must always return the default value in the event of abnormal execution." Every typed evaluation method takes a required default value argument the SDK returns when the provider can't be reached. https://openfeature.dev/specification/sections/flag-evaluation/. ↩

11. The Upstash rate-limit SDK's documented timeout option fails open by design for this same reason. https://upstash.com/docs/oss/sdks/ts/ratelimit/overview. ↩

12. NerdLevelTech, "API Rate Limiting with Upstash Redis: 2026 Node Tutorial" — covers the IP-extraction and IETF-headers side of the rate-limiter in detail: https://nerdleveltech.com/api-rate-limiting-upstash-redis-sliding-window-tutorial. ↩

13. IETF draft draft-ietf-httpapi-ratelimit-headers — RateLimit + RateLimit-Policy structured fields. https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/. ↩