Retries and failure

Backoff, dead-letter handling, and how to make retries safe.

Default behaviour

A job that throws is retried with exponential backoff: 1s, 2s, 4s, 8s, and so on, up to 5 attempts. After the final attempt it moves to the dead-letter state and stops.

Configure per job:

export const chargeCard = defineJob({
  name: "charge-card",
  retries: { attempts: 3, backoff: "exponential", base: "2s", jitter: true },
  async handler({ chargeId }) { /* ... */ },
});

Always use jitter

jitter: true is the default and you should leave it on.

Without jitter, every client that failed at the same moment retries at the same moment. A brief downstream blip produces a synchronised retry wave that is often worse than the original failure — the retries themselves become the outage.

Jitter spreads retries across the backoff window, which desynchronises clients that failed together.

At-least-once delivery

Halyard guarantees at least once, not exactly once. A job can run twice if a worker dies after completing the work but before marking the row done.

This means handlers must be idempotent. In practice:

async handler({ chargeId }) {
  const charge = await charges.find(chargeId);
  if (charge.status === "captured") return;   // already done, safely exit
  await paymentProvider.capture(charge.token, { idempotencyKey: chargeId });
  await charges.markCaptured(chargeId);
}

The guard at the top and the provider’s own idempotency key both matter. The guard handles the common case cheaply; the key handles the race where two workers pass the guard simultaneously.

Non-retryable failures

Some failures will never succeed on retry — malformed input, a deleted record, a permanently rejected card. Throw PermanentError to skip straight to dead-letter:

import { PermanentError } from "halyard";

if (!order) throw new PermanentError(`order ${orderId} no longer exists`);

Retrying these wastes capacity and delays the jobs behind them.

Replaying dead letters

npx halyard dead-letter list
npx halyard dead-letter replay --id 01J2X...
npx halyard dead-letter replay --job send-receipt --since 2h

Replay resets the attempt counter. Fix the cause before replaying in bulk, or you will simply refill the dead-letter queue.