Skip to content
Halyarddocs
Halyarddocs

Core concepts

Retries and backoff

Failures are normal: networks drop, partners rate-limit, deploys restart workers. Halyard retries failed attempts with exponential backoff and keeps what still fails in a dead-letter queue.

On this page

How retries work

When an attempt fails, Halyard checks two things: is the error retryable, and are attempts left? If both are true the job returns to queued with a run_at in the future. Otherwise it moves to failed and into the dead-letter queue.

A retry is the same job with the same ID and payload. Each attempt is recorded with its own start time, duration, logs and error, so you can see exactly how a job got where it is.

Exponential backoff

The delay doubles after each failure, so a brief outage costs a few quick retries while a long one doesn't hammer the service that is down. The default policy starts at 10 seconds and caps at one hour.

Backoff formula
delay(n) = min(initial × factor^(n − 1), max_delay)actual    = delay(n) × random(1 − jitter, 1 + jitter) # defaults: initial = 10s, factor = 2, max_delay = 1h, jitter = 0.25

Delay before each retry

base ±25% jitter
  1. Retry 110s
  2. Retry 220s
  3. Retry 340s
  4. Retry 41m 20s
  5. Retry 52m 40s
  6. Retry 65m 20s
  7. Retry 710m 40s
Default policy: 10s initial delay, factor 2, capped at 1h. Log scale.
AttemptWaits before itJitter windowTime since first attempt
1——0s
210s7.5s–12.5s≈ 10s
320s15s–25s≈ 30s
440s30s–50s≈ 1m 10s
51m 20s1m–1m 40s≈ 2m 30s
62m 40s2m–3m 20s≈ 5m 10s
75m 20s4m–6m 40s≈ 10m 30s
810m 40s8m–13m 20s≈ 21m 10s
With the default 8 attempts, a job that keeps failing is given up after about 21m 10s.

Why jitter

If a dependency fails for a minute, every job that hit it fails at about the same moment. Without jitter they would all retry at the same moment too, and knock it over again. Jitter spreads each delay randomly by ±25%, so retries arrive as a smooth trickle instead of a wave.

Maximum attempts

max_attempts counts every attempt, including the first. The default is 8; the allowed range is 1 to 50. Set it to 1 to never retry. Attempts lost because a worker disappeared (no heartbeat) are retried without counting, up to 5 times, because the handler may never have run.

Which errors retry

Any error a handler throws is retried unless you mark it as permanent.

OutcomeRetriesNotes
Handler threw / returned an errorYesThe default for any exception.
Attempt timed outYesLease expired after the queue's timeout.
Worker lost (no heartbeat)YesDoes not count toward max_attempts.
NonRetryableErrorNoGoes straight to failed and the dead-letter queue.
Payload failed validationNoRejected at creation with 422, never queued.
Job cancelledNoMoves to cancelled; no further attempts.

Throw a non-retryable error when trying again can't possibly help, such as a missing record or invalid input:

import { NonRetryableError } from "@halyard/node"; worker.task("send-welcome-email", async (job) => {  const user = await db.users.find(job.payload.user_id);  if (!user) {    // Retrying won't make the user exist    throw new NonRetryableError(`User ${job.payload.user_id} not found`);  }  await sendEmail({ to: user.email, template: "welcome" });});

Custom retry policies

Policies resolve from the most specific level: a value on the job wins over the task, the task over the queue, and the queue over the project default. Set a policy on the task when it reflects how that work behaves, such as a partner API with strict rate limits.

worker.task(  "sync-crm-contact",  {    retry: {      max_attempts: 12,      initial: "30s",      factor: 3,      max_delay: "6h",      jitter: 0.3,    },  },  async (job) => {    const res = await crm.upsertContact(job.payload);    if (res.status === 429) {      // Honour the partner's Retry-After instead of the policy      throw job.retryIn(res.headers["retry-after"] + "s");    }  },);

job.retryIn overrides the computed delay for this one retry and still counts as an attempt. Override a single job when you create it:

JavaScript
await halyard.jobs.create({  queue: "media",  task: "resize-image",  payload: { image_id: "img_2Lq8" },  retry: { max_attempts: 3 }, // overrides task and queue defaults});

Dead-letter queue

Jobs that fail for good are kept in the dead-letter queue of their queue for 30 days with their payload, every attempt and the last error. Halyard publishes a job.failed event for each one, so you can alert on it with a subscription.

Once you've fixed the cause, retry dead letters one at a time or in bulk. The job keeps its ID and history and starts again with a full set of attempts.

# List dead letters on the emails queuecurl "https://api.halyard.example/v3/jobs?queue=emails&status=failed&dead_letter=true" \  -H "Authorization: Bearer hy_test_51Hq8Zx2rTmV9cKpL3d" # Retry one with a fresh set of attemptscurl -X POST https://api.halyard.example/v3/jobs/job_8Vn4Hc2Lp6/retry \  -H "Authorization: Bearer hy_test_51Hq8Zx2rTmV9cKpL3d"

Suggested