Make jobs idempotent

Why jobs run twice

Halyard guarantees that every job runs at least once. “Exactly once” isn't something any queue can promise, because the failure that matters happens between your side effect and the acknowledgement. A job runs again when:

• the worker crashes or is deployed after charging the card but before reporting success;
• the job exceeds its timeout while the slow call it was waiting on actually succeeded;
• someone retries a failed job from the dashboard, or the same work is queued twice.

An idempotent job produces the same outcome no matter how many times it runs. This guide makes a charge-subscription job idempotent in four layers, from the queue down to the payment provider.

Build it

1. Deduplicate at the source

   Build the idempotency key from what makes the work unique: the subscription and the billing period. A renewal script that runs twice then creates one job, not two.

   EnqueueNode.jsPython

   // billing/renewals.jsawait halyard.jobs.create({  queue: "billing",  task: "charge-subscription",  payload: { subscription_id: sub.id, period: "2026-11" },  idempotencyKey: `charge:${sub.id}:2026-11`,});

2. Make writes idempotent

   Let the database enforce uniqueness on the business key, then write with an upsert. The second run finds the existing invoice instead of creating another.

   migrations/0007_invoices_unique_period.sql

   -- migrations/0007_invoices_unique_period.sqlalter table invoices  add constraint invoices_subscription_period_key unique (subscription_id, period);

   Upsert the invoiceNode.jsPython

   const invoice = await db.one(  `insert into invoices (subscription_id, period, amount, status)   values ($1, $2, $3, 'open')   on conflict (subscription_id, period) do update set amount = excluded.amount   returning *`,  [sub.id, period, sub.price],);

   The handler in the next step calls this query through a small db.invoices.upsert() helper.

3. Pass the job ID downstream

   Your database is safe now, but the card charge isn't. Most payment, email and messaging APIs accept an idempotency key; job.id is stable across every attempt of the same job, so it is exactly the right value. If the worker dies after the charge, the retry sends the same key and gets the original charge back.

   HandlerNode.jsPython

   // billing/charge-subscription.jsimport { NonRetryableError } from "@halyard/node";import { db } from "./db.js";import { payments } from "./payments.js"; export async function chargeSubscription(job) {  const { subscription_id, period } = job.payload;  const sub = await db.subscriptions.find(subscription_id);  if (!sub) throw new NonRetryableError("Subscription not found");   const invoice = await db.invoices.upsert({ subscription_id, period, amount: sub.price });  if (invoice.status === "paid") return { skipped: "already paid", invoice: invoice.id };   const charge = await payments.charge(    { customer: sub.customer_id, amount: invoice.amount, currency: "eur" },    { idempotencyKey: job.id },  );   await db.invoices.update(invoice.id, { status: "paid", charge_id: charge.id });  return { invoice: invoice.id, charge: charge.id };}

4. Checkpoint long jobs

   A job with several independent side effects should record its progress. job.checkpoint(name) stores a marker on the job; on the next attempt job.checkpoints lists every marker saved so far, so the retry resumes where the last attempt stopped.

   CheckpointsNode.jsPython

   worker.task("close-month", async (job) => {  const done = new Set(job.checkpoints);   if (!done.has("invoices")) {    await finalizeInvoices(job.payload.period);    await job.checkpoint("invoices");  }  if (!done.has("ledger")) {    await postToLedger(job.payload.period);    await job.checkpoint("ledger");  }  if (!done.has("statements")) {    await sendStatements(job.payload.period);    await job.checkpoint("statements");  }});

   Each step still has to be idempotent on its own: the worker can crash after a step's work and before its checkpoint is saved.

5. Prove it with a test

   The simplest test of idempotency is the honest one: run the handler twice with the same job and check the world only changed once.

   TestNode.jsPython

   // billing/charge-subscription.test.jsimport { test, expect } from "vitest";import { chargeSubscription } from "./charge-subscription.js";import { payments } from "./payments.js";import { seedSubscription, fakeJob } from "./test-helpers.js"; test("charging twice charges once", async () => {  const sub = await seedSubscription({ price: 4800 });  const job = fakeJob({ id: "job_3Rk9Tq7Xw2", payload: { subscription_id: sub.id, period: "2026-11" } });   const first = await chargeSubscription(job);  const second = await chargeSubscription(job);   expect(second).toEqual({ skipped: "already paid", invoice: first.invoice });  expect(payments.charges).toHaveLength(1);});

Review checklist

Before shipping a new task, walk through its side effects one by one.

• Is there a unique constraint for every row the job creates?
• Does every external call carry job.id (or a key derived from the payload) as its idempotency key?
• Does the handler check current state (“already paid”) before acting, instead of trusting the payload?
• Is each email or notification guarded by a sent-flag or a provider-side key?
• For long jobs: does every checkpoint come after the step's work, never before?

When retries should stop

Idempotency makes retries safe; it doesn't make them useful. A declined card won't succeed on attempt four. Throw NonRetryableError for outcomes that need a human or a different code path, and let Halyard retry only failures that can clear on their own: timeouts, 5xx responses and rate limits.

Recap

• Halyard delivers at least once; a job can run more than once, so every side effect must tolerate it.
• Idempotency keys stop duplicate jobs; they don't make the handler safe on their own.
• Writes become upserts keyed on business IDs, and external calls reuse job.id as their key.
• Long jobs record checkpoints so a retry resumes instead of starting over.

Next steps

Keep going with a related guide, or look up every option used here in the Jobs API reference.