Process image uploads

How it works

Resizing a 12-megapixel photo takes a full CPU core for a second or two. Do that in a web request and a handful of simultaneous uploads will slow every other page. The pipeline below keeps your web servers out of it entirely.

1. The browser uploads the original straight to your object storage with a signed URL.
2. Your app records the upload and queues a process-image job with the storage key.
3. A worker on the media queue downloads the original and writes three WebP variants.
4. The page polls a tiny status endpoint and shows progress until the job succeeds.

Build it

1. Create a media queue

   Give image work its own queue. A low concurrency stops one worker machine from trying to resize 50 photos at once, and a timeout catches the rare file that makes the encoder hang.

   Create a queuecURLNode.jsPython

   curl https://api.halyard.example/v3/queues \  -H "Authorization: Bearer hy_test_51Hq8Zx2rTmV9cKpL3d" \  -H "Content-Type: application/json" \  -d '{ "name": "media", "concurrency": 4, "timeout": "2m" }'

2. Queue a job after the upload completes

   The browser calls this endpoint once storage confirms the upload. The payload carries the storage key, never the image itself: job payloads are limited to 256 KB.

   Upload completeNode.jsPython

   // app/api/uploads/complete/route.jsimport { Halyard } from "@halyard/node";import { db } from "@/lib/db"; const halyard = new Halyard(process.env.HALYARD_KEY); export async function POST(request) {  const { key } = await request.json();  const upload = await db.uploads.insert({ key, status: "processing" });   const job = await halyard.jobs.create({    queue: "media",    task: "process-image",    payload: { upload_id: upload.id, key },    idempotencyKey: `image:${upload.id}`,  });   return Response.json({ uploadId: upload.id, jobId: job.id });}

3. Resize in the worker

   The worker rotates the image according to its EXIF data, scales it down to each width without ever enlarging it and reports progress after each variant. Variant keys are derived from the upload ID, so a retry overwrites the same files instead of creating new ones.

   WorkerNode.jsPython

   // worker.jsimport { Worker } from "@halyard/node";import sharp from "sharp";import { storage } from "./lib/storage.js";import { db } from "./lib/db.js"; const SIZES = [  { name: "thumb", width: 160 },  { name: "medium", width: 800 },  { name: "large", width: 1600 },]; const worker = new Worker(process.env.HALYARD_KEY, { queues: ["media"], concurrency: 4 }); worker.task("process-image", async (job) => {  const { upload_id, key } = job.payload;  const original = await storage.get(key);   for (const [i, size] of SIZES.entries()) {    const output = await sharp(original)      .rotate()      .resize({ width: size.width, withoutEnlargement: true })      .webp({ quality: 82 })      .toBuffer();     await storage.put(`images/${upload_id}/${size.name}.webp`, output, "image/webp");    await job.progress(Math.round(((i + 1) / SIZES.length) * 100));  }   await db.uploads.update(upload_id, { status: "ready" });}); worker.start();

4. Show progress in the browser

   Add a status route on your server, then poll it from the page once a second until the job is finished.

   app/api/jobs/[id]/route.js

   // app/api/jobs/[id]/route.jsimport { Halyard } from "@halyard/node"; const halyard = new Halyard(process.env.HALYARD_KEY); export async function GET(_request, { params }) {  const { id } = await params;  const job = await halyard.jobs.retrieve(id);  return Response.json({ status: job.status, progress: job.progress });}

   components/upload-status.tsx

   // components/upload-status.tsx"use client"; import { useEffect, useState } from "react"; type JobState = { status: string; progress: number }; export function UploadStatus({ jobId }: { jobId: string }) {  const [job, setJob] = useState<JobState | null>(null);   useEffect(() => {    const timer = setInterval(async () => {      const next: JobState = await fetch(`/api/jobs/${jobId}`).then((r) => r.json());      setJob(next);      if (next.status !== "queued" && next.status !== "running") clearInterval(timer);    }, 1000);    return () => clearInterval(timer);  }, [jobId]);   if (!job || job.status === "queued") return <p>Waiting for a worker…</p>;  if (job.status === "failed") return <p>We couldn't process this image.</p>;  return <progress value={job.progress} max={100} aria-label="Processing image" />;}

Large files and timeouts

The queue's two-minute timeout covers photos up to about 40 megapixels on a 2-vCPU worker. If you accept panoramas or scans, raise the timeout on the individual job rather than for the whole queue, so a stuck regular photo still fails fast.

await halyard.jobs.create({  queue: "media",  task: "process-image",  payload: { upload_id: upload.id, key },  timeout: upload.bytes > 50_000_000 ? "10m" : undefined,});

Watch memory, not just CPU

Decoding a large image needs roughly width × height × 4 bytes. With a concurrency of 4, a worker processing four 40-megapixel images at once needs about 640 MB just for pixels. Size your worker machines for the concurrency you set, or lower the concurrency when you raise the size limit.

Recap

• CPU-heavy work gets its own media queue with a low concurrency and a job timeout.
• The job payload holds the storage key, never the image bytes.
• job.progress() feeds a small status endpoint; the browser never sees your API key.
• Re-running the job overwrites the same variant keys, so retries are safe.

Next steps

Keep going with a related guide, or look up every option used here in the Jobs API reference.