Skip to content

Background jobs

BullMQ queues, the worker process, retries and dead letters.

On this page

Slow or scheduled work runs in BullMQ on Redis. The API enqueues and the worker consumes. They share one codebase and one build, with different entrypoints (dist/main.js and dist/worker.main.js).

bash
pnpm start:dev            # API
pnpm start:worker:dev     # worker — without it, queued work just waits

What runs as a job

QueueJobsWhat for
ingestionknowledge.ingestParse, chunk and embed a source
crawlcrawl.run, crawl.resync-dueCrawl websites; weekly resync
mailmail.invitationInvitation emails
storagestorage.sweep, storage.purge-organization, …Remove abandoned uploads and deleted workspaces' files
privacyprivacy.retention, privacy.account-export, privacy.workspace-export, …Retention deletes and data exports
catalogcatalog.syncDaily OpenRouter model sync
analyticsanalytics.insights, analytics.rollup, …Topic/sentiment reading; nightly rollups
webhookswebhooks.deliverSigned webhook delivery with retries
metameta.reply, meta.sendAnswer and send on WhatsApp, Messenger and Instagram

Account emails (verification, password reset) are deliberately not queued: their links carry live single-use tokens, which must not sit in Redis or on a dashboard.

The registry: src/queue/jobs.ts

Every job is declared in one place:

  • JobPayloads is the typed payload. Payloads are ids only, never content.
  • JOB_POLICIES gives each job's queue, attempts, backoffMs (doubling each retry) and timeoutMs. The timeout fails a hung call so it cannot hold a worker slot forever.
  • JOB_SCHEDULES lists the recurring jobs. The worker upserts each at boot under its own name, so a restart replaces the schedule rather than doubling it.
  • QUEUE_CONCURRENCY is set per queue, so slow crawls cannot starve mail.

Rules every job follows

  1. Idempotent. It can run twice safely. Use the row it works on, such as a delivery id or a source revision, as the idempotency key.
  2. Carries organizationId and does its work inside withTenant, exactly like a request. See Tenancy.
  3. Retries with bounded exponential backoff.
  4. Fails visibly. After its last attempt a job stays in the queue's failed set, our dead letters, until someone retries or removes it. A job that cannot be retried safely records a terminal state row instead of throwing forever.

Watching the queues

Platform admins have two views:

  • /admin/jobs in the web app (Platform → Jobs): each queue's counts, connected workers, the oldest waiting job, and every job with its error and stack trace, to retry or remove. A queue with waiting jobs and no worker is flagged at the top. That is the signature of a worker that is not running.
  • /admin/queues on the API: Bull Board.

Everyone else gets a 404 on both.

Get started free — 50 credits a month