Background jobs
BullMQ queues, the worker process, retries and dead letters.
On this page
Slow or scheduled work runs in BullMQ on Redis. The API enqueues and the worker consumes. They share one codebase and one build, with different entrypoints (dist/main.js and dist/worker.main.js).
bash
pnpm start:dev # API
pnpm start:worker:dev # worker — without it, queued work just waits
What runs as a job
| Queue | Jobs | What for |
|---|---|---|
ingestion | knowledge.ingest | Parse, chunk and embed a source |
crawl | crawl.run, crawl.resync-due | Crawl websites; weekly resync |
mail | mail.invitation | Invitation emails |
storage | storage.sweep, storage.purge-organization, … | Remove abandoned uploads and deleted workspaces' files |
privacy | privacy.retention, privacy.account-export, privacy.workspace-export, … | Retention deletes and data exports |
catalog | catalog.sync | Daily OpenRouter model sync |
analytics | analytics.insights, analytics.rollup, … | Topic/sentiment reading; nightly rollups |
webhooks | webhooks.deliver | Signed webhook delivery with retries |
meta | meta.reply, meta.send | Answer and send on WhatsApp, Messenger and Instagram |
Account emails (verification, password reset) are deliberately not queued: their links carry live single-use tokens, which must not sit in Redis or on a dashboard.
The registry: src/queue/jobs.ts
Every job is declared in one place:
JobPayloadsis the typed payload. Payloads are ids only, never content.JOB_POLICIESgives each job's queue,attempts,backoffMs(doubling each retry) andtimeoutMs. The timeout fails a hung call so it cannot hold a worker slot forever.JOB_SCHEDULESlists the recurring jobs. The worker upserts each at boot under its own name, so a restart replaces the schedule rather than doubling it.QUEUE_CONCURRENCYis set per queue, so slow crawls cannot starve mail.
Rules every job follows
- Idempotent. It can run twice safely. Use the row it works on, such as a delivery id or a source revision, as the idempotency key.
- Carries
organizationIdand does its work insidewithTenant, exactly like a request. See Tenancy. - Retries with bounded exponential backoff.
- Fails visibly. After its last attempt a job stays in the queue's failed set, our dead letters, until someone retries or removes it. A job that cannot be retried safely records a terminal state row instead of throwing forever.
Watching the queues
Platform admins have two views:
/admin/jobsin the web app (Platform → Jobs): each queue's counts, connected workers, the oldest waiting job, and every job with its error and stack trace, to retry or remove. A queue with waiting jobs and no worker is flagged at the top. That is the signature of a worker that is not running./admin/queueson the API: Bull Board.
Everyone else gets a 404 on both.