Skip to content

Knowledge pipeline

Ingestion, chunking, embeddings and the SSRF-safe crawler.

On this page

Knowledge sources (files, texts, Q&A pairs and websites) become chunks with embeddings in document_chunks, a normal tenant table with a vector(1024) column, an HNSW index on cosine distance, and btree indexes on agent_id and source_id.

Ingestion: knowledge.ingest

One job on the ingestion queue does the whole pipeline:

  1. 1Read the sourceupload, text or Q&A
  2. 2Extract textPDF by page, Word with headings
  3. 3Chunk along headings~500 tokens, title + heading path prefixed
  4. 4Embed in batches
  5. 5Replace the source's chunksone transaction
  • Status: pending → syncing → synced, or failed with a message the customer can act on ("this PDF has no text layer — it is probably a scan").
  • Revisions: each edit bumps the source's revision, and a job for an older revision does nothing. Two quick edits never leave the older text behind. Until the new chunks land, the agent keeps answering from the old ones.
  • Limits: the workspace's training size (from its plan) is checked when text is added and again on every training run. Embedding tokens are recorded per source, cumulatively, including failed runs.
  • Deleting a source deletes its chunks and, for a file, the upload.

Uploads

Files never pass through the API. The browser gets a signed PUT that fixes the key, type and exact length, and uploads straight to the bucket. The API then checks what arrived (size, and the real type from the file's first bytes) before anything may use it. Objects sit under org/<organization-id>/. An hourly storage.sweep removes uploads that were started but never completed.

The crawler

A website source is crawled from a start page, a sitemap or a list of up to 500 URLs, with optional exclude rules.

Safety first: these are customer URLs fetched from our server. src/crawler/safe-fetch.ts:

  • resolves each hostname and refuses anything not on the public internet (private, loopback, link-local, cloud metadata, carrier-grade NAT, IPv6 equivalents, IPv4-mapped forms)
  • connects to the address it checked, so there is no second DNS lookup for a rebinding attack to exploit
  • follows redirects itself and re-checks every hop
  • allows only http and https on standard ports, with no credentials in URLs
  • caps each response with a hard deadline and a byte limit after decompression

Its tests run a real local server and try every one of those routes in.

Politeness: robots.txt is obeyed (RFC 9309), including Crawl-delay up to 10 seconds; otherwise one second between requests. The user agent identifies us.

How a crawl runs:

  1. 1crawl.run20 pages per job, chained
  2. 2Normalise URLno fragment, no query
  3. 3robots.txt + exclude rules
  4. 4Fetch, strip nav/footer/banners
  5. 5Content hashunchanged → keep chunks
  6. 6Unreached pages deleted at the end

website_pages is the frontier, so a crawl that dies partway resumes where it stopped. Unchanged pages are never re-embedded, and duplicates are skipped. JavaScript-only pages are reported as failed, with that reason.

Weekly resync: the hourly crawl.resync-due starts a crawl for each website source past its date.

Get started free — 50 credits a month