AI runtime
How a reply is assembled, retrieved, streamed and charged.
On this page
The AI layer lives in src/ai and uses Mastra as a library, in-process. It is not a separate service and adds no network hop.
The boundary
- Nothing outside
src/aiimports@mastra/*or the OpenRouter provider. The rest of the server talks to our own service types. That boundary lets Mastra's API change without touching thirty call sites. MastraModuleis imported last inAppModuleand mounted under its own prefix. It registers a catch-all controller that would otherwise swallow unrelated routes.- Prefer Mastra's own primitives (processors for guardrails, workflows for resumable work, channel adapters) over reimplementing them.
Agents are built per request
An agent is a database row, not a module singleton. For every reply, the runtime assembles a Mastra Agent from that row: instructions, model, temperature, enabled actions and procedures, retrieval settings, and contact attributes visible to the agent.
How one reply works
- 1Refuse earlypermission, model, rate limit, credits
- 2Store the user message + a streaming placeholder
- 3Retrieveembed the question, nearest chunks above the floor
- 4Assemble the agent+ numbered knowledge, last 10 messages
- 5Stream over SSEstart, delta…, done or error
- 6Store the reply with its usage event
- Everything refusable is refused first, in the error contract: permission, an unavailable model, rate limits, and the credit reservation. A reply reserves its model's credits in the same transaction that stores it. With none left, it answers
402 OUT_OF_CREDITS. - The user's message and a
streamingplaceholder are stored. - Retrieval: the question is embedded and the agent's nearest chunks fetched (pgvector, HNSW, cosine), filtered by workspace and agent in the query, keeping those above the similarity floor (
RETRIEVAL_SIMILARITY_FLOOR). If nothing clears the floor, the agent is told to say it does not know instead of guessing. - The agent sees the numbered knowledge (for citations) and at most the last 10 messages, always starting on a user turn.
- The reply streams as Server-Sent Events (
startwith citations,activity,ui,delta,doneorerror), with a heartbeat for proxies. Closing the tab aborts the model call, so no tokens are spent on nobody. - The reply is stored
completewith its usage event, orfailed, in which case the reservation is refunded once.
Models and credits
- One gateway: OpenRouter, addressed as
openrouter/<vendor>/<model>. model_catalogis synced daily from OpenRouter's public list (catalog.sync). Only chat models are kept. A model OpenRouter drops is marked unavailable, never deleted.- Credits per reply default to
ceil(blended price / $0.50 per million tokens), minimum 1, with input weighted 3:1 over output. Platform admins can override this per model. :freemodels are offered outside production only.- Every cost-incurring call writes a
usage_eventsrow before it counts as complete. The app role can insert and read that table but never update it.
Embeddings
Embeddings go through the EmbeddingProvider port (OpenRouter today, EMBEDDING_MODEL), at 1024 dimensions. Every chunk records embedding_model and embedding_version. Changing the model means re-embedding everything, never mixing vectors from two models.
Guardrails and insights
Spam, moderation, prompt-injection and topic checks run in one call on the platform's GUARD_MODEL, recorded as guard usage at no credits. They fail open: if the guard call fails, the message goes through. Topic and sentiment reading for analytics uses the same model, recorded as insight usage.
Treat model output as untrusted
A tool argument the model produced is validated like anything that arrives over the network. Custom API actions go through the guarded dialler, and their runs are recorded without their arguments.