Using the on-device RAG agent (browser)

The site's Testing page (reached from the landing page's Stop 06 CTA) Agent search answers questions about ingested rows with a grounded, streamed response — retrieval, embedding, and generation all run in the page; nothing leaves the device. This doc is the usage guide for reusing that pattern outside the site: what ships where, the module API, the exact recipe, and the constraints that keep it honest.

What ships where

The engine knows nothing about the agent — generation is host-side composition over the same three retrieval calls every other consumer gets.

Models (downloaded on demand, cached, never bundled)

openbmb/MiniCPM5-1B was the default and is gone — a ~688 MB download every visitor paid for before typing anything. Removing it left one preset, which is why there is no picker anymore.

Both hosts now run the same runtime. The embedder stays on the vendored wllama build (site/vendor/wllama/) — transformers.js cannot run that custom bidirectional arch, and the approved embedding model must not change. The agent runs on that same vendored wllama: both shipped answer models are single-file GGUFs (the gguf arch), so llama.cpp is the one generation path and the ONNX/WebGPU answer tier has been retired. A GGUF agent runs on the main thread, NOT the ONNX agent worker: wllama resolves URLs via document.baseURI (a Worker has no document), and it offloads the actual compute to its own sub-worker anyway — exactly how the embedder already runs. Each runtime's cache manager stores its weights after the first fetch.

Both run on the GPU via llama.cpp's WebGPU backend, which is compiled into the vendored builds: every layer is offloaded (n_gpu_layers) whenever a real adapter answers, with CPU as the fallback. Note this is why a GGUF model must still run on the main thread — WebKit exposes navigator.gpu there only, so the existing bridge is what puts generation where the GPU is reachable.

Do not reintroduce an unconditional navigator.gpu stub. The vendored wrapper carries a local patch that makes requestAdapter() answer null, which is how a caller pins CPU (ggml's WebGPU backend can abort mid-decode). It used to be unconditional — and since it replaced navigator.gpu page-wide at import, and the embedder imports wllama on every page, it silently disabled WebGPU for transformers.js and the reranker too. It is now gated on globalThis.__SPEEDYDB_WLLAMA_CPU__, held only for the duration of a CPU load (withWllamaCpuPin) and restored afterwards. The version-lock test asserts the guard token is present.

Both answer models are hybrid reasoners, and handling that is not cosmetic — it decides whether you get an answer at all. This build of wllama parses the reasoning itself and streams it as delta.reasoning_content, never as <think> tags inside delta.content. A model left thinking therefore spends its entire n_predict budget on chunks that never reach the answer, and the reply comes back empty (observed with both models before the fix). So a preset carrying thinking: true makes generateWllama send llama.cpp's chat_template_kwargs: { enable_thinking: false }, which turns reasoning off at the template — measured on Qwen3-0.6B as 39 reasoning chunks + empty content becoming 4 content chunks + no reasoning.

Two cheap backstops ride along in case a build (e.g. the older compat wasm) ignores that kwarg: the preset's optional prompt-level soft switch (noThink, Qwen's /no_think, appended to the system turn), and stripThink — a stream-aware filter (visibleAfterThink) for a model that emits literal <think> tags as content, which re-derives the visible tail from the full raw text on every chunk so nothing leaks even when </think> arrives split across tokens. Any reasoning_content is dropped outright: it is not an answer.

The transformers.js causal arch (site/vendor/transformers/ — the library plus the version-matched ONNX Runtime wasm, self-hosted — inside the module Web Worker agent-worker.js) remains wired for an explicitly-configured ONNX text-generation repo, with the same WebGPU probing and shader-f16 handling as before; no shipped preset uses it. transformers.js is still loaded for the reranker and for the answer models' token-budget tokenizers.

Generation is greedy (temperature: 0) with cache_prompt: false, so each ask evaluates a clean context — without it, llama.cpp's prompt-cache prefix reuse bleeds a stray token from the previous answer into the next. GGUF weights are a single quantized file wllama fetches from modelUrl; like every other model they are never bundled.

The agent module API (site/agent.js)

import { loadAgent, answer, isLowConfidence, NOTHING_FOUND,
         AGENT_MODEL_ID, AGENT_MODEL_BYTES } from "./agent.js";

// Idempotent load; onProgress(loadedBytes, totalBytes) reports the download.
// A cached model reports completion immediately. A failed load may be retried.
await loadAgent((loaded, total) => render(loaded / total));

// Grounded answer: `passages` are plain chunk texts, MOST RELEVANT FIRST.
// The system prompt (applied inside — don't add your own) instructs the model
// to answer ONLY from the numbered passages and to reply exactly
// "nothing found" when they don't answer. onText receives the FULL text so
// far on every streamed token. Resolves to the final trimmed answer.
const text = await answer("is anything coastal?", passages, (soFar) => show(soFar));

// Low-confidence classifier: true for empty answers, anything containing
// NOTHING_FOUND, or hedges ("the passages do not mention…", "cannot find…").
// Deliberately tight: a grounded NEGATIVE ("No, the coastal line does not run
// on Sundays [2]") is an answer, not a hedge.
if (isLowConfidence(text)) { /* widen and retry once — see the recipe */ }

Generation defaults: max_new_tokens: 300, greedy decoding (do_sample: false — deterministic grounded QA), chat template from the model repo. buildMessages and parseGenerated are exported pure, so hosts and tests can exercise the exact prompt and output handling without loading the runtime. Run one generation at a time.

Configuring the models (importing frontends)

Both hosts are configurable by the frontend that imports them — before the first load; configuring after a load throws. Defaults are exactly the shipped models, so zero-config behavior never changes. Each configure* call starts from the defaults (no accidental stacking) and returns the resolved config; agentInfo() / embedderInfo() return the ACTIVE config (the legacy AGENT_* / EMBED_* constant exports always name the defaults).

import { configureAgent, agentInfo } from "./agent.js";
configureAgent({
  modelId: "onnx-community/your-model-ONNX", // any transformers.js text-generation repo
  device: "webgpu",       // a PREFERENCE: the WORKER probes for a real GPU
                          // adapter in its own scope before any weights
                          // download and falls back to wasm+q4, reporting the
                          // final device back (ensureDevice covers only the
                          // inline no-module-workers path) — `navigator.gpu`
                          // existing is not the same as an adapter answering
  dtype: "q4f16",         // default: derived from device (q4f16 gpu / q4 wasm)
  maxNewTokens: 300,
  systemPreamble: "…",    // keep the grounding + "nothing found" contract intact
});
// modelBytes / sizeLabel derive from dtype unless given (they feed the UI)

// Shipped presets (MODELS) set modelId + arch + modelUrl + sizes together — pick one instead
// of spelling out the repo. There is exactly ONE now (MiniCPM5-1B was removed), so this is the
// default and there is nothing to pick between:
configureAgent({ model: "qwen3-0.6b" });   // Qwen3-0.6B · GGUF via wllama · ~0.4 GB
// It flags `thinking` (a hybrid reasoner): the <think> block is disabled via the preset's
// `noThink` soft switch (Qwen's "/no_think") and stripped from the output otherwise — so an
// answer is always just the grounded, cited prose.

// Two model ARCHITECTURES load: "gguf" (a GGUF file run through wllama on CPU — both shipped
// presets) and "causal" (the transformers.js text-generation pipeline in the agent worker,
// for an ONNX repo you configure yourself). A raw modelId with no preset/arch is "causal":
configureAgent({ modelId: "you/your-rag-ONNX" }); // → causal (needs WebGPU in practice)

// The "gguf" arch is opaque (any GGUF file), so it is NEVER name-inferred — set arch:"gguf"
// and a modelUrl. The shipped presets carry their own public HF URLs; point modelUrl at your
// own host to avoid depending on that host's CORS:
configureAgent({ arch: "gguf", modelId: "you/your-model",
  modelUrl: "https://huggingface.co/unsloth/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q4_K_M.gguf" });
// gguf runs on wllama IN THE AGENT WORKER, the same place the ONNX path runs, and uses WebGPU
// where the worker can see it. It used to be pinned to the MAIN THREAD — the vendored bundle
// resolved asset URLs through `document.baseURI`, which throws off-window, so importing it in a
// worker failed outright. Generating with WebGPU blocks whichever thread it runs on, so that pin
// put the block on the UI thread and froze the tab; the bundle now falls back to
// `self.location.href`. Placement follows GPU reachability, not architecture: worker when the
// worker has a GPU, main-thread bridge only when the host has one the worker cannot see (WebKit).
// Greedy (temperature:0), cache_prompt:false for a clean context per ask. To self-host, mirror
// the weights with scripts/mirror-models.sh and let site/model-config.js's agentGgufByKey map each
// preset key to its own file — a single shared URL would send every pick to the same weights.
// ONNX (causal) weights self-host through modelBase like any other model.
//
// isGpuExecError (agent-models.js) classifies a WebGPU/ORT execution failure (a GPU OOM
// mid-generation on the causal path) so the client can say "the GPU couldn't run this
// model" and point at a smaller one, instead of a generic "generation failed".

import { configureEmbedder, embedderInfo } from "./embedder.js";
configureEmbedder({
  modelId: "your/embedding-model", // engine pins the vector space to this id
  modelUrl: "https://…/model.gguf", // a wllama-runnable GGUF
  dim: 1024,
  queryPrefix: "query: ",     // MATCH the model card — wrong prefixes silently
  documentPrefix: "document: ", // degrade retrieval
  poolingType: "cls",
  nCtx: 2048,
});

Two warnings that don't move: the demo itself never overrides the approved embedding model, and the engine wipes stored vectors when modelId changes (ensure_embedding_model — vector spaces must never mix). Swap the agent model only for a checkpoint documented to answer strictly from supplied context.

The recipe

The reference implementation is runAgent in site/app.js; this is the same flow reduced to its engine calls:

const db = /* your WasmBrowserDb (in-memory or durable — see durable.js) */;

// 1 · retrieve — hybrid: word-ranked hits, plus semantic when the embedder is on
let hits = db.findText(question, 50).slice(0, 5);          // {row_id, source, ord, text}
if (embedderReady) {
  const vec = await embedQuery(question);                   // embedder.js
  mergeSemantic(hits, db.findSemantic(vec, 50));            // dedupe by row; cap ~8; keep scores
}

// 2 · nothing to ground on → say so WITHOUT invoking the model
if (!hits.length) return NOTHING_FOUND;

// 3 · first pass — the chunk texts themselves, most relevant first (≤ 5)
let text = await answer(question, hits.map(h => h.text), onToken);

// 4 · low confidence → add breadth: widen each passage to its ±1 neighboring
//     rows (the same expansion the result cards' breadth buttons use), dedupe
//     overlapping windows by chunk text, budget the total (≤ 10 000 chars —
//     the 4k-token window must also fit the preamble, question, and answer),
//     and ask EXACTLY once more. A second hedge is final.
if (isLowConfidence(text)) {
  const widened = hits.map(h => db.expandText(BigInt(h.row_id), BigInt(1)).join("\n"));
  text = await answer(question, dedupeAndBudget(widened), onToken);
}

expandPassages in site/app.js is the dedupe-and-budget reference: it walks hits in relevance order, drops rows an earlier window already contributed, and truncates once the 10k-char budget is spent.

Honesty contracts (keep these if you reuse the pattern)

• Empty retrieval never reaches the model. "nothing found" costs zero tokens and zero download — a nonsense query must not trigger the 700 MB fetch.
• Sources shown = grounding. Render exactly the passages the model received, with their provenance (row · source · #ord), beneath the answer.
• One widening retry, then stop. Each pass costs ~30–60 s of single-thread CPU; a loop would hang the page chasing an answer that isn't there.
• Download on deliberate engagement, never on bare page load. The demo starts it when the search step opens (so the model is there by the first question); a landing-page visit alone must not cost the user ~1 GB.
• Grounded negatives are answers. Don't widen just because the answer says "no" — that's isLowConfidence's job; keep its patterns tight.
• No phase may hang forever. Both the load and the answer phases carry a silence watchdog (agent.js): if the worker stops sending forward-progress messages for the window — a browser can OOM-kill a dedicated worker with no error event, and building a ~1 GB session right after the download is the peak-memory moment — the watchdog rejects, terminates the wedged worker (the only thing that frees a deadlocked GPU session), and clears the load promise so the next ask retries against the cached weights. Measure silence, not elapsed time, so a legitimately slow multi-hundred-MB download is never killed.

Reusing it in another app (e.g. a FloatPlan-style adapter)

1. Copy site/agent.js, site/agent-worker.js, and site/vendor/transformers/ (the library plus the ONNX Runtime wasm — the wasm must match the library's pinned onnxruntime-web version, so keep them together and self-hosted). In a bundled app, npm install @huggingface/transformers in the worker and drop the import shim — the shape is exactly FloatPlan's rag-worker.
2. Bring your own retrieval: any list of passage strings works, but the widening step needs row-addressable neighbors (expandText here; your index's equivalent elsewhere).
3. Keep the module's system preamble, the NOTHING_FOUND contract, the confidence classifier, and the context budget. Swap the model only for another checkpoint documented to answer strictly from supplied context.
4. Keep the pipeline in the worker (copy agent-worker.js alongside; the proxy expects it at ./agent-worker.js). Session load and CPU decode block whatever thread runs them — the main thread is never the right place; the inline path exists only for browsers without module workers.

Limits (honest)

• The GPU is not optional in practice. Same model, same prompt, same output, measured in-browser on a ~450-token RAG context (Apple metal-3, Chromium):

  Qwen3-0.6B	first token	total
  WebGPU	0.96 s	1.3 s
  wasm (CPU)	126 s	145 s

  That is ~130× on first token — and MiniCPM5-1B (measured before it was removed) landed at 0.86 s on the GPU versus ~230 s on CPU. Without cross-origin isolation this host can't thread wasm, so CPU is single-threaded and prefill dominates: usable as a fallback, painful as a default. The GPU is used whenever a real adapter answers.

• The CPU fallback is a real path, not a formality: ggml's WebGPU backend can abort mid-decode on some browsers/devices, so a GPU failure at load or mid-generation retries once on CPU (weights are cached, so it rebuilds the session rather than re-downloading). Force it with configureAgent({ device: "wasm" }) if a device turns out to be flaky.

• findText is an O(rows) demo scan; at real scale put a proper index in front and keep the same passage contract.

• The classifier is pattern-based English; if your product answers in other languages, extend HEDGES accordingly.

• The demo answers from ≤ 5 passages (± widening). It's a context demo, not a long-document summarizer.