The basic package: insert and retrieval, corrected

The smallest shippable SpeedyDb: no chat, no data transformations. It ingests documents, retrieves against them, and hands back assembled context. It has to run in three places with no server of its own — a browser, a WordPress plugin, and a local app.

This is the corrected version of the hand-drawn flows, with the eighteen decisions folded in. Every rule below is normative: where the code disagrees, the code is wrong and the status table says so.

The interactive version of these walkthroughs is on /engine.html, which marks each step as implemented or planned.

The rules that are easy to get wrong

Four of these produce silence rather than errors, which is why they lead.

1. A vector's identity is the whole runtime, not the model name

Two vectors are comparable only if every one of these matches. Measured on nomic-embed-text-v1.5 — same model, same weights, three runtimes:

The declared tolerance is 0.998. Two of the three fail, and nothing errors — a store mixing them just quietly returns worse neighbours. The same sentence lands 0.984 from itself, which is inside the range where genuinely different documents live.

So the model record carries a full space identity, not a name:

nomic-ai/nomic-embed-text-v1.5@0188c9bf…/nomic-embed-text-v1.5.Q4_K_M.gguf
  :Q4_K_M:dim768:mean:l2:cosine:retrieval-prefix-v1:sv1

repo · revision · file · quantization · dimension · pooling · normalization · distance metric · prefix scheme · schema version.

The package embeds on the GPU runtime only. One runtime, one quantization, one dimension — chosen once and pinned. The CPU path exists for development and its vectors are not written to a store the GPU path will read. An insert whose space identity differs from the store's is refused, not converted.

That used to be a rule writers had to remember — ensure_embedding_model(model_id, dim) declared the space, and put then accepted anything of the right length. Dimension is not identity: 768 is shared by nomic-embed-text, bge-base, e5-base and all-mpnet, so vectors from a different model of the same size were accepted in silence and simply returned worse neighbours. Every vector write now names the space that produced it, and the store refuses a mismatch — the caller passes the embedder's identity, never the store's, because reading it back off the store would make the check pass by construction.

2. Scope is a set, not a bit

scope 0|1 cannot express the store tree. The real model is already in rust/src/scope.rs: a Scope enum with a ScopeSet of what a caller may read, plus an optional DocumentSet to narrow further. A row carries its scope; a reader carries its permissions; the intersection is what exists.

Every read materializes rows through store.reader(&scopes). A row the caller may not read comes back as None — indistinguishable from a row that is not there. The unscoped fetch is deprecated precisely so nobody can forget.

For WordPress that maps to: draft / private / published, author id, and post-type — which is exactly the multi-dimensional case a bit cannot hold.

3. Direct file retrieval is scoped like everything else

The file-retrieval path was the one flow with no permission check on it. It gets the same scoped reader as the vector path, applied before rows are read, not as a filter on the way out.

4. Reads are only shared within one scope

Two callers asking the same question may share the query embedding — it is derived from the prompt and nothing else. They may never share retrieved rows unless their ScopeSet is identical. Grouping across identities is how one user's private documents end up in another's answer.

Insert

What changed. Extraction is its own stage rather than hidden inside the parser, because for these targets PDFs and scanned images are most of the real input. The content address is taken before parsing so an identical file under a new name costs nothing. Chunks carry an explicit parent_id assigned by the parser — the breadth process cannot "grab parents and children" without it. .spdb is written before the vector index, and the index is derived from it.

Identity hashes are 128-bit

uint32 is not an identity. Birthday bound on 2³²:

A collision means a chunk is dropped as "already exists" — data loss with no error. A modest WordPress site passes 50,000 chunks without trying. Use hash128 (xxh3-128) from rust/src/dedup.rs, which the dedup path already relies on for exactly this reason.

Write queue

What changed.

• The retry check moved to the failure edge. It was in front of the first attempt.
• "Remove" became "attach". A deduped request that is merely dropped leaves its caller waiting forever. It joins the queued item and receives that item's result.
• "Update existing chronological order" is now explicit: replace the payload, keep the original queue position. Freshest content, fair ordering — the two readings of the old label behave differently under load.
• Admission control. The queue is bounded in bytes, not items, because the constraint is RAM, and a browser tab is where this hurts. A few MB is the budget; a folder upload that exceeds it is throttled at the producer rather than accepted and forgotten.
• Failures are visible. After three attempts a request lands in a durable failure log that the plugin/app surfaces to the user, with the reason and a retry action — not a console line.

The queue nothing called

write_queue.rs was complete and had ten passing tests, and WriteQueue had zero callers outside its own module. Decisions 11 and 18 were both marked implemented on the strength of it. Neither was true in the way that matters: nothing bounded what could be in flight, and no user had ever seen a dead letter, because nothing produced one.

That is a specific kind of wrong worth naming — not a bug in the code, which was fine, but a claim about the product resting on a module the product does not reach. A test suite cannot catch it, because the tests exercise the module directly and pass.

Wired to POST /ingest, it answers three questions before any work happens:

The ceiling is real only if the accounting is. Bytes stay charged from admission until complete/fail, so work in flight counts too — a bound that released on dequeue would bound only the waiting, which is not the hazard. Every exit from the handler settles its ticket; one that did not would leak bytes until the server refused everything, which is a leak that looks like load.

Why this server needs it at all: it is thread-per-connection behind a single service mutex, so every concurrent /ingest holds its whole document in memory while it waits its turn. A hundred simultaneous 50 MB uploads is 5 GB parked on thread stacks, and the failure was the process dying rather than any request being told no.

Dead letters are readable at GET /ingest/dead-letters and never carry the payload — a failure log that quotes the document is a second copy of it, in a place with different retention and different eyes on it.

Insert steps

What changed.

• chunk yes, embedding no is no longer an error. That is precisely the state after a model change or a failed embed, and raising there made migration impossible without manual repair. It enqueues for embedding.
• chunk no, embedding yes is an orphan, not an error either: drop the vector and let the index rebuild. It cannot be authoritative, because .spdb is.
• .spdb is written first. The vector index is derived, so a crash between the two leaves a recoverable state — rebuild the index — rather than an orphan vector nothing points at.
• Locks are leases with a TTL. The old flow unlocked only on success, so a crashed worker held a row forever and the retry counter turned that into a lost document. A lease that has expired may be stolen; the previous holder's write is rejected on return.

Update and remove

Missing entirely from the original, and the first thing a WordPress plugin needs — edit a post, trash a post.

Two rules worth stating out loud:

• An edit re-embeds only what changed. Chunks are content-addressed, so a typo fix in paragraph nine costs one chunk, not the document.
• The Active filter lives inside the index query. Applied afterwards it is one forgotten WHERE away from serving deleted content, and a soft-deleted row still costs search time.

Model transition

Its own process, deliberately not a queue job — nothing is rewritten in place.

The live index answers every query until the new one is complete. There is no window where a query is served from a half-migrated store — which is the failure the 0.984 measurement predicts, and the reason a rolling in-place re-embed is not an option. Different dimensions are fine: the two indexes are separate until the flip, so a 768-dim store can transition to a 1024-dim one with no schema change and no downtime.

Retrieval

The query embedding step was missing. Insert embeds; retrieval hashed the prompt and went straight to the store. Vector search needs a query vector, and the prefix asymmetry is part of the contract the goldens pin — a query embedded with the document prefix lands somewhere else.

Read queue

The old flow grouped requests from different users. Sharing retrieved rows across scopes is a data leak; only an identical ScopeSet may share a result. The query embedding is derived from the prompt alone and may always be shared.

Read steps

What changed.

• Top-k, not a similarity floor. "60%+ semantic match" is per-corpus fragile — cosine distributions differ by corpus and by model, so one cutoff returns everything on one and nothing on another. This has already bitten once here: a fixed keep-threshold of 0.7 dropped every passage and had to move to 0.35 once the actual score distribution was measured. A rank is stable where a similarity number is not.
• Term selectivity is worth naming. Skipping query terms that appear in more than ~15 % of documents took one prose query from 158 s returning 0 passages to 2.84 s returning 40.
• Breadth after rerank, and this ordering is load-bearing. Anything added before the cross-encoder is judged on whether it resembles the query — measured: a passage that answered the question but shared no vocabulary with it scored 0.015, against 0.015 for an unrelated control. Passages added after the reranker survive; the same passages added before it do not.
• Neighbours inherit the parent's rerank score. The token-budget stage drops "the lowest quality matches", which needs every passage to have a score — breadth-added ones have none of their own.
• The basic package returns context and stops. No chat means no generate/validate/regenerate loop. (When that loop does exist it needs an iteration cap; nothing else in the design lacks one.)

Read steps — direct file retrieval

Order corrected: the manifest was being read after the rows, but it is what declares BytesPerRow and SeekPoints — you need it to read them. And scope resolution happens at the key lookup, so an unauthorized file is "not found" rather than filtered later.

Storage is an interface, not Postgres

Pg Vector cannot be the store for any of the three targets — a browser has no Postgres, and neither does a shared-host WordPress install. One interface, three backings:

The lock semantics are identical everywhere — a lease with a TTL that may be stolen after expiry. Only the mechanism differs.

Schema

One vector table, one space at a time. Different models transition by building a second index, whether or not the dimensions match.

vector

space (was "model lookup") — space_id, repo, revision, file, quantization, dim, pooling, normalization, metric, prefix_scheme, schema_version, runtime, active. Every field participates in comparability; a store refuses a write whose space_id differs from its own.

spdb_key, metadata, manifest are unchanged from the drawing.

Open: chunk size

Not decided, because it should be measured rather than argued. Chunk size affects retrieval quality more than most of the rest of this document, and the right answer is document-type dependent — a transcript, a contract and a code file do not want the same boundaries.

The experiment: ingest the same corpus at ~200 / ~400 / ~800 tokens with ~15 % overlap, score each with graph-eval/rerank_eval on a labelled set, and check whether one size wins outright or whether two indexes (fine for precision, coarse for context) beat either alone. The harness for this already exists; only the run is outstanding.

Implementation status

Honest as of this document. "Design" means the rule above is the intended behaviour and the code does not do it yet.

A region that cannot be placed is not a citation

The extraction stage has a second phase behind it: show the document, let the user select part of it, and map that selection to a value that feeds a table. That is a requirement on the first phase, because the geometry has to be captured while the document is being extracted — a text-only extractor discards it, and adding it later means re-extracting everything ever ingested.

SourceRange::Region { page, x, y, w, h } already existed for OCR. It was unplaceable: pixel coordinates with no record of what they were measured against. The same paragraph is at x=300 when a page was rasterised at 150 DPI and x=600 at 300, and a viewer handed the number alone cannot tell which it got. Nothing errors — the highlight just lands somewhere plausible and wrong. This is the vector-identity failure in a different costume: the numbers survive, their meaning does not.

So a region now carries the frame it was measured in, and normalized_region() returns fractions of the page — the form a viewer can draw at any zoom:

The fields are Option only so regions written before this still load; anything writing one now must fill them, which the compiler enforces at every construction site. normalized_region() returns None rather than assuming a page size, because a highlight drawn in the wrong place is worse than no highlight.

The OCR path did not have the dimensions to record — OcrRegion never carried them — so it gained a frame field too. Its existing regions are therefore unplaceable and honestly report themselves as such.

The index did not rebuild from .spdb

Decision 8 says .spdb is the source of truth and the index is derived from it, "so a crash between the two leaves a recoverable state — rebuild the index". The write ordering was right; the claim about recovery was not, and it took four lines to disprove:

ingest a document → close → delete the provenance sidecar → reopen → search
rows in .spdb: [1]        hits: 0

rebuild_index walked the sidecar, not the content store. A row whose provenance never landed was present in .spdb, invisible to every reader, and unrecoverable — a document the store held and could not find, which is the exact failure the decision was written to prevent.

Recovery now runs on open: rows in an active source with no provenance are re-indexed from the content store, which still knows each source's dedupe key — the source_file_id, and therefore the citation. What cannot be reconstructed (file hash, MIME, the derivation's range) is left empty rather than guessed, and the derivation is marked provider = "recovered-from-spdb" so a reader can tell a reconstructed row from an ingested one. The store also says so on stderr, because a store that quietly repairs itself hides the crash that made it necessary.

A soft-deleted source is not an orphan and is never resurrected — otherwise a deletion would come back on the next restart.

The browser extractor guessed the page size

The same failure, found while wiring the extractor to it. extractDocumentGeometry already returned positioned text runs per page — and a width/height beside them that were not the page's size. They were the bounding box of the text plus 24pt of padding, floored at 200:

return { width: Math.max(200, Math.ceil(maxX + 24)), … }   // "page box from the run extents"

A page whose text sits in its top half reported half a page, and a highlight placed against it landed in the wrong place while looking entirely plausible. Nothing caught it because nothing consumes the function yet — and its own doc comment already promised the page box, so the documentation had been right and the code wrong for as long as both existed.

The declared box now comes from CropBox, else MediaBox, inherited through /Parent (producers routinely set it on the tree node rather than every leaf), and is null when the PDF declares none. The text extent survives under its own name, contentWidth/contentHeight, so a renderer with no declared box can still lay something out without being able to mistake one for the other.

Where PDF extraction runs

There is one PDF parser in this repo and it is in the browser (site/extract.js: page tree, content streams, filter chains, encryption, CMap glyph decoding, positioned runs). Native has none, and docfile.rs says so. That is a decision, not an omission, and it is worth writing down because the omission and the decision look identical from the outside.

Today extraction is the client's job. POST /ingest accepts pages — an array of page strings — so anything that can read a PDF can feed the server without the server knowing what a PDF is. The browser does it; so did a 30-line node script, against a real 74-page manual.

Server-side upload is planned. When it lands, the server accepts the bytes and extracts them itself. Two things follow from that, and both are constraints on work done now:

• pages stays the ingest contract. A server-side extractor becomes one more producer of pages, not a second ingest path. Nothing about the route needs to change to allow it.
• The engine choice stays open, and it is not "port extract.js to Rust". The browser cannot use a native engine (wasm), so extract.js survives every option — porting it means maintaining two implementations of a format whose whole difficulty is its long tail, with each one's job being to agree with the other. The cheaper shapes are a node sidecar running the extractor we already have (one implementation, no new parser) or pdfium (a native library in the image, but real glyph metrics). Pick pdfium only if the select-a-region-and-map-it-to-a-column feature needs tight boxes — chars × 0.5em is fine for highlighting a line and not fine for a table cell. That is a requirement question, not an ingest question.

The running head was in every chunk

Chunking a paginated document per page put each page's furniture into every one of its chunks. On the 74-page radio manual the first line of all 74 pages is GX1280SPage 41 — so every passage /context returned opened with the model number and the page number, where it matched every keyword query, diluted every embedding, and spent part of the cross-encoder's 512 tokens before reaching any content.

Nothing is lost by removing it. The page number is already recorded structurally, in SourceRange::Pages, and surfaced on the passage — it is better as a field than as a sentence the retriever has to learn to ignore.

Detecting it is two rules, and the second one is the interesting one:

That second rule is what makes the strip safe to run automatically. Without it, a table continued down the edge of every page has exactly the shape of a running head. With it, Page 5 of 12, Page 6 of 12 (offset from the PDF index, as in any document with unnumbered front matter) is recognised and Channel 16, 68, 9 is left alone.

Three more guards, because a stripper that is wrong deletes content and says nothing: only the outermost 3 lines of a page are candidates and stripping stops at the first line of real content; a line must open or close 60 % of at least 5 pages; and what was removed is reported in the ingest note, not done quietly.

Implemented in both document_normalize.rs and speedydb-client.js, which have to agree — the same document ingested in the browser and on the server must produce the same chunks.

Chunk size, measured

Decision 17 asked what the parser should chunk to, and answered "that will depend on the document types — test some sizes." Tested: the full 5,183-document SciFact corpus re-ingested at each size into its own store, same labelled eval each time. Separate stores rather than separate scopes in one store, because the keyword index's term statistics are global and four copies of the corpus would have changed the scoring for every arm.

Monotonic up to 1500 on every quality axis at once — bigger chunks retrieve more gold, hand the model fewer passages, and produce fewer answers built on no gold at all — and then it stops. 1500 → 3000 moves gold-in-context not at all (47/60 both) and misleading answers not at all (13 both), for 10% more latency.

That plateau is the informative part. If the rule were "bigger is better" the curve would keep climbing; it flattens exactly where a chunk starts holding a whole document (1.0 chunks per doc at 3000), because past that there is nothing further to stop splitting.

The mechanism is not "bigger is better", it is "don't split the unit of evidence". SciFact documents are abstracts averaging ~1,300 characters, so at 1500 chars a document is 1.5 chunks and at 200 it is nine. The gold label is attached to the abstract; fragmenting it hands the reranker a piece of the evidence and asks whether that piece answers the question. It ranks lower, because honestly it does. Nothing about this says a 50-page PDF wants 1500-character chunks — it says the chunk should be the document's natural unit of meaning, which is exactly the "depends on the document types" the decision called for. A corpus of long documents needs its own run; this one cannot answer that, because its documents are shorter than the largest chunk tested.

Two costs worth stating with it:

• Latency scales with chunk size, 1.22 s → 3.79 s across the range, because the cross-encoder reranks longer sequences. The quality win is real and so is the 3×.
• The reranker truncates at 512 tokens (MAX_SEQ_TOKENS), and 1500 characters is already ~375. Past roughly 2,000 the reranker stops seeing the end of a chunk. The 3000 arm does not test that, and should not be read as evidence it is harmless: SciFact abstracts are ~1,300 characters, so a 3000-char chunk still holds ~1,300 characters of text and truncation never engages. Locating that ceiling needs a corpus of genuinely long documents.

These arms are keyword + rerank only — embedding 5,183 documents four times over was the expensive part. The vector leg was then confirmed separately on the 1500 store, and it is now worth having: 47/60 keyword-only → 50/60 with vectors fused. Notably it added nothing on the same corpus with the previous reranker, which is a second place the reranker choice showed up.

Ingest, as deployed

POST /ingest — {source_id, text | chunks, scope?, chunk_chars?, overlap_chars?} — is live on api-rag-small. Idempotent by source_id: a repeat writes nothing and returns the rows it already has. POST /ingest/remove deactivates every row of a document, and reports removed / already_removed / unknown_source_id rather than a bare count that reads as success either way. Verified end to end by scripts/check-ingest-lifecycle.py: round trip, idempotency, scope in both directions, malformed scope refused, and removal checked through both retrieval legs.

That last clause is the whole point of the check, and it was added after a bug. Removal was previously called "verified" on the strength of the endpoint returning 200. It did — while the document stayed retrievable: keyword search filtered soft-deleted rows and the single-row fetch behind the vector leg did not, so a removed document stopped matching on words and kept coming back semantically. A deletion is only verified by asking for the thing afterwards, by every route that could answer.

Two limits worth stating plainly.

Retrieval is now hybrid. POST /ingest embeds what it stores, POST /ingest/backfill embeds rows that predate the embedder, POST /ingest/vectors accepts client-embedded vectors and refuses any whose space_id is not this store's (409). The vector file is the one the browser writes, so a store moves between them.

And on SciFact it changed nothing. All 6,628 rows embedded (118 s, ~56/s), then the same labelled eval:

The vector leg is live — it retrieves more candidates — but the same gold documents reach the context. The binding constraint is downstream, and it is the same one graph RAG hit: a passage that answers the question without sharing vocabulary with it scores 0.088 on the cross-encoder, against a keep_threshold of 0.35, and is dropped. Demonstrated directly: a query for "what does the impeller do" returns a lexically-matching document and discards the semantically correct one the vector leg surfaced.

So first-stage recall is not what limits this pipeline. Two independent retrieval mechanisms now say so.

The gate was the bottleneck, and it is fixed

With retrieval cleared of suspicion, the next question was whether the gate — the rule deciding which reranked passages reach the model — could be fixed at all, or whether the cross-encoder itself was wrong for the domain. scripts/rerank-gate-sweep.py answers that before tuning anything, by asking where the gold document sits in the reranked pool:

The cross-encoder was ranking the answer first and the absolute keep_threshold was then discarding it. 15 of 55 reachable golds, thrown away after being correctly identified. The ordering was never the problem; the claim that its scale is meaningful was.

Three replacements were measured on the whole pipeline, plus a fourth thing no IR set can measure. Every labelled set — SciFact, BEIR, MS MARCO — has an answer in the corpus for every query, because that is what makes it relevance-labelled. So a gate can be tuned to a perfect score on any of them while having quietly lost the ability to say "nothing found." scripts/check-grounding-refusal.py asks ten ordinary questions from domains this corpus has no coverage of, and counts refusals.

Nine queries that used to be answered "nothing found" — from a store that held the answer — are now answered correctly, and the number of queries handed irrelevant context did not move. Both columns matter: driving "returned nothing" to zero is trivial and worthless if it just converts correct refusals into confident wrong answers, which is exactly what min_keep 1 does (11 → 18).

Two things had to be fixed for this to be safe, and both were latent bugs rather than tuning:

• Weighting::Relative structurally disabled the grounding guard. Min-max maps a pool's best member to exactly 1.0, so every keep_threshold ≤ 1 keeps it and the pipeline can never refuse. floor_threshold now reads Passage::abs_weight — the calibrated sigmoid, kept alongside the ranking weight — so "this whole pool is irrelevant" stays expressible on either scale. The floor of 0.15 sits between the highest an off-topic query scored (0.128) and the 10th percentile of gold (0.161); at 0.10 it leaked 2 of the 10 off-topic questions back through.
• A pool with one candidate was min-maxed to 0.0 and dropped by any threshold — a store with exactly one matching passage answered "nothing found" while holding the answer. It now falls back to the absolute sigmoid when the pool has no spread. Rare on 7,626 chunks; routine in the browser package, where a user has uploaded four files.

Both defaults live in PipelineConfig::default and DEFAULT_PIPELINE_CONFIG, and the two are pinned to each other by a test — a browser package that gates differently from the server is one product giving two answers to one question.

These numbers are a calibration, not a constant. They were measured on SciFact with jina-reranker-v1-turbo-en. Another reranker has another scale, and the floor in particular is a claim about this model's sigmoid. Both scripts print exactly the numbers they were chosen from. Set floor_threshold to 0 when running a reranker out of its domain — that is the case Weighting::Relative was built for, and there the absolute scale carries no information at all.

The floor is over-fitted to SciFact's genre, and no threshold fixes it

The floor was derived from two measured numbers: SciFact gold at p10 = 0.161, and off-topic queries topping out at 0.128. That gap is real for SciFact, and it is an artefact of the genre — SciFact pairs a scientific claim with a scientific abstract, so gold passages overlap the query lexically by construction. Probed with correct answers that share no content word with the question, on a store scoped so each was the only candidate:

Retrieval surfaced all five — the vector leg works. The gate discarded four. And the scores sit inside the off-topic range (0.015–0.128): a correct Mona Lisa answer scores 0.039, while an off-topic query's best match scores 0.128. The two distributions overlap, so no floor keeps correct paraphrases and refuses off-topic questions. This is the same structural finding as graph RAG: when the model scores the right answer and an irrelevant passage the same, the problem stopped being calibration.

Widening that probe from 5 pairs to 30 made it worse, not better: 26 of 30 correct answers scored at or below the best an unanswerable question could pull up. There is no refusal threshold on this reranker. So the floor was retired as a refusal mechanism (floor_threshold now defaults to 0) and replaced by the two changes below.

Resolution: answer, cite, and say how sure you are

The product now answers with the best passage it has, cites it, and reports confidence rather than refusing. Refusing requires a bar the measurement says cannot be drawn; labelling does not. /context and /generate return confidence per passage and for the answer, the assembled prompt always instructs the model to cite, and a low pool adds an instruction to open by saying it is not confident.

Confidence bands are measured per reranker (scripts/measure-confidence-bands.py) against two populations — correct answers sharing no content word with the question, and ordinary questions the corpus cannot answer. Where they overlap is what low has to mean. That measurement also settled which reranker to run:

So the ceiling was never inherent to cross-encoders — it was that model. bge separates the two populations almost completely, which is what makes a confidence label mean something; jina does not, and cannot support the label at all. Deployed: bge on the server path (its own GPU), jina in the browser, bands per model and printed at startup next to the model they were measured for. Verified live — 50/60 on labelled SciFact, and 10/10 unanswerable questions answered and labelled low.

The browser leg is the open problem. It runs jina, whose bands are degenerate, so it will label nearly everything low — honest, but close to useless as a signal. Either a browser-capable reranker that separates, or the browser states confidence only when it can.

A reranker that fails must not return numbers

Found while measuring: bge placed on the GPU that already held the answer model and the embedder (6.7/8.2 GB) could not allocate a cuBLAS handle — CUBLAS failure 1: the library was not initialized. The pipeline kept going, because a failed batch scored every passage 0.0 on the reasoning that the keep threshold would drop them. It does not: sigmoid(0.0) is 0.5, which clears most thresholds. A dead GPU produced a pool of confidently mid-ranked passages, and a 60-query calibration run came back as sixty identical 0.500s that looked exactly like data.

Reranker::rerank now returns Result. A failure produces no passages, a degraded reason, and a 503 — deliberately not folded into Confidence::Low, because "this passage is a weak match" and "the GPU stopped working" are different facts with different fixes, and only the first is about the caller's question.

A scope on a request is not an authenticated identity. /context and /generate now read with the scope the body names, so an ingested user:alice document is invisible to a query scoped elsewhere — but nothing proves the caller is alice. A multi-tenant deployment must put authentication in front and derive the scope from that, never from the body. SPDB_REQUIRE_SCOPE=1 at least makes an unscoped query a 400 rather than a full read.