CPU embedding decision — 2026-09-09

Ship sentence-transformers/all-MiniLM-L6-v2, using the pinned Qdrant ONNX conversion. The package includes its files and Apache-2.0 license. It works without an account, API key, GPU, or inference-time download. New nodes select this profile by default. This is an English short-memory baseline, not a claim about multilingual retrieval, source-code search, or all CPU architectures.

Measured comparison

AMD Ryzen Threadripper PRO 3955WX, Linux x86_64, Python 3.13.11. ONNX CPU provider, two inference threads, document batches of 16, fresh processes. No GPU was used. Two threads constrain parallelism; they do not simulate an old laptop or ARM board. Query latency uses 96 warmed single-query calls. Model-session loading excludes imports and downloads and uses files already in the OS cache.

The selection plan preceded evaluation. It set a 150 MiB artifact budget and 750 MiB peak benchmark-process memory budget. BGE crossed the memory budget during SciFact, so its large-corpus run was stopped. The benchmarks/results/bge-small-resource-limit.json (generated locally) records that termination. A separate completed memory-only BGE run supplies its latency and memory-fixture metrics. Its memory-only process peaked at 244.9 MiB; that is not comparable to MiniLM's full-corpus peak.

This eliminates that BGE configuration under this declared batch/context budget; it does not establish that BGE cannot run on CPU. Smaller batches or shorter contexts could lower memory use and merit a future separately declared run. There is no BGE SciFact quality score because its corpus encoding did not finish. MiniLM is the sole eligible completed candidate, so the weighted-quality tie rule is not used to claim a head-to-head quality win.

Retrieval results

The frozen memory fixture has 64 original hand-authored snippets, 32 queries, and one relevant snippet per query. It includes confusable distractors and paraphrases. It is a small synthetic regression set, not an independent estimate of real-user accuracy. SciFact is the full BEIR test split: 5,183 scientific abstracts and 300 judged queries. It is external data, but its domain differs from agent memory and model training overlap is possible.

Hybrid uses independent cosine and BM25 rankings with RRF constant 60, matching the implementation's scoring formula. Benchmark IDs break ties deterministically; actual signed-record IDs can order exact ties differently. Hybrid is not uniformly better: it helps on SciFact and hurts on the paraphrase-heavy memory fixture. The CLI exposes --semantic-only and --lexical-only; default text search retains the system's hybrid policy. Weight changes require broader real memory evaluation, rather than tuning against these 32 queries.

The benchmark preserves each downloaded artifact's native truncation. Inspection found that this Qdrant MiniLM tokenizer sets max_length=128, despite the upstream model's commonly documented 256-token setting. The report uses the effective value, not the initial candidate metadata's 256. SciFact abstracts are truncated for this standard whole-document benchmark. Runtime memory ingestion rejects oversized input instead, so these scores do not measure a future chunk-and-aggregate pipeline.

Bundling and operation

Model: upstream MiniLM. Conversion: Qdrant snapshot. Revision: 5f1b8cd78bc4fb444dd171e59b18f3a3af89a079.

The weights are 90,387,630 bytes; all tested model/tokenizer artifacts total 91,334,522 bytes. Licenses and the profile add a small amount. Python and ONNX runtime dependencies are additional disk/memory costs. The generated wheel includes the model, tokenizer, notices and license; it does not include the benchmark dataset or alternative model. Dependency installation still needs packages or an offline wheelhouse; offline inference alone needs no connection.

The embedding namespace is a SHA-256 profile of model files, revision, pooling, prefixes, normalization, dimensions and runtime versions. It uses 384 dimensions, attention-mask mean pooling, L2 normalization, and no query/passage prefix. FastEmbed 0.8.0, ONNX Runtime 1.29.0, and tokenizers 0.23.2 are pinned and checked at load. Every listed asset is checksummed before loading. The encoder uses CPUExecutionProvider, two threads, batch size one, and a specific local model path. Runtime telemetry events are disabled before model construction.

.venv/bin/intertexum --data /tmp/notes init
.venv/bin/intertexum --data /tmp/notes write --text 'Restore the database from backups after disk failure.'
# Publish the returned private ID to make it eligible for shared search:
.venv/bin/intertexum --data /tmp/notes publish PRIVATE_ID --audience '*'
.venv/bin/intertexum --data /tmp/notes search --text 'Recover data after losing a drive'

The 128-token limit includes special tokens. Overlength input fails with a clear split/shorten message; mesh_write_document provides bounded, atomic chunking. Encoding errors never cause fallback to another model. Existing custom-vector nodes retain their configured namespace and require explicit vectors. A future model change requires a new namespace and re-embedding; equal dimension alone does not mean compatible vectors. Remote record signatures do not prove that the origin actually used the claimed embedding model; quarantine approval remains necessary.

Reproduction and evidence

uv sync --locked --extra test
.venv/bin/python -m benchmarks.prepare
.venv/bin/python -m benchmarks.run minilm
.venv/bin/python -m benchmarks.run bge-small --memory-only
.venv/bin/python -m benchmarks.work

prepare downloads the committed revisions and verifies hashes from benchmarks/MODEL_SOURCES.json; it does not select the latest model silently. The exact SciFact archive SHA-256 is 536e14446a0ba56ed1398ab1055f39fe852686ecad24a6306c80c490fa8e0165. The memory fixture SHA-256 is 4f40811ad4ab948495804f0c30ebef02121393d35ec222ee3b4a90288a240cbd.

Raw metrics and per-query rankings are generated under benchmarks/results/, which is excluded from Git and distribution archives. Reproduce them with the commands above; the measured selection summary remains in this document.

The BGE full run can be repeated with benchmarks.run bge-small; the runner does not automatically stop on the memory threshold. In this session it was explicitly interrupted once the observed process memory exceeded the budget. It is not necessary to finish a disqualified configuration to reproduce the decision.

Dataset sources: BEIR and SciFact. Test on the weakest intended CPU and on real, consented memory examples before promising a device-level SLO.