Dicklesworthstone/frankensearch
Dicklesworthstone/frankensearch has 94 GitHub stars and 23 forks. GitHot measured 0 stars in 24 hours and +2 in 7 days. Last checked 2026-11-15 16:30 UTC. Last push 2026-11-15.
What you can do
- Progressive search phases (Initial, Refined, RefinementFailed; library searchers with a reranker also emit Reranked)
- Agent-friendly streaming (--stream) with machine-readable output
- Result explanation surfaces (fsfs explain <rank|R-id|path> against the last search)
- Multiple output formats: table, json, jsonl, toon, csv
Measured star growth
- in 24h
- 0
- in 7 days
- +2
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Two-tier hybrid local search for Rust and the fsfs CLI: sub-millisecond first-pass results from a Potion/Model2Vec fast tier, then MiniLM quality refinement. Fuses native pure-Rust BM25 (Quill engine) and semantic vector search with Reciprocal Rank Fusion. Progressive iterator API, f16 SIMD vector index, feature-gated compilation.
Developer ToolsREADME.md
Originalfrankensearch
Two-tier hybrid local search for Rust and the fsfs standalone CLI: fast first-pass results, then quality refinement.
Quick Navigation
- Install In One Line
- Quick Start
- How It Works
- Architecture Breakdown
- Algorithms Used
- Library Integration Quickstart (Rust)
- Feature-Flag Decision Table
- Config Recipes
- Troubleshooting by Symptom
- Reference Appendix
- Key Source Files
- Glossary
- Knob Impact Matrix
Install In One Line
curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/frankensearch/main/install.sh | bash -s -- --easy-mode
The installer verifies every downloaded archive before replacing a binary. A
missing or malformed checksum, unavailable SHA-256 tool, or mismatch is a hard
failure. If the platform has no full semantic release artifact, the ordinary
path builds the loader-capable default from source; it never silently installs
the model-free lite profile. Use --lite only when that reduced capability is
intentional: lite indexes, watches and searches by keyword only (its index
has no vector tiers, so a running fsfs watch does not block searches from
other processes). Intel macOS is the explicit exception:
the pinned ONNX Runtime has no supported x86_64 Darwin distribution, so
ordinary semantic installation fails with unsupported_platform and points to
--lite instead of attempting a source build that cannot succeed.
The Linux x86-64 full archive requires glibc 2.39 or newer (Ubuntu 24.04 and
later) from fsfs 1.12.2, which is built inside Ubuntu 24.04 by
scripts/build_fsfs_gnu_full_release.sh; 1.10.0 through 1.12.1 required 2.43.
A source build works from glibc 2.38, the floor of the bundled ONNX Runtime
archive; older systems have no semantic build yet and the installer points
them to --lite
(#62). The standard installer
provisions and verifies the two default search models, Potion and MiniLM,
requiring roughly 621 MB. Install optional models separately with
fsfs download-models MODEL. Offline installation requires the default pair
to be cached and verified; it never downloads missing models.
For networks that require a proxy, export the proxy settings
before running the installer so its model-provisioning subprocess inherits them.
When upgrading a Linux full installation from 1.9.0 or earlier, use the installer
above: those executables' fsfs update selects the MUSL lite archive and loses
semantic support. Older Apple Silicon lite installations should use
install.sh --lite to retain their profile. The 1.9.1 updater preserves both
the compiled ABI and the full/lite profile for subsequent updates.
Semantic indexes written by 1.7/1.8 require an explicit rebuild with the original
configuration: fsfs index /original/source --index-dir /existing/index.
Indexes written by 1.9.0 can be opened directly by 1.9.1.
The 1.10.0 release changes the embedder adapter version and therefore
requires that explicit rebuild for 1.9.x semantic indexes too. Keep the original
documents and configuration available; opening an old generation with the new
producer is refused. Model artifact checksums and native producer fingerprints
are unchanged, but the fast Potion tier has a new adapter fingerprint.
Installer goals:
- zero-friction first run
- auto-configured model cache path
- sane defaults for interactive usage
No Windows archive has been published yet: every release through v1.12.3
ships Linux and macOS assets only. When the Windows x86-64 full archive
ships, it will require the
Microsoft Visual C++ v14 x64 Redistributable.
Install the current supported x64 package if the required MSVCP140 or
VCRUNTIME140 DLLs are missing. Keep the archive's DirectML.dll beside
fsfs.exe; that runtime and its license are included in the archive.
Cargo Install (Developer Path)
fsfs currently builds from this workspace and uses the pinned nightly toolchain
(rust-toolchain.toml). Its default feature set compiles the Model2Vec and
FastEmbed loaders, but keeps the large model bytes outside the crate. Normal
workspace and crates.io builds therefore need no model artifacts at compile
time. The documented default path is loader-capable without any feature flags:
cargo +nightly install --path crates/frankensearch-fsfs
fsfs download-models potion-multilingual-128m
fsfs download-models all-minilm-l6-v2
fsfs download-models potion-multilingual-128m --verify
fsfs download-models all-minilm-l6-v2 --verify
fsfs version
fsfs status --format json
fsfs doctor --format json
For an uninstalled development binary, use
cargo build --locked -p frankensearch-fsfs --bin fsfs with the repository's
pinned toolchain, then run target/debug/fsfs with the same model setup.
The Linux DSR quality lane builds in a clean checkout, installs with
cargo install --locked --debug --path crates/frankensearch-fsfs --root <private-root>,
and verifies that the installed executable matches the Cargo build byte for byte.
It uses fresh HOME/XDG directories and checks repeat indexing, both vector
generations, ranked hybrid and vector-only search, and actual Initial/Refined
result records. Its executable quick-start
driver retains each run's artifacts and exercises model, result, authority,
timeout, child/listener and provenance failures. It needs Python 3.11+ and
strace for the warm-query network check.
Replay it with scripts/check_fsfs_executable_quickstart.sh --negative-probes --require-source.
--binary /absolute/path/to/fsfs checks a supplied executable as a smoke test;
its SHA-256 is recorded, while its source revision remains explicitly unknown.
The checker checkout revision cannot establish where that binary was built.
The two downloads use revision-pinned manifests and verify every file before
atomic promotion. Explicit --verify is fail-closed: a missing or corrupt
registered cache returns a typed nonzero error rather than a successful payload.
fsfs status reports manifest states (missing, incomplete, mismatch, or
verified); only fsfs doctor also opens each verified cache through the
compiled Model2Vec/FastEmbed loader. A hard doctor verdict exits nonzero with
one stable subsystem_error report with the failing checks in its context. If the cache is absent or offline mode
forbids acquisition, indexing fails with an actionable typed error; it never
substitutes hash control embeddings for semantic results. Use
--no-default-features alone produces the model-free lite binary. Add
--features semantic-native for the native semantic profile described below.
Native quality embeddings in fsfs
The default fsfs build can use pure-Rust F32 MiniLM for its quality tier:
fsfs download-models all-MiniLM-L6-v2-native
Set quality_model = "all-MiniLM-L6-v2-native" in the [indexing] section of
your fsfs configuration, then rebuild the index with that configuration.
Indexing, search, append and watch use the selected native producer;
fsfs status verifies its artifacts and fsfs doctor loads the native model.
The CLI supplies and drains its existing blocking pool. Library users of
FsfsRuntime attach their pool with with_native_blocking_pool.
Use an optimized (--release) executable for native inference. Cold model
loading can exceed the default 500 ms quality deadline even in a release build,
depending on the host, and return Initial results with a refinement-timeout
explanation.
In a daemon or TUI, successful initialization is retained even if that first
query times out, so later queries can reuse the loaded model. Each query still
has its own deadline and must match the index's producer identity.
Native and ONNX models have separate installation directories and producer identities. Changing this setting requires re-embedding the corpus. A missing or invalid native quality model leaves a new index fast-only; an existing native quality generation rejects incompatible models and preserves Initial results when refinement fails. The standard build's default quality model remains ONNX.
For an ONNX-free source build, use the explicit native profile:
cargo build --release -p frankensearch-fsfs --locked --no-default-features --features semantic-native
target/release/fsfs download-models
target/release/fsfs doctor
target/release/fsfs index ./documents
target/release/fsfs search "your query"
This profile includes Model2Vec, native quality inference and native reranking,
without the fastembed or ort dependency. It defaults to English native F32
MiniLM; bare download-models provisions the configured fast and quality models.
The optional reranker still needs fsfs download-models ms-marco-minilm-l-6-v2.
Existing ONNX quality generations need re-embedding with the native producer.
Enabling both semantic-native and semantic-loaders keeps the standard ONNX
default. Native-only builds update from source; fsfs update refuses to replace
them with a standard/lite release archive or an unclassified rollback backup.
fsfs update --check and rollback listing remain available. The profile cannot
initialize ONNX models; existing valid cached answers or an explicitly selected
daemon can still serve results from their attested producer.
Opt-in multilingual native embeddings
The pure-Rust native feature also supports the 384-dimensional
paraphrase-multilingual-MiniLM-L12-v2 model for CJK and mixed-language
corpora. Acquisition and activation are both explicit; it is not part of the
default download set and is never selected merely because it is installed:
fsfs download-models paraphrase-multilingual-minilm-l12-v2
fsfs download-models paraphrase-multilingual-minilm-l12-v2 --verify
For fsfs, select it in your configuration and build a fresh index:
[indexing]
quality_model = "paraphrase-multilingual-minilm-l12-v2"
The default fsfs build uses the native multilingual loader for indexing, search, append, watch and doctor with this selection. Use an optimized executable and daemon-backed search: cold initialization can exceed the 500 ms quality budget, while the daemon retains the completed model for subsequent queries.
Library callers compiled with --features native load the verified model
directory with NativeEmbedder::load_multilingual(...) (or the corresponding
NativeEmbeddingModel variant). This producer has a distinct frozen identity
from all-MiniLM-L6-v2, despite sharing its 384-value output dimension. An
existing semantic index therefore cannot be reused or mixed: switch the model,
re-embed the complete corpus, and atomically publish that backfilled generation
before serving queries from it. Identity checks fail closed if vectors from the
two spaces are combined.
Opt-in smaller fast models
The fast tier defaults to potion-multilingual-128M (256 dimensions, 512 MB of
weights, a 500k-piece tokenizer). Two English-only Model2Vec models are
registered as smaller, explicitly selected alternatives: potion-base-8M
(256 dimensions, 30 MB) and potion-base-32M (512 dimensions, 129 MB). Like
the opt-in quality models, they are never downloaded or selected implicitly:
fsfs download-models potion-base-8m
fsfs download-models potion-base-8m --verify
[indexing]
fast_model = "potion-base-8M"
Then rebuild the index with that configuration. Each model is a distinct
vector space with its own producer identity: an index built with one is refused
by another, and an unrecognized fast_model is a configuration error rather
than a fallback. Both apply the reference model2vec rule of dropping the
WordPiece [UNK] token before pooling.
Cold fsfs search --no-daemon on a 56-file corpus, release build, both tiers
delivering REFINED, models installed with verification receipts (as
fsfs download-models leaves them), median of seven interleaved runs with
distinct queries (2026-09-23, Threadripper PRO 5975WX):
| Fast model | Cold search | Peak RSS |
|---|---|---|
| potion-multilingual-128M | 0.75 s | 0.83 GB |
| potion-base-32M | 0.47 s | 0.42 GB |
| potion-base-8M | 0.41 s | 0.31 GB |
These are English models. On three English BEIR sets they rank as well as the multilingual default once results are refined, and as well or better before that: nDCG@10, INITIAL / REFINED, one release build and otherwise identical configuration, paired over each set's test queries (2026-09-24):
| Fast model | SciFact | NFCorpus | ArguAna |
|---|---|---|---|
| potion-multilingual-128M | 0.590 / 0.679 | 0.291 / 0.331 | 0.336 / 0.357 |
| potion-base-32M | 0.655 / 0.678 | 0.307 / 0.332 | 0.342 / 0.358 |
| potion-base-8M | 0.604 / 0.672 | 0.306 / 0.333 | 0.340 / 0.358 |
No REFINED difference from the default is significant (95% paired bootstrap;
the largest is 8M's -0.007 on SciFact). In INITIAL, 32M is ahead on SciFact
(+0.066) and both are ahead on NFCorpus (about +0.015). Non-English text was
not tested, and it is what the multilingual default is for. The comparison
is docs/quality_harness/fsfs_beir_product_eval.py compare.
Library callers select one with
Model2VecEmbedder::load_registered(dir, &RegisteredModel2Vec::potion_base_8m()?).
To build the full binary profile with Potion and MiniLM embedded, provision the revision-pinned inputs and select the feature explicitly:
scripts/rch-ensure-deps.sh --models-only
scripts/rch-ensure-deps.sh --models-only --check
cargo +nightly install --path crates/frankensearch-fsfs \
--no-default-features --features embedded-models
Provisioning validates every artifact's byte length and SHA-256. Cargo's build script performs no network access. The embedded profile changes distribution, not retrieval semantics: it uses the same loaders and registered model identities as the default source build.
Quick Start
The standard installer provisions the two semantic search tiers, using roughly 621 MB of pinned model artifacts. Other registered models are optional. First setup time depends on network speed. Later starts reuse the verification receipt while the exact manifest and file states remain unchanged.
# 1) Install
curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/frankensearch/main/install.sh | bash -s -- --easy-mode
# 2) Acquire and independently verify both registered semantic tiers
fsfs download-models potion-multilingual-128m
fsfs download-models all-minilm-l6-v2
fsfs download-models potion-multilingual-128m --verify
fsfs download-models all-minilm-l6-v2 --verify
# 3) Index a directory
fsfs index ./my-project
# 4) Search
fsfs search "how does retry backoff work" --limit 5
Plain fsfs index <path> is a one-shot operation: it seals the generation and
exits. Use fsfs watch <path> or fsfs index <path> --watch only when you
explicitly want a long-running incremental watcher. A new index uses the
complete-generation layout (see "Complete generations" below), which serves
searches while it is watched: a watched change is searchable from another
process within seconds, and retention keeps the active generation and one
predecessor, which share their unchanged keyword segments (a 19,480-file
store held 186 MB after 8 watched edits, about one fresh build). On that tree a
one-line edit is published in about 1.8–2.0 s (median per round, lite release
build), against about 0.7 s for the legacy watcher to commit it. On Linux, running
fsfs index, fsfs watch, or fsfs index --watch against a completed legacy
index (index_sentinel.json with generation_complete = true) automatically
promotes it into a complete-generation store while searches keep serving from
the legacy predecessor until FSFS-CURRENT publishes the first generation. An
index that remains on the legacy layout (an incomplete/interrupted legacy root,
a non-Linux platform without FSFS_COMPLETE_GENERATIONS=1, a store inside a
non-hidden directory of the indexed tree, or FSFS_COMPLETE_GENERATIONS=0) has
a known limit (bd-z2nfa): its watcher holds the vector files' exclusive writer
lock for its whole life, so fsfs search from another process (and the query
daemon) is refused with "another fsfs process is writing this index's vector
files" until the watcher stops. Build it once with FSFS_COMPLETE_GENERATIONS=1
to force conversion.
Example output:
PHASE REFINED: 5 hit(s) for "how does retry backoff work"
vector generation: potion-multilingual-128M class=semantic
1. src/retry.rs:14 score=0.033 [both]
Recover transient network failures with exponential backoff, bounded retries…
2. docs/failures.md score=0.016 [semantic]
…
5 results in 21ms
fsfs index writes two vector generations from the same documents:
vector/index.fsvi (the fast tier, potion) and vector/quality.fsvi (the
quality tier, all-MiniLM-L6-v2, its own embedding space). A search first
fuses the fast tier with Quill BM25 and emits INITIAL, then re-ranks the
head against the quality tier and emits REFINED; with --stream both
phases arrive as separate frames (query.stream.initial_ready, then
query.stream.refined_ready). If the quality model is not installed when
you index, the generation is built fast-only, fsfs doctor reports
semantic.quality_generation as a warning with the recorded reason, and
searches stop at INITIAL until you re-index with the model present.
fsfs status shows both generations (vector_generation_id,
quality_generation_id). Deletes, append-batch, compact, and watch mode
keep the two tiers in step. Both generations carry RaptorQ repair sidecars
(index.fsvi.fec, quality.fsvi.fec): fsfs doctor
verifies them as durability.vector_sidecars, fsfs compact restores a
generation whose bytes drifted before merging, and an in-place delete drops
the sidecar until the next compaction re-protects the file. Keyword segments
carry no sidecars, since they rebuild from your files in seconds: every
keyword writer verifies its segments when it opens, quarantines one that
fails (*.fslx.quarantine), and reindexes the documents it held. The first search in a shell pays the model load
(about 3 s for potion, plus the MiniLM session for the quality stage); the
query daemon that fsfs search starts by default keeps later searches to tens
of milliseconds and exits on its own after ten idle minutes; fsfs daemon --stop
stops it sooner, and the next search starts a fresh one.
--stream also uses that warm daemon: Initial can arrive while quality
refinement is still running, followed by Refined or RefinementFailed and one terminal
event. JSONL and TOON announce complete cached replays with daemon_cache_hit.
Use --no-daemon for direct execution. A daemon policy or producer mismatch is
an error requiring a daemon restart or explicit direct execution; a request
that has delivered Initial is never silently retried.
Warm daemon requests share immutable reader allocations and admitted model services. Generation and producer admission are serialized briefly; retrieval and socket delivery proceed concurrently. Bursts can share native embedding batches. A disconnected request stops its own delivery while an executing native batch retains its bounded worker capacity until completion.
The complete-generation daemon drives at most eight searches within a total of nine admitted connections, including queued and partially framed requests. Each response is bounded to 4 MiB. The legacy flat-index socket transport admits at most 16 clients, bounds each phase frame to 4 MiB and each response to 8 MiB plus 64 KiB, and reports capacity failures explicitly. Slow readers do not hold shared search state while writes drain. Its queued and currently written response bytes are bounded by 16 × (8 MiB + 64 KiB).
The shared query cache retains at most eight entries of at most 4 MiB of encoded phases each; concurrent requests also retain their own working payloads. Model/index memory and active query candidate sets depend on the corpus, requested limits and concurrency. These transport budgets are not a process memory cap.
What It Does
frankensearch combines lexical and semantic retrieval with progressive delivery:
- lexical BM25 for exact keyword precision (native Quill engine via the
quillfeature; Tantivy retained behindlexical-tantivyas the conformance oracle/migration lane) - fast semantic tier for immediate relevant hits
- quality semantic tier for reranked refinement
- reciprocal rank fusion (RRF) to combine sources robustly
Result: responsive first answers plus better final ranking without blocking the UI.
Core Features
- Release binaries ship the Model2Vec/FastEmbed loaders; the installer downloads and verifies the two default models (
potion-multilingual-128M+all-MiniLM-L6-v2) on first run. Theembedded-modelsbuild profile additionally embeds those bytes for a zero-download first run; it is a source-build option, and every published full asset uses the loader + verified-download path - Progressive search phases (
Initial,Refined,RefinementFailed; library searchers with a reranker also emitReranked) - Agent-friendly streaming (
--stream) with machine-readable output - Result explanation surfaces (
fsfs explain <rank|R-id|path>against the last search) - Multiple output formats:
table,json,jsonl,toon,csv - Watch/incremental indexing mode for local corpus updates
- Portable SIMD vector search + quantized FSVI storage
- Opt-in cross-encoder reranking of the refined head (
fsfs search --rerank,search.rerank); ANN path via feature flags
CLI At A Glance
fsfs index <dir> --format json emits one completion envelope after durable
publication; --format jsonl emits it as one line. The payload includes actual
file counts, vector model identities, and generation_complete. If embedding
retries are exhausted, semantic_deferred_files and a warning explain why the
published artifacts still need indexing resumed before semantic search.
Completed ordinary index runs retain evidence for reusing their work on later
runs. Reuse normally requires matching executable, configuration, producer identities
and artifact bytes. fsfs index <dir> --upgrade-reuse compatible-contract explicitly
permits a different executable when its versioned indexing contract also matches.
The same opt-in is available as [indexing] upgrade_reuse = "compatible_contract"
or FRANKENSEARCH_INDEXING_UPGRADE_REUSE=compatible_contract; disable it with
--upgrade-reuse same-executable. Source, producer, topology and artifact checks
still apply, and old receipts without the contract cannot use this path.
Reuse skips document embedding; model verification and startup health probes
still run.
On complete-generation stores, compact and delete preserve admitted reuse
evidence for surviving documents without running embedding inference. A later
index reuses those documents and embeds changed or restored source rows.
Exact JSON number parsing advances the indexing contract to v2. A v1 completion
receipt starts cold once, including with this opt-in; later v2-compatible binary
upgrades can reuse its work. Older query caches and daemons are also invalidated
so forwarding preserves score bits and explanations.
A file whose device, inode, length and modification and
change times still match the stamp recorded when it was read is reused without
being opened (a stamp is recorded only once its change time is older than the
filesystem's timestamp tick, as git does); a file whose stamp moved is reused
if its content hash still matches. A reused file keeps its keyword rows and
vectors, and the payload counts such files as unchanged_files, so a rerun
over an unchanged tree opens no source file. Changed and newly discovered
files are read and indexed again; removed files leave the index. fsfs index --force (also --full) rebuilds all discovered files and clears stale
lexical rows.
Existing indexes acquire this evidence on their next successful build.
Cross-process reuse is supported on Linux; other platforms keep the existing
process-scoped executable evidence policy.
Each search hit carries its path, a plain-text snippet of the best-matching
fragment, and line: the 1-based line of the file where the snippet's first
query word sits (the table prints path:line). line is absent for hits
without a snippet and when the file on disk no longer holds the fragment.
Source code in mainstream languages (Rust, Python, TypeScript/JavaScript, Go,
Java, Kotlin, Swift, C/C++, C#, Ruby, PHP, Scala, Elixir, Haskell, Lua, Vue,
Svelte and others) and prose or configuration files (Markdown, MDX, reStructuredText,
AsciiDoc, Org, TeX, TOML, YAML, JSON) get keyword and semantic search. Other
text files get keyword search only, as do files larger than
discovery.max_file_size_mb (10 MB by default). Log, lock, temporary and
backup files, and files between two and four times that size limit, are
counted in the index but cannot be found yet; larger and binary files are
skipped.
# Basic search
fsfs search "structured concurrency" --limit 10
# Stream for agents/pipelines
fsfs search "query" --stream --format jsonl
# TOON mode
fsfs search "query" --stream --format toon
# Narrow results: lang:/type: take a language name or an extension,
# ext: one literal extension, path: (or a bare word) a path fragment.
# Extension clauses widen each other; every path clause must match.
fsfs search "retry backoff" --filter "lang:rust path:src"
fsfs search "retry backoff" --filter "ext:rs ext:md"
# Explain one result of the last search: by rank, R0-style id, or path
fsfs explain 1
# Re-score the refined head with the cross-encoder (one-time model install)
fsfs download-models ms-marco-minilm-l-6-v2
fsfs search "query" --rerank --format json
# Keep index fresh
fsfs index ~/projects --watch
# Health checks
fsfs doctor
Reranking selects one bounded, query-relevant passage from each candidate's indexed text. A match deep inside a long file can therefore contribute its surrounding context to the cross-encoder. When Quill holds the document body, this also preserves the indexed version after a source edit or deletion and uses extracted PDF text. Vector-only indexes use a bounded source-text read.
Fast semantic windows (on by default)
The fast tier embeds up to 128 overlapping windows of each file, not just its
opening. The cap is indexing.fast_window_max_per_file in your project
fsfs.toml or user configuration; 1 restores prefix-only embedding:
[indexing]
fast_window_max_per_file = 1
The accepted range is 1..=128. An index keeps the policy it was built with
until the next fsfs index, which rebuilds it under the configured one.
Windows are bounded 2,000-character passages of the full lexical text,
including code blocks. Adjacent windows overlap by 200 characters. The cap retains a final
window at the end of a long file, but capped files can still have gaps in
semantic coverage. More windows increase fast-tier embedding work and index
size. The quality tier keeps one bounded prefix per source file; for a
windowed source its blend weight shrinks with the share of the file that
prefix covers.
Windows are why hybrid search is the default. On the repository's 253 judged
code queries (docs/quality_harness/fsfs_code_product_eval.py), prefix-only
vectors put hybrid search far below keyword-only search: nDCG@10 0.310
Initial and 0.349 Refined against 0.590. With 128 windows hybrid matches it
(0.577 and 0.581, within the confidence interval) and recalls more (R@100
0.93 against 0.89). On BEIR SciFact, NFCorpus and ArguAna, Refined beats
keyword-only search either way (0.690, 0.339 and 0.365 against 0.653, 0.305
and 0.314), and windows change it by less than 0.002. On that 579-file,
35 MB tree the index grows from 89 to 106 MB, a cold build takes about 28 s
instead of 23 s, and a search answers in about 26 ms instead of 21 ms
(median, both phases).
The index completion JSON includes fast_window_coverage: completed source
and window counts, covered and total canonical characters, and the number of
capped files. Overlap is counted once. Use these figures to choose a cap for
your corpus; a completed publication does not imply every character was
embedded when a file exceeded its window budget.
Search uses the best-scoring fast window for each source before fusion and applies path filters to source IDs. A long file cannot consume the entire result budget with duplicate window hits. Semantic-only hits receive context from the winning indexed passage; keyword-ranked hits keep their lexical fragment. Queries use the window layout saved with the index, so the indexing setting is not required merely to search it.
Reindexing verifies every window before reusing a source's embeddings and
retires obsolete rows when a file shrinks or disappears. fsfs watch keeps
the windows of a changed file current the same way, and a deleted file loses
all of its rows. Complete-generation stores support windowed append and
deletion through successor publication.
Query syntax
The word-matching (BM25) tier matches query words exactly, ignoring case,
without stemming: quantize does not match quantized. A file needs only
some of the words; more and rarer matches rank higher, and matches in the
file name are boosted 2x over the body. The vector tiers read the
query as plain text, which is how related wording still surfaces. An
exclusion (-fusion, -"rank fusion" or NOT "rank fusion") applies to
every tier: the vector tiers read the query without it and skip each file
the BM25 tier matches for it. The BM25 tier also understands:
| Query | Meaning |
|---|---|
"rank fusion" |
the words adjacent and in order; "rank fusion"~2 allows two positions of slack |
title:rrf, content:rrf |
match in the file name only, or in the body only |
+rrf -fusion |
rrf required in the word match; files containing fusion excluded from all results |
quantize AND f16, rrf OR fusion |
both words required, or either |
rrf^3 fusion |
count rrf three times as much |
* is not a wildcard: quantiz* searches for the word quantiz. To see how
each word scored for a hit, run fsfs explain <rank> after the search.
Complete generations (the default for new indexes)
The complete-generation store publishes lexical data, vector tiers, and the
catalog as one retained bundle, so readers in other processes, including a
search while fsfs watch runs, always see one whole generation. On Linux a
build into a directory with no index creates this store, and later commands
recognize it automatically; on macOS, where the store is not yet qualified on
release hosts, a build creates it only with FSFS_COMPLETE_GENERATIONS=1. The
store may sit inside the tree it indexes only beneath a hidden directory, like
the default in-project .frankensearch that fsfs index . uses; a build into
any other directory inside the indexed tree keeps building a legacy index, and
the store may never contain that tree. On Linux, a directory that already holds a
completed legacy index (index_sentinel.json with generation_complete = true)
is automatically migrated into this store on its next fsfs index, fsfs watch,
or fsfs index --watch while readers keep using the legacy predecessor until
FSFS-CURRENT publishes the first generation; an incomplete/interrupted legacy
root stays on the legacy recovery path unless FSFS_COMPLETE_GENERATIONS=1
forces conversion, and FSFS_COMPLETE_GENERATIONS=0 keeps building legacy
indexes. When the legacy index's completion evidence comes from the same fsfs
build and configuration, the conversion copies its keyword segments (shared by
hard link) and vector files into the first generation and carries every
unchanged file instead of recomputing it; otherwise, as on the first build after
an upgrade, it rebuilds. Once FSFS-CURRENT is published, the old vector/,
lexical/ and index_sentinel.json stay behind, unread:
fsfs index ~/projects --index-dir ./search-store
fsfs search "structured concurrency" --index-dir ./search-store --no-daemon
fsfs search "structured concurrency" --explain --index-dir ./search-store --format json
fsfs tui --index-dir ./search-store
fsfs explain 1 --index-dir ./search-store
fsfs append-batch --file updates.jsonl --index-dir ./search-store
fsfs delete src/obsolete.rs --index-dir ./search-store
fsfs compact --index-dir ./search-store
fsfs flush --index-dir ./search-store --format json
fsfs search "ownership rules" --expand --index-dir ./search-store --format json
For a relative configured index directory, commands also find the nearest existing store in a parent directory when run from a project subdirectory. Corrupt selections and interrupted first publications produce errors at that store instead of redirecting the command to a different legacy index.
Current source builds can finish a pressure-deferred quality tier. Clear forced deferral and fast-only policy, enable a pressure profile that permits quality, and make the original registered quality model available, then run:
fsfs index --complete-quality --index-dir ./search-store
Completion uses the generation's exact retained embedding inputs, even after
source files are moved, preserves its fast vectors, and publishes a complete
successor. A generation missing those exact inputs requires a full source reindex.
For a generation retaining all pending quality inputs, append-batch also keeps
the latest replacement and added bodies for this completion. Clearing forced
deferral leaves that work pending until --complete-quality succeeds; deleted
documents are excluded. The original quality producer must still match.
Deleting the last pending document clears current work. Later appends can still
queue their inputs for the original quality producer and explicit completion.
Stores first indexed from an empty corpus under forced deferral preserve that
producer choice too, so their later appends can complete quality normally.
append-batch accepts JSONL objects with string id and text fields, from
--file or standard input. It inserts or replaces the supplied documents in
lexical data, every present vector tier, and an existing catalog, then publishes
one complete successor. Repeated IDs use their final body in the batch; the
reported count is the number of unique inserted or replaced IDs. Lexical search
retains the whole canonical body, including long-document suffixes and code;
both embedding tiers use the ordinary indexer's bounded canonical input.
It does not scan, create, or edit source files.
If the store has a quality tier, its matching producer must be available too.
Malformed input, incompatible embedding responses, and cancellation before
publication leave the selected generation unchanged. An empty batch publishes
nothing, and readers already holding the predecessor keep that generation.
delete supports exact IDs and --prefix; it updates lexical membership, every
present vector tier, and the catalog in a new generation. Source files remain
on disk and can be indexed again by a later rebuild. compact also publishes a
successor. Existing readers keep their original bundle throughout either
operation. A delete with no matches leaves the selected generation unchanged.
search --expand pins one generation across all variants and counts each
variant's final ranking once. Expansion executes directly even when --daemon
is set; streamed searches deliver Initial before waiting for optional expansion.
The expansion provider is a remote LLM API: the query text goes to Anthropic
or OpenAI when the matching API key is set (see the FAQ). Without a key, the
original query's normal phases are retained.
Ordinary direct and forwarded searches save explanation context beside the
store; explain refuses context from a different generation or from a multi-query
fusion whose component explanations are unavailable.
search --explain also attaches per-hit evidence directly to the returned
ranking in table, JSON, JSONL, and TOON output, including daemon-forwarded
queries. Complete-generation streams emit explanation frames for the final
ranking before the terminal frame. The evidence uses the generation retained
by the query, so another search cannot replace its explanation context.
Reranking reads the generation's stored document text, so edits or removals in
the source tree cannot silently change the content scored for retained results.
tui opens the search dashboard against one admitted generation for the entire
session. Lexical, fast, and quality stages, snippets, and status refreshes keep
using that generation while a watcher or another command publishes successors.
Reopen the dashboard to select the latest generation. Without an interactive
terminal, the command reports the selected generation's status.
flush revalidates and synchronizes the selected generation, its inventory,
and its selection descriptor without changing sealed engine files. Its receipt
identifies the exact generation and reports durability: confirmed,
scope: selected_generation, generation_changed: false, and
watch_queue_drained: false. The barrier works for every selected complete
bundle, including vector-only bundles, without loading models, opening search
engines, rescanning source files, or draining queued watch changes. An active
publisher, corrupt bundle, cancellation before completion, or failed
synchronization prevents a success receipt. This also provides an explicit
durability retry after a publication whose final directory synchronization failed.
Every command that publishes a successor (index, watch, append-batch,
delete, compact) then reclaims older generations: it keeps the selected
generation and its immediate predecessor (the rollback target), removes
everything older plus abandoned unsealed builds, and never removes a generation
that any live reader, in any process, still holds. The receipt reports what was
kept and removed (retention in JSON). FSFS_GENERATION_RETENTION=report
reports what would be removed without removing anything, and
FSFS_GENERATION_RETENTION=off keeps every generation.
This is the cooperative local store. The complete FSVI v2 authority migration remains unfinished. Commands refuse to create a nested index inside a sealed generation.
Reproducible Showcase Suite
Use this sequence to reproduce the core demo + benchmark evidence bundle:
# Progressive CLI behavior and machine-output surfaces
fsfs search "structured concurrency" --limit 10 --format table
fsfs search "structured concurrency" --limit 10 --stream --format jsonl
# Statistical benchmark regression lane (Tier-3 reproducibility anchor)
cargo test -p frankensearch-fsfs --test benchmark_baseline_matrix -- --nocapture
# Graceful-degradation/fault demonstration lane
cargo test -p frankensearch-fsfs --test pressure_simulation_harness \
scenario_long_run_soak_fault_injection_stays_within_drift_thresholds -- --nocapture
The benchmark lane validates deterministic artifact identity fields (dataset_sha256, matrix_sha256, samples_sha256) plus a fixed replay command contract.
Configuration
Configuration precedence (highest first; fsfs config prints the resolved
source_precedence_applied for the current process):
- CLI flags
- environment variables
- config files (project
fsfs.tomlover user~/.config/fsfs/config.toml) - built-in defaults
--fast-only, FRANKENSEARCH_FAST_ONLY=true, and [search] fast_only = true
disable quality work under the default performance pressure profile. The
highest-precedence value wins, including an explicit false. The strict
and degraded profiles keep quality disabled: a winning fast_only=false
returns a configuration error naming the profile. A hard pause disables quality
after source precedence and records the safety clamp in profile diagnostics.
fsfs search --quality-only "query" searches the quality vector tier directly,
without running lexical or fast-vector retrieval. Ordinary searches return one
Initial phase with skip_reason="quality_only"; --expand can additionally
produce a Refined ranking from quality-only query variants. The mode supports the same filters, limits,
streaming output and daemon transport as ordinary search. The selected quality
model and its compatible index must be available; missing models, incompatible
vectors, cancellation and quality timeouts are errors. This mode does not fall
back to another tier. It conflicts with --fast-only or a resolved
search.fast_only=true; use --no-fast-only when the configuration disables
quality, with a pressure profile that permits it. Lite builds and forced
lexical-only pressure mode refuse quality-only requests. Raw fsfs serve
requests select it with "mode":"quality_only" (or "quality").
fsfs search --ranking-priors "identifier" opts into bounded path/module
signals for identifier and short keyword queries. The equivalent config is
[search] ranking_priors = true, or FRANKENSEARCH_RANKING_PRIORS=true;
--no-ranking-priors explicitly restores neutral ranking. Natural-language
queries and degraded retrieval keep neutral priors. The policy travels with
daemon requests and separates result-cache entries. --explain reports an
applied fusion.prior_boost before optional reranking. The default stays off;
this opt-in does not establish a general relevance improvement.
Common environment variables:
| Variable | Purpose | Example |
|---|---|---|
FRANKENSEARCH_INDEX_DIR |
Override index/data directory | ~/.local/share/frankensearch |
FRANKENSEARCH_MODEL_DIR |
Override model location | ~/.cache/frankensearch/models |
FRANKENSEARCH_INDEXING_FAST_WINDOW_MAX_PER_FILE |
Maximum fast semantic windows per file; 1 keeps prefix indexing, 2..128 enables passages |
32 |
FRANKENSEARCH_RERANK |
Re-score the refined head with the cross-encoder (same as --rerank); needs fsfs download-models ms-marco-minilm-l-6-v2 once |
1 |
FRANKENSEARCH_RERANK_TIMEOUT_MS |
Rerank-stage deadline (search.rerank_timeout_ms, default 300). The head is up to 30 documents, so abstract-length documents need about a second; at 300 ms the stage times out and the fused order stands |
1000 |
FRANKENSEARCH_FAST_ONLY |
Disable quality work; follows CLI > environment > config precedence. false requires a profile that permits quality |
true |
FRANKENSEARCH_QUALITY_WEIGHT |
Blend quality vs fast tier | 0.7 |
FRANKENSEARCH_RRF_K |
RRF constant | 60 |
FRANKENSEARCH_LOG |
Tracing filter | info |
For full contracts and knobs:
docs/fsfs-config-contract.mddocs/fsfs-dual-mode-contract.mddocs/fsfs-packaging-release-install-contract.mddocs/fsfs-packaging-release-install-contract.md#host-migration-playbooks-priority-projectsdocs/fsfs-packaging-release-install-contract.md#staged-rollout-and-deterministic-fallback-protocoldocs/fsfs-packaging-release-install-contract.md#upgrade-and-migration-compatibility-verification-strategydocs/architecture/
Network proxies
Model downloads, installer model provisioning, and hosted embedding APIs share the same proxy policy. Export settings before invoking the installer or CLI:
# HTTP CONNECT proxy for HTTPS destinations, including Hugging Face.
export HTTPS_PROXY=http://127.0.0.1:3128
fsfs download-models potion-multilingual-128m
# Alternatively, use a SOCKS5 proxy for destinations without a scheme-specific proxy.
export ALL_PROXY=socks5h://127.0.0.1:1080
The first nonempty value in each row wins; surrounding whitespace is ignored:
| Destination | Proxy lookup order |
|---|---|
| HTTPS | https_proxy, HTTPS_PROXY, all_proxy, ALL_PROXY |
| HTTP | http_proxy, all_proxy, ALL_PROXY |
| Bypass list | no_proxy, then NO_PROXY |
Uppercase HTTP_PROXY is deliberately ignored because CGI can populate it from
an untrusted HTTP request header (httpoxy). Proxy URLs require a scheme:
http://, https://, socks5://, or socks5h://. Both SOCKS5 spellings resolve
destination hostnames at the proxy with the current transport. Proxy credentials
are passed literally: percent-encoded userinfo is not decoded by asupersync.
The bypass list accepts comma-separated hostnames, domain suffixes, IPv4/IPv6
literals, literal IP CIDR ranges, and * for all destinations. Hostnames match
case-insensitively at a dot boundary, with optional leading or trailing dots;
example.com also matches api.example.com, but not otherexample.com.
An optional port restricts an entry to that destination port (for example,
localhost:8080 or [::1]:8080). CIDR matching uses literal destination IPs
only; it does not perform DNS lookups. To force direct CLI connections regardless
of other proxy settings:
no_proxy='*' NO_PROXY='*' fsfs download-models potion-multilingual-128m
Model downloads reevaluate proxy selection and bypasses at each redirect, keep
the configured redirect limit, and reject HTTPS-to-HTTP downgrades. Hosted
embedding redirects must stay on the same origin to protect request data and
API credentials. --offline continues to forbid downloads regardless of proxy
settings; installer offline mode only verifies cached models.
Library callers can set DownloadConfig::proxy or ApiEmbedderConfig::proxy to
ProxySetting::FromEnvironment (default), Disabled (always direct), or
Url(...) (explicit proxy, ignoring environment settings and bypass lists).
Invalid selected proxy URLs fail without falling back to a direct connection.
Proxy configuration debug output redacts the URL, including credentials.
Model acquisition also bounds stalled network operations. By default,
DownloadConfig::request_timeout allows 60 seconds for connection setup, proxy
negotiation, and response headers at each redirect hop. Once streaming starts,
DownloadConfig::idle_timeout allows 60 seconds without receiving model bytes.
Only byte progress resets that inactivity limit, so large downloads can keep
running while a silent proxy or server fails promptly. Caller cancellation and
earlier caller deadlines apply during network waits and retry backoff; incomplete
artifacts are discarded before a retry or error is returned. Library callers can
adjust both limits for their network.
How It Works
Pipeline summary:
Query
-> canonicalize
-> classify
-> fast embed + Quill BM25 (default lexical)
-> RRF fusion (initial)
-> quality embed (top candidates)
-> blend (and optional rerank)
-> refined results
Model path used in the default quality lane:
- fast tier: potion-128M (or fallback)
- fusion: RRF over lexical + semantic ranks
- quality tier: MiniLM
- optional final rerank (
fsfs search --rerank/search.rerank; libraryrerankfeature): pure-Rust frankentorch cross-encoder (native, ms-marco-MiniLM-L-6-v2 or jina-reranker) re-scores the refined head oncefsfs download-models ms-marco-minilm-l-6-v2has installed the weights; without a verified model the stage is skipped with a typed reason (query.stage.rerank.disabled.unavailable). A FastEmbed/ONNX alternative sits behindfastembed-reranker
Architecture Breakdown
frankensearch is split into focused crates so each concern can evolve independently:
| Crate | Responsibility |
|---|---|
frankensearch-core |
Shared types/traits/errors/config, query canonicalization/classification, metrics/eval helpers |
frankensearch-embed |
Embedding backends and fallback stack (hash, model2vec, fastembed) |
frankensearch-index |
FSVI vector storage, SIMD dot products, top-k search, optional native HNSW ANN |
frankensearch-lexical |
Tantivy schema/index/search for BM25 lexical retrieval (conformance oracle + cass-compat interop lane) |
frankensearch-quill |
Native pure-Rust BM25 lexical engine (FSLX segments, delta-visible indexing) |
frankensearch-quill-gauntlet |
Differential conformance/perf gauntlet certifying Quill against the pinned Tantivy oracle |
frankensearch-fusion |
RRF fusion, two-tier orchestration, blending, optional rerank integration |
frankensearch-rerank |
Cross-encoder reranking (pure-Rust frankentorch native backend + optional FastEmbed) |
frankensearch-storage |
FrankenSQLite metadata persistence, dedup/content-hash tracking, embedding queue |
frankensearch-durability |
Repair/protection primitives for index artifacts and segment health |
crates/frankensearch-fsfs |
Standalone CLI product around the library stack |
crates/frankensearch-tui |
Shared TUI shell/input/theme/replay framework used by fsfs/ops |
crates/frankensearch-ops |
Fleet observability/control-plane TUI and telemetry materialization. Experimental (decision 2026-09-02, bd-p6k61): no shipped telemetry source yet; its only producer is its own simulator, nothing depends on it, and no release lane builds it |
This separation gives you two options:
- use the top-level library as a drop-in engine in your own app
- run the full standalone
fsfsbinary with progressive CLI/TUI workflows
Query Lifecycle (Detailed)
At execution time, the system follows this shape:
- Canonicalize and parse the query
- Normalize text, strip noise, preserve user intent.
- Classify query type (
identifier,short keyword,natural language) for adaptive budgets.
- Run Phase 1 retrieval
- Generate fast embedding (or fallback).
- Run lexical BM25 search (if configured).
- Run vector top-k on the fast index.
- Fuse lexical + semantic candidates with RRF.
- Emit
Initialresults quickly.
- Run Phase 2 refinement (unless
fast_only)
- Generate quality embedding.
- Re-score or refine top candidates with quality tier.
- Blend fast and quality semantic scores.
- Optionally rerank with cross-encoder.
- Emit
RefinedorRefinementFailed(graceful degradation path).
Algorithms Used
1. Reciprocal Rank Fusion (RRF)
RRF is rank-based and model-agnostic. It does not require score calibration across systems:
RRF(doc) = Σ_sources 1 / (K + rank(doc, source) + 1)
Default K is 60 (configurable with FRANKENSEARCH_RRF_K / rrf_k).
Why RRF:
- robust to score-scale mismatch between BM25 and vector similarity
- simple, stable, and strong in practice
- rewards documents that appear in multiple retrieval channels
2. Two-Tier Score Blending
During refinement, fast and quality semantic scores are normalized then blended:
blended_score = alpha * quality_score + (1 - alpha) * fast_score
alpha is controlled by quality_weight (default target 0.7).
3. Deterministic Ordering
When ties happen, ranking remains deterministic through stable tie-break logic
(including lexical comparison and doc_id ordering), which helps replayability
and makes diff-based evaluation much cleaner.
Index Format and Retrieval Internals
FSVI Storage
Vector data is stored in FSVI files with memory-mapped access:
- primary default quantization:
f16(good memory/quality tradeoff) - optional
f32paths where needed - explicit metadata and format checks to catch corruption early
Why this matters:
- lower memory footprint than full
f32storage in common workloads - fast startup and read paths via mmap
- predictable, portable on-disk format for local search use
Vector Top-K Implementation
The brute-force search path is optimized around:
- SIMD dot products
- bounded heap selection for top-k
- NaN-safe total ordering
- two-phase work: score first, materialize doc ids for winners
This gives strong baseline behavior while ANN remains optional for larger corpora.
Runtime and Concurrency Model
The async model uses asupersync and capability context (Cx), not Tokio.
Important implications:
- host-controlled runtime ownership: your app provides runtime/cancellation context
- cancellation-aware search phases and timeouts
- no hard Tokio coupling in public contracts
This is useful if you need to embed search inside existing non-Tokio runtimes or strictly controlled execution environments.
Design Principles
Core engineering principles in this project:
Progressive delivery first
Fast initial answer, then quality refinement, instead of blocking on best possible ranking.
Graceful degradation
If quality tier/reranker/model loading fails, search still returns useful initial results.
Determinism and reproducibility
Stable ordering and artifact-driven evaluation support regression tracking and CI gates.
Explicit tradeoffs over hidden magic
Key knobs (
rrf_k, blend weight, fast-only mode, candidate multipliers) are visible and tunable.Practical hybrid retrieval
BM25 and embeddings are treated as complementary signals, not mutually exclusive choices.
Why This Is Useful
frankensearch is especially strong when you need:
- intent-level recall beyond exact grep matching
- low-latency first results for interactive workflows
- local-first operation with optional model download only
- agent-friendly output contracts (
jsonl,toon) and explainability hooks - one reusable engine across multiple products instead of bespoke search stacks
In short: it closes the gap between exact text lookup and semantic retrieval without forcing you into remote services or heavyweight distributed systems.
Tuning Playbook
Common tuning patterns:
Need lower tail latency:
- run fast-only:
FRANKENSEARCH_FAST_ONLY=trueor--fast-only; both work with the defaultperformanceprofile - reduce candidate budget and rerank depth
- keep lexical enabled for exact-match recovery
- run fast-only:
Need higher relevance quality:
- keep quality tier enabled
- increase candidate multiplier
- enable reranking for top-N only
Need memory efficiency:
- stay on f16 index quantization
- use ANN selectively above practical corpus thresholds
Need operational clarity:
- emit JSONL/TOON for automation
- monitor phase timings and drift through test harness artifacts
Quality and Evaluation Discipline
The repository includes explicit quality harnesses and statistical checks:
- IR metrics:
nDCG@K,MRR,Recall@K, plus bootstrap confidence intervals - profile comparisons with deterministic manifests
- benchmark/e2e artifact contracts for repeatable regression analysis
This keeps tuning decisions evidence-driven rather than anecdotal.
nDCG@10 measured through the fsfs product (release build, fsfs serve
modes, default settings, held-out BEIR test queries, 2026-09-22;
docs/quality_harness/fsfs_beir_product_eval.py):
| Corpus (documents / queries) | Lexical only (Quill BM25) | Initial (BM25 + potion-128M, RRF) | Refined (+ MiniLM) |
|---|---|---|---|
| SciFact (5,183 / 300) | 0.653 | 0.590 | 0.689 |
| NFCorpus (3,633 / 323) | 0.301 | 0.291 | 0.332 |
| ArguAna (8,674 / 1,406) | 0.315 | 0.336 | 0.363 |
Refined is significantly better than both lexical-only and Initial results on
all three corpora (paired bootstrap, 95% intervals excluding zero). Initial
against BM25 alone is mixed: significantly worse on SciFact, indistinguishable
on NFCorpus and significantly better on ArguAna, so on some corpora the first
results and fast_only search rank no better than plain lexical search.
The cross-encoder (--rerank, ms-marco-MiniLM-L-6-v2) applies to these
abstract-length documents only with a raised rerank_timeout_ms. With it, at
limit 10, reranking changed nDCG@10 by +0.021 on SciFact (not significant),
+0.016 on NFCorpus (significant) and −0.013 on ArguAna (significantly worse),
so reranking stays opt-in and its 300 ms default deadline is unchanged.
Limits and Tradeoffs
Being explicit about scope helps set expectations:
- Hybrid search is more complex than plain
rgand has model/runtime overhead. - Quality-tier refinement improves ranking but increases latency.
- ANN helps at larger scale but adds index lifecycle complexity.
- Semantic quality depends on corpus characteristics and embedding model fit.
Use rg/grep for strict exact matching and frankensearch when ranking by
intent and contextual relevance matters.
Library Integration Quickstart (Rust)
If you want to embed frankensearch directly in your Rust app, this is the
minimum end-to-end flow:
use std::path::Path;
use std::sync::Arc;
use frankensearch::{
EmbedderStack, IndexBuilder, TwoTierConfig, TwoTierIndex, TwoTierSearcher,
};
asupersync::test_utils::run_test_with_cx(|cx| async move {
// 1) Resolve a verified semantic embedder. HashEmbedder is a control
// double, not a semantic engine; auto_detect without models still
// returns that control stack.
let stack = EmbedderStack::auto_detect_semantic_with(Some(Path::new("./models")))
.expect("production search needs a verified semantic embedder");
// 2) Build an index from documents
IndexBuilder::new("./my_index")
.with_embedder_stack(stack)
.add_document("doc-1", "Rust ownership and borrowing")
.add_document("doc-2", "Structured concurrency with asupersync")
.build(&cx)
.await
.expect("index build should succeed");
// 3) Open and search with the same semantic family
let stack = EmbedderStack::auto_detect_semantic_with(Some(Path::new("./models")))
.expect("search must use the same semantic family the index was built with");
let index = Arc::new(
TwoTierIndex::open(Path::new("./my_index"), TwoTierConfig::default()).unwrap(),
);
let mut searcher = TwoTierSearcher::new(index, stack.fast_arc(), TwoTierConfig::default());
if let Some(quality) = stack.quality_arc() {
searcher = searcher.with_quality_embedder(quality);
}
let (results, metrics) = searcher
.search_collect(&cx, "ownership rules", 10)
.await
.expect("search should succeed");
println!("results={} phase1_ms={:.2}", results.len(), metrics.phase1_total_ms);
#[cfg(feature = "quill")]
{
// `open_hybrid` opens every arm of the index directory and attaches
// the active lexical reader (blue-green roots resolve to their
// active engine dir; foreign or damaged layouts are typed errors).
use frankensearch::LexicalRead;
let parts = frankensearch::open_hybrid(
&cx,
"./my_index",
frankensearch::TwoTierConfig::default(),
)
.await
.expect("open hybrid index");
let lexical = parts.lexical.expect("lexical arm attached");
let lexical_hits = lexical
.search(&cx, "ownership", 10)
.await
.expect("search lexical arm");
assert!(!lexical_hits.is_empty());
}
});
Notes:
- Use
EmbedderStack::auto_detect_semantic_with(or explicitmodel2vec+fastembed).auto_detect/auto_detect_withstill return a hash-only stack when no model is present; that is a control policy, not a working semantic engine.IndexBuilderwithout an explicit stack already refuses that hash-only fallback. - Enable
quillfor the native bulk-built lexical index; uselexical-tantivyonly for the explicit Tantivy oracle/comparator lane during migration. - Enable
cass-compatonly when interoperating with the external CASS tool's schema-v8 Tantivy index at<base>/index/v8/. This foreign-format adapter is not a Quill fallback and remains outside default builds. - Use
search_collect_with_textor fullsearch(...)when you need negation filtering (-term) and rerank text access. - Keep
TwoTierConfigexplicit in code for reproducible behavior across environments. - This path is proven with the real models, not doubles: the gate's
facadestage runsintegration.rs::real_models_two_tier_search_yields_refined_through_the_public_api(potion fast tier + MiniLM quality tier from the registered cache; INITIAL then REFINED with the quality tier searched). Reproduce locally withFRANKENSEARCH_REQUIRE_SEMANTIC_E2E=1 cargo test -p frankensearch --features hybrid --test integration -- real_models.
Native ANN on admitted v2 indexes (opt-in)
An index opened through TwoTierIndex::open_admitted_v2_with_paths can use
independent native HNSW graphs for fast and quality retrieval. Enable either
or both before sharing the index with a TwoTierSearcher:
use frankensearch_index::native_hnsw::HnswParams;
// `index` is an already admitted FSVI v2 TwoTierIndex; `cx` is the caller's &Cx.
index.enable_native_fast_hnsw(cx, HnswParams::default(), 42)?;
index.enable_native_quality_hnsw(cx, HnswParams::default(), 43)?;
Each graph shares its retained vector owner. The progressive fast phase retrieves native ANN candidates and exactly rescores them before fusion. Quality retrieval searches independently of the fast candidate pool and exactly rescores its own candidates. Successful refinement retains both tiers' coverage, including the candidate counts after exclusions; failed refinement preserves the fast receipt.
Query admission checks the identity that actually accompanies the provider's
bound embedding, including through cached providers. Bound requests bypass the
raw-vector cache because that cache does not retain response identities.
An explicit SearchParams override uses an exact fast-tier scan after identity
admission, bypassing native ANN and MRL candidate selection. Native retrieval is
available through ActivatedTierSearch and TwoTierSearcher; the raw-vector
TwoTierIndex::search_fast API keeps its existing path.
Graph construction through these low-level index methods is synchronous.
For document-to-index construction, NativeIndexBuilder uses the blocking pool
carried by the caller's Cx, or an explicit with_blocking_pool override, for
vector sealing, admission, graph construction and persistence. The caller owns
and drains that pool; callers without one provide a suitable synchronous lane.
Generation replacement builds every requested successor graph before installing new state; failure leaves the current index intact. The quality policy survives a successor that temporarily has no quality tier. To retain the graphs across process restarts, explicitly save both graphs in a generation-specific staging directory before publishing that generation:
let fast_graph = staging.join("fast.fshnsw");
let quality_graph = staging.join("quality.fshnsw");
index.save_native_fast_hnsw(&fast_graph)?;
index.save_native_quality_hnsw(&quality_graph)?;
// After publishing, dropping the old index and reopening the same admitted
// FSVI v2 generation, load its graphs before sharing with TwoTierSearcher.
// Publication may move the staged bundle, so use its final directory.
let fast_graph = published_generation.join("fast.fshnsw");
let quality_graph = published_generation.join("quality.fshnsw");
reopened.load_native_fast_hnsw(&fast_graph, HnswParams::default(), 42)?;
reopened.load_native_quality_hnsw(&quality_graph, HnswParams::default(), 43)?;
Each .fshnsw.receipt binds the graph to the complete FSVI image, generation,
embedding identity, live document set, construction parameters and seed. Loading
requires exact agreement with the admitted tier and requested policy. Missing,
stale, corrupt, wrong-tier or policy-mismatched sidecars return errors and leave
the previous graph installed. Loading performs no writes or automatic rebuilds;
a fresh open stays exact until a graph is explicitly built or loaded.
Saving writes the graph and adjacent receipt separately. Callers must serialize
writers and finish both files before selecting the containing generation; save
does not publish a generation or authorize mutation of a sealed bundle. Legacy
v1 opens are ineligible. The fsfs defaults remain unchanged; ANN recall and
latency still need qualification at representative corpus sizes.
Baseline Performance Envelope (Reference)
These are practical CPU-only reference numbers for a healthy local setup.
Treat them as orientation targets, not hard SLAs. Rows marked ledger are
the newest committed measurements in docs/PERF_LEDGER.md; rows marked
receipt come from the committed latency receipt
docs/evidence/perf/library-two-tier-latency-20260903-thinkstation1.json
(release profile, AMD Ryzen Threadripper PRO 5975WX / 64 threads, registered
potion-multilingual-128M + all-MiniLM-L6-v2, a 1,000-document / 659 KB
synthetic prose corpus, 50 timed queries after 5 warm-ups); rows marked
product receipt come from docs/evidence/perf/fsfs-latency-20260903-thinkstation1.json,
the same corpus as files driven through the release fsfs binary (index,
cold no-daemon searches, and one-request-per-connection queries against the
query daemon). Both are regenerated by QUALITY_GATE_STAGES=perf scripts/quality-gate.sh. Rows marked target have no committed measurement
and are design budgets:
| Operation | Typical Envelope | Basis |
|---|---|---|
Fast hash embedding (hash_embed_fnv, tokenize + FNV) |
~2 μs | ledger, 2026-07-04 |
| Fast model embedding (potion-128M, short query) | ~0.1 ms | receipt, 2026-09-03 (p50 0.10 ms, p95 0.17 ms) |
| Quality model embedding (MiniLM, short query) | ~5 ms | receipt, 2026-09-03 (p50 5.1 ms, p95 6.1 ms); at index time both tiers together cost ≈16 ms per document |
| Vector search (fast tier, top-10; 1K and 10K docs) | ~0.1 ms / ~0.3 ms | receipt, 2026-09-03 (p50 0.07 ms on 1K docs; p50 0.29 ms on 10K docs from library-two-tier-latency-10k-20260903-thinkstation1.json, where INITIAL is p50 1.0 ms, REFINED delivery p50 7.6 ms / p99 11.0 ms, and indexing 10,000 docs / 6.5 MB costs 190 s, 16.8 ms per document for both tiers, with 5.4 MB fast, 7.9 MB quality and 91 MB lexical artifacts) |
| Lexical search (Quill, 1K docs) | ~0.2 ms | receipt, 2026-09-03 (p50 0.23 ms, p95 0.38 ms) |
| RRF fusion (1,000 + 1,000 candidates) | ~23 μs | ledger, 2026-07-04 |
Phase 1 initial delivery (library TwoTierSearcher, hybrid Quill + potion, warm) |
< 1 ms |
receipt, 2026-09-03 (p50 0.40 ms, p95 0.58 ms, p99 2.4 ms) against the < 15 ms target; the fsfs in-process path measured 18–19 ms on a 65-file corpus on 2026-09-01 and is not yet receipted |
Phase 2 refined delivery (library TwoTierSearcher, INITIAL + REFINED) |
~6 ms |
receipt, 2026-09-03 (p50 5.5 ms, p95 6.8 ms, p99 7.8 ms over the 40 of 50 queries that refined; the 10 short-keyword queries were answered by the lexical arm alone) against the ~150 ms target |
| Index build, both vector tiers + Quill (1,000 docs, 659 KB) | ~17 s | receipt, 2026-09-03 (17.2 s wall: 15.8 s embedding both tiers, 1.3 s Quill; artifacts 536 KB fast, 792 KB quality, 74 MB lexical; 1.29 GB RSS at the end) |
fsfs index (1,000 files / 660 KB, both vector tiers + Quill + catalog) |
~14 s | product receipt, 2026-09-03 (14.1 s wall, 26.9 MB on disk) |
Daemon-served fsfs search (warm query daemon, one request per connection, INITIAL + REFINED) |
~13 ms | product receipt, 2026-09-03 (p50 12.7 ms, p95 13.9 ms, p99 14.8 ms over 50 queries, 0 cache hits; :ready round trip 1.1 ms). Before the adaptive accept poll landed the same run measured p50 50 ms: the daemon slept 50 ms between empty accept polls. The receipt predates ec85748b, which removed a per-request re-hash of every lexical segment, a cost that grew with index size: on a 524-file code repository (80 MB lexical index), warm fsfs serve median went from 24.0 to 9.6 ms (not yet receipted) |
Daemon-served fsfs search --rerank (cross-encoder over the refined head) |
~250–400 ms | product receipt, 2026-09-03 (p50 412 ms over 20 queries, all applied, with the host at a 15-minute load of 63; an earlier run of the same lane at load 9 measured p50 233 ms, p95 320 ms: the int8 cross-encoder is CPU-bound and shares the box). Those documents were short synthetic prose: on BEIR SciFact abstracts (2026-09-22, limit 10) the default 300 ms deadline expired on every query, and with rerank_timeout_ms = 60000 a served request took p50 554 ms, p95 722 ms |
Watch mode: file written → ingested into both tiers (fsfs index --watch) |
~0.7 s | product receipt, 2026-09-03 (20 files written one at a time: event-to-applied p50 725 ms, p95 848 ms, max 886 ms = the 500 ms debounce plus p50 224 ms ingest; all 20 searchable from a fresh process after the watcher's graceful exit; host-pressure sampling pinned for the measurement, since a saturated host pauses the watcher by design) |
Cold process start (fsfs search without a running daemon, INITIAL + REFINED) |
~0.75 s | product measurement, 2026-09-23 (0.75 s median of 7 on a 56-file corpus, receipted models, 0.83 GB peak RSS; the same protocol measured 2.25 s and 1.40 GB before the compact Unigram tokenizer and the parallel matrix read, and the 2026-09-03 receipt 3.3–3.6 s, when building the 500,353-piece potion vocabulary dominated) |
What changes the envelope the most:
- query class and candidate budget
- corpus size and document length distribution
- enabled features (
lexical,rerank,ann) - model tier selection and cache state
Failure Modes and Degradation Behavior
The engine is intentionally designed to degrade gracefully:
| Condition | Behavior | What Caller Sees |
|---|---|---|
| Quality refinement timeout | Phase 2 aborts safely | SearchPhase::RefinementFailed { error: SearchTimeout, ... } |
| Quality embedder errors | Initial results preserved | SearchPhase::RefinementFailed { ... } |
fast_only=true |
Skip quality phase by design | only Initial phase + skip_reason="fast_only" |
| No quality embedder configured | Skip quality phase | only Initial phase + skip_reason="no_quality_embedder" |
fsfs query over 4,096 characters, or holding a control character |
cut to its first 4,096 characters; control characters read as spaces | every phase, on the cleaned query (payload query shows it) |
fsfs query the planner reads as low-signal (punctuation, no words) |
lexical-biased, no refinement | only Initial phase + skip_reason="query_low_confidence" |
| Fast embedder fails but lexical succeeds | lexical-only fallback | valid Initial results from lexical path |
| Fast embedder fails and no lexical fallback | hard failure | search returns error |
| Lexical backend failure | semantic continues | search continues without lexical contribution |
Practical implication: phase-1 UX can still stay responsive even when higher-cost quality paths fail.
The shipping fsfs CLI applies a stricter mode contract than the reusable
fallback-capable library primitive: Full and FastOnly require an admitted
vector generation and matching real fast embedder before Initial is emitted.
Only explicitly selected LexicalOnly bypasses that readiness boundary. A
quality-tier failure remains progressive and emits actionable
RefinementFailed output while preserving the admitted Initial results.
Production Deployment Checklist
Use this as a pragmatic hardening pass before rollout:
- Pin explicit feature set (
semantic,hybrid,full, etc.) and toolchain. - Decide runtime mode per environment (
fast_onlyin latency-critical paths, full two-tier where quality matters). - Set
FRANKENSEARCH_MODEL_DIRto a stable writable path with enough disk. - Enable structured logs/metrics (
FRANKENSEARCH_LOG) and capture phase timings. - Run quality gates:
scripts/quality-gate.sh(ordsr quality --tool frankensearch), which coverscargo fmt --checkcargo check --workspace --all-targetscargo clippy --workspace --all-targets -- -D warningscargo test --workspace --lib --exclude frankensearch-quill-gauntletplus the fsfs test binaries- the Quill native witness against real Quill and pinned Tantivy, plus its oracle-validator negatives
- the real-model quick-start lane and the executable quick-start gate against the built binary
- Run benchmark and quality harnesses on representative corpora before release.
- Validate degradation behavior by intentionally forcing quality timeout/failure.
- For large corpora, evaluate ANN thresholding and memory budget explicitly.
- Keep reproducible artifacts for before/after tuning comparisons.
Crates.io Publishing
Publication runs on a real host with configured Cargo registry credentials.
GitHub Actions does not publish this repository. The two release lines are
versioned independently: v* tags identify fsfs binary releases. Verified on
2026-10-29, the latest downloadable binary is
fsfs 1.12.3
(October 5), while crates.io carries
frankensearch 0.7.1
(October 3) and frankensearch-fsfs 1.12.3
(October 5). The 0.7.1 library bundle is tagged crates-v0.7.1.
The changelog lists every member, source revision and semantic
index rebuild requirement. A crate publication or git tag does not establish
cross-platform binary qualification.
Run scripts/check_crates_publish_contract.sh --mode gate --scope workspace
against the clean intended revision and the live registry. Build-verify every
package archive and complete the workspace publish dry run before uploading:
cargo publish --locked --workspace --dry-run \
--exclude optimize-params --exclude frankensearch-quill-gauntlet
cargo publish --locked --workspace \
--exclude optimize-params --exclude frankensearch-quill-gauntlet
Cargo publishes in dependency order and waits for registry availability. Verify the public archive checksums and source revisions, then run consumers without workspace paths or source replacement. An occupied version is not evidence that the intended source was published; verify it before reusing it. Crate-bundle GitHub releases must not replace the binary release as GitHub's latest release, because the installer and updater use that endpoint.
Quality Gate (dsr, not GitHub Actions)
GitHub Actions is not used as a gate for this repository: every CI workflow under
.github/workflows/ is disabled (owner decision 2026-09-01). The one active workflow,
quill-gh51-landing.yml, runs only on manual dispatch and gates nothing. The gate that must pass
before any release lives in the repository and runs on a real host:
export FSFS_UPGRADE_REUSE_PREVIOUS_BINARY=/absolute/path/to/previous/fsfs
scripts/quality-gate.sh # fmt, check, clippy -D warnings, lib tests,
# Quill/Tantivy witness, fsfs tests, real-model e2e,
# executable quick-start gate
dsr quality --tool frankensearch # the same gate plus the packaging/installer
# contract checks, driven by dsr
The full e2e stage runs the two-executable upgrade-reuse regression. Supply a
previous fsfs built from a different source revision with the same registered
producer protocols and indexing contract, and retain its source and SHA-256
receipt. The test requires a different executable hash and exercises compatible
reuse, default cold rebuilding, and independent source, producer, and artifact drift.
Stages can be selected with QUALITY_GATE_STAGES=fmt,check,clippy,tests and
the registered model cache with QUALITY_GATE_MODEL_DIR. The end-to-end stage
fails closed when the two registered models are absent unless
QUALITY_GATE_ALLOW_MODEL_SKIP=1 is set. Release validation requires the real
models and does not use that skip. Releases use dsr or the documented real-host
build and publication commands with the same gates;
docs/fsfs-packaging-release-install-contract.md and
docs/crates-publishing-contract.md are the authoritative recipes.
The default quill stage runs the existing native engine witness and all tests
beside its independent oracle, plus the typed-query seed and replay regressions.
It inventories the complete all-feature gauntlet
and checks executed names and terminal counts against that inventory. The
90-second execution budget excludes compilation; missing Tantivy, ignored
required tests, an empty selection, timeout or incomplete output fails the stage.
This is bounded native correctness coverage, not full Quill conformance or a
performance verdict.
Before releasing changes to Quill, its analyzer/contracts or the gauntlet, also run the complete default and all-feature nonignored suites on the validation host:
QUALITY_GATE_STAGES=quill-full scripts/quality-gate.sh
# When changing the bounded driver or its selection, prove its failure paths:
QUALITY_GATE_STAGES=quill-probes scripts/quality-gate.sh
The full lane keeps the expensive evidence-assembly tests and their original assertions. Existing ignored tests retain the nightly, isolated-process or fixture-review prerequisites printed in the inventory; they are never counted as passes. E6's scaled-corpus and real-model quality campaigns and the QG performance ratchet remain separate requirements. A passing bounded lane does not override a failing full lane.
Troubleshooting by Symptom
| Symptom | Likely Cause | What To Check |
|---|---|---|
| Initial results are slow | Candidate budget too high, cold cache, oversized corpus | candidate multipliers, model cache warm-up, corpus and index size |
| No refined phase arrives | fast_only enabled, no quality embedder, timeout |
FRANKENSEARCH_FAST_ONLY, quality model availability, quality_timeout_ms |
RefinementFailed appears often |
quality model unstable/slow, timeout budget too tight | model path/logs, timeout config, CPU contention |
| Results feel exact-match heavy | lexical overweight via candidate mix, weak semantic model tier | embedder stack selection, query class budgets, quality tier availability |
| Results feel semantically off | embedding model mismatch for corpus domain | switch model tier, tune blend weight, add rerank |
Negation queries (-term) behave unexpectedly |
missing text provider in convenience path | use search_collect_with_text/search(...) with text callback |
| Output parsing issues in automation | wrong format for downstream parser | use --format jsonl or --format toon consistently |
| High memory usage | large index + quality/rerank/ANN enabled | feature set, f16 defaults, ANN config, corpus scope |
| Legacy or foreign lexical layout detected at open | pre-flip Tantivy directory or damaged FSLX segment | rebuild-on-detect re-derives the Quill index from canonical storage; foreign layouts are typed errors, fsfs doctor reports the lexical subsystem |
fsfs doctor reports semantic.quality_generation as a warning |
quality model absent at index time, so the generation is fast-only | install and verify all-minilm-l6-v2, then re-index (Quick Start); searches stop at INITIAL until then |
Sequence Diagram (Mermaid)
sequenceDiagram
autonumber
participant U as User/Caller
participant S as TwoTierSearcher
participant C as Canonicalizer+Parser
participant F as Fast Embedder
participant L as Lexical Backend
participant V as Vector Index
participant R as RRF Fusion
participant Q as Quality Embedder
participant B as Blend Stage
participant X as Optional Reranker
U->>S: search(query, k)
S->>C: canonicalize + classify + parse
par Fast semantic path
S->>F: embed(query)
F-->>S: fast query vector
S->>V: search_fast(top_k * multiplier)
V-->>S: semantic candidates
and Lexical path (if enabled)
S->>L: BM25 search(top_k * multiplier)
L-->>S: lexical candidates
end
S->>R: fuse(lexical, semantic, rrf_k)
R-->>S: initial ranked results
S-->>U: SearchPhase::Initial
alt fast_only OR no quality embedder
S-->>U: done (initial only)
else quality refinement enabled
S->>Q: embed(query) with timeout
alt quality success
Q-->>S: quality query vector
S->>V: quality_scores_for_indices(...)
V-->>S: quality scores
S->>B: blend(quality_weight)
opt rerank feature enabled
S->>X: rerank(top_n)
X-->>S: reranked results
end
S-->>U: SearchPhase::Refined
else quality timeout/failure
S-->>U: SearchPhase::RefinementFailed
end
end
Feature-Flag Decision Table
These are crate feature flags from frankensearch/Cargo.toml:
| Goal | Recommended Feature Set | Why |
|---|---|---|
| Fastest dev loop / CI smoke checks | default (hash) |
zero model downloads, minimal deps |
| Better semantic quality without lexical | semantic |
enables hash + model2vec + fastembed |
| Hybrid retrieval (semantic + BM25) | hybrid |
adds the Quill BM25 lexical arm (the post-flip lexical default) on top of semantic recall |
| Persistent local indexing | persistent |
hybrid + storage for durable metadata/queues |
| Durable + self-healing stack | durable |
persistent + durability |
| Lexical-only library build | quill |
pure-Rust Quill BM25 engine alone; since the flip, the lexical feature already selects Quill |
| Tantivy oracle/migration lane | lexical-tantivy |
explicit Tantivy-backed lexical path; cass-compat aliases it for external CASS schema-v8 interop |
| Full capability surface | full |
durable + rerank + ann + download + graph + api |
| Full stack + FTS5 storage backend | full-fts5 |
full + fts5 for advanced local SQL FTS paths |
Quick examples:
# Hybrid local search library build
cargo build -p frankensearch --features hybrid
# Full stack with ANN + rerank + download
cargo build -p frankensearch --features full
# Full stack plus FTS5
cargo build -p frankensearch --features full-fts5
Config Recipes
1) Latency-First Profile
Best for interactive UX where fast first answer matters most.
export FRANKENSEARCH_FAST_ONLY=true
export FRANKENSEARCH_QUALITY_WEIGHT=0.7
export FRANKENSEARCH_RRF_K=60
export FRANKENSEARCH_QUALITY_TIMEOUT=250
Operational effect:
- returns
Initialquickly and skips/limits expensive refinement behavior - predictable low-latency tail for chat/assistant loops
2) Quality-First Profile
Best for offline analysis, report generation, or high-precision ranking.
export FRANKENSEARCH_PRESSURE_PROFILE=performance
export FRANKENSEARCH_FAST_ONLY=false
export FRANKENSEARCH_QUALITY_WEIGHT=0.85
export FRANKENSEARCH_RRF_K=40
export FRANKENSEARCH_QUALITY_TIMEOUT=1200
Operational effect:
- gives quality tier and rerank more room to improve ordering
- increases median and tail latency
3) Memory/CPU-Conservative Profile
Best for constrained laptops or multi-tenant CI hosts.
export FRANKENSEARCH_PRESSURE_PROFILE=strict
export FRANKENSEARCH_FAST_ONLY=true
export FRANKENSEARCH_QUALITY_TIMEOUT=200
export FRANKENSEARCH_HNSW_THRESHOLD=200000
Operational effect:
- reduces expensive quality-phase work
- avoids ANN build/search overhead for smaller corpora
Optional: TwoTierConfig::optimized() TOML Recipe
For library consumers using TwoTierConfig::optimized(), place a file at
data/optimized_params.toml:
quality_weight = 0.8
rrf_k = 50.0
candidate_multiplier = 4
quality_timeout_ms = 800
fast_only = false
explain = false
hnsw_ef_search = 100
hnsw_ef_construction = 200
hnsw_m = 16
hnsw_threshold = 50000
mrl_search_dims = 0
mrl_rescore_top_k = 30
Use this when you want deterministic, checked-in tuning presets instead of host-specific env var drift.
Reference Appendix
Key Source Files
| Area | Source File | Purpose |
|---|---|---|
| Facade crate | frankensearch/src/lib.rs |
Top-level public API surface and re-exports |
| Index build workflow | frankensearch/src/index_builder.rs |
High-level corpus-to-index pipeline |
| Progressive orchestration | crates/frankensearch-fusion/src/searcher.rs |
Phase 1/2 flow, fallback paths, telemetry |
| Rank fusion | crates/frankensearch-fusion/src/rrf.rs |
Reciprocal Rank Fusion implementation |
| Two-tier blending | crates/frankensearch-fusion/src/blend.rs |
Fast/quality score normalization and blending |
| Two-tier index wrapper | crates/frankensearch-index/src/two_tier.rs |
Fast/quality index alignment and lookup |
| Top-k vector search | crates/frankensearch-index/src/search.rs |
Heap-based top-k selection and scoring paths |
| On-disk vector format | crates/frankensearch-index/src/quantization.rs |
FSVI quantization; mmap via mapped_file.rs |
| Core config knobs | crates/frankensearch-core/src/config.rs |
TwoTierConfig, defaults, env overrides |
| Core result types | crates/frankensearch-core/src/types.rs |
SearchPhase, ScoredResult, hit structs |
| Query classification | crates/frankensearch-core/src/query_class.rs |
Query-type detection and adaptive budgets |
| Eval/statistics | crates/frankensearch-core/src/metrics_eval.rs |
nDCG/MRR/Recall/MAP + bootstrap helpers |
| Embedder auto-detect | crates/frankensearch-embed/src/auto_detect.rs |
Fast/quality model discovery and stack setup |
| Storage ingest/queue | crates/frankensearch-storage/src/pipeline.rs |
Storage-backed ingestion, queue processing, embedding sinks |
| Durability repair layer | crates/frankensearch-durability/src/fsvi_protector.rs |
Protect/verify/repair flows for vector artifacts |
| FSFS CLI entry | crates/frankensearch-fsfs/src/lib.rs |
Standalone CLI product wiring |
| FSFS runtime orchestration | crates/frankensearch-fsfs/src/runtime.rs |
Command dispatch, search/index execution, stream emission |
| Shared TUI shell | crates/frankensearch-tui/src/shell.rs |
Reusable shell loop/navigation/overlay plumbing |
| Ops telemetry storage | crates/frankensearch-ops/src/storage.rs |
Control-plane telemetry persistence/materialization |
| Native lexical engine | crates/frankensearch-quill/src/index.rs |
Quill index lifecycle: seal/flush, FSLX segments, MaxScore/block-max WAND clause thresholds |
| Quill query execution | crates/frankensearch-quill/src/argus.rs |
BM25 query execution, sealed cursors, block-max pruning |
| Quill postings codec | crates/frankensearch-quill/src/quiver.rs |
Posting blocks and per-block block-max entries (encode_with_block_max) |
| Quill differential gauntlet | crates/frankensearch-quill-gauntlet/src/runner.rs |
Conformance/perf witness certifying Quill against the pinned Tantivy oracle |
Glossary
| Term | Meaning |
|---|---|
| Two-tier search | Progressive retrieval: fast initial pass, quality refinement pass |
Phase 1 / Initial |
First emitted result set, optimized for low latency |
Phase 2 / Refined |
Optional improved ranking after quality embedding |
RefinementFailed |
Graceful degradation event when Phase 2 errors/times out |
| RRF | Reciprocal Rank Fusion combining lexical + semantic rank lists |
| BM25 | Lexical ranking function used by the lexical backends (Quill and Tantivy) |
| FSLX | Quill's on-disk segment format: framed sections (TERMDICT, POSTINGS, POSITIONS, BLOCKMAX, DOCLEN, IDMAP, IDHASH) with reference validation |
| Delta segment | Quill lightweight segment carrying a generation's upserts/deletes against a base; visibility is delta-resolved at read time |
| Concat-merge | Quill segment merge mode that concatenates same-shape term streams without re-encoding (vs compacting tombstoned rows) |
| FSVI | On-disk vector index format used by frankensearch-index |
f16 quantization |
Half-precision storage mode reducing memory footprint |
TwoTierIndex |
Wrapper over fast and optional quality vector indexes |
TwoTierSearcher |
Main orchestrator that runs retrieval/fusion/refinement |
TwoTierConfig |
Primary tuning config for latency/quality behavior |
TwoTierMetrics |
Per-search diagnostics (phase timings, candidate counts, skip reason) |
EmbedderStack |
Paired fast + optional quality embedder selection object |
Cx |
asupersync capability context passed into async operations |
Knob Impact Matrix
| Knob | Where Set | Primary Impact | Increase Tends To | Decrease Tends To |
|---|---|---|---|---|
quality_weight |
TwoTierConfig, FRANKENSEARCH_QUALITY_WEIGHT |
Blend balance | Favor quality-tier ranking signal | Favor fast-tier ranking signal |
rrf_k |
TwoTierConfig, FRANKENSEARCH_RRF_K |
RRF rank sensitivity | Flatten rank differences across sources | Emphasize top ranks more strongly |
candidate_multiplier |
TwoTierConfig |
Candidate pool size | Improve recall headroom, increase latency/work | Reduce latency/work, may reduce recall |
quality_timeout_ms |
TwoTierConfig, FRANKENSEARCH_QUALITY_TIMEOUT |
Phase 2 budget | More chances to finish refinement | More RefinementFailed timeouts |
fast_only |
TwoTierConfig, FRANKENSEARCH_FAST_ONLY |
Phase behavior | Skip Phase 2 entirely (true) |
Enable Phase 2 when quality embedder exists (false) |
hnsw_threshold |
TwoTierConfig, FRANKENSEARCH_HNSW_THRESHOLD |
ANN activation point | Use brute-force for more corpus sizes | Use ANN earlier for large corpora |
hnsw_ef_search |
TwoTierConfig |
ANN query beam width | Better ANN recall, more latency | Lower latency, potentially lower recall |
mrl_search_dims |
TwoTierConfig |
MRL scan dimensionality | Better first-pass quality, more compute | Faster first-pass, potentially less quality |
mrl_rescore_top_k |
TwoTierConfig |
Full-dim rescore scope | Better refined ordering, more compute | Less compute, potentially weaker refinement |
lexical feature |
Cargo feature | Hybrid retrieval capability | Better exact-match precision and fallback paths | Semantic-only behavior |
rerank feature |
Cargo feature | Cross-encoder rerank | Better top-result precision, higher latency | Lower latency, less fine-grained top ordering |
ann feature |
Cargo feature | Approximate nearest-neighbor path | Better scale behavior at large corpus sizes | Simpler exact brute-force behavior |
Why Not Just grep/ripgrep/ctags?
grep/ripgrep/ctags are excellent for exact text and symbol lookup. frankensearch solves a different problem: semantic intent search over mixed corpora.
| Tool | Strong At | Limitation vs frankensearch |
|---|---|---|
grep |
exact substrings | no semantic similarity |
ripgrep |
very fast regex search | no embedding-based recall |
ctags |
symbol navigation | not document-level semantic ranking |
frankensearch/fsfs |
hybrid semantic + lexical, progressive refinement | higher complexity/runtime footprint |
Use both: keep rg for exact matches and use fsfs for intent-level retrieval.
FAQ
Does it run fully local?
Yes. Search/indexing runs on your machine. Network access is only needed for model downloads, update checks, and fsfs search --expand. Expansion sends the query text to api.anthropic.com when ANTHROPIC_API_KEY is set, otherwise to api.openai.com when OPENAI_API_KEY is set; with neither key it warns and searches the original query only. Documents and index contents are never sent.
Can I use only the library and skip fsfs?
Yes. Add frankensearch as a dependency and wire your own app/runtime.
What if the quality model is unavailable?
Search still works using fast-tier and lexical paths; you get RefinementFailed or fast-only behavior.
Which output format should agents use?
Use jsonl for streaming automation and toon if your downstream stack expects TOON semantics.
Add --compact to fsfs search for the token-lean shape: each hit is id (the R0-style id
fsfs explain takes), doc, s (score), r (0-based rank), snip and line, without the fusion
and freshness diagnostics. A 10-hit result shrinks from about 8 KB to 3 KB of JSON; with
--format toon the hits become one tabular row each.
Is this tied to Tokio?
No. Async/concurrency is built around asupersync and Cx.
Contributing
Project policy is no direct external merges, but issues and PRs are still useful for bug reports and proposal clarity.
If you are working inside this repository as an internal/automation agent, run the gate before every push (it wraps the individual commands below):
scripts/quality-gate.sh
# or the pieces it runs:
cargo fmt --check
cargo check --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --lib --exclude frankensearch-quill-gauntlet
The real-model CLI lane also checks explicit native quality selection. Provision
all-MiniLM-L6-v2-native alongside the standard Potion and ONNX models before
running the gate. Its native test accepts MINILM_FIXTURE_DIR,
POTION_FIXTURE_DIR, and FASTEMBED_MINILM_FIXTURE_DIR for separate verified
fixture directories; absent or invalid fixtures fail the test.
To exercise an optimized executable through this lane, set FSFS_E2E_BINARY
to the absolute path of the release fsfs built from the source being checked.
The gate still builds and checks the workspace; its executable tests log the
selected binary explicitly.
Useful docs:
AGENTS.mddocs/e2e-artifact-contract.mddocs/dependency-semantics-policy.mddocs/planning/UPGRADE_LOG.md— upgrade history (relocated from repo root)docs/planning/BRIDGE_PLAN_2026-09-02.md— the gap-closing plan from the 2026-09 reality check (what is left between the README and the code, per goal)
License
MIT License (with OpenAI/Anthropic Rider). See LICENSE.