Local-LLM-Backed Conversational Search (Design Spike)

Note: This is a design spike, not an implementation plan. It documents the current search surface, investigates what exists today for "related notes finding", and proposes a phased data flow so a future /big-plan run can open concrete issues. No search-runtime code is changed by this spike.

Motivation

The user has asked whether the Search page can accept conversational queries such as:

maybe there's a note that is the prompt for making tag based UI. on foo-bar project

Today the Search page only accepts keyword-level queries. The user's request is to let the app resolve a conversational sentence into a ranked filename list, using a local model so there is no network round-trip to a hosted LLM.

The user also stated "we are using local LLM for related notes finding. so same strategy can be used for searching?" — which is the starting point for this design. The first task of this spike was to verify that statement against the codebase. See Existing "local LLM" mechanism below.

Current State

AiSearchPanel

File: tauri-app/renderer/components/search/ai-search-panel.tsx

A small React panel embedded in the Search page sidebar (tauri-app/renderer/components/search/sidebar.tsx). It exposes a textarea, a "Search with AI" button, and a results region. On submit it calls:

const titles = messages.map((m) => ({
  filename: m.filename,
  title: m.title,
}));
const res = await getBackend().aiSearchMessages(trimmed, titles);
setResult(res);
onResultFilenames(res.filenames);

The response shape is:

{
  filenames: string[];
  summary: string;
}

The panel passes filenames back up to the parent sidebar, which narrows the visible message list to only those filenames. The summary is rendered as descriptive text above the list. Only message titles are sent to the backend — bodies are not shipped through this path today.

aiSearchMessagesFallback

File: packages/backend-bridge/src/ai-search-fallback.ts

This is the only implementation of aiSearchMessages in the repo today. All three adapters route through it:

The algorithm is a plain lowercase substring match on titles:

const keywords = prompt.toLowerCase().split(/\s+/).filter(Boolean);
const matched = titles.filter((t) =>
  keywords.some((kw) => t.title.toLowerCase().includes(kw)),
);

Consequences for the conversational-query use case:

• Body text is never consulted. "note that is the prompt for making tag based UI" can only hit a filename whose title literally contains one of those tokens.

• Conversational filler wrecks recall. Each word — "maybe", "there's", "a", "note", "that", "is" — becomes a keyword, so any title containing a stopword-like fragment matches. Precision is low.

• Signature name is misleading. The surface is called aiSearchMessages but no AI is involved today; only the name and the UI spinner suggest otherwise.

Existing "Local LLM" Mechanism — What Actually Exists

The user's statement "we are using local LLM for related notes finding" is a mis-memory of what the Similar Notes Finder actually does. This spike could not find any local-LLM inference anywhere in the repository. Specifically:

• No candle, ort / ONNX Runtime, llama.cpp / llama-cpp-2, tch / libtorch, or rust-bert dependency in tauri-app/Cargo.toml or tauri-app/core/Cargo.toml.

• No gguf / .onnx / .safetensors assets in tauri-app/, packages/, or tauri-app/core/defaults/.

• No model download, first-run fetch, or model-path setting in packages/app-defaults/src/types.ts / defaults.ts.

What does exist, and what the user is presumably thinking of, is the Similar Notes Finder. The full stack is:

The header comment in similar_docs.rs confirms the design: "Core business logic for reading and writing similar-docs archive content." There is no model, no embedding, no tensor runtime. BM25 + a Japanese morphological tokeniser is, in practice, good enough for related-note retrieval given a draft body (the draft itself carries many content-specific tokens). It is much weaker when the query is a single English sentence full of function words, which is exactly the conversational-search case.

Archive rename (historical note)

Archive rename previously used an ai_summarize_title Rust command that shelled out to the Claude CLI for a filename-sized title. That command was removed in the archive-rename-no-ai refactor; filename derivation now uses a pure heuristic (heading extraction, frequent-word scoring, numbered fallback) implemented in tauri-app/renderer/utils/derive-archive-filename.ts. This note is kept to prevent confusion if reviewing old git history.

What the user appears to believe

The user's request frames "local LLM" as already-paid-for infrastructure that "same strategy can be used for searching." Based on the gap above, the design needs to be honest: there is no shared local-LLM runtime to reuse. Any local-LLM path will be new tech for this repo. The design can still absolutely use the existing BM25 index (which is legitimately local and very fast) as one of its layers.

Proposed Data Flow for Conversational Search

The target return shape must stay { filenames, summary } so the existing AiSearchPanel consumer is unchanged. What changes is what goes into producing it.

Option A — BM25-enriched query rewrite, no local LLM (cheapest)

User query ──► tokenise with Lindera
            ──► drop common function words / short particles
            ──► optional: synonym expansion (static dictionary)
            ──► BM25 query over archive bodies (not just titles)
            ──► top-K paths
            ──► summary = canned string or score breakdown

No model, no new runtime. Reuses SimilarityIndex. The fix for the user's example mostly comes from (a) querying bodies, not just titles, and (b) tokenising with Lindera instead of ASCII split(/\s+/). This is the option to ship first.

Option B — Local embeddings + vector search

Per archive note (offline, indexed):
  chunk body ──► embedding model (local, e.g. multilingual-E5-small ~120 MB)
             ──► store as f32 vectors in a local file next to archives

Per query (online, interactive):
  query text ──► same embedding model (≤ ~50 ms on M-series for a 50-tok sentence)
             ──► cosine similarity over ~1–10k stored vectors (few ms)
             ──► top-K paths
             ──► summary = "Ranked by semantic similarity"

Viable and fast once indexed. The user's conversational query is exactly the sweet spot of embeddings (semantic > lexical). Corpus sizes in the user's workspace are low-thousands of notes at most, so brute-force cosine over f32 vectors is fine — no FAISS/HNSW required in phase 1.

Option C — Full local chat model round-trip

User query + top-N BM25 candidates (title + snippet)
         ──► local instruct model (e.g. Qwen2.5-3B-Instruct Q4_K_M ~2 GB GGUF)
         ──► structured JSON { filenames: [...], summary: "..." }

A conversational round-trip from a 3B-class Japanese-capable instruct model on an Apple-silicon laptop is roughly 200–800 ms first-token + 30–60 tok/s throughput, so a short JSON response is ~1–3 s wall-clock. The first call after app start is materially slower because of model load and KV-cache setup (multi-second). This is too slow to drive a live search page on every keystroke, but it is acceptable for a click-to-search interaction like the one AiSearchPanel already has.

Disk/RAM budget is the bigger problem. See Packaging.

Recommendation: A → B → (optionally) C

Phase the work:

1. Phase 1 — A. Land body-aware BM25 behind aiSearchMessages. Cheap, high-ROI, covers 60–80 % of conversational queries in practice.

2. Phase 2 — B. Add local embeddings as a second ranker. Embedding model is small enough (~100–200 MB) to consider bundling. Phase 1 paths remain available as fallback when the embedding model is absent or disabled.

3. Phase 3 — C. Only if users still hit recall issues that embeddings don't fix, add a local instruct model to synthesise the summary field and optionally re-rank embedding top-N. Model is not bundled; first-run download.

Each phase is independently shippable and each earlier phase becomes the fallback path for the next one.

Latency budget

Backend Surface

The existing aiSearchMessages(prompt, titles) signature is sufficient for Phase 1 and nearly sufficient for Phase 2 and Phase 3 — with two caveats:

1. titles is body-less. For phases that search bodies or embed documents, the backend must read archive bodies itself rather than relying on the frontend's titles argument. That is already how similar_docs_query works (the Rust side walks archives/). Recommend: in the Tauri adapter, ignore the titles argument for A/B/C paths and drive retrieval from the Rust-side corpus. Keep the argument in the signature for Mock/REST adapters that still use the fallback.

2. summary is free text. Phase 3 wants to put a model-written sentence here. No schema change needed; just document in the type comment that the string may be longer on LLM-backed paths.

No breaking change to the BackendAdapter type is required. A new Rust invoke command is recommended — ai_search_messages (matches the TODO already in tauri-adapter.ts) — so the fallback keyword behaviour remains as a default the other two adapters can fall back to.

Per-adapter behaviour

Packaging Considerations

Anything larger than BM25 introduces a distribution question that the current app has never had to answer. Current binary is ~5 MB (see architecture/index.mdx). Rough budget impact per option:

Runtime choice

For Rust on macOS (primary target), the realistic options are:

• candle (HuggingFace, pure Rust) — Metal acceleration on Apple Silicon. Good for embeddings (Option B). Instruct-model support is working but less mature than llama.cpp.

• llama-cpp-2 (Rust bindings for llama.cpp) — Metal-accelerated, wide GGUF support. Best for Option C. Pulls a C/C++ toolchain into the build and links against a llama.cpp static lib.

• ort (ONNX Runtime) — Works for embeddings; instruct models typically require conversion. Less common in recent pure-Rust desktop apps.

Recommendation: candle for Phase 2, llama-cpp-2 for Phase 3. Phases stay independent so this choice is not locked in by Phase 1.

macOS entitlements and signing

• Loading a model from ~/Library/Application Support/zudotext/<app>/models/ does not require any new Tauri capability (the app already has FS access to its own data directory via Tauri's scoped FS plugin / Rust fs calls).

• No network entitlement is needed for inference. A network fetch to download the model does go through reqwest or similar, which already has network access.

• com.apple.security.cs.disable-library-validation is not required as long as the runtime (candle / llama-cpp-2) is linked into the app binary rather than loaded as a separate .dylib at runtime.

• llama.cpp's Metal backend loads a Metal shader at runtime; this works under standard code signing in our current Tauri setup.

• JIT entitlements are not required for the common Rust/Metal runtimes.

iOS packaging

Out of scope for Phase 1 and Phase 2 — iOS already restricts background CPU and memory. Phase 3 on iOS would require a substantially smaller model (phi-2-quantised / Qwen-1.5-0.5B-class) and should be a separate spike.

Bundled vs first-run download — recommendation

• A: nothing to bundle.

• B: first-run download by default; expose a bundled variant only for offline-kit builds if we add that later. Protects the default .app size from a 30× jump.

• C: first-run download, opt-in toggle in Settings → AI ("Enable local chat-grade search"). Default off.

Follow-Up Issues

The items below are each self-contained. They assume this design doc stays as the reference, so the titles and scope can be pasted into /big-plan directly.

1. Enable body-aware retrieval for aiSearchMessages ✓ Shipped (Phase 1)

Status: Complete. TauriAdapter.aiSearchMessages now calls invoke("ai_search_messages", { prompt, topK }) — the fallback path is no longer used. The ai_search_messages Rust command is wired.

Original scope: Add a new Rust command ai_search_messages(prompt, options) that runs a Lindera-tokenised BM25 query over archive bodies (not just titles) via the existing SimilarityIndex. Wire it into the Tauri adapter. Keep aiSearchMessagesFallback as the default for Mock and REST adapters. Update the AiSearchPanel UI copy to reflect that it searches both titles and bodies.

Acceptance:

• invoke("ai_search_messages", { prompt: "tag based UI prompt", ... }) returns paths from archive bodies that mention "tag" / "prompt", even when the archive title does not contain those tokens.

• packages/backend-bridge/src/tauri-adapter.ts no longer falls through to aiSearchMessagesFallback.

• Mock and REST adapters still return fallback keyword results.

• Unit tests in tauri-app/core/ cover: common-word query, Japanese query, empty corpus, prompt with only function words.

2. Scope creep test — BM25 summary string and top-K config ✓ topK implemented

Status: The topK parameter is implemented end-to-end. aiSearchMessages(prompt, titles, topK?: number) accepts an optional topK argument (see packages/backend-bridge/src/types.ts), TauriAdapter passes it through as invoke("ai_search_messages", { prompt, topK }), the aiSearch.topK: number AppSettings field exists in packages/app-defaults/src/types.ts, and the Settings UI exposes it in the AI Search section (tauri-app/renderer/components/settings/sections/ai-search-settings.tsx).

Original scope: Now that aiSearchMessages is no longer trivially-fake, decide what summary should say and expose the top-K as a setting. Candidates: "Found N notes that best match your query" / "Ranked by body similarity (BM25)". Add aiSearch.topK (default 20) to AppSettings. Surface the setting under an existing settings section (likely "Search" or "General").

Acceptance:

• New field aiSearch.topK: number in packages/app-defaults/src/types.ts with validation (range 5–100) and default.

• Settings UI control renders and persists.

• The new Rust command clamps K to the setting.

• aiSearchMessagesFallback is unchanged (mock/rest keep their current summary).

3. Local-embedding spike — prototype multilingual-E5-small via candle

Scope: A research spike (no user-facing change): prototype a Rust function that loads intfloat/multilingual-e5-small via candle, embeds the archive corpus on-demand, and caches vectors next to the BM25 cache. Benchmark indexing time and query latency against the real workspace size. Output: a follow-up doc section in this file (or a sibling doc) capturing numbers + a go/no-go recommendation.

Acceptance:

• A proof-of-concept branch (not merged) demonstrates cosine-sim top-K over an archive corpus of ≥ 500 notes.

• Indexing time, p50/p95 query latency, peak RSS, and cold-start time are recorded.

• A write-up under doc/src/content/docs/architecture/ updates this spike with a recommendation: bundle vs first-run download, fp16 vs int8, candle vs ort.

4. Capability-flagged Phase 2 feature — embedding-ranked search

Scope: Behind the existing capabilities.aiSuggestions-style pattern, add a new capabilities.localEmbedding flag. When on, ai_search_messages uses the embedding index (from issue 3) as the primary ranker and BM25 as a fallback when the index is absent or fails to load. Add a first-run download flow with a progress UI.

Acceptance:

• Settings → AI → "Use local embedding search" toggle.

• First-run download writes a checksummed model file into the app's data dir.

• If the download fails, the feature silently degrades to Phase 1 BM25 behaviour and shows a one-time toast.

• E2E test (mock adapter) covers the degraded path.

5. Decision doc — llama-cpp-2 vs candle for Phase 3 chat inference

Scope: Before any instruct-model wiring, produce a short ADR comparing llama-cpp-2 and candle for desktop-Rust chat inference. Must cover: build-time toolchain impact, binary size delta, Metal-backend status, model-format flexibility (GGUF vs safetensors), packaging implications for macOS code signing, and licence compatibility.

Acceptance:

• ADR lives under doc/src/content/docs/architecture/ and ends with a chosen runtime plus a one-paragraph justification.

• Two small throw-away benchmark binaries (one per candidate) are run on the author's Mac; timings are in the ADR.

• Next issue (Phase 3 implementation) cites this ADR by path.

6. Optional — conversational-query heuristic layer

Scope: Between the raw user prompt and BM25/embedding retrieval, add a conservative heuristic layer that strips conversational filler ("maybe there's a note that is"), detects named-entity fragments ("on foo-bar project" → boost matches mentioning foo-bar), and lightly rewrites the query. No model — just rules. Stays inside Phase 1 / Phase 2. Gated by a setting.

Acceptance:

• Heuristic module lives under packages/backend-bridge/src/ (not in Rust, so the

  Mock/REST adapters also benefit) and is unit-tested.

• Toggle aiSearch.conversationalRewrite (default on) in settings.

• Regression test: "maybe there's a note about tag based UI" rewrites to query terms that hit a fixture note with title "tag-based UI prompt" via the existing keyword fallback.

Out of Scope (Explicitly)

• Touching tauri-app/renderer/components/search/ai-search-panel.tsx, packages/backend-bridge/src/ai-search-fallback.ts, or any search runtime code in this spike.

• Changing the similarDocs Related Notes panel. That subsystem is healthy for its current use case (query-by-draft-body) and should not be re-plumbed through an LLM on this spike's watch.

• iOS support for Phases 2 and 3.

• Any hosted-API-backed fallback (the user's stated requirement is "no API call for AI services. can serve locally.").

References

• Current AI search panel: tauri-app/renderer/components/search/ai-search-panel.tsx

• Current keyword fallback: packages/backend-bridge/src/ai-search-fallback.ts

• Similar Notes Finder BM25 core: tauri-app/core/src/similarity/index.rs, tauri-app/core/src/similarity/tokenizer.rs

• Similar Notes Finder Tauri command: tauri-app/src/commands/similarity.rs

• Similar Notes Finder UI: tauri-app/renderer/components/similar-notes-finder/

• Heuristic archive rename (replaced former Claude-CLI-backed title summariser): tauri-app/renderer/utils/derive-archive-filename.ts

• Epic issue: zudolab/zudo-text#937

• This sub-issue: zudolab/zudo-text#944