Back to Blog
LoreDocs

Finding the doc you can't remember the name of

LoreDocs Pro's hybrid semantic search finds documents by meaning, not filename -- so the spec you half-remember surfaces on the first query, every time.

There is a particular kind of friction that accumulates slowly and invisibly: you know you wrote something down, you remember roughly what it said, but you cannot recall the filename. You search your vault by keyword and get nothing, because the word you remember is not the word you used when you wrote the note. You scroll through a list of files hoping the title jogs your memory. Ten minutes later you have either found it or given up and started over.

LoreDocs Pro's semantic search is built to solve exactly this. It finds documents by what they mean, not what they are named.

How keyword search fails at the edges

Full-text search is fast and reliable for known queries. If you are looking for a document that contains the phrase "schema migration" and you type "schema migration," FTS5 will find it immediately. LoreDocs includes FTS5 keyword search on the free tier for exactly this use case.

The gap appears when the query and the document use different vocabulary for the same idea. You are thinking "how we decided on the data model" but the document says "entity-relationship design rationale." You are thinking "the onboarding checklist" but the spec is titled "new-hire workflow guide." Keyword search has no way to bridge that gap. The document exists; the query just does not share its vocabulary.

This is the everyday case for knowledge accumulated over months. The longer a vault has existed, the less reliable keyword recall becomes, because the language you use when you write and the language you use when you search diverges over time.

Hybrid retrieval: meaning plus keywords

LoreDocs Pro's semantic search uses a hybrid retrieval pipeline: small-scale dense embeddings for dense vector search, BM25 full-text scoring, and RRF (Reciprocal Rank Fusion) to merge the two rankings. The result is a retrieval system that rewards both semantic similarity and exact-term matches, giving you a result set that captures documents you remember precisely and documents you recall just vaguely.

The search is chunk-aware. Documents are split at paragraph boundaries (up to 256 tokens per chunk) before indexing, so a long specification does not need to match a query at the document level. If one paragraph of a twenty-page architecture doc is exactly what you are looking for, the chunk-level index surfaces that paragraph. The full document comes back with its most relevant section ranked at the top.

To use it, run vault_rebuild_index once after installing the Pro dependencies, then call vault_search with semantic=true. From any MCP-compatible surface -- Claude Code, the Codex desktop app, Cursor IDE, or Hermes Agent -- you can query your vault by asking the model to search for a concept. The model calls vault_search with the natural-language query directly.

What this looks like in practice

Suppose you are working in Claude Code on a data pipeline and you need to find the document where you outlined the partition strategy for a large table. You do not remember the filename or the exact phrasing, but you remember the decision involved weekend data and late-arriving events. A semantic query like "partition strategy for late-arriving events" will find the spec even if it is titled "ingest-layer design v2" and refers to the same problem as "weekend batch processing with out-of-order records."

The same applies across vault types. A product spec written in one session becomes findable from any future session, regardless of how the vocabulary drifts between when it was written and when it needs to be recalled. The spec stays where it is. The search adapts to how you think about it now.

The free tier and the Pro layer

LoreDocs gives you three vaults and FTS5 keyword search on the free tier, no signup required. If your vault is small and your queries are precise, the free tier covers most needs. Semantic search is a Pro feature, available once you install loredocs[pro] and run the index build. Pro also unlocks unlimited vault creation and access to the full 48-tool MCP suite.

The local-first architecture means both tiers store everything on your machine. There is no cloud backend, no data uploaded to a third-party indexer, no retention policy to navigate. The vector index lives in a LanceDB file in your local vaults directory alongside the document content, queryable at any time with or without network access.

If you have a knowledge vault that has grown past the point where you can recall filenames reliably, semantic search gives it back. Explore LoreDocs or get posts like this weekly: Dispatches from the Labyrinth.

PS: We are accepting early access signups for a lifetime deal -- join the waitlist to be notified when it opens.

Related reading:

DS

Written by Debbie Shapiro

Principal at Labyrinth Analytics Consulting. Data engineer with 35+ years across six technology generations, from mainframes to AI agents. She designs LangGraph pipelines, data warehouses, and the memory tooling behind LoreConvo and LoreDocs. Based in Washington State.

The torchlight, delivered.

One email when a new post is published: agentic AI, data engineering, and memory tools. No spam, no upsell, no AI summaries. Unsubscribe anytime.

Subscribe

Labyrinth Analytics Consulting helps organizations navigate the dark corners of their data. Learn more at labyrinthanalyticsconsulting.com.

More from the blog