When a prompt ends, the model forgets. That is why LoreConvo was built -- to keep the thread alive while you iterate on a design, debug a pipeline, or explore a new model. LoreConvo's session memory saves and recalls conversation context automatically, so your model can pick up where it left off. However, session memory is only part of the picture. For data engineers and AI practitioners who spend hours curating datasets, tuning embeddings, and documenting model behavior, the real challenge is structured artifacts: data schemas, experiment results, and code snippets that need to survive beyond individual conversations, be versioned over time, and be queryable by both humans and machines. Recalling a session summary is not the same as retrieving a specific schema version or a ranked set of experiment logs. That gap compounds over time.
Session memory and structured artifacts are different retrieval problems, and bolting the second onto the first would have compromised both. What the work needed was a place to store the structured knowledge that survives beyond a single chat: versioned, searchable, and accessible to the model without manual copying. So we built a dedicated document store that complements LoreConvo's session memory rather than duplicating it. LoreDocs is a local knowledge vault that lets you keep, version, and retrieve the artifacts of your AI work without ever leaving the tools you already use.
From fleeting chat to lasting insight
A typical day for a data engineer might start with a quick conversation with an LLM to sketch a data pipeline, then move on to writing a Spark job, testing a feature store, and finally documenting the results. Each step generates artifacts: SQL snippets, configuration files, experiment logs, and design notes. With LoreConvo, you can ask the model to summarize the pipeline or suggest improvements, but once the session ends the model no longer has access to those details. You end up copying the summary into a wiki, a ticket, or a notebook, and you lose the direct link between the model's reasoning and the original source.
LoreDocs bridges that gap. It treats every document -- a markdown note, a JSON schema, a plain-text log -- as a first-class citizen in a vault that lives on your own machine. The vault is a single SQLite file you own, back up, and move wherever you need. Because the data never leaves your disk, you retain full control over privacy and compliance, a requirement that many teams cannot ignore.
Why a dedicated knowledge store matters
Durable knowledge is more than a backup of chat transcripts. It is a structured, searchable, and versioned repository that can be queried by both humans and machines.
The multi-vault system lets you create named vaults that reflect the logical boundaries of your work -- for example, one for your feature-store schemas, another for model experiment logs, a third for architecture notes. Each vault can be tagged and searched independently. The workspace-scoped auto-vault feature mirrors the way you organize projects on disk: opening a vault bound to a directory path is idempotent, so onboarding a new project means a single call that follows your existing folder structure.
Full-text search powered by SQLite's FTS5 engine lets you find documents across vaults with simple keyword queries. When you need to locate a specific transformation step or a hyper-parameter setting, a search returns matching documents and the surrounding context. For practitioners who need deeper retrieval, the Pro tier adds semantic search: a hybrid LanceDB index combining dense embeddings with BM25 ranking, chunk-aware at paragraph boundaries, so results rank by meaning rather than exact keyword match.
Document versioning records every change you make. If a schema evolves or a model configuration is tweaked, you can restore a previous version with a single command -- an audit trail that matters for reproducibility and post-mortems. When you need to give an LLM access to a whole vault at once, the vault-prime operation injects all vault context in a single call, which is especially useful in automated pipelines where the model must reason over a collection of documents before producing output.
Finally, LoreDocs integrates with the same MCP ecosystem that powers AI-enhanced IDEs. By placing a configuration file in your project, tools like Claude Code, OpenAI Codex, Cursor IDE, and Hermes Agent can discover the vault automatically. No per-client setup is required. For environments that cannot use MCP, a Python fallback script provides the same query access against the local SQLite store.
How LoreDocs fits a data engineer's workflow
A data engineer's workflow is already a blend of command-line tools, notebooks, and IDE extensions. LoreDocs adds a thin layer that feels native to that mix.
When you start a new project, your agent opens a vault scoped to that workspace: one idempotent call binds the vault to the directory, so there is nothing to wire up by hand. Adding knowledge from there is direct -- point LoreDocs at a file on disk, or have the agent write an ad-hoc note straight into the vault. If you already keep a rich set of markdown notes in Obsidian, a single call imports the entire vault directory, walking the folder tree and extracting YAML frontmatter tags so your existing knowledge base is searchable from any MCP client. For automation that runs without an MCP client -- a scheduled job or a CI step -- a bundled Python fallback script performs the same add and search operations directly against the local SQLite file.
When you need to find something, a keyword search returns matching documents with context. Semantic search (Pro) handles the cases where you remember the idea but not the exact wording. Because every vault lives in a single file, moving a project to a new machine is as simple as copying that file -- no hidden cloud service, no vendor lock-in.
The free tier gives you three vaults at no cost, which covers most personal experiments. When you need unlimited vaults, the Pro tier adds semantic search, auto-discovered document relationships, and the full MCP tool suite.
A durable store for a durable future
The conversation model solved the problem of short-term context, but the real value of AI in data engineering comes from accumulating knowledge over weeks, months, and years. By giving that knowledge a dedicated store, LoreDocs lets you treat every experiment, schema change, and model insight as a reusable asset. The combination of multi-vault organization, full-text and semantic search, version control, and zero-config integration with popular AI-enhanced IDEs creates a workflow where the model can both generate new ideas and retrieve the exact pieces of prior work that inform those ideas.
LoreDocs is available now from the Anthropic marketplace and PyPI: see /tools for install steps and the rest of the Lore suite. The Lifetime Deal for early supporters opens August 4, so if you want to lock in early pricing, join the waitlist at labyrinthanalyticsconsulting.com/lifetime.
If you are new to the Lore suite, start with why your Claude sessions start from zero for the session memory context, and then see how LoreDocs vaults are designed for AI projects for the architecture that makes durable knowledge retrieval work.