Add persistent disk cache, bump to 0.5.0

Save index to .codebase-index-cache.pkl after every build. On startup,
load from cache if git ref matches (instant) or incrementally update if
≤20 files changed. Eliminates cold-start penalty on server restarts,
context compaction, and new sessions.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Author: RecoDemo

.gitignore

@@ -52,6 +52,9 @@ htmlcov/
 # ── Claude Code local state ───────────────────
 .claude/
 
+# ── Codebase index cache ─────────────────────
+.codebase-index-cache.pkl
+
 # ── Misc ───────────────────────────────────────
 *.log
 *.bak

README.md

@@ -16,6 +16,8 @@ Indexes codebases by parsing source files into structural metadata -- functions,
 
 **Automatic incremental re-indexing:** In git repositories, the index stays up to date automatically. Before every query, the server checks `git diff` and `git status` (~1-2ms). If files changed, only those files are re-parsed and the dependency graph is rebuilt. No need to manually call `reindex` after edits, branch switches, or pulls.
 
+**Persistent disk cache:** The index is saved to a pickle cache file (`.codebase-index-cache.pkl`) after every build. On subsequent server starts, the cache is loaded and validated against the current git HEAD — if the ref matches, startup is instant. If a small number of files changed (≤20), the cached index is loaded and incrementally updated instead of rebuilt from scratch. This eliminates the cold-start penalty when restarting Claude Code sessions, restarting the MCP server, or resuming work after context compaction.
+
 ## Language Support
 
 | Language | Method | Extracts |
@@ -55,6 +57,16 @@ PROJECT_ROOT=/path/to/project python -m mcp_codebase_index.server
 
 `PROJECT_ROOT` specifies which directory to index. Defaults to the current working directory.
 
+### Persistent Cache
+
+In git repositories, the server automatically caches the index to `.codebase-index-cache.pkl` in the project root. On startup:
+
+1. **Cache hit (exact match):** If the cached git ref matches the current HEAD, the index loads instantly from disk — no parsing, no file walking.
+2. **Cache hit (small changeset):** If ≤20 files changed since the cached ref, the cached index is loaded and incrementally updated on the first query.
+3. **Cache miss:** If the changeset is large or no cache exists, a full rebuild runs and saves a new cache.
+
+Add `.codebase-index-cache.pkl` to your `.gitignore` — it's a local-only build artifact.
+
 ### Configuring with OpenClaw
 
 Install the package on the machine where OpenClaw is running:
@@ -99,7 +111,7 @@ openclaw mcp list
 
 All 18 tools will be available to your agent.
 
-**Performance note:** The server automatically detects file changes via `git diff` before every query (~1-2ms) and incrementally re-indexes only what changed. However, OpenClaw's default MCP integration via mcporter spawns a fresh server process per tool call, which discards the in-memory index and forces a full rebuild each time (~1-2s for small projects, longer for large ones). This is a mcporter process lifecycle limitation, not a server limitation. For persistent connections, use the [openclaw-mcp-adapter](https://github.com/androidStern-personal/openclaw-mcp-adapter) plugin, which connects once at startup and keeps the server running:
+**Performance note:** The server automatically detects file changes via `git diff` before every query (~1-2ms) and incrementally re-indexes only what changed. However, OpenClaw's default MCP integration via mcporter spawns a fresh server process per tool call, which discards the in-memory index and forces a full rebuild each time (~1-2s for small projects, longer for large ones). With persistent caching, these cold starts are now significantly faster — the server loads from the disk cache instead of re-parsing the entire codebase. For persistent connections (avoiding even the cache load overhead), use the [openclaw-mcp-adapter](https://github.com/androidStern-personal/openclaw-mcp-adapter) plugin, which connects once at startup and keeps the server running:
 
 ```bash
 pip install openclaw-mcp-adapter
@@ -138,6 +150,39 @@ Or using the Python module directly (useful if installed in a virtualenv):
 }
 ```
 
+#### Reinforcing Tool Usage with Hooks
+
+Claude Code tends to default to built-in Glob/Grep/Read tools even when codebase-index is available. In addition to CLAUDE.md instructions (see below), you can add hooks that fire on every prompt to reinforce the behavior. Add this to `.claude/settings.local.json`:
+
+```json
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "echo 'CRITICAL REMINDER: Use codebase-index MCP tools FIRST for ALL code navigation (find_symbol, get_function_source, search_codebase, get_dependencies, etc). Only fall back to Glob/Grep/Read for non-code files.'"
+          }
+        ]
+      }
+    ],
+    "UserPromptSubmit": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "echo 'Use codebase-index MCP tools first for code navigation.'"
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+Hook stdout is injected as context Claude sees before responding. `SessionStart` fires on startup, resume, and context compaction. `UserPromptSubmit` fires on every turn.
+
 ### Important: Make the AI Actually Use Indexed Tools
 
 By default, AI assistants will ignore the indexed tools and fall back to reading entire files with Glob/Grep/Read. Soft language like "prefer" gets rationalized away. Add this to your project's `CLAUDE.md` (or equivalent instructions file) with **mandatory** language:
@@ -193,6 +238,8 @@ Tested across four real-world projects on an M-series MacBook Pro, from a small
 | Django | 3,714 | 707,493 | 29,995 | 7,371 | 36.2s | 126 MB |
 | **CPython** | **2,464** | **1,115,334** | **59,620** | **9,037** | **55.9s** | **197 MB** |
 
+With persistent caching, subsequent startups bypass the full build entirely. Cache load time is negligible compared to parsing — a cache hit on CPython restores the full index in under a second instead of 56s.
+
 ### Query Response Size vs Total Source
 
 Querying CPython — 41 million characters of source code:

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mcp-codebase-index"
-version = "0.4.6"
+version = "0.5.0"
 description = "Structural codebase indexer with MCP server for AI-assisted development"
 requires-python = ">=3.11"
 readme = "README.md"

src/mcp_codebase_index/server.py

@@ -32,6 +32,7 @@
 import json
 import os
 import sys
+import pickle
 import time
 import traceback
 
@@ -55,6 +56,10 @@
 _query_fns: dict | None = None
 _is_git: bool = False
 
+# Persistent cache
+_CACHE_FILENAME = ".codebase-index-cache.pkl"
+_CACHE_VERSION = 1  # Bump when ProjectIndex schema changes
+
 # Session usage stats
 _session_start: float = time.time()
 _tool_call_counts: dict[str, int] = {}
@@ -156,33 +161,116 @@ def _format_duration(seconds: float) -> str:
     return f"{hours}h {mins}m"
 
 
+def _cache_path(project_root: str) -> str:
+    """Return the path to the pickle cache file for this project."""
+    return os.path.join(project_root, _CACHE_FILENAME)
+
+
+def _save_cache(index: "ProjectIndex") -> None:
+    """Persist the project index to a pickle cache file."""
+    try:
+        root = index.root_path
+        path = _cache_path(root)
+        payload = {"version": _CACHE_VERSION, "index": index}
+        with open(path, "wb") as f:
+            pickle.dump(payload, f, protocol=pickle.HIGHEST_PROTOCOL)
+        print(f"[mcp-codebase-index] Cache saved → {path}", file=sys.stderr)
+    except Exception as exc:
+        print(f"[mcp-codebase-index] Cache save failed: {exc}", file=sys.stderr)
+
+
+def _load_cache(project_root: str) -> "ProjectIndex | None":
+    """Load a cached project index if it exists and is compatible."""
+    path = _cache_path(project_root)
+    if not os.path.exists(path):
+        return None
+    try:
+        with open(path, "rb") as f:
+            payload = pickle.load(f)
+        if not isinstance(payload, dict) or payload.get("version") != _CACHE_VERSION:
+            print("[mcp-codebase-index] Cache version mismatch, ignoring", file=sys.stderr)
+            return None
+        index = payload["index"]
+        from mcp_codebase_index.models import ProjectIndex as PI
+        if not isinstance(index, PI):
+            return None
+        return index
+    except Exception as exc:
+        print(f"[mcp-codebase-index] Cache load failed: {exc}", file=sys.stderr)
+        return None
+
+
 def _ensure_index() -> None:
     """Build the project index on first use (lazy initialization).
 
+    Tries to load from a pickle cache first. If the cache is valid and
+    the git ref matches (or the changeset is small enough for incremental
+    update), skips a full rebuild.
+
     This is called on the first tool call rather than at startup so that
     the MCP server can complete its initialization handshake immediately.
     Without this, large projects would cause Claude Code to timeout waiting
     for the server to become ready.
     """
+    global _project_root, _indexer, _query_fns, _is_git
+
     if _indexer is not None:
         return
+
+    _project_root = os.environ.get("PROJECT_ROOT", os.getcwd())
+    _is_git = is_git_repo(_project_root)
+
+    cached_index = _load_cache(_project_root)
+    if cached_index is not None and _is_git and cached_index.last_indexed_git_ref:
+        current_head = get_head_commit(_project_root)
+        if current_head == cached_index.last_indexed_git_ref:
+            # Exact match — use cache directly
+            print("[mcp-codebase-index] Cache hit (git ref matches)", file=sys.stderr)
+            _indexer = ProjectIndexer(_project_root)
+            _indexer._project_index = cached_index
+            _query_fns = create_project_query_functions(cached_index)
+            return
+
+        # Check if changeset is small enough for incremental update on cache
+        changeset = get_changed_files(_project_root, cached_index.last_indexed_git_ref)
+        total_changes = len(changeset.modified) + len(changeset.added) + len(changeset.deleted)
+        if not changeset.is_empty and total_changes <= 20:
+            print(
+                f"[mcp-codebase-index] Cache hit with {total_changes} changed files, "
+                f"applying incremental update",
+                file=sys.stderr,
+            )
+            _indexer = ProjectIndexer(_project_root)
+            _indexer._project_index = cached_index
+            _query_fns = create_project_query_functions(cached_index)
+            # _maybe_incremental_update will handle the rest on first tool call
+            return
+
+        print(
+            f"[mcp-codebase-index] Cache stale ({total_changes} changes), full rebuild",
+            file=sys.stderr,
+        )
+
     _build_index()
 
 
 def _build_index() -> None:
     """Build (or rebuild) the project index and query functions."""
     global _project_root, _indexer, _query_fns, _is_git
 
-    _project_root = os.environ.get("PROJECT_ROOT", os.getcwd())
+    if not _project_root:
+        _project_root = os.environ.get("PROJECT_ROOT", os.getcwd())
     print(f"[mcp-codebase-index] Indexing project: {_project_root}", file=sys.stderr)
 
     _indexer = ProjectIndexer(_project_root)
     index = _indexer.index()
     _query_fns = create_project_query_functions(index)
 
-    _is_git = is_git_repo(_project_root)
+    if not _is_git:
+        _is_git = is_git_repo(_project_root)
     if _is_git:
         index.last_indexed_git_ref = get_head_commit(_project_root)
+        _save_cache(index)
 
     print(
         f"[mcp-codebase-index] Indexed {index.total_files} files, "
@@ -256,6 +344,8 @@ def _maybe_incremental_update() -> None:
         file=sys.stderr,
     )
 
+    _save_cache(idx)
+
 
 # ---------------------------------------------------------------------------
 # Tool definitions