@karmaniverous/jeeves-watcher

Filesystem watcher that keeps a Qdrant vector store in sync with document changes. Extract text from files, chunk it, generate embeddings, and query your documents with semantic search.

Requirements

• Node.js ≥ 22.13

Features

• Filesystem watching — monitors directories for file changes via chokidar
• Multi-format extraction — PDF, HTML, DOCX, Markdown, JSON and plain text (other extensions are read as plain text)
• Configurable chunking — character-based text splitting with overlap control
• Embedding providers — Gemini (default) or mock (for testing); extensible via provider registry
• Qdrant sync — automatic upsert/delete keeps the vector store current
• Rules engine — glob-based inference rules for metadata enrichment
• REST API — Fastify server for search, status, config, and management, with optional bearer API key (api.apiKey)
• Source documents — index records owned by another application, retrieved by HTTP callback on notification (sources config, POST /documents); see the Source Documents guide
• SDK — @karmaniverous/jeeves-watcher-sdk, a typed TypeScript client for the application-facing API
• CLI — jeeves-watcher init, config validate, start, and more

Version Control (VCS)

The watcher supports optional git-backed version control of watched content. When enabled, every file change under a watch root is automatically committed, providing full history, diff, and revert capabilities.

Configuration

Add a vcs block to your config:

{
  "vcs": {
    "enabled": true,
    "commitThrottleMs": 30000,
    "maxBatchSize": 1000,
    "branch": "master",
    "commitMessage": {
      "enabled": true,
      "provider": "anthropic",
      "model": "claude-haiku-4-0",
      "apiKey": "${ANTHROPIC_API_KEY}"
    },
    "retention": {
      "maxAgeDays": 30,
      "maxVersions": 100,
      "squashCron": "0 0 * * *"
    },
    "defaultAccessToken": "${GIT_TOKEN}"
  }
}
Copy

Per-Root Overrides via watch.paths

Watch path entries can be objects with per-root VCS overrides:

{
  "watch": {
    "paths": [
      "/data/notes/**/*.md",
      {
        "path": "/data/tickets/**/*.json",
        "vcs": {
          "enabled": false
        }
      },
      {
        "path": "/data/email/**/*.json",
        "vcs": {
          "remote": "https://github.com/org/email-archive.git",
          "accessToken": "${GITHUB_TOKEN}",
          "retention": {
            "maxAgeDays": 7,
            "maxVersions": 50
          }
        }
      }
    ]
  }
}
Copy

Per-root properties (remote, accessToken) are only available in per-root overrides. All other vcs.* properties can be overridden per root.

Deployment Notes

• Git dependency: git must be on the system PATH. The watcher initializes a git repository in each VCS-enabled watch root automatically.
• stateDir: With VCS enabled, stateDir (default .jeeves-metadata) must not overlap any watch root (equal, parent or child): startup fails otherwise. The .gitignore the watcher maintains in each VCS root always lists .git/, node_modules/, .jeeves-watcher/ and .jeeves-metadata/. VCS state (git repos) lives inside the watch roots themselves, not in stateDir.
• Filesystem constraints: Each VCS-enabled watch path entry must resolve to a single directory root (not a glob with wildcards in directory segments).

Remote Push

When remote is set on a per-root VCS config, the watcher pushes to the remote after every commit. Configure authentication via accessToken (per-root) or defaultAccessToken (global fallback):

{
  "vcs": {
    "enabled": true,
    "defaultAccessToken": "${GIT_TOKEN}"
  },
  "watch": {
    "paths": [
      {
        "path": "/data/notes",
        "vcs": {
          "remote": "https://github.com/org/notes-archive.git"
        }
      }
    ]
  }
}
Copy

Push failures are logged and recorded in /vcs/status but do not block commits.

Squash Retention

To prevent unbounded history growth, the squash retention system periodically compresses old commits into a single baseline. Controlled by vcs.retention:

• maxAgeDays and maxVersions define retention boundaries. The tighter constraint wins — if maxAgeDays: 30 would keep 200 commits but maxVersions: 100 caps at 100, only 100 are retained.
• squashCron controls when squash runs (e.g., "0 0 * * *" = daily at midnight). Checked every 60 seconds.
• After squash, old commits are replaced with a single "historical baseline" commit. If a remote is configured, squash triggers a force push.

API Endpoints

JsonMap Built-in Helpers

The following helpers are available in every JsonMap lib context:

fetchSiblings

Retrieves text from sibling files (same directory, same extension) around the current file, useful for contextual embedding (e.g., injecting surrounding email thread messages). JSON siblings use the JSON text derivation; other files are read raw.

Options: { before?: number (default 3), after?: number (default 1), sort?: "name" | "mtime" (default "name") }

Returns string[] of sibling text. Files that cannot be read are silently skipped. For a source document there is no file to read around, so fetchSiblings returns [].

Install

npm install @karmaniverous/jeeves-watcher
Copy

Quick Start

# Generate ./config/config.json (--output is a directory)
npx jeeves-watcher init --output ./config

# Validate it
npx jeeves-watcher config validate --config ./config/config.json

# Start the watcher
npx jeeves-watcher start --config ./config/config.json
Copy

Documentation

Full docs, guides, and API reference:

docs.karmanivero.us/jeeves-watcher

License

BSD-3-Clause