v0.20.0+
Files are not the only thing the watcher can index. A source document is a record that lives in another system (a database row, a chat thread, a ticket) and that the watcher retrieves by calling back to that system over HTTP. The owning application tells the watcher that a record changed; the watcher fetches the current record, renders it with an inference rule, embeds it and keeps its Qdrant points in sync, exactly as it does for a modified file.
The watcher keeps no copy of a source document beyond the Qdrant points themselves (vectors, chunk_text and projected metadata). The source system remains the single copy of the record.
This guide covers the model, configuration and callback contract. For the HTTP endpoints, see the API Reference. For a typed client, see the SDK.
Every point carries exactly one identity field:
| File point | Source point | |
|---|---|---|
| Identity field | file_path |
source: a reference <name>://<id> |
| Retrieved by | Reading the file from disk | GET to the source's configured callback URL |
| Triggered by | Filesystem events | POST /documents notifications |
| Point ID | UUIDv5 of the normalised path + chunk index (case-insensitive) | UUIDv5 of the exact reference + chunk index (case-sensitive) |
| Extra system fields | none | source_rule (the designated rule) |
File points are unchanged by source support: same point IDs, same payload fields, no reindex on upgrade. A source point never carries file_path, and the watcher never writes source onto a file point. Consumers that need "the document key" can read file_path ?? source.
The watcher treats a point as a source point only when it carries a non-empty string source and no file_path. A legacy file point whose rule wrote a source metadata value (before source became reserved) still carries file_path, so it is a file point: prune may delete it, and it is never discovered by a source reindex or deleted by POST /documents/delete.
source and source_rule are reserved payload fields, alongside file_path, chunk_index, chunk_text, total_chunks, content_hash, matched_rules, line_start, line_end, created_at and modified_at. Rule and enrichment metadata can never overwrite them (reserved names are dropped from metadata before every write), and a rule schema that declares a property with a reserved name produces a warning: on the console at startup, in the log on config reload, and under warnings from POST /config/validate.
A reference is <name>://<id>:
<name> is a configured source name: a lowercase letter followed by 1 to 31 of [a-z0-9-] (at least two characters, so a reference can never be mistaken for a Windows drive letter).<id> is opaque to the watcher: 1 to 512 non-whitespace characters. /-separated segments are allowed, but an empty segment (leading, trailing or doubled /) or a . / .. segment is rejected as invalid_id.Example: app://chat/thread/6710c3a9.
References are compared exactly. app://chat/thread/ABC and app://chat/thread/abc are two different documents with two different sets of point IDs.
Sources are declared in config under sources. Callers never send URLs; they send references, and the watcher builds the URL from config.
{
"sources": {
"app": {
"url": "https://app.internal.example/watcher/documents/{id}",
"headers": {
"Authorization": "Bearer ${APP_CALLBACK_TOKEN}"
},
"timeoutMs": 10000,
"concurrency": 8
}
}
}
| Field | Type | Default | Description |
|---|---|---|---|
url |
string |
Required | Callback URL template. Must contain {id}, which is replaced with the document id: each /-separated segment is URL-encoded and the slashes are kept. Must be an absolute http/https URL with {id} outside the scheme, host and port. |
headers |
Record<string, string> |
none | Headers sent with every callback request (for example Authorization). |
timeoutMs |
number |
10000 |
Callback timeout in milliseconds. The clock starts when the request is sent, not while it waits for a concurrency slot. |
concurrency |
number |
8 |
Maximum concurrent callbacks to this source, across notify processing, reindex and render. |
For the reference app://chat/thread/6710c3a9, the example above requests https://app.internal.example/watcher/documents/chat/thread/6710c3a9.
Rules for sources:
url and headers support ${ENV_VAR} substitution. If any of them still contains an unresolved ${...} after substitution, loading the config fails and the service does not start. The error names the config path, never the value.url that is not an absolute http(s) URL, or that places {id} in the scheme, host or port, also stops the service at startup.sources is read at startup only. Changing it requires a restart.sources.<name> (for example timeout instead of timeoutMs) is ignored with a warning, logged at startup and on config reload and returned by POST /config/validate under warnings.sources never appears in GET /config (with or without a JSONPath path) or in the config echoed by POST /config/apply, and neither does the api block, so URLs, header values and the API key cannot leak through the API.unknown_source) and no request is made. The watcher only ever requests URLs built from this config, and never follows a URL to another origin.The source owner implements one read-only endpoint. For each reference, the watcher sends:
GET <url with {id} replaced>
<configured headers>
| Response | Outcome |
|---|---|
200, Content-Type: application/json (parameters such as charset allowed), body is a JSON object |
The document. Processed with its designated rule. |
200 with a JSON object that has no keys ({}) |
Failure no_content; points untouched. |
200 with any other content type, or JSON that is not an object (an array, a string, ...) |
Failure unsupported_content_type; points untouched. |
200 with a body that is not valid JSON |
Failure source_unavailable; points untouched. |
A body larger than 10 MiB (by Content-Length or while streaming) |
Failure source_unavailable (response too large); points untouched. |
404 |
The document no longer exists. Its points, enrichment metadata and issues are removed (deleted). |
Anything else: 3xx, 401, 403, other 4xx, 5xx, timeout, network error |
Failure source_unavailable; points untouched. |
Notes:
GET is used. Redirects are not followed: a 3xx is a failure.404 ever deletes. A misconfigured or failing source (wrong token, outage, timeout) records an issue and leaves existing points alone, so a source outage can never empty the index.timed out / network error (<code>)), never the URL or header values.concurrency.The SDK exports the SourceDocument type and the SOURCE_DOCUMENT_GONE_STATUS constant (404) for implementing the callback.
Every notification names a designated rule: { "source": "app://chat/thread/6710c3a9", "rule": "chat-thread" }.
match would accept it.rule_missing). If the rule disappears later (for example after a config change), processing records a rule_missing issue and leaves the points alone.source_rule and reused by reindex, POST /render, POST /metadata and POST /reindex { scope: "path", source }. Re-notifying a reference with a different rule switches it.match is evaluated and acts as the input contract: if the retrieved record does not match, nothing is written and a match_failed failure is reported with the AJV errors in details. The designated rule is applied alone, with no file reader: a map that calls fetchSiblings gets an empty list, since a source document has no directory and a rule must not read server files. A failed match is never an HTTP error on POST /documents: asynchronously it becomes a match_failed issue, and with ?wait=true a failed result.rule), plus details where there is structured context. This is how POST /reindex { "scope": "issues" } can retry a reference whose very first notification failed (it has no points, so no stored rule).Source rules match and resolve against these attributes:
interface SourceAttributes {
source: {
ref: string; // "app://chat/thread/6710c3a9"
name: string; // "app"
id: string; // "chat/thread/6710c3a9"
rule: string; // designated rule name
retrievedAt: string; // ISO 8601, server clock
};
json: Record<string, unknown>; // the callback body
}
There is no file key, and file attributes have no source key. Write source rules so their match requires source (and json), so the rule can never match a file during normal file processing.
A record served by the callback:
{
"threadId": "6710c3a9",
"userId": "u-123",
"title": "Resetting a password",
"updatedAt": "2026-10-12T09:30:00Z",
"messages": [
{ "role": "user", "text": "How do I reset my password?" },
{ "role": "assistant", "text": "Open Settings, then Security." }
]
}
A designated rule for it:
{
"name": "chat-thread",
"description": "One chat thread from the application.",
"match": {
"type": "object",
"required": ["source", "json"],
"properties": {
"source": {
"type": "object",
"properties": {
"name": { "const": "app" },
"id": { "type": "string", "pattern": "^chat/thread/[^/]+$" }
}
},
"json": {
"type": "object",
"required": ["userId", "messages"],
"properties": {
"userId": { "type": "string", "minLength": 1 },
"messages": { "type": "array", "minItems": 1 }
}
}
}
},
"schema": [
{
"properties": {
"type": { "type": "string", "set": "chat_thread" },
"user_id": { "type": "string", "set": "{{json.userId}}" },
"title": { "type": "string", "set": "{{json.title}}" },
"updated_at": { "type": "integer", "set": "{{toUnix json.updatedAt}}" }
}
}
],
"template": "# {{title}}\n{{#each messages}}\n\n**{{role}}:** {{{text}}}\n{{/each}}\n"
}
set expressions resolve against the attributes, so they read json.*. Named schemas, map, template and render work exactly as for JSON files; see the Inference Rules Guide.
The embedded text is chosen like this:
template or render: its output is embedded and split with the Markdown splitter. The template context holds the JSON record's top-level fields, plus source (the identity object above), json (the whole record) and the rule's metadata, in that order of precedence (later wins). A record field named source or json, or one with the same name as a metadata property, is shadowed; read it as json.<field>..json file would be (an empty template or render output is logged and falls back to this).A document with no text to embed (JSON that extracts to nothing) fails with no_content; nothing is written. If a set expression, template or render fails, the document fails with interpolation_error (one details entry per failed step) and nothing is written. (A file in the same situation is still indexed without the failed values, with an interpolation_error issue.)
Chunking follows the instance-wide embedding.chunkSize and embedding.chunkOverlap. A long record becomes several points, all carrying the same source; when it shrinks, the surplus chunks are removed.
POST /render { "source": "..." } shows the result without writing anything (see the API Reference).
Dates use host time. Template and
sethelpers such asdateFormat(andtoUnixon date strings without an offset) use the host's local timezone. Run the watcher withTZ=UTC(or another fixed zone) if rendered dates must not depend on the host. See Deployment.
Each run retrieves the current record and ends in one of four outcomes (the status reported by POST /documents?wait=true):
| Status | When | Effect |
|---|---|---|
indexed |
Content changed (or the document is new); or the content is unchanged but the projected metadata differs. | Content change: re-chunk, re-embed, upsert, remove surplus chunks. Metadata-only change: setPayload on every chunk (and delete-payload for keys no longer projected), no embedding call. |
unchanged |
Same rendered-text hash (content_hash) and same metadata. |
Nothing written, no embedding call. Clears the reference's open issues, unless one of them is an upsert_error (then all are kept: a partial upsert can leave the new hash on chunk 0, so a matching hash does not prove success). |
deleted |
The callback answered 404. |
Points, enrichment metadata and issues for the reference are removed. |
failed |
Any failure (see Issue codes). | Nothing written; points untouched. An issue is recorded under the reference. A failed Qdrant read (of the stored chunk 0) is a failure too, never treated as "not indexed": the document is not re-embedded on the strength of a read error. |
Details:
upsert_error issue.matched_rules on a source point is always the designated rule; rule or enrichment metadata cannot override it.GET /search/facets and the values of GET /config: those list values seen in files only.created_at is kept from the existing chunk 0 when present, otherwise set to the processing time; modified_at is the time of the last write (a content or metadata change; an unchanged run leaves it alone). Business dates belong in rule metadata (as updated_at in the example).POST /metadata { source } is validated against the stored designated rule's schema (400 invalid_request on a type mismatch), stored under the exact reference and merged on every later run, as for files. A reference with no stored point is 404 source_not_indexed and nothing is stored; if Qdrant cannot be read the request fails with 500 internal_error (never 404).GET /issues. A later successful run clears them.POST /documents validates each notification synchronously and, by default, answers 202 with accepted and rejected after queuing the accepted references. A request with 0 or more than 100 documents, a missing or empty rule, or an unknown key fails as a whole with 400 invalid_request; a bad reference, unconfigured source, unsafe id or unknown rule rejects only its own document (invalid_reference, unknown_source, invalid_id, rule_missing). Processing happens on an in-memory notify queue keyed by reference:
embedding.concurrency references (default 5) are processed at once.shutdownTimeoutMs. Stopping drops whatever is still pending (including owed re-runs) and waits for in-flight runs; a request still being handled when the queue stops gets queue_stopped for its documents. The enrichment store is closed only when both stops finished; if a stop timed out (a run still in flight) it is left open so that run can finish its writes. Re-notify, or run a reindex, after a restart if that matters.extraction_error issue under the reference, with the designated rule, so it is never lost silently.With ?wait=true, the same queue processes the notifications (so concurrency and coalescing still apply) and the response carries one result per document. Use it for backfills and tests; the asynchronous path is the normal one, and its failures surface in GET /issues.
There is no push mode: the watcher always retrieves content through the callback.
Reindex discovers source documents from Qdrant: chunk-0 source points (non-empty source, no file_path). A document whose chunk 0 has no source_rule is counted but skipped. A document that was never notified is never found: notifications are the only entry point, as filesystem events are for files.
POST /reindex scope |
Source documents |
|---|---|
full |
Every indexed source document is retrieved and re-embedded, even when its content is unchanged. |
rules (no path) |
Every indexed source document is retrieved and its stored rule re-applied. Unchanged content: metadata is updated with setPayload, no embedding call. If the rule now renders different text, the document is re-embedded. |
rules with path |
Files only. |
issues |
Source references with open issues: indexed documents with their stored rule; references that were never indexed (their first notification failed) with the rule recorded on their issue. A reference whose stored-rule lookup fails is reported as failed (callback errors, sources.failed) and the run continues. |
path with source |
The listed references (at most 1000 per request), re-processed with their stored rule. References that are not indexed are skipped (counted in total, not in toProcess). The references are enqueued on the notify queue like notifications: the request answers at once, and the work is not a tracked reindex run (no GET /status progress, no completion callback). |
prune |
Never selects source points. Prune only deletes file points (including legacy file points that carry a rule-written source). |
Retrieval outcomes are the same in every scope: 404 deletes the document; any other failure records an issue and leaves its points untouched. In full, rules and issues, source documents run after the files. Up to reindex.concurrency (default 50) documents are submitted to the notify queue at once; the queue's own concurrency (embedding.concurrency) and each source's concurrency still cap the runs and callbacks. A failing document never stops the run. The completion callback (reindex.callbackUrl) then carries sources (counts of indexed, unchanged, deleted, failed) and an errors list naming the failed file paths first, then the failed source references.
Every source run of a reindex goes through the notify queue: a reindex and a notification of the same reference never run in parallel, and when they meet they coalesce into one run (forced re-embedding if either asked for it) that retrieves the latest record.
A config-triggered reindex (configWatch.reindex) covers files only. To re-apply changed rules to source documents, call POST /reindex explicitly (rules or full).
To remove source documents, use POST /documents/delete with exactly one of sources (1 to 1000 references; more is 400 invalid_request) or filter (a Qdrant filter); other keys are rejected. The filter is always constrained to source points (non-empty source, no file_path), so it can never delete a file point, including a legacy file point carrying a rule-written source. A filter without any condition ({}, { "must": [] }, ...) is rejected with 400 invalid_request rather than deleting every source document, as is a filter Qdrant rejects. The response is { "deleted": <documents removed> }. Enrichment metadata and issues are removed for the deleted references and, in the sources form, for every listed reference. A notification still queued for a deleted reference re-indexes it when it runs. See the API Reference.
The watcher always creates keyword payload indexes on source and source_rule. When many users' documents share one collection, index the tenant field as well and filter on it in every search:
{
"vectorStore": {
"url": "http://localhost:6333",
"collectionName": "app_documents",
"payloadIndexes": [
{ "field": "user_id", "type": "keyword", "isTenant": true },
{ "field": "updated_at", "type": "integer", "isPrincipal": true }
]
}
}
See vectorStore.payloadIndexes. A field may be listed once; source and source_rule are indexed by the watcher and may not be listed. If an index already exists with the same type but different isTenant / isPrincipal flags, the watcher logs a warning and leaves it as it is (it never recreates an index); drop it in Qdrant to apply the new flags. The watcher does not know who the end user is: the calling application is responsible for adding the tenant filter to every search.
When the API is reachable from another host, set api.apiKey. Every route except GET /status then requires Authorization: Bearer <key>. See API Authentication.
@karmaniverous/jeeves-watcher-sdk is a thin, typed TypeScript client for the application-facing API (status, search, scan, render, metadata, reindex, issues, documents.notify, documents.delete), plus the callback-contract types. Request and response types come from @karmaniverous/jeeves-watcher-core, the same schemas the service validates with.
import { createWatcherClient, f } from '@karmaniverous/jeeves-watcher-sdk';
const watcher = createWatcherClient({
baseUrl: 'http://watcher.internal.example:1936',
apiKey: process.env.WATCHER_API_KEY,
});
await watcher.documents.notify([
{ source: `app://chat/thread/${threadId}`, rule: 'chat-thread' },
]);
const hits = await watcher.search({
query: 'password reset',
limit: 8,
filter: f.and(f.eq('type', 'chat_thread'), f.eq('user_id', userId)),
});
POST /documents, POST /documents/delete and the source forms of /render, /metadata and /reindexsources, api.apiKey, vectorStore.payloadIndexes and rate limitingset, map, templates