@karmaniverous/entity-managerThis guide is written for STAN assistants working in codebases that use Entity Manager. It is intended to be self-contained: it explains the library’s runtime model, configuration rules, and the type-level contracts you must satisfy when integrating a provider (e.g., DynamoDB) or writing app code.
Entity Manager is provider-agnostic: it does not talk to your database. Instead it:
addKeys, getPrimaryKey, removeKeys).query).pageKeyMap).This repo uses a strict generic-parameter dictionary. In this guide, acronyms appear primarily in generic type parameters.
config.indexes).const attrs = ['userId'] as const).Entity Manager assumes a table with a global hash key and range key property name shared across all entity types:
config.hashKey: e.g. "pk" or "hashKey2".config.rangeKey: e.g. "sk" or "rangeKey".For each entity token:
"<entityToken><shardKeyDelimiter><shardSuffix>", e.g. user!0a.
chars === 0, the suffix is empty: user!."<uniqueProperty><generatedValueDelimiter><uniqueValue>", e.g. userId#abc123.Each entity can define a time-based scale-up schedule via shardBumps:
{ timestamp: number; charBits: 1..5; chars: 0..40 }
timestampProperty on the item (e.g., created).charBits controls radix: radix = 2 ** charBits.chars controls width; full shard space is radix ** chars.uniqueProperty (string-hash) modulo the full shard space.Generated properties are additional string-valued tokens used as index key components:
"<hashKeyValue>|k#v|k#v..."null/undefined, encoding returns undefined."k#v|k#v..." (missing values become empty strings)config.indexes describes secondary index key tokens (not provider-specific index definitions):
indexes: {
[indexToken]: {
hashKey: config.hashKey | shardedGeneratedKey;
rangeKey: config.rangeKey | unshardedGeneratedKey | transcodedProperty;
projections?: string[]; // optional, validated (must not include key tokens)
}
}
Entity Manager uses index definitions for:
createEntityManagerUse createEntityManager(config) with a values-first config literal (as const) to capture tokens and index names:
import { z } from 'zod';
import {
createEntityManager,
defaultTranscodes,
} from '@karmaniverous/entity-manager';
const userSchema = z.object({
userId: z.string(),
created: z.number(),
firstNameCanonical: z.string(),
lastNameCanonical: z.string(),
});
const config = {
hashKey: 'hashKey2',
rangeKey: 'rangeKey',
generatedProperties: {
sharded: { userPK: ['userId'] as const },
unsharded: {
firstNameRK: ['firstNameCanonical', 'lastNameCanonical'] as const,
},
},
propertyTranscodes: {
userId: 'string',
created: 'timestamp',
firstNameCanonical: 'string',
lastNameCanonical: 'string',
},
indexes: {
created: { hashKey: 'hashKey2', rangeKey: 'created' },
firstName: { hashKey: 'hashKey2', rangeKey: 'firstNameRK' },
userCreated: { hashKey: 'userPK', rangeKey: 'created' },
} as const,
entities: {
user: { uniqueProperty: 'userId', timestampProperty: 'created' },
},
// compile-time shape capture only; stripped from runtime config
entitiesSchema: { user: userSchema },
transcodes: defaultTranscodes,
} as const;
const em = createEntityManager(config);
Notes:
entitiesSchema is compile-time only; the runtime config is still validated by Zod, and entitiesSchema is stripped before parsing.as const (especially on indexes and generated property element arrays) to preserve literal unions.The constructor parses config with Zod and rejects many mismatches, including:
generatedKeyDelimiter, generatedValueDelimiter, shardKeyDelimiter)propertyTranscodesIf config parsing fails, treat it as a schema error and fix the config, not the code.
Entity Manager’s public item/record types are by-token (narrowed by entity token).
EntityItem<CC, ET>
EntityItemPartial<CC, ET, K = unknown>
K omitted: permissive partial.K provided: a Pick<...> restricted to K.EntityRecord<CC, ET>
EntityRecordPartial<CC, ET, K = unknown>
K).Token-agnostic storage helpers (exported for reference and advanced use):
StorageItem<CC> / StorageRecord<CC> (property-level, flattened across entities)EntityKey<CC>: Record<CC['HashKey'] | CC['RangeKey'], string>PageKeyByIndex<CC, ET, IT, CF> narrows the page key object to only the key components for that index when CF carries indexes.PageKey<CC> shape.addKeys, removeKeys, getPrimaryKeyaddKeys(entityToken, item, overwrite?)EntityItemPartial).EntityRecordPartial) with:
overwrite=false)Use it before writes.
const record = em.addKeys('user', {
userId: 'u1',
created: Date.now(),
firstNameCanonical: 'lee',
lastNameCanonical: 'zhang',
});
// record.hashKey2 and record.rangeKey are present (strings)
removeKeys(entityToken, recordOrRecords)EntityRecord input yields strict EntityItem output.EntityRecordPartial<..., K> yields EntityItemPartial<..., K>.Use it after reads if your app layer should not see storage keys.
getPrimaryKey(entityToken, itemOrItems, overwrite?)EntityKey<CC>[]).overwrite=false, returns exactly that pair.rangeKey from uniqueProperty.timestampProperty is present, computes exactly one hash key.timestampProperty is missing, enumerates hash key space across bumps (deterministic per bump if uniqueProperty is present).Use it to generate keys for point reads / batch reads.
EntityManager.query() doesEntity Manager’s query API is orchestration only. You provide shard query functions (provider integration); it runs them across shards and indexes and merges results.
ShardQueryFunctionShardQueryFunction<CC, ET, IT, CF, K> is:
(hashKey: string, pageKey?: PageKeyByIndex<...>, pageSize?: number)
=> Promise<{ count: number; items: EntityItemPartial<CC, ET, K>[]; pageKey?: PageKeyByIndex<...> }>
Rules for a correct implementation:
hashKey value.pageKey and pageSize as “resume token” and per-shard page size.pageKey: undefined when that shard/index is exhausted.When you call em.query(options):
pageKeyMap into a 2-layer structure:
indexToken -> hashKeyValue -> pageKey | undefinedentityToken, hashKeyToken, and [timestampFrom, timestampTo].throttle).pageKey === undefined (no pages remain), orlimit (note: fan-out can exceed limit).uniqueProperty and sorts by sortOrder.result.pageKeyMap) you pass into the next call.QueryOptions fields you must understandKey options:
entityToken: selects entity config (uniqueProperty, timestampProperty, shard bumps, defaults).item: a partial item used to generate alternate hash key spaces (if needed); may be {} for global hash key indexes.shardQueryMap: map of indexToken -> ShardQueryFunction (single- or multi-index).pageKeyMap?: string: the returned token from a prior query call.limit?: number | Infinity: target maximum total items across shards (positive int or Infinity).pageSize?: number: max per-shard page size (positive int).timestampFrom?: number, timestampTo?: number: bounds which shard bumps/shards are queried.sortOrder?: SortOrder<...>: used after merge/dedupe.throttle?: number: max shard queries in flight.import type {
QueryOptionsByCF,
ShardQueryFunction,
ShardQueryMapByCF,
} from '@karmaniverous/entity-manager';
const cf = {
indexes: {
firstName: { hashKey: 'hashKey2', rangeKey: 'firstNameRK' },
lastName: { hashKey: 'hashKey2', rangeKey: 'lastNameRK' },
},
} as const;
type CF = typeof cf;
type CC = /* your ConfigMap<...> */;
const firstNameSQF: ShardQueryFunction<CC, 'user', 'firstName', CF> =
async (hashKey, pageKey, pageSize) => {
// provider-specific query here
return { count: 0, items: [], pageKey };
};
const lastNameSQF: ShardQueryFunction<CC, 'user', 'lastName', CF> =
async (hashKey, pageKey, pageSize) => {
return { count: 0, items: [], pageKey };
};
const shardQueryMap: ShardQueryMapByCF<CC, 'user', CF> = {
firstName: firstNameSQF,
lastName: lastNameSQF,
};
const options: QueryOptionsByCF<CC, 'user', CF> = {
entityToken: 'user',
item: {},
shardQueryMap,
pageSize: 25,
limit: 200,
};
const result = await em.query(options);
// result.pageKeyMap is a compact token: pass it back into the next call
K is a type-only way to express “I projected a subset of attributes at runtime”.
If you pass a projection tuple type through your shard query functions/options/results, then:
Pick<EntityItem<...>, KeysFrom<K>>[].Example:
const attrs = ['userId', 'created'] as const;
type K = typeof attrs;
const sqf: ShardQueryFunction<CC, 'user', 'firstName', CF, K> = async () => ({
count: 0,
items: [],
});
Critical invariant:
uniqueProperty and sorts by sortOrder.uniqueProperty, andOtherwise, runtime dedupe/sort may be incorrect.
Entity Manager supports two complementary inference routes:
If you have a literal like:
const cf = { indexes: { firstName: {...}, lastName: {...} } } as const;
Then:
IndexTokensOf<typeof cf> becomes 'firstName' | 'lastName'ShardQueryMapByCF<..., typeof cf> only allows those keysPageKeyByIndex<..., 'firstName', typeof cf> narrows the page key shape to that index’s componentsIf you want ITS derived from a captured config literal type (commonly your config value type):
ShardQueryMapByCC<CCMap, ET, CCLiteral>QueryOptionsByCC<CCMap, ET, CCLiteral>These derive ITS from IndexTokensFrom<CCLiteral> and pass CCLiteral through the CF channel for page-key narrowing.
findIndexToken convenienceEntityManager.findIndexToken(hashKeyToken, rangeKeyToken, suppressError?) returns an index token, and when the manager was constructed with a values-first config literal, its return type narrows to IndexTokensOf<CF>.
This package includes abstract helpers meant for provider adapters.
BaseEntityCliententityManager and a logger (debug/error).batchProcessOptions (used by downstream adapters).BaseQueryBuilderBaseQueryBuilder helps adapters construct a ShardQueryMap fluently while keeping the provider-specific logic inside getShardQueryFunction(indexToken).
Key points:
IndexParams type (your per-index mutable query state).indexParamsMap: Record<ITS, IndexParams>.build() returns a ShardQueryMap by mapping index tokens to shard query functions.query() forwards to entityManager.query(...) with the builder’s pageKeyMap and built shardQueryMap.The shared query options type for builder query() is:
QueryBuilderQueryOptions<CC, ET, CF> = QueryOptions minus entityToken, pageKeyMap, shardQueryMap.When queries behave unexpectedly, check:
propertyTranscodes includes every property referenced by generated properties.hashKey value as the partition key value.pageKey into provider pagination correctly.pageKey: undefined when exhausted.result.pageKeyMap as an opaque token and pass it unchanged.limit is a target cap across all shards. Fan-out may overshoot because it queries shards in batches of pageSize.as const) and avoid broad annotations that widen literal unions.That’s the full contract: if your shard query functions behave like “query one shard page”, Entity Manager can do the rest.