compose
Synopsis
Compose index definition file(s) from a sample list produced by list. Building a new index requires a profile from profile.
USAGE
| Argument | Description |
|---|---|
INPUT_FILE |
JSONL sample list produced by list |
-o, --output-dir COMPOSE_DIR |
Output directory for index definitions |
-n, --name TEXT |
Name of created or updated index |
-pf, --profiles-file FILE |
Profiles YAML file with index configuration (required to build a new index) |
-S, --session-id SESSION |
Session name: subdirectory of COMPOSE_DIR/NAME/ and tag appended to index names (default: timestamp) |
I/O
Input: JSONL sample list (from list), profiles YAML required only for new index creation (from profile)
Output: index definition files in COMPOSE_DIR/NAME/SESSION/, with NAME.yaml as the entry point (INPUT_FILE for plan, build and apply)
Advanced Options
| Option | Description |
|---|---|
-pr, --profile TEXT |
Profile name to use (default: default_profile from profiles file) |
-p, --partition-count INT |
Partitions per index, fixed for its whole life; 0 lets the first build size it (default: 0) |
-si, --shard-size SIZE |
Max size of one shard (e.g. 256GB, 500000MB); omit for a single unlimited index per span |
-m, --partition-min-size SIZE |
Minimum partition file size (e.g. 500MB, 1GB) |
-P, --partition-count-limit INT |
Upper bound on auto partition count (default: 256) |
Description
Takes a JSONL sample list (produced by list) and generates index definition
files that can be passed to plan, build or apply.
Output files are written to COMPOSE_DIR/NAME/SESSION/, where SESSION defaults to the
current timestamp if --session-id is not provided. Pass the NAME.yaml file in that
directory as the input to plan, build or apply to process the index.
Building a new index - provide --profiles-file (produced by profile).
A layout file is written to COMPOSE_DIR/NAME_layout.yaml for future updates.
Updating an existing index - omit --profiles-file. The layout file at
COMPOSE_DIR/NAME_layout.yaml is detected and loaded automatically.
If --profile is not specified, the default_profile field in the profiles file is used.
Partitioning - each Bloom filter is split into N partition files. Two indexes can only be merged when they share that count, and an update merges new samples into an existing shard, so the count is fixed once for the life of the index and stored in the layout file.
Set it with --partition-count, or leave it at 0: the first plan, apply or build then
sizes it from the resource limits (--limits), for a full shard when sharding is on, and writes
it to the layout. Later runs reuse the stored value and warn when -p asks for another one.
The minimizer size (--minim-size) is stored the same way.
The layout also records, per span, the samples composed so far (total_samples) and the number
of sessions that added samples to it (updates).
Sharding - with --shard-size, a span is split into independent shards of at most that
size, instead of one index growing without bound. Shards are never merged together: each one
is built and registered on its own, and a query hits them all.
The per-span sample limit is derived from the Bloom filter size of the span, since a shard
costs about bf_size x samples / 8 bytes, and is rounded down to a multiple of 8 samples
(minimum 8). It is written to the layout file, together with the shards and their sample
counts, so a later session knows where to continue.
An update fills the last shard of the span until that limit is reached, then opens a new one. Filling a shard means merging the session's new samples into it; only chunks (a build-time split, see plan) are merged as well.
Without --shard-size, a span keeps a single index named NAME_g<i>. With it, shards are
named NAME_g<i>_p<k>.
Examples
# Build a new index (writes layout to ./db/my_index_layout.yaml)
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml
# Build with a session tag (output goes to ./db/my_index/my_session/)
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml -S my_session
# Use a specific profile
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml --profile baseline
# Override partition count
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml --partition-count 4
# Set minimum partition size
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml --partition-min-size 500MB
# Split large spans across multiple sub-indices
kmhelpers compose samples.jsonl -o ./db -n my_index -pf profiles.yaml --split-size 10GB
# Update an existing index (auto-detects ./db/my_index_layout.yaml)
kmhelpers compose samples.jsonl -o ./db -n my_index