Search

One query, three lanes

The search service (role search, gRPC port 9094) is backed by Apache Lucene, the Java search library. It serves a fixed set of mapping subjects: each subject names an index mapping, and optionally a chunk lane, that the service was mounted with. Every query must pick one of three lanes, explicitly:

• Lexical. Analyzed term queries over the mapping's TEXT fields. This is classic keyword search: it matches the words and their grammatical forms.
• Vector. k-nearest-neighbor search (kNN, find the k most similar vectors) over the embedding vectors of the document's chunks. The query text is embedded by the same provider that embedded the chunks, so both sides live in the same vector space. Shipped providers: model2vec (in-process), TEI (gRPC), and OVMS (KServe v2 gRPC).
• Hybrid. Run both lanes and merge the two ranked lists with reciprocal-rank fusion (RRF): each result's score is the sum of 1 / (k + rank) over the lists it appears in, with k = 60. RRF only looks at ranks, not raw scores, so a document that ranks well in both lanes rises above a document that ranks first in only one.

There is no default lane. An unset lane, an unknown subject, a field outside the subject's mapping, a vector query against a subject with no chunking policy, or a k over 10,000 is refused by name, never silently defaulted. The contract's header puts it in one line: "the service never guesses what a caller meant." A reranking extension point, with providers and a certification harness, exists but is not wired into the query lanes.

The fusion constant and the whole merge fit in two lines:

dev-tools/protomolt search/service, LuceneSearchStore.java (abridged)

/** Reciprocal-rank fusion constant for the hybrid lane. */
private static final int RRF_K = 60;
...
scores.merge(key, 1.0 / (RRF_K + rank + 1), Double::sum);

The mapping is declared on the schema, not in a config file

Most search systems keep their index definitions (which fields are text, which are keywords, which are vectors, which analyzers to apply) in a separate mapping file or a database, maintained next to the schema. ProtoMolt puts them on the schema itself, as a field option, index, from the ai.protomolt.proto.index.hints.v1 package. The option says the field's index type (keyword, text, integer, date, vector, and friends), the analyzer for text, whether numeric fields should be sortable or facetable (doc values for sorting and grouping), how a range or a taxonomy path should be emitted, and, for vectors, the dimensions, the similarity measure (cosine, dot product, L2, max inner product), and engine-neutral HNSW tuning (the m and ef_construction graph parameters, which engines map onto their native knobs). A TREE_PATH field is indexed as its ancestor chain, so a document filed under a/b/c also matches a query for a. The mounted Lucene store does not yet honor these tuning knobs and builds vector fields with Lucene's default parameters; the OpenSearch mapping generator does honor them.

Three pieces cooperate, in a separation the docs state explicitly: a hint says how a field should be indexed, a mapping resolves hints for a concrete message type, and an engine plugin interprets the mapping for one backend. Hints come from a chain of sources, composed so the first hit wins:

flowchart LR
          cat["CatalogIndexingHintSource, side-car entries for schemas you cannot annotate"] --> chain["composed with orElse"]
          opt["ProtoOptionsIndexingHintSource, descriptor options"] --> chain
          chain --> inf["InferringIndexingHintSource, falls back to the proto type"]
          inf --> factory["IndexMappingFactory.defaults walks the descriptor"]
          factory --> map["IndexMapping: paths, engine field names, hints, repeated flag, sensitivity class"]
          map --> plugins["SearchEngineIndexerProvider plugins, discovered via ServiceLoader"]
          plugins --> lucene["lucene"]
          plugins --> dist["distributed-lucene, the sibling distributed engine"]
          plugins --> more["opensearch, solr, qdrant"]

IndexMappingFactory.defaults(hintSource).create(descriptor) walks a message descriptor and produces an IndexMapping: the message's full name plus a list of indexed fields, each carrying two naming vocabularies (the dot-separated protobuf path and the name written into the engine document), the resolved hint, a repeated flag, and the field's sensitivity class. A schema you own declares its hints inline; a schema you cannot touch gets entries in the programmatic catalog, keyed by message and field name; and whatever neither covers is inferred from the proto type. Declaring a real index, then, looks like this:

message Doc {
  string doc_id = 1 [(ai.protomolt.proto.index.hints.v1.index) = { type: INDEX_FIELD_TYPE_KEYWORD }];
  string title  = 2 [(ai.protomolt.proto.index.hints.v1.index) = {
    type: INDEX_FIELD_TYPE_TEXT
    analyzer: "english"
    sub_fields: [{ name: "raw", type: INDEX_FIELD_TYPE_KEYWORD }]
  }];
  repeated float embedding = 3 [(ai.protomolt.proto.index.hints.v1.index) = {
    type: INDEX_FIELD_TYPE_VECTOR
    vector_dims: 768
    vector_similarity: VECTOR_SIMILARITY_COSINE
    hnsw: { m: 16, ef_construction: 128 }
  }];
}

Chunking and embedding, declared the same way

A text field can also carry a chunking policy: at index time the service splits the field's text into chunks (the shipped chunker packs sentences up to a token budget, over one of two pinned boundary rule sets, chosen by id) and embeds each chunk, so the same Lucene index holds keyword fields, analyzed text, doc values, and HNSW vectors. There is no separate metadata file: the .proto is the metadata. The policy computes a SHA-256 digest over a canonical rendering of every component, and chunk identity is doc_id#digest#ordinal. Change any policy value and you get a new digest, a new chunk generation, and a re-index via replay. Two indexes agree on chunk boundaries and vector spaces exactly when their digests agree.

// A chunking policy: derive chunks and vectors from the annotated source text
// field at index time (chunk + embed on the server). Attach to a string field.
// The policy digest identifies the derivation pipeline; index consumers pin it
// the way they pin a schema.
message ChunkingPolicy {
  ChunkingSpec chunking = 1;
  EmbeddingSpec embedding = 2;
  string vector_field = 3;   // empty = engine convention ("<field>#<model>")
  bool store_chunk_text = 4;
}

One declaration, rendered for each engine

The payoff of putting hints on the schema is that engine configuration becomes a rendering problem. The render-index-mappings verb takes a message type and an engine name and emits that engine's native index definition, derived entirely from the declarations: for OpenSearch, a mapping JSON whose vector fields come out as knn_vector with the hint's dimensions, similarity, and HNSW parameters applied; for Solr, a managed-schema fragment using DenseVectorField; for Qdrant, a collection definition with named vectors and payload indexes; and for Lucene, a field-spec report describing what the mounted store will build. The same message type yields all four, so moving a corpus between engines starts from the schema, not from a hand-ported mapping file.

{ "mappings": { "properties": {
    "doc_id":  { "type": "keyword" },
    "title":   { "type": "text", "analyzer": "english",
                 "fields": { "raw": { "type": "keyword" } } },
    "embedding": { "type": "knn_vector", "dimension": 768,
                   "method": { "name": "hnsw", "space_type": "cosinesimil",
                               "parameters": { "m": 16, "ef_construction": 128 } } }
} } }

<fieldType name="knn_768_cos" class="solr.DenseVectorField"
           vectorDimension="768" similarityFunction="cosine"/>
<field name="doc_id"    type="string"      indexed="true" stored="true"/>
<field name="title"     type="text_en"     indexed="true" stored="true"/>
<field name="title.raw" type="string"      indexed="true" stored="false"/>
<field name="embedding" type="knn_768_cos" indexed="true" stored="true"/>

{ "vectors": {
    "embedding": { "size": 768, "distance": "Cosine" }
  },
  "payload_indexes": [
    { "field": "doc_id", "schema": "keyword" },
    { "field": "title",  "schema": "text" }
  ] }

doc_id     KEYWORD   StringField + SortedDocValues
title      TEXT      analyzer=english; sub-field raw: KEYWORD
embedding  VECTOR    KnnFloatVectorField(dims=768, COSINE)
           note: Lucene graph defaults; the hint's hnsw {m, ef_construction}
           is not applied by the mounted store

The plugin seam is deliberately narrow. Engine plugins implement a write-side indexer interface, discovered from the classpath; there is no query-side index SPI, so the SPI says how documents land but not how they are asked for. The shipped read paths are the mounted Lucene service and, for OpenSearch, a kNN search plus a recall-then-rerank composition in the OpenSearch module. The Solr and Qdrant plugins are write shape only today, with no read side, and an ndjson plugin exports documents as newline-delimited JSON without consulting hints at all. The sibling distributed engine already plugs into the same seam under the engine id distributed-lucene, and the Rust engine the search sub-pages cover is being wired in as the next provider behind it.

Sensitivity classes declared in the schema's metadata carry into the rendered mappings as index security. A field marked for encryption renders as store-only ciphertext, a keyword field the engine neither indexes nor keeps doc values for, so semantic search runs over content the engine never sees in the clear: the ciphertext is findable by vector but a term search on it refuses. Mask and exclude classes render OpenSearch security-plugin role fragments (masked_fields and field-level security), up to a complete role body ready to install. A live integration test proves the encrypted-store pattern end to end against a real OpenSearch: index the ciphertext, find it by vector, watch the engine refuse a term search on it. The docs state the boundary plainly: a vector is derived from plaintext and leaks through neighborhood and inversion channels, so treat the vectors as confidential in their own right.

How a document lands in the index

Indexing is workflow-driven. A durable run's index step calls SearchIndexService/IndexDocument after parsing, fetching the document's core-plus-parsed planes over the repository wire contract (with a 30-second call deadline). The service maps the parent document under the subject's mapping, derives the chunk children from the chunk lane, and writes the whole family as one Lucene block: children first, parent last, which is Lucene's block contract. Each chunk child gets its vector field, named by the <field>#<model> convention. At query time a vector hit lands on a chunk child, and the block join relates it back to the parent document, so the caller gets the document rather than a bare chunk.

dev-tools/protomolt search/service, LuceneSearchStore.java (abridged)

// Children first, parent last: Lucene's block contract. The
// update deletes every previous block member carrying the
// identity term and adds the new block in one atomic step.
subject.writer().updateDocuments(
        new Term(served.docIdField(), docId), block);

The write is an atomic replace by identity: updateDocuments deletes every previous block member carrying the document's identity term and adds the new block in one step, so replays never duplicate. The out-of-the-box subject, repo-document, indexes identity plus folded search metadata, skipping the storage and provenance planes, with an optional chunk lane over the folded body.

Visible now, durable soon after

Lucene distinguishes visibility from durability, and the service exploits the split. Every write refreshes the NRT (near-real-time) searcher, so a document is queryable immediately. Fsync commits, the expensive part, batch up: every 64 writes, every 2 seconds, or on close. The service's own docs state the worst case: a crash loses at most the last interval's writes, and a replay re-derives them from the repository, which stays the source of truth.

Snapshots, and the cheap read-only node

The store has a SnapshotStore seam, with an S3 implementation. A snapshot copies a commit point to a bucket, keyed subject/mapping-digest/policy-digest, with the segments_N file written last as the atomic marker: a reader either sees the whole commit point or none of it. A read-only variant of the service opens no IndexWriter, answers UNIMPLEMENTED on the indexing RPC, restores from snapshots, and can follow the writer on a refresh interval. That is what makes a remote read-only analytics node cheap to run: mount the search and metric roles together, point them at the snapshot bucket, and the node serves queries and aggregates with no local indexing at all. The docs summarize the relationship: the repository stays the source of truth, and a snapshot is a cache. The S3 implementation is still a prototype.

One guard sits upstream of all of this: a VectorizationPolicy (by default, unclassifiedOnly()) rejects, at mount time, fields to embed that carry a sensitivity classification.

Metrics over the same index

The metric service (role metric, gRPC port 9095) answers aggregate questions, group-by-style measures and dimensions, over the same index. Its members are declared on the schema too, via the metric field option and metric_message message option from ai.protomolt.proto.metric.v1: each declares its role (dimension or measure), its aggregate (count, sum, average, min, max, count-distinct), and the public name a query may use. What is queryable is exactly what the schema declares; a request for anything else is rejected by name.

The bridge between the two services is Lucene doc values, the columnar per-document storage that the search service already writes when a hint says sortable or facetable. The Lucene metric executor runs a single pass of collectors over those doc values, so aggregation is a read path over existing storage. The README's phrase is "no copy and no ETL." Concretely, the metric role reaches into the co-mounted search role and borrows its live searchers:

dev-tools/protomolt metric/lucene, MetricServiceModule.java (abridged)

// The search role must wire first so its store contribution exists
...
List<LuceneSearchStore> stores =
        context.contributions().all(LuceneSearchStore.class);

So the metric role will not start without the search role on the same node: its fast path is literally the search role's index. A second executor runs one DuckDB SELECT ... GROUP BY over the Parquet files an Iceberg sink wrote to the document lake, for lake-scale aggregates. Backend selection is explicit: an unspecified backend resolves to the mount's engine on a single-engine mount, and on a multi-engine mount it is refused, naming the mounted engines.

Three rules keep the numbers trustworthy. COUNT_DISTINCT counts exactly, up to 100,000 tracked values per measure, and rejects a request past that bound instead of estimating. Declared rollups rebuild a whole aggregate table and atomically replace it: exact or refused, a result that fills the group budget never lands. And security is rewritten into the compiled query, not bolted on after: a principal's denied members refuse by name, and injected row filters (equality conditions from the access policy) are ANDed in at compile time. A principal carrying metric access rules is refused outright from rebuilding rollups, because a rebuilt rollup is an unrestricted reduction they could then read around their own row filters.

int64 amount_cents = 4 [
  (ai.protomolt.proto.index.hints.v1.index) = {
    type: INDEX_FIELD_TYPE_INT64
    sortable: true
  },
  (ai.protomolt.proto.metric.v1.metric) = {
    role: MEMBER_ROLE_MEASURE
    aggregate: AGGREGATE_SUM
    name: "revenue"
  }
];

Read side and read side agree because the declaration is shared: the same field that declares sortable: true is the one the metric service sums, and every metric response carries a physical_plan (the collector description or the rendered SQL) as evidence of how the answer was produced.

Request limits on the search and metric listeners

Both services admit unary calls when the headers arrive, before gRPC has finished receiving or decoding the request. Each admitted call reserves room for two maximum requests (4 MiB per request and 64 concurrent calls by default, set through withAdmission), and a call the remaining capacity does not cover is refused at once with RESOURCE_EXHAUSTED. On a server started with an API token, every service is mapped to a scope (search-query, search-index, metrics-query, metrics-rebuild), and a scoped caller whose budget for that scope sets a positive maxPayloadBytes is measured frame by frame as gRPC reads it. The size checked is the uncompressed size, gzip frames included, before protobuf parsing and before the request finishes arriving, and a request over the cap is refused with RESOURCE_EXHAUSTED. A compressed frame on a transport where the interceptor cannot read the decompressed stream is refused the same way rather than passed. The per-caller cap does not apply to callers with no budget for the scope, to a budget with maxPayloadBytes of 0, or to a server started without an API token. Separately, gRPC's maxInboundMessageSize bounds the compressed length of each frame. These are limits on these two listeners, not a guarantee against compression bombs across every surface.

The search console

Ships with the service: a small web console (role search-console, HTTP port 8096) that is one pure-JDK HTTP server rendering one page, no framework and no build step. Its subject and lane pickers are populated from ListSubjects, so the page offers exactly what the service serves and no more. On guarded nodes it requires a browser login (an HttpOnly cookie, 12-hour session) and the principal must hold the search-query scope.

Pipestream Search: the distributed engine

Pipestream Search is a separate engine written in Rust, under active daily development. Most engines that search by meaning rather than by keyword build a graph of each item's nearest neighbours and walk it towards the question, which is fast and which can miss a good answer sitting in a part of the graph the walk never reached. This engine stores every vector in a compact four-bit form and reads all of them on every query, with an optional second pass that rescores the best candidates against the exact original numbers. Its keyword side is a word-list index scored by BM25, using counts gathered from the whole corpus and sent to every machine, so that scores computed in different places can be compared with each other.

The coordinator keeps the only ranked list in a query and tells the shards the score of its current tenth-best answer, over the gRPC stream and a signed UDP channel beside it, so the machines can stop considering hopeless candidates together and the answer the fleet returns is the answer one machine holding everything would have returned. Measured on a corpus of 86.6 million court-opinion chunks across eight shards, two of them Raspberry Pi boards, a vector query returns in 280 ms at the median.

It reads the same index mappings declared on the descriptor as everything above, because the declaration is the contract rather than any one backend's configuration, and it is being integrated into ProtoMolt as a search engine provider behind the same plugin seam the engines above use. Three pages cover it in depth: the engine overview, the feature tour, and an interactive demo with real explain trees.