Types & Metadata

One declaration, every subsystem reads it

Protobuf messages are described by descriptors: the runtime blueprint of a message type, available even when no generated code is present. ProtoMolt adds custom options to that blueprint, so a field can state, in the schema file itself, how it should be validated, displayed, indexed, and even how a language model should fill it. Everything operates on the descriptor (a dynamically resolved message gets identical treatment to a compiled one), so the declarations are the single source of truth for every consumer.

The example below is a real field from the toolkit's example corpus: the name of a court in a legal opinion. It carries three option families at once. Try switching between the subsystems on the right, and switching option families off on the left, to see who reads what.

(ai.protomolt.proto.meta.v1.field) = {
  display_name: "Court"
  description: "The issuing court, as it appears in the caption."
  sensitivity: "pii"   // a free-form class string
}

(ai.protomolt.proto.validate.v1.field) = {
  required: true
  string: { max_len: 200 }
}

(ai.protomolt.proto.index.hints.v1.index) = {
  type: INDEX_FIELD_TYPE_TEXT
}

(ai.protomolt.proto.llm.v1.field) = {
  instruction: "Name the court exactly as it appears in the caption."
  safeguards: "Do not abbreviate."
}

The full declaration, exactly as it appears in the example corpus:

dev-tools/protomolt protobuf llm examples, court/v1/opinion_metadata.proto

string court = 1 [
  (ai.protomolt.proto.validate.v1.field) = { required: true  string: { max_len: 200 } },
  (ai.protomolt.proto.meta.v1.field)     = { description: "The issuing court, as it appears in the caption." },
  (ai.protomolt.proto.llm.v1.field)      = { instruction: "Name the court exactly as it appears in the caption."
                                            safeguards: "Do not abbreviate." }
];

The validator: 26 named formats, zero regular expressions

The validate.v1 option dialect is enforced by ProtoValidator, a dialect-neutral engine. It never reads an annotation dialect directly: every dialect implements a rule-source interface and translates into one neutral constraint model, and all violations from all sources merge. The built-in dialect ships with the toolkit, and a vendored copy of buf.validate (the protovalidate schema) plugs into the same engine just by being on the class path. The compatibility claim is at the annotation level: a .proto file already carrying (buf.validate.field) and (buf.validate.message) options works as-is, with no rule rewriting. ProtoMolt implements the supported Buf Protovalidate specification in its own Java engine: the buf.validate module supplies the annotation reader and Buf's attributed schema definition, and rule evaluation runs inside ProtoMolt. Each message type's rules are translated and compiled once, when its constraints are first assembled, and cached; regex patterns and CEL programs compile at that point, so a malformed rule fails on first use of its type, not partway through checking a message.

The string rule surface has 37 knobs. Eleven are structural (min and max length, pattern, prefix, suffix, contains, and friends). The other 26 are named format switches, and each is backed by a hand-written parser in the core/formats package, a zero-dependency library with no java.util.regex anywhere in it. This is a deliberate choice: a regular expression checks shape, a parser checks meaning. The calendar parser refuses 2026-02-30, a date any digit-pattern would accept. The gtin format verifies the GS1 check digit, not just the digit count:

// Value must be a GS1 Global Trade Item Number: 8, 12, 13, or 14
// digits whose final digit is the GS1 mod-10 check digit. Stricter
// than a digit-count pattern: an invalid check digit refuses.
optional bool gtin = 33;

The 26 named formats:

Most vocabulary formats use data that ships with the JDK itself: strict BCP 47 language tags, ISO 4217 currency codes, ISO 3166 region codes via CLDR, so nothing needs bundling or attribution. The one exception is link_relation, which bundles two sorted name tables, attributed in the toolkit's NOTICE: it accepts a Dublin Core relation term (isPartOf), a name from the IANA Link Relations registry (canonical), or an RFC 8288 extension relation written as an http or https URI with a non-empty authority. A relation is rendered as an href, so javascript:, data:, vbscript:, and file: values refuse, and locator-less urn: and tag: URIs refuse as extension relations. Names match exactly as registered, so next passes and Next refuses, and lookups are binary searches with no regular expression. On its own it is a string format: a field can check that a value names a relation. The repository's typed document links use it for the relation each link field declares, and record static links in the catalog so they can be followed in both directions (see document links). SKOS concept schemes are well-known types too (see concept schemes below). endpoint_address is a precedence grammar: it parses host:port before considering URI forms, so localhost:99999 refuses on the out-of-range port instead of passing as a plausible URI. Violation rule ids deliberately align with protovalidate's naming (string.min_len, timestamp.within), so error vocabulary transfers.

Conformance is measured, not asserted. The toolkit drives ProtoValidator against buf's full protovalidate conformance suite with the upstream runner pinned at commit 63347b8: 2880 of 2880 cases pass in the default comparison mode with no skip list. The suite's own runner scores it, and CI runs it on every pull request and every push to main, so the score cannot silently regress.

One declaration fires at every boundary: the gRPC interceptor (server and client side), the Kafka serde's write and read paths, the registry's config gate, the search service's document gate, and the Connect transforms. A message that validates at intake cannot go stale on its way through the pipeline.

Types that extend the basics: ranges and paths

Sometimes a string or an integer is the wrong abstraction. A contract period, a page span, a shelf location in a category tree: these want their own types, but a new message type should never force every client to upgrade. The governing philosophy is wire-plain, server-strict: every value stays a plain protobuf scalar or message on the wire, so any client in any language reads it untouched, and the platform's gates enforce the semantics server-side with structured INVALID_ARGUMENT rejections.

The library lives in protobuf/types: DateRange, LongRange, and DoubleRange share one four-field shape (begin, include_head, end, include_tail), over ISO date strings, 64-bit integers, and doubles. TreePath carries a taxonomy position as an ordered list of segments. The include flags end the closed-versus-half-open ambiguity that bites every range API: an unset flag means the bound is included, and an unset bound means the range is open on that side. Move the sliders:

The range types carry their invariants with them, as message-level CEL rules that enforce at-least-one-bound, begin-not-after-end, and refusal of an include flag without its bound:

// A calendar-day range. Bounds are strict ISO dates, so the
// whole range stays wire-plain: any proto client can read it.
message DateRange {
  option (ai.protomolt.proto.validate.v1.message) = {
    cel: { id: "date_range.ordered"
           message: "begin must not be after end"
           expression: "!has(this.begin) || !has(this.end) || this.begin <= this.end" }
  };
  optional string begin = 1 [(ai.protomolt.proto.validate.v1.field) = { string: { date: true } }];
  optional string end   = 2 [(ai.protomolt.proto.validate.v1.field) = { string: { date: true } }];
  optional bool include_head = 3;  // Absent = included
  optional bool include_tail = 4;
}

The indexing SPI recognizes these types by name: a field of ai.protomolt.proto.types.v1.LongRange indexes as a LONG_RANGE with no hint options at all, and TreePath likewise needs nothing extra. Any other message that happens to declare a matching (gte, lte) or (min, max) bound pair duck-types into a range too.

Taxonomies, postal codes, concept schemes: operator-loaded, never bundled

Some of the well-known types are not values at all but config documents: a Taxonomy (the allowed tree a TreePath may point into), a PostalCodePack (per-country postal code formats), and the concept schemes described below. None ships with data in the tree, deliberately. Reference lists like these tend to carry licenses that forbid bundling (schema.org, for one, is CC-BY-SA and is never bundled), and freezing them into an enum would break wire-plain: every new category or country would become a schema migration for every federated consumer. So an operator loads them as config documents, and the documents carry no name and no version of their own. The config subject (taxonomy:<name>, postal-codes) is the identity, and the config source's version, a git commit or a topic offset, is the version; a copy of either inside the document could only agree or lie.

A TreePath field binds to its taxonomy in the schema, with a taxonomy: "products" entry in its field rules, and enforcement is fail-closed: a declared taxonomy with no mounted document refuses the message rather than waving it through, because "no list loaded" and "every value allowed" must never look the same. Postal formats are UPU-style fixed templates (N for a digit, A for an uppercase letter, anything else literal), deliberately never a pattern, so operator-supplied data cannot smuggle in a regular expression. And where Google already defined the type, ProtoMolt adopts rather than reinvents: the invariants of google.type.* messages (Money, Date, Interval, LatLng, PostalAddress, PhoneNumber) are enforced by a built-in rule source on the default validator lineup.

Concept schemes: SKOS facets, checked before they mount

A ConceptScheme models one facet (subject, department, region, product line) after W3C SKOS, in types/v1/concepts.proto. Each Concept has a stable opaque id independent of its labels, preferred, alternate and hidden labels per language, broader parents, related concepts, cross-scheme exact_match, close_match, broad_match, narrow_match and related_match mappings, notes, notations, and a deprecated flag with replaced_by successors. A document tags a value with a ConceptRef (scheme id plus concept id).

Like a taxonomy, a scheme is an operator-loaded config document with no id or version inside it: its config subject, concept-scheme:<slug>, is the identity. Mounting is verify-then-swap. The scheme's own field rules run first, then the graph checks: unique ids, label integrity, references that resolve, a broader hierarchy with no cycles (stricter than SKOS itself, because a cycle has no well-defined descendants), related kept outside the hierarchy, and replaced_by chains that end at a live concept. A concept's round-trip iri and each notation's datatype_iri must use the http or https scheme with a non-empty authority host, or the urn scheme (schemes compare case-insensitively). Anything else, including javascript:, data:, file: and an authority-less http: reference, is refused as concept_scheme.iri_scheme, so a value that could act when a renderer turns it into a link never rides along on a round trip. A scheme failing any of them is refused with its reasons, and the previously mounted version keeps serving.

// Ids of the concept schemes a value must belong to. Legal only on a
// singular or repeated field of ai.protomolt.proto.types.v1.ConceptRef;
// anywhere else fails rule compilation.
// Enforcement is fail-closed: a bound scheme with no mount refuses
// rather than passes, and a deprecated concept is still a member.
repeated string concept_schemes = 20;

Membership is fail-closed in the same way as taxonomies: a field bound to a scheme that is not mounted refuses with concept.unmounted, and the mounted version rides every membership violation as evidence. A deprecated concept stays a member, so documents tagged before its retirement keep validating. Query expansion is defined by ConceptExpansion: a query on a concept always includes its exact_match closure across mounted schemes and the deprecated concepts whose replaced_by chain reaches it, and can opt into in-scheme descendants and one hop of close_match. An expansion reads one immutable snapshot of the mounts, so it is deterministic for the versions it reports, and one that would pass 10,000 concepts is refused rather than truncated.

The toolkit holds the types, the validator rule and its catalog interface, and the config-lane mount with expansion, ConceptSchemeMounts. The repository builds on them. It checks every bound ConceptRef in a document against the host's mounted schemes at save, indexes the admitted tags per document, and expands DocumentFilter.concept through the mounted schemes when it filters listings and dynamic links, returning the mounted versions it read as evidence (see concept tags). A stock repository host mounts its schemes once at boot from DOCUMENT_PLATFORM_CONCEPT_SCHEMES, a list of directories, files or schemeId=path pairs of binary ConceptScheme files, each validated as it mounts, and an embedding host can pass a ConceptSchemeMounts to RepoServices.withConceptSchemes. With none mounted every bound tag is refused as concept.unmounted and a concept filter answers FAILED_PRECONDITION. Search queries do not expand concepts, and a validator built without a catalog refuses every bound scheme.

Sensitivity: one string, three subsystems act on it

meta.v1 metadata includes a sensitivity class: a free-form string ("public", "internal", "pii", "secret" are examples, not an enum). Three independent consumers read that one string, and each acts on it in a different direction.

flowchart LR
          s["meta.v1 sensitivity: a free-form class string"] --> e["Embedding gate: refuses classified TEXT fields by default"]
          s --> r["Index rendering: encrypt, mask, or exclude classes"]
          s --> c["Screening: model-driven spans, masked and tagged"]

The embedding gate. The default VectorizationPolicy refuses to embed any TEXT field that carries a sensitivity class. The reasoning is physical: an embedding leaves the process, is retained by whatever serves the model, and cannot be un-sent. A vector is also not the harmless summary it looks like, since inversion attacks recover a usable approximation of the source text. Deployments opt named classes in, or declare the question moot for a field that carries no class.

The index security renderer. The render-index-mappings tool takes a sensitivity object such as {"encrypt": ["pii"]} and renders those fields as store-only ciphertext containers (type: keyword, index: false, doc_values: false): semantic search runs over content the engine never sees in the clear. Sibling rules render mask and exclude fragments, or a complete security role ready to install. The boundary is stated plainly in the docs: the vector is derived from plaintext and leaks through neighborhood and inversion channels, so treat vectors as confidential in their own right.

The screening layer. A mounted ScreeningConfig (class, model reference, threshold, policy) is applied to every string field whose sensitivity equals the screened class: findings are masked with *** and tagged, or, with explicit opt-in only, the document is refused. Below-threshold detections are dropped and never recorded, because an unactioned finding would itself be evidence of the text's content. One design line matters here: screening is not validation. Schema validity is deterministic, the same input always gives the same verdict. Model detection is probabilistic and carries its model version as evidence on every mask, so it must never masquerade as schema truth.

Index hints: 19 field kinds and the vector lane

The index.hints.v1 option family tells the index layer what each field becomes. There are 19 field kinds: full-text TEXT, exact-value KEYWORD, the numeric point kinds, DATE, BINARY, VECTOR, OBJECT, NESTED, SKIP, the five *_RANGE kinds (which the canonical range types select by name), and TREE_PATH. A hint with no explicit kind is not an error: the hint source infers the field type from the proto descriptor, and any attributes you do set still win over the inferred defaults.

dev-tools/protomolt search index SPI, the 19 field kinds

INDEX_FIELD_TYPE_TEXT  = 1;   // full-text, analyzed
INDEX_FIELD_TYPE_KEYWORD = 2; // exact value
INDEX_FIELD_TYPE_VECTOR = 10; // KNN vector over a repeated numeric field
// ... INT32, INT64, FLOAT, DOUBLE, BOOLEAN, DATE, BINARY,
//     OBJECT, NESTED, SKIP, five *_RANGE kinds ...
INDEX_FIELD_TYPE_TREE_PATH = 19; // taxonomy path, ancestor-chain terms

Beyond the kind, one FieldIndexHint can carry stored and indexed flags, a name override, vector dimensions, similarity function, and element type, HNSW graph parameters (m, ef_construction), sub-fields, analyzers, a null value, sortable and facetable flags, map and date-resolution modes, and a per-engine escape hatch keyed by engine id for the cases the common model does not cover. The chunk-and-embed policy that turns TEXT into vectors has a deterministic digest: two indexes agree on their chunk boundaries and vector spaces exactly when their policy digests agree, so a changed policy is visible as a changed digest rather than silent drift.

Further option families have their own readers: metric.v1 declares measures and dimensions read by ProtoOptionsMetricHintSource in metric/spi, and repo.v2.link marks DocumentLink fields with a link relation and target types read by DocumentLinks in repo/admission. mesh.v1 declares default processing profiles and field roles on an application contract and has no runtime reader.