AI-Friendly Output Formats

Every stage of the platform produces structured, machine-readable output. The results are typed protobuf messages, deterministic chunk and vector sets, and standard archival formats — designed to be consumed directly by AI pipelines for RAG, search, and training, without scraping text back out of a database.

Protobuf-typed end to end

The document model is not a blob of extracted text with a bag of string keys. The payload that travels between modules is a PipeDoc, and parsed metadata is typed per format family: there is one proto each for PDF, Office, image, email, media, HTML, EPUB, RTF, font, database, WARC, and climate (NetCDF) documents, plus dedicated protos for the Dublin Core and Creative Commons vocabularies. The mapping from Tika's metadata interface onto these protos covers more than 1,000 fields, documented in TIKA_INTERFACE_MAPPING.md.

The proto definitions live under core-services/pipestream-protos/common/proto/ai/pipestream/parsed/data/. Because the schema is protobuf, every consumer — Java, C++, Python, Go — works against generated types rather than guessing at key names, and a field's meaning is fixed by the schema rather than by convention at each call site.

Structure that plain text extraction loses is preserved explicitly: PDF bookmark outlines (via PDFBox), EPUB tables of contents, HTML heading hierarchies, and a full CommonMark AST for Markdown files, carried in parsed_metadata["commonmark"].

Deterministic chunk sets

Chunking is deterministic by construction: the same document, source label, and configuration produce the same chunks on every run. Each chunk carries its start and end offsets into the source text and a SHA-256 content_hash of the sanitised chunk text — which the proto itself describes as enabling "reprocessing dedup, content-addressed embedder cache keys, and byte-verification of alternative chunker backends" (pipeline_core_types.proto). Chunk IDs and directive keys are derived, not assigned:

modules/module-chunker/docs/architecture.md (§1, §6.1)

chunk_id = {docHash}:{sourceLabel}:{chunkerConfigId}:{chunkNumber}:{start}:{end}

directive_key = sha256b64url(
    source_label + "|" + cel_selector + "|" +
    join(",", sorted(chunker_config_ids)) + "|" +
    join(",", sorted(embedder_config_ids)))

Because the hash covers exactly the inputs that decide the output, identical work converges on identical keys — retries and re-runs deduplicate themselves instead of accumulating copies. Each chunk also carries a ChunkAnalytics record (roughly twenty fields: POS densities, vocabulary density, a potential-heading score, offsets), so downstream consumers can filter or rank chunks without re-running NLP.

Re-index honesty. Chunk IDs are not stable across chunker versions and were never intended to be. Changing a boundary-moving option orphans persisted embeddings, and the chunker documents this plainly: "the old ids name chunks whose text no longer exists." There is deliberately no silent migration.

Embedding vectors as named output sets

Vectors are not a side table attached to chunks after the fact. Chunker and embedder output accumulates on the document as named SemanticProcessingResult sets, under a strict three-stage shape contract:

The chunker emits result sets with chunks and empty vectors — the empty embedding_config_id / unset vector pair is the explicit "chunked, not embedded" marker.
The embedder replaces those sets with fan-out copies, vectors filled, everything else copied byte-for-byte.
The semantic-graph module appends centroid and boundary result sets; the embedder's sets are preserved unchanged, enforced by a deep-equal self-check.

What gets chunked and embedded is decided upstream by vector_set_directives carried on the document — each directive pairs a source label and selector with named chunker and embedder configs. Because every result set is named, multiple chunker × embedder combinations coexist on one document (the design documentation's example: two chunkers and three embedders yield six vector sets in one index), and a consumer picks sets by name rather than re-deriving them.

Quality indexes

The quality module writes a QualityIndex into SearchMetadata.quality_index: a composite score plus per-dimension records with weight snapshots and skip explanations. Dimensions are CEL expressions (compiled once, cached, coerced and clamped to [0,1]) or native QualityChecker beans — MinHash duplication, host-reputation authority, cosine topic relevance. Two dimensions from the bundled default profile:

modules/module-quality/src/main/resources/quality-default-profile.json

{
  "dimension_id": "recency",
  "weight": 0.5,
  "expected_inputs": ["search_metadata"],
  "cel": "hasDate ? exp(-0.693 * ageDays / 365.0) : 0.5"
},
{
  "dimension_id": "readability",
  "weight": 0.75,
  "expected_inputs": ["document_analytics"],
  "cel": "clamp(1.0 - (avgSentenceLength > 20.0 ? (avgSentenceLength - 20.0) / 40.0 : (avgSentenceLength < 8.0 ? (8.0 - avgSentenceLength) / 8.0 : 0.0)), 0.0, 1.0)"
}

The scoring doctrine is "judge only what was witnessed": a dimension whose expected_inputs are absent is skipped, not zeroed, and the composite is computed over witnessed evidence only. Five profiles ship in the box (default, scientific-papers, news-articles, technical-docs, web-content), selected per document by a category map.

Advisory, not enforcing. The below_threshold_action field (skip_indexing, route_to_review) is logged advice carried for consumers; the module never drops, rewrites, or reroutes a document. Cross-document duplication detection is future work — the MinHash checker accepts corpus signatures, but nothing supplies them yet, so today it measures internal redundancy only.

Renderings: one document, many formats

The gRParse service (C++, GPU-accelerated OCR and layout) parses PDFs and raster images into a single merged, page-streamed Document — the additive output of its collectors — and renders every OutputFormat the wire declares from that same document:

TEXT, MARKDOWN, HTML, HTML_SPLIT_PAGE
JSON, YAML
DOCTAGS, DOCLANG, VTT

An empty to_formats keeps the plain-text default; populated options the service does not implement, and unrenderable format values, are rejected with INVALID_ARGUMENT naming the offender. A request can also name a result Target — zip or s3 — which delivers one canonical bundle, identical whichever target carries it:

grpc-services/gRParse/README.md ("Result targets")

manifest.json           every other member with its SHA-256 and byte size,
                        plus the generator and schema version
document.pb             the Document, deterministically serialized
document.json           the canonical JSON dialect of the same document
exports/<name>.<ext>    one file per output format the request asked for
pages/page_NNNN.png     each page image the document embeds
pictures/pic_NNNN.png   each picture image the document embeds

The bundle is byte-for-byte reproducible: members are sorted by path, archive timestamps are fixed at the MS-DOS epoch, the compressor is held to one setting, and the manifest carries no whitespace, no clock, and sorted keys. The same document and the same requested formats produce a byte-identical archive on every machine and every run — so renderings can be content-addressed and cached like everything else in the system.

Current scope. ConvertSource currently accepts one FileSource containing base64-encoded PDF, PNG, JPEG, or TIFF bytes; asynchronous jobs and remote sources are deliberately left unimplemented. The put and presigned_url targets return UNIMPLEMENTED, named in the status message.

Archival and interchange

For crawl output, the Confluence connector writes interchange formats rather than only feeding the pipeline. A completed Sync run can land as an Open Knowledge Format v0.2 bundle — a directory tree plus a zip — with a sibling WARC 1.1 archive (ISO 28500): one resource record per live web_url, a conversion record for the OKF markdown, and an HTML collection page whose links are those same URIs. The zip is not stored inside the WARC.

grpc-services/grpc-confluence/README.md

OUTPUT_DIR=/data/okf/confluence-run   # alias: OKF_DIR
# OUTPUT_STORE=filesystem             # default; set s3 when grpc-output-s3 is loaded
# OUTPUT_FORMATS=okf                  # also: protobuf,json,microsoft-connector
# OUTPUT_PREFIX=run-2026-08-23

The protobuf output format is the same binary the Kafka Connect source plugins publish; json is a file export, not the gRPC wire. For change capture, the Connect sources emit raw protobuf bytes — ConfluenceChange / MicrosoftChange — onto a topic, so downstream consumers get the typed change stream directly:

grpc-services/grpc-confluence/README.md ("Kafka Connect")

{
  "name": "confluence-source",
  "config": {
    "connector.class": "ai.pipestream.connect.ConfluenceSourceConnector",
    "topic": "confluence.changes",
    "grpc.target": "localhost:9095",
    "grpc.plaintext": "true",
    "include.bodies": "false"
  }
}

The same OKF/WARC output is wired for the Microsoft Graph proxy, and S3 object keys follow the source hierarchy ({space}/pages/{id}.md and similar), so an archived crawl is navigable without any tooling beyond a file listing.