ProtoMolt

An open-source platform for turning document collections — file shares, buckets, databases, wikis — into structured, searchable, AI-ready data.

Documents enter through connectors, are admitted by a hardened intake service, and travel through a versioned graph of processing modules — parsing, chunking, embedding, quality scoring — orchestrated by an engine that never calls its modules: they pull work from it. Document bodies live in S3-compatible object storage while small claim-check pointers ride Kafka, so the bus stays fast no matter how large the files are. Finished documents are indexed into OpenSearch under governed, versioned index plans.

Every contract in the system is a Protocol Buffers definition in one buf workspace. Every per-hop decision — filters, field mappings, routing — is a pre-compiled CEL expression stored in the graph itself.

The shape of the system

connectors (JDBC · S3 · filesystem · Confluence · SharePoint)
      │  push, API-key authenticated
      ▼
connector-intake  ──XADD──▶  Redis stream  ──▶  engine
 (admission only)                               │  owns the data path
                                                ▼
                          versioned pipeline graph (DAG of module nodes)
                          hops: parse → chunk → embed → score → …
                          edges: CEL conditions + PERSIST/CACHE/MEMORY
                                                │
              modules dial IN and pull work ◀───┘  (engine never dials out)
              (JVM runtime, or any language behind module-proxy)
                                                │
                                                ▼
                      repository-service (Postgres ledger + S3 bodies)
                                                │
                                                ▼
                      opensearch-manager (sole writer) → OpenSearch

One job = one short-lived bidirectional gRPC stream between a module and the engine: Hello → WorkUnit → Heartbeat → WorkAck → AckConfirmed. The engine commits the Kafka offset only after the ack, so crashes redeliver instead of losing work.

The platform, piece by piece

Engine — jobs & orchestration

The service that owns a document’s journey end to end. Demand-pull work distribution over gRPC, versioned graph execution, three named durability behaviors per hop.

Repository — the document store

Documents split into four typed parts on S3-compatible object storage, a Postgres ledger as the card catalog, partial saves by server-side copy, and a delete doctrine with proof.

Modules — agent communication

Processing agents written in any language pull work from the engine; the engine never calls them. A small gRPC contract, a leased catalog, and a certification test battery.

Document graphs

Pipelines are versioned, self-contained DAG snapshots. Documents are pinned to a graph version at admission and every stored byte is graph-qualified.

Data processing

Chunking with offset-anchored deterministic IDs, embedding through pluggable inference backends, semantic centroids and topic boundaries, CEL-scored quality profiles.

Streaming parsers

A fleet of standalone gRPC parsers — PDF/OCR/layout in C++ on GPU, Office, email, EPUB, WARC, EBCDIC, spreadsheets — all streaming typed protobuf, nothing touching disk.

Search

A single-writer OpenSearch indexing plane with governed index plans, three physical vector layouts, hybrid BM25/semantic search with A/B experiments, and an experimental distributed Lucene engine.

Mapping & validation

Every hop can carry declarative CEL filters, mappings, and routing conditions — pre-compiled at graph load, validated against the literal runtime environment, gated by a 12-rule deploy check.

Connectors & intake

A hardened front door with API-key authentication and deterministic document identity, fed by connectors for databases, S3, filesystems, Confluence, and SharePoint — with web crawling and Google Drive on the way.

AI-friendly output formats

Typed protobuf end to end, renderable to text, Markdown, HTML, JSON, YAML, DocTags, and VTT — plus WARC archives and Open Knowledge Format bundles for crawl output.

Design positions

Pull, not push. Modules ask the engine for work. There is no service discovery and no per-module load balancer; scaling out is opening more streams, and scaling to zero just lets the queue accumulate.
Claim-check everywhere. Kafka carries pointers, not payloads. Bodies are staged in object storage and hydrated on demand — only the parts a module actually needs.
Durability is named per edge. Each hop declares its recovery guarantee — PERSIST, CACHE, or MEMORY — and a right-to-be-forgotten document can force every edge to MEMORY so its bytes are never written down.
Deterministic identity. Document IDs, work-unit IDs, and Kafka keys are all derived from content and coordinates, so retries converge instead of duplicating state.
Honest telemetry. Eventually-consistent traces are for observation only; decisions are always made from authoritative stores. The codebase treats this as a rule, not a guideline.

A note on names. ProtoMolt is the platform's name; the code's published coordinates keep their original pipestream identity — Maven group ai.pipestream, buf modules under buf.build/pipestreamai, container images under docker.io/pipestreamai, and Kafka topics like pipestream.module.*. Those names are stable and are not changing.

Status. ProtoMolt is open source and under active development. The core loop — intake, engine, modules, repository, indexing — runs in production-like deployments today; some surfaces described on these pages are marked experimental where they are. Nothing on this site is marketing copy: each section cites the contracts and code it describes.