Roadmap: logfwd implementation phases

[strawgate]

Roadmap

Living tracking issue for logfwd implementation phases. Check items off as they land; link PRs in comments.

---

Completed phases (click to expand)

Proven Core (Phases 0–1)

• Phase 0: Kani prototype — 31 proofs, CI working
• Phase 1a: Create logfwd-arrow (PR feat: create logfwd-arrow crate, move builders + scanner structs (Step 1) #307)
• Phase 1b: Create logfwd-input (Create logfwd-input crate, move file tailer + IO #265)
• Phase 1c: Move remaining impure code (Move remaining impure modules (enrichment, compress, diagnostics) #266)
• Phase 1d: Tighten core to no_std (Tighten logfwd-core to no_std + forbid(unsafe_code) #267)
• Phase 1.5: Framer + Aggregator + Proofs (PR feat: Framer + CriAggregator — zero-copy, Kani-proven line splitting (Steps 4-5) #311) — NewlineFramer, CriAggregator, byte_search, all Kani-proven

Performance (Phases 2–3)

• Phase 2: Unified StructuralIndex (Unified SIMD StructuralIndex: extend ChunkIndex to detect all structural characters #313) — StreamingClassifier in `structural.rs`, all sub-issues closed
• Phase 3: Zero-copy Bytes pipeline (Refactor: zero-copy pipeline — Bytes as universal currency, Framer/Aggregator model #303) — StreamingBuilder uses `bytes::Bytes`

Scanner + Pipeline (Phases 4–5)

• Phase 4: Scanner restructure + FieldSink (Phase 3: Scanner restructure + FieldSink trait #269)
• Phase 5: Pipeline state machine + BatchTicket (Pipeline state machine extraction + BatchToken #270) — typestate, Kani-proven, wired into pipeline.rs (PR feat: Phase 5c — checkpoint wiring (#584, #585, #586, #587, #589) #637, feat: wire FileCheckpointStore into pipeline — persist and restore file offsets (#588) #784), FileCheckpointStore, instance lock (feat: exclusive instance lock — prevent duplicate file processing (#737) #814)

Verification (Phase 6 + 7a)

• Phase 6: proptest state machines + CI hardening (proptest state machines + CI hardening #271)
• Phase 7a: TLA+ PipelineMachine.tla (PR feat: TLA+ formal spec for PipelineMachine drain + checkpoint ordering #651) — 8 safety invariants + 4 liveness properties

Core Hardening (Phase 8) + Column Naming (Phase 10)

• Phase 8: Tighten logfwd-core (Tighten logfwd-core to no_std + forbid(unsafe_code) #267) — `#![no_std]` + `#![forbid(unsafe_code)]`, SIMD via `wide`
• Phase 10: Type suffix redesign (Column type design: suffix only on conflict, bare names by default #445) — bare names, StructArray conflicts, rewriter deleted (PRs fix: output sinks dispatch on Arrow DataType, not column name suffix #568, feat: suffix column names only on type conflict, delete dead rewriter (#445) #684, feat: replace type-conflict suffixed columns with Arrow StructArray format #760)

Production Readiness (completed items)

• Async output worker pool — MRU dispatch, ES/Loki async sinks (PR feat: async output worker pool (MRU dispatch) + ES/Loki async sinks #701)
• Sink trait unification — unified OutputSink/Sink (PR Refactor to unify OutputSink and Sink traits by removing sync outputs #857)
• OTLP wire format fix — raw zstd + correct resource attribute field tag (PR fix: OTLP wire format — raw zstd + correct resource attribute field tag #887)
• File tailing reliability — truncation, OOM, glob dedup, checkpoint validation (PR fix: file tailing reliability — truncation, OOM, glob dedup, checkpoint validation, warnings (#796, #800, #799, #656, #730) #808)
• FileCheckpointStore persistence (PR feat: wire FileCheckpointStore into pipeline — persist and restore file offsets (#588) #784)
• Exclusive instance lock (PR feat: exclusive instance lock — prevent duplicate file processing (#737) #814)
• JSON escape sequence decoding (PR fix(scanner): decode JSON escape sequences in string values (#410) #885)
• Config validation hardening (PR Fix config validation to catch typos and invalid values at load time #854)

---

Open work

I/O Correctness Refactor (#1315) 🆕

Comprehensive refactor of file tailer, OTLP receiver, OTLP exporter, and new file appender — correct by construction via typestate, Kani proofs, unified error types, and ecosystem libraries. Research: #1283.

Phase 0 — Shared Infrastructure:

• Shared semantic field definitions (logfwd-types) #1302 — Shared semantic field definitions
• IO_CONTRACTS.md — living contract for I/O and control-plane semantics #1303 — IO_CONTRACTS.md living contract doc
• Unified IoAction error type + HTTP response classifier #1304 — Unified IoAction error type + HTTP response classifier
• Replace hand-rolled retry with backon crate #1305 — Replace hand-rolled retry with backon (fixes thundering herd)
• meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306 — Component lifecycle trait + ComponentMetrics
• Replace glob with globset + walkdir in file tailer #1307 — Replace glob with globset + walkdir
• Use sonic-rs for OTLP JSON parsing #1308 — Use sonic-rs for OTLP JSON (4-6.5x faster)
• Extract atomic_write_file() helper from checkpoint code #1309 — Extract atomic_write_file() helper

Phase 1 — File Tailer: #1310 — split monolith, typestate, per-file correctness (fixes #797, #798, #799, #800, #801, #796)

Phase 2 — OTLP Receiver: #1311 — axum, streaming decompression, unified receivers (supersedes #689)

Phase 3 — OTLP Exporter: #1312 — ⚠️ P0 panic recovery, checkpoint fix, jitter, fanout (fixes #1233, #1232, #1225)

Phase 4 — File Appender: #1313 — new component with rotation, cleanup, path templates (fixes #1231)

Phase 5 — Verification: #1314 — TLA+, Turmoil, OTLP loopback, otelcol oracle, benchmarks

Phase 7: Complete Verification (#895)

Comprehensive TLA+ modeling + simulation testing + spec-to-code alignment.

• 7b: Implement `force_stop()` — align code with TLA+ spec (ForceStop action)
• 7c: Pipeline testability — `with_input()`, `with_checkpoint_store()` builders for test injection
• 7d: Extract `BatchAccumulator` — pure state machine from run_async, Kani proofs
• 7e: TLA+ ShutdownProtocol.tla — N input threads + channel + consumer + pool drain coordination (extended by I/O verification suite — TLA+, Turmoil, loopback, otelcol oracle, benchmarks #1314)
• 7f: TLA+ PipelineBatch.tla — multi-source batching, checkpoint merge, error handling, backpressure (extended by I/O verification suite — TLA+, Turmoil, loopback, otelcol oracle, benchmarks #1314)
• 7g: TLC in CI — run all TLA+ specs on every PR
• 7h: Simulation test suite — 11 deterministic scenarios (extended by I/O verification suite — TLA+, Turmoil, loopback, otelcol oracle, benchmarks #1314)
• 7i: Archive FileCheckpoint.tla — move to research/, update gap analysis

Core Hardening

• Phase 9: Allocation-free kernel (Remove extern crate alloc from logfwd-core (allocation-free kernel) #358) — remove `extern crate alloc`, stack-only buffers

Crate Restructure (#851)

• Phase A: Add `default-members` to skip bench crates (PR refactor: add default-members to skip bench crates during dev #867)
• Phase B: Extract `logfwd-types` crate (refactor: extract logfwd-types crate — decouple SIMD from pipeline types (#851) #861) — decouple SIMD from pipeline types (relates to Shared semantic field definitions (logfwd-types) #1302)
• Phase C: Remove logfwd-io dep from logfwd-output and logfwd-transform (refactor: remove logfwd-io dependency from logfwd-output and logfwd-transform (#851) #862)
• Phase D: Split logfwd-io into input + checkpoint + diagnostics (refactor: optional logfwd-io split after Phase C and module-first cleanup (#851) #863) (relates to File tailer refactor — split monolith, typestate, per-file correctness #1310)
• Phase E: Extract logfwd-runtime crate from binary (refactor: extract logfwd-runtime crate from binary (#851) #933)

Production Readiness

• AsyncFanout with circuit breaker (Design: AsyncFanout — independent sink tasks with retry, circuit breaker, per-sink isolation #319) → addressed by Output delivery contract — retry, reject, checkpoint, fanout #1312 (I/O refactor Phase 3)
• CRI silent truncation fix (Fix: CRI reassembler silently truncates lines exceeding max_line_size #275)
• Structured logging with tracing (P1: Structured logging with tracing crate #191)
• Failure metrics — dropped batches, per-sink errors (P1: Add failure metrics (dropped batches, per-sink errors, buffer size, parse errors) #192) → addressed by meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306 (ComponentMetrics)
• TCP/UDP backpressure (P0: TCP/UDP inputs have no backpressure — will OOM under sustained load #576)
• TCP/UDP TLS support (bug: TCP ingress needs TLS/mTLS support (UDP DTLS out of scope) #579)
• Async OTLP sink — migrate from ureq to reqwest (Convert OtlpSink to async reqwest (migrate from ureq) #689) → superseded by OTLP receiver refactor — axum, streaming decompression, unified receivers #1311 (receiver refactor)
• Async TCP/UDP sinks — migrate from std::net to tokio (Convert TcpSink to async tokio::net::TcpStream #690, Convert UdpSink to async tokio::net::UdpSocket #691)
• Enrichment config wiring (Enrichment types k8s_path, host_info, static, csv, jsonl have code but no config wiring #540)

[strawgate]

PR #1320 lands benchmark correctness fixes: adds std::hint::black_box() to prevent the compiler from optimizing away measured work in batch_formation, file_io, pipeline, and output_encode benches, and fixes the sender/receiver startup race in _bench-pair by replacing the fixed sleep 5 with a /ready readiness poll + 1 s grace period for socket binding.

[strawgate]

PR #1320 implements the benchmark race fix and black_box measurement improvements for this phase. See #1320

[strawgate]

Superseded by #1426 — new roadmap with committed path, planned work, and research tree.

[strawgate]

Cleanup fan-in from 2026-04-06 has been turned into an execution meta: #1450.

This keeps the new work aligned with the existing umbrellas instead of creating a competing roadmap tree.

New execution issues created from the fan-in:

• bug: enforce strict FileTailer read budget and add fairness regression coverage #1441 strict FileTailer read-budget enforcement
• test: add multi-source checkpoint and remainder interleaving regression suite #1442 multi-source checkpoint/remainder regression suite
• refactor: shared output retry/status helpers and richer worker ack failure taxonomy #1443 shared output retry/status helpers and richer worker ack failure taxonomy
• docs: make config reference canonical and remove config status drift #1444 config docs truth / status drift cleanup
• meta: diagnostics contract hardening and dashboard metric cleanup #1445 diagnostics contract hardening and dashboard metric cleanup
• meta: pipeline seam extraction — checkpoint and shutdown #1449 pipeline checkpoint/shutdown seam extraction

Suggested roadmap interpretation:

• I/O correctness refactor — correct by construction, not by patch #1315 remains the umbrella for I/O/output/control-plane architecture
• Phase 7: Complete verification — TLA+ specs, simulation testing, spec-to-code alignment #895 remains the umbrella for verification/runtime seam work
• refactor: crate restructure — extract logfwd-types, reduce blast radius #851 remains the umbrella for crate/module structure decisions
• meta: cleanup wave 2 — temporary execution meta for contract-first, module-first slices #1450 is the concrete cleanup execution meta for the next mergeable slices