Release/v0.0.4

.gitignore

@@ -6,3 +6,4 @@ logs/
 # Service runtime archives
 services/*/data/
 data/ojo/
+.codex

CHANGELOG.md

@@ -2,6 +2,36 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.0.4] - 2026-04-25
+
+### Changed
+
+- Added a Windows-specific default config for `ojo-syslog` (`services/syslog/syslog.windows.yaml`) so Windows uses ETW-oriented defaults while Linux continues to use `services/syslog/syslog.yaml`.
+- Updated `ojo-systrace` to namespace Windows-derived trace components as `windows.*` instead of reusing Linux `kernel.*` names, keeping the Windows trace shape compatible without sharing Linux naming.
+
+#### Efficient Archiving
+
+- Archives are now significantly more compressed and lightweight. Each service can be independently tuned via the YAML config, with full control over format, compression, retention, and archival strategy.
+
+```yaml
+storage:
+  archive_enabled: true
+  archive_dir: data/ojo
+  archive_format: parquet         # columnar storage for efficient reads
+  archive_compression: zstd       # high-ratio compression
+  archive_mode: trend             # archive strategy  (`trend`, `lossless`, `forensic`)
+  archive_window_secs: 60         # rolling window per archive cycle
+  archive_max_file_bytes: 67108864  # 64 MiB per file
+  archive_retain_files: 8         # number of archive files to retain
+  archive_file_stem: ojo-snapshots
+```
+
+### Fixed
+
+- Fixed trend aggregation first-sample accounting in `host-collectors` so the first point in a window is no longer double-counted.
+- Fixed parquet archive persistence in `host-collectors` so repeated writes preserve previously written active-file batches instead of truncating earlier archive data.
+
+
 ## [0.0.3] - 2026-04-15
 
 ### Added

Cargo.toml

@@ -2,6 +2,7 @@
 name = "ojo"
 version = "0.0.3"
 edition = "2021"
+default-run = "ojo"
 
 [workspace]
 resolver = "2"
@@ -27,6 +28,10 @@ ctrlc = "3"
 serde = { version = "1", features = ["derive"] }
 serde_yaml = "0.9"
 serde_json = "1"
+arrow-array = "58"
+parquet = { version = "58", default-features = false, features = ["arrow"] }
+prost = "0.13"
+snap = "1"
 tokio = { version = "1", features = [
   "macros",
   "rt-multi-thread",

README.md

@@ -64,7 +64,7 @@ These binaries are separate workspace crates under `services/<name>/`. Each runs
 | Systrace | `ojo-systrace` | `services/systrace/systrace.yaml` | `system.systrace.*` |
 | Syslog | `ojo-syslog` | `services/syslog/syslog.yaml` | `system.syslog.*` |
 
-Shared OTLP and filtering helpers live in `crates/host-collectors`. Grafana dashboards for each extension are under `grafana/` (`docker.json`, `gpu.json`, `sensors.json`, `mysql.json`, `postgres.json`, `nfs-client.json`, `nginx.json`, `redis.json`, `systemd.json`, `systrace.json`, `syslog.json`).
+Shared OTLP, filtering, and archive helpers live in [`crates/host-collectors`](crates/host-collectors/README.md). Grafana dashboards for each extension are under `grafana/` (`docker.json`, `gpu.json`, `sensors.json`, `mysql.json`, `postgres.json`, `nfs-client.json`, `nginx.json`, `redis.json`, `systemd.json`, `systrace.json`, `syslog.json`).
 
 Release archives include the main `ojo` agent and all sidecar services for Linux (`-unix`) and Windows (`-win`).
 
@@ -258,6 +258,43 @@ cd services/mysql && cargo run -- --config mysql.yaml
 cd services/postgres && cargo run -- --config postgres.yaml
 cd services/nfs-client && cargo run -- --config nfs-client.yaml
 cd services/nginx && cargo run -- --config nginx.yaml
+
+## Archive Storage Modes
+
+Archive behavior is configured in each YAML `storage` section:
+
+- `archive_enabled`
+- `archive_dir`
+- `archive_max_file_bytes`
+- `archive_retain_files`
+- `archive_file_stem`
+- `archive_format` (`parquet`)
+- `archive_mode` (`trend`, `lossless`, `forensic`)
+- `archive_window_secs` (trend mode window)
+- `archive_compression` (`zstd`)
+
+Mode guidance:
+- `trend`: lossy, very compact, best for long-range dashboards and baselines.
+- `lossless`: full-fidelity records with efficient parquet+zstd compression.
+- `forensic`: compatibility row mode for deeper incident-level archives.
+
+## Archive Replay
+
+Replay archived trend parquet files to OTLP or Prometheus remote-write (Mimir compatible):
+
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir services/sensors/data \
+  --endpoint http://localhost:4320/otlp/v1/metrics \
+  --protocol otlp
+```
+
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir services/sensors/data \
+  --endpoint http://localhost:4320/mimir/api/v1/push \
+  --protocol remote-write
+```
 cd services/redis && cargo run -- --config redis.yaml
 cd services/systemd && cargo run -- --config systemd.yaml
 cd services/systrace && cargo run -- --config systrace.yaml
@@ -399,3 +436,10 @@ Cross-checking to `x86_64-pc-windows-gnu` from Linux requires a MinGW toolchain
 - Ensure `collection.include_process_metrics: true`
 - Verify process permissions
 - Check that `metrics` filters are not excluding `process.`
+
+Shell log output (no endpoint required):
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir <archive_dir> \
+  --protocol shell-logs
+```

crates/host-collectors/Cargo.toml

@@ -7,6 +7,9 @@ edition = "2021"
 anyhow = "1"
 gethostname = "1"
 serde_json = "1"
+arrow-array = "58"
+arrow-schema = "58"
+parquet = { version = "58", default-features = false, features = ["arrow", "zstd"] }
 opentelemetry = { version = "0.31", features = ["metrics", "trace"] }
 opentelemetry_sdk = { version = "0.31", features = ["metrics", "trace", "rt-tokio", "experimental_trace_batch_span_processor_with_async_runtime"] }
 opentelemetry-otlp = { version = "0.31", features = [

crates/host-collectors/README.md

@@ -0,0 +1,57 @@
+# Host Collectors
+
+Shared runtime library used by `ojo` and all sidecar services.
+
+## What it provides
+- OTLP meter/tracer provider setup helpers.
+- Prefix-based metric filtering helpers.
+- Unified archive writer implementation used across services.
+
+## Archive configuration contract
+The shared archive config is modeled by `ArchiveStorageConfig` and consumed by all services:
+
+- `archive_enabled`
+- `archive_dir`
+- `archive_max_file_bytes`
+- `archive_retain_files`
+- `archive_file_stem`
+- `archive_format` (currently `parquet`)
+- `archive_mode` (`trend`, `lossless`, `forensic`)
+- `archive_window_secs` (trend mode)
+- `archive_compression` (`zstd`)
+
+## Archive modes
+- `trend`: compact lossy summaries by time window (`min/max/avg/count/first/last`).
+- `lossless`: full-fidelity row archival with parquet + zstd compression.
+- `forensic`: compatibility row mode with row-level payload retention.
+
+## Typical outputs
+- trend: `<archive_file_stem>-trend.parquet`
+- lossless: `<archive_file_stem>-lossless.parquet`
+- forensic: `<archive_file_stem>-forensic.parquet`
+
+## Replay
+Use the workspace replay tool:
+
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir <archive_dir> \
+  --endpoint http://localhost:4320/otlp/v1/metrics \
+  --protocol otlp
+```
+
+Remote-write example:
+
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir <archive_dir> \
+  --endpoint http://localhost:4320/mimir/api/v1/push \
+  --protocol remote-write
+```
+
+Shell log output (no endpoint required):
+```bash
+cargo run --bin archive-replay -- \
+  --archive-dir <archive_dir> \
+  --protocol shell-logs
+```