Benchmarks in non-Rust SDKs

[hubcio]

Currently, iggy doesn't have benchmark for other SDKs. We only have iggy-bench Rust binary, which has tons of options.

It would be great if we could bench other SDKs performance. We should decide between two approaches:

• (A) separate native benchmark binaries per SDK (Go bench, Java bench, etc.),
• (B) extend iggy-bench (Rust binary) to call other SDK functions via FFI. Trade-offs around consistency, performance measurement accuracy, and maintenance burden.

Pros/cons:

• Option A (separate binaries): idiomatic per language, no FFI overhead in measurements, but duplicated logic and harder to keep methodology consistent
• Option B (iggy-bench calls SDKs via FFI): single tool, unified reporting and methodology, but FFI layer adds complexity and may skew benchmarks

Either way, we want to share a common report.json schema so results are comparable across SDKs - this way we could upload these results to https://benchmarks.iggy.apache.org.

Questions:

• Does FFI overhead invalidate benchmark results, or is it negligible for I/O-bound workloads?
• How much bench logic can realistically be shared vs needs per-SDK customization?

Opening discussion.

[slbotbm]

How much bench logic can realistically be shared vs needs per-SDK customization?

Currently, the iggy-bench binary runs as follows: The orchestrator (BenchmarkRunner) starts a benchmark, waits for the tasks spawned by the benchmark to finish, and compiles the final report. The benchmark implementation (Benchmarkable types) configures the workload and spawns actors. The actors do the actual work, take measurements, and send them to the orchestrator. From this point of view, if we were to use FFI bindings to benchmark other SDKs, creating the actor code in other languages would be required as a minimum. Furthermore, the benchmark layer would have to be refactored to allow it to spawn actors in other laguages as well.

The above only applies to non-FFI SDKs (Node, Java, Go, C#).