C++ Client Design and Implementation Details

[slbotbm]

Overview

The C++ client uses CXX to build a low level API from the files in src/*.rs, which are then used to create a high-level client with the header file being include/iggy.hpp and implementations living in src/*.cpp. Bazel is used as the build-system.

Concepts

Shared vs opaque structs: In CXX, “shared” structs are structs whose internals are transparent to both the C++ as well as Rust side (the Message struct in src/lib.rs is an example of this). “Opaque” types are types that are only visible to one side, and we need to define functions so that the other side can fetch data from these (the Client or Identifier in src/lib.rs are examples of this).

Design

I iterated on the C++ client a few times and finally settled on the following ideas:

• Having as few shared structs as possible. This is because I would like to entrust the validation of input to the rust side. Further, shared structs introduce latency and memory overhead since they need to be converted between rust and cpp, with data being copied multiple times.
• Expose a low-level as well as a high-level client. The high level client being built from src/*.cpp aims to present a C++-native interface to the user, but does this by introducing latency by converting data between Rust and C++. If the user is alright with working with non-native C++ code, they can use the low-level client.
• For the opaque types, instead of Box’ing them, I prefer to handle raw pointers. This is because I would like to give users using the low-level client explicit control of these types’ lifetimes. For the high-level client, I wrap these pointers in RAII classes that safely handle the destruction when the type goes out of scope.
• The current client is a blocking one. This is because CXX requires C++20 and up for asynchronicity. Looking at the data, C++17 still has 43% of the market share. If we utilize C++20, there may be compatibility issues and could end up alienating some users. Hubert has suggested some libraries in the PR’s comments, and I’ll take a look at them as well.
• We use Bazel 9 as the build system for generating the final binaries, but for generating the CXX-related header files, we do not use rules_rust as the primary build path; instead BUILD.bazel invokes cargo build with a Bazel-provisioned Rust toolchain. Since Bazel requires an isolated environment, this approach helps us preserve the local path dependency iggy = { path = "../../core/sdk" } in Cargo.toml without having to convert the whole project into one managed using Bazel instead of cargo.

Implementation

• include/iggy.hpp is the main header file defining all of the structs and methods available to the user. The implementations live in src/*.cpp for the high-level client, and src/*.rs for the rust bindings.
• On the C++ side, CompressionAlgorithm, Expiry, MaxTopicSize and PollingStrategy provide higher-level structures that enable the user to specify their preferred configurations. On the Rust side, some of these structs accept inputs for certain options like Expiry::duration(value), and do not for some like Expiry::server_default(). To avoid defining a shared struct for each of them, the Rust side accepts two values (_kind and _value) wherever such a pattern is observed and the code builds the required type internally. For options that do not accept a value like Expiry::server_default, the C++-side client sends 0 as the default value, and it is discarded when building the required type on the Rust side.
• The IggyMessage class on the C++ side utilizes the shared struct Message and presents a unified class for receiving and sending messages. The send_message_ flag controls what the user can access when sending or receiving messages. Further, since message ids are 128-bit numbers, the client uses abseil to work with them. The PolledMessages class is similarly defined and utilized the PolledMessages shared struct to handle messages.
• The four opaque types defined on the Rust side (Client, Identifier, StreamDetails, and TopicDetails) are handled on the C++-side using RAII classes to safely deal with raw pointers. These also expose various methods for the user to interact with the Rust side, which are similar to the Python ffi.
• IggyException is the unified error class that our library presents to the user. All rust:Error and std::exception are cast to IggyException before being presented to the user.

Limitations

• The client is currently blocking, preventing users from taking full advantage of Iggy.
• The generated binaries are very large (iggy-cpp is ~500 MB, the example in examples/example.cpp is ~200 MB)

Future Development / Ideas

• Make the client asynchronous
• Bring the client up to parity with the canonical Rust version
• Currently, the client assumes a lot for the user. For example, it uses tcp by default, and the user cannot use a different method. To increase its flexibility, I was thinking of implementing a shared struct that carries commands from the C++ side, is parsed on the Rust side, and the required configurations are created accordingly.

Credits

The initial iteration of the client and the build system settings were largely inspired by fluss-rust’s C++ bindings

[hubcio]

See the discussion under #2661.