A single-node message broker in Rust, loosely modeled on the original Kafka paper — Kafka: a Distributed Messaging System for Log Processing.
A learning project: the fundamentals (binary encoding, framing, log index
structures) are hand-rolled rather than pulled from crates, and performance
is validated with criterion benchmarks (see BENCHMARKS.md).
- A topic is split into partitions; each partition is an append-only log of segment files on disk.
- Messages are addressed by their byte offset in the partition log (the paper's original scheme — no per-message index, just a sorted list of segment start offsets).
- The broker is stateless: consumers track and commit their own offsets; messages are deleted by time-based retention, not acknowledgement.
- Pull-based consumers with long-poll fetch; batched produce requests;
batched
fsync(page-cache-friendly sequential I/O). - Per-message CRC-32; on startup the log truncates at the first torn or corrupt entry.
- Delivery guarantee: at-least-once, ordered within a partition.
frame := length(u32) | api_key(u8) | correlation_id(u32) | body
message := length(u32) | crc32(u32) | magic(u8) | attributes(u8) | payload
APIs: CreateTopic, Metadata, Produce, Fetch (long-poll), OffsetBounds.
cargo test --workspace
cargo clippy --workspace -- -D warnings
cargo bench -p ferrum-log
cargo run --bin ferrum-broker