pubsub_itc_fw

A low-latency, multi-threaded, event-driven application framework for C++17, built around the reactor pattern. It provides inter-thread communication, inter-process communication, pub/sub messaging, timers, high availability, and a binary serialisation DSL — all designed for environments where heap allocation on the hot path is not acceptable.

Features

• Inter-thread communication (ITC) via lock-free MPSC queues
• Inter-process communication (IPC) via unicast TCP with zero-copy PDU paths
• Pub/sub messaging via unicast fanout
• Timers via timerfd and epoll
• High availability via primary/secondary instance pairs with external arbiter pool and automatic leader election
• Binary serialisation DSL — a Python code generator producing C++17 encode/decode headers; sub-100ns round-trip on typical messages

Design Principles

• CPU-pinned threads with lock-free fast paths throughout
• No heap allocation on any hot path — pool allocators, bump allocators, and slab allocators used exclusively
• Zero-copy on all inbound and outbound PDU paths
• Deterministic shutdown
• Message ordering preserved

High Availability

The framework provides a built-in leader-follower protocol for deploying resilient application pairs. Two application instances are deployed — primary and secondary — and leader election is deterministic: the node with the lowest configured instance_id wins.

A separate pool of up to three dedicated arbiter processes (arbiter_primary, arbiter_secondary, witness) provides external arbitration to prevent split-brain when both nodes are undecided. Once elected, the peer-to-peer connection between the two application nodes is maintained with heartbeats. If the leader fails, the follower promotes itself and increments the epoch, ensuring that any restarting node can immediately recognise it is stale and rejoin as follower without requiring further arbitration.

The protocol is intentionally simple — there is no need for a full consensus algorithm such as Raft or Paxos given the fixed two-node-plus-arbiter topology.

Serialisation DSL

Messages are defined in a lightweight DSL and compiled to C++17 headers by a Python code generator:

message StatusQuery (id=100, version=1)
    i64 instance_id
    i32 epoch
end

Supported field types include i8, i16, i32, i64, bool, datetime_ns, string, array<T>[N], list<T>, optional T, and named enum and message references. The wire format is little-endian binary. On little-endian hosts, list<primitive> decode is zero-copy.

Requirements

Item	Detail
Language	C++17
Target compiler	gcc-8.5 / RHEL 8
Build system	CMake + build.py
Logging	Quill v11.x
Test framework	GoogleTest (C++), pytest (DSL tests)

Developer Quick-start (devsetup.sh)

The fastest path from source to a running sandbox is the convenience wrapper, which runs all three steps — build, release, deploy — in sequence:

./devsetup.sh                        # first time (creates DB)
./devsetup.sh --skip-create-db       # subsequent runs (DB already exists)

Once setup completes, start the stack:

python3 devenv.py start

devsetup.sh sets the required environment variables (third-party library paths and versions) and forwards all arguments to devsetup.py. Any flag accepted by the build or deploy steps can be passed through — see ./devsetup.sh --help.

Building (build.py / build.sh)

./build.sh

Builds both the C++ components and the Java admin service, runs all tests, and stages the result into build/installed/. This staging directory is what release.py reads from — it is not the runtime location.

Unit tests and integration tests run automatically. The build script reports signal-based failures (SIGABRT, SIGSEGV, etc.) by name.

Common options:

Flag	Effect
--no-java	Skip the Java admin service build
--no-cpp	Skip the C++ build; build Java only
--clean	Clean before building (C++: deletes build/; Java: runs mvn clean)
--no-tests	Skip all tests (C++ unit/integration tests and Maven Surefire)
--valgrind	C++ build with Valgrind-compatible options (disables lock-free optimisations)
--doxygen	Generate Doxygen documentation after the C++ build
-j N	C++ build parallelism (default: all CPUs)

build.sh is a thin wrapper that sets the platform-specific environment variables required by CMake and then calls build.py.

Packaging (release.py)

Assembles a versioned deployment artefact from the build staging area:

python3 release.py

Reads the version from project(... VERSION x.y.z ...) in CMakeLists.txt and the git short hash from git rev-parse. Reads binaries and the admin-service JAR from build/installed/. Outputs build/release/pubsub-<version>-<hash>.tar.gz containing bin/, lib/, etc/ (config templates with unexpanded ${placeholder} values), db/, environments/, devenv.py, deploy.py, and a release.json manifest.

Options: --install-dir (staging dir, default: build/installed), --env, --version, --output-dir, --no-git-hash.

Deployment (deploy.py)

Unpacks a release artefact and prepares it for launch:

python3 deploy.py --env environments/prod.toml \
                  --artefact pubsub-<version>-<hash>.tar.gz \
                  --install-dir /opt/pubsub \
                  --skip-certs

Steps performed in order:

1. Unpack the artefact into the install directory, stripping its top-level directory.

2. Expand config templates — substitutes ${placeholder} values in all etc/**/*.toml files. Placeholder names are derived mechanically from the environment TOML by flattening every section and key into a single string: [section] key → ${section_key}. For example, [arbiter_primary] peer_host in the env TOML becomes ${arbiter_primary_peer_host} in the component template. A small number of placeholders are injected programmatically by deploy.py itself rather than read from the env TOML (currently ${paths_install_dir}, ${shared_reactor_cpu_registry_shm_path}, and ${shared_reactor_cpu_registry_lock_file}). An undefined placeholder causes a hard exit naming the file and the missing key — there are no silent failures.

   Tracing a placeholder: if you see ${foo_bar_baz} in an application template and cannot find its value, either (a) open the env TOML and look for a [foo] section with key bar_baz, or (b) search deploy.py for namespace["foo_bar_baz"].

3. Generate TLS certificates — self-signed via openssl req -x509 for each [tls.*] section. Pass --skip-certs when placing CA-signed certificates for production.

4. Create the database — delegates to db/create_db.py.

5. Export SCRAM credentials — delegates to db/export_credentials.py.

The install directory defaults to paths.install_dir from the env TOML (installed/ for dev, /opt/pubsub for prod).

Options: --skip-certs, --force-certs, --skip-db, --skip-create-db, --drop-db, --sudo-postgres, --liquibase-contexts.

Developer Sandbox (devenv.py)

devenv.py starts, stops, and monitors the full component stack on a developer machine. It reads component definitions and paths from an environment TOML (default: environments/dev.toml).

Prerequisite: run devsetup.sh (or the three steps manually) before the first start.

Starting everything:

python3 devenv.py start

Components are started in the order defined in [startup_order] in the env TOML, with a 1-second delay between each. Logs go to installed/log/<name>.log (application log) and installed/log/<name>.stdout (stdout/stderr). PID files go to /var/tmp/pubsub/run/<name>.pid.

Checking status:

python3 devenv.py status

Stopping everything:

python3 devenv.py stop

Components are stopped in reverse startup order. Stale PID files are cleaned up automatically.

Restarting a single component (useful during development iteration):

python3 devenv.py restart sequencer
python3 devenv.py restart               # restarts everything

Skipping HA components (run without arbiters, witness, and secondary instances):

python3 devenv.py --no-ha start

Using a different environment:

python3 devenv.py --env environments/test-1.toml start

Options summary:

Flag	Default	Effect
--env PATH	environments/dev.toml	Environment TOML to use
--no-ha	off	Skip components with ha_only = true
--delay SECONDS	1.0	Pause between component starts

Namespace

All framework classes live in the pubsub_itc_fw namespace.

License

Apache-2.0