A modern, high-performance fixed-size thread pool implementation in C++20 designed for efficient concurrent task execution. This project demonstrates fundamental concepts in concurrent systems design, including thread synchronization, work-stealing algorithms, and modern C++ concurrency primitives.
Thread pools are a core building block in concurrent applications, enabling efficient utilization of system resources by reusing a fixed set of worker threads to execute numerous tasks. Rather than creating a new thread for each task—an expensive operation—thread pools maintain a pre-allocated set of workers that process incoming tasks from shared queues. This project implements a sophisticated work-stealing scheduler that significantly improves load balancing and CPU cache efficiency compared to traditional global-queue approaches. Each worker thread maintains its own task deque, allowing lock-free execution in the common case while stealing work from underutilized peers during idle periods.
A pre-allocated pool of worker threads sized at construction time. Rather than creating threads on-demand, this design reduces allocation overhead and enables predictable resource management. The pool is ideal for server applications, real-time systems, and other scenarios where thread creation overhead must be minimized. The number of workers is typically set to the number of logical CPU cores to maximize throughput without incurring context-switch overhead.
Each worker thread owns a thread-local std::deque for storing pending tasks. This design eliminates contention from a single global queue that would become a bottleneck as the pool scales. With per-worker queues, most task scheduling requires no synchronization—the owning thread pushes and pops tasks from its queue without locks. This dramatically reduces lock acquisition frequency and improves cache locality by keeping task state local to the worker.
When a task is submitted to the pool, it is distributed to workers in round-robin fashion. This ensures even distribution of work across threads and prevents any single worker from being overloaded immediately upon submission. The scheduling decision is made using a simple atomic counter that is incremented for each submission, keeping the overhead minimal.
When a worker becomes idle, it does not spin or block; instead, it attempts to steal pending tasks from other workers. The thief selects a random starting worker and probes neighbors in circular fashion, examining their queues to find work. Stealing from the front (FIFO end) of a victim's deque minimizes synchronization contention, as the owner primarily operates on the back (LIFO end). This enables true data parallelism: idle threads actively help underutilized peers, keeping the entire pool productive and improving overall system utilization.
The submit() method returns a std::future<T> that can be used to retrieve task results asynchronously. Internally, tasks are wrapped in std::packaged_task objects that capture their return values and exceptions. This follows standard C++ patterns and integrates seamlessly with existing code expecting std::future semantics, including blocking waits, timeouts, and exception propagation.
The shutdown() method ensures all pending tasks are executed before the pool terminates. Workers are signaled to exit their main loops only after the task queue is empty, providing strong guarantees that no work is lost. This is critical for applications requiring orderly termination and cleanup.
The project uses CMake 3.15+ with clean, idiomatic build configuration. Targets are properly exposed for linking by downstream projects, and dependencies (Google Benchmark, CTest) are declaratively specified. The build supports both Debug and Release configurations, with the latter enabling aggressive optimizations for performance-critical benchmark runs.
Correctness is verified through a comprehensive test suite built with CTest. Tests cover basic scheduling, future semantics, exception handling, stress conditions, graceful shutdown, and work-stealing behavior. All tests pass successfully on modern compilers (GCC 11+, Clang 13+, MSVC 2019+).
Performance is quantified using Google Benchmark, a production-grade benchmarking framework. Multiple benchmarks measure throughput under various conditions: pure scheduling overhead, public API latency, CPU-bound workloads, and comparisons against std::async. This enables data-driven optimization and clear performance communication.
The thread pool implements a sophisticated two-level scheduling hierarchy: a global round-robin scheduler distributes incoming tasks, and a distributed work-stealing network keeps idle workers productive.
submit(task)
↓
┌──────────────────────┐
│ Random +Round-Robin Scheduler │
│ (Atomic Counter) │
└──────────────────────┘
↓
┌──────────────────────┐
│ Worker Threads │
│ (num_threads × N) │
└──────────────────────┘
↙ ↓ ↓ ↘
┌────────────────────────┐
│ Per-Worker Task Deques │
│ (std::deque<Task>) │
└────────────────────────┘
↙ ↓ ↓ ↘
Owner pops LIFO ↔ Thief steals FIFO
(Back of deque) (Front of deque)
When the owner thread pops tasks from the back of its deque (LIFO order), it operates on the most recently added task. This maximizes temporal locality: the task was just enqueued, its data is likely still in L1/L2 cache, and any work-local state remains warm. In contrast, FIFO access (popping from the front) retrieves the oldest task, whose data has likely aged out of cache. By prioritizing recent tasks, LIFO minimizes cache misses and improves single-thread performance.
The owner thread operates on the back of its deque, and thieves operate on the front. This separation creates a "race-free zone" in the middle of the deque: as long as the deque contains multiple elements, the owner and thief never access the same element simultaneously. Contention only arises when the deque is nearly empty, and in those cases a failed steal is cheap—it signals that the victim actually has very little work remaining. This design allows lock-free implementations (via CAS-based operations) with minimal synchronization cost.
In a global-queue design, every task submission and completion requires acquiring a central mutex. With N workers and high submission rates, this lock becomes a severe bottleneck. Per-worker queues shift the synchronization burden to theft operations, which occur less frequently (only when a worker is idle). In the common case—a steady stream of work—the owning thread's local operations require zero synchronization, and the lock is acquired only during occasional work-stealing, dramatically reducing contention.
work-stealing-thread-pool/
├── include/
│ ├── thread_pool.hpp # Public thread pool interface
│ └── work_stealing_deque.hpp # Per-worker task deque implementation
├── src/
│ └── thread_pool.cpp # Implementation details
├── bench/
│ ├── enqueue_bench.cpp # Raw scheduling throughput
│ ├── submit_bench.cpp # Public API latency
│ ├── cpu_bench.cpp # CPU-bound workload scaling
│ └── async_bench.cpp # std::async comparison
├── tests/
│ ├── test_basic.cpp # Basic enqueue and execution
│ ├── test_futures.cpp # Future semantics and result retrieval
│ ├── test_exceptions.cpp # Exception propagation
│ ├── test_stress.cpp # High-concurrency stress testing
│ ├── test_shutdown.cpp # Graceful shutdown behavior
│ └── test_work_stealing.cpp # Work-stealing validation
├── CMakeLists.txt # Build configuration
└── README.md # This file
include/ — Public headers defining the thread pool interface. Users include thread_pool.hpp to create and interact with the pool.
src/ — Implementation files containing worker loops, synchronization logic, and scheduling algorithms.
bench/ — Benchmark programs using Google Benchmark. Each measures a specific performance aspect to guide optimization and document baseline behavior.
tests/ — Comprehensive test suite verifying correctness under normal and pathological conditions. Tests are run via CTest and should pass on all supported platforms.
CMakeLists.txt — Modern CMake configuration with dependency management, compiler flags, and target definitions.
#include "thread_pool.hpp"
#include <iostream>
int main() {
// Create a thread pool with 8 worker threads
ThreadPool pool(8);
// Submit a task and get a future for the result
auto future = pool.submit([]() {
return 42;
});
// Block until the task completes and retrieve the result
std::cout << future.get() << std::endl; // Output: 42
// Submit multiple tasks
std::vector<std::future<int>> futures;
for (int i = 0; i < 100; ++i) {
futures.push_back(pool.submit([i]() {
return i * i;
}));
}
// Collect all results
for (auto& f : futures) {
std::cout << f.get() << " ";
}
// Shutdown is called automatically when pool is destroyed
return 0;
}The primary API method. Accepts any callable (lambda, function pointer, std::function) and returns a std::future<T> where T is the callable's return type. The task is wrapped in a std::packaged_task to capture its result. Returns immediately after enqueueing; the task executes asynchronously on a worker thread.
Use when: You need the task's return value, or you want to handle exceptions from the task, or you need to wait for completion.
A lower-level API for fire-and-forget task submission. Does not return a future; the task simply executes asynchronously. Slightly faster than submit() because it avoids the overhead of wrapping the task in std::packaged_task.
Use when: The task returns void, you don't care about the result, or you're in a latency-critical path where the extra overhead is measurable.
| Method | Returns | Exception Handling | Use Case |
|---|---|---|---|
submit() |
std::future<T> |
Propagates via future.get() |
General-purpose; results needed |
enqueue() |
void |
Lost (logs if enabled) | Fire-and-forget; throughput critical |
The work-stealing scheduler keeps the thread pool productive even when work is unevenly distributed across workers.
-
Owner Operations (Lock-Free): Each worker thread owns a deque. It pushes new tasks onto the back and pops completed tasks from the back (LIFO). With no contention, these operations require no synchronization.
-
Thief Operations (Synchronized): When a worker has no pending tasks, it becomes a thief. It selects a random victim worker and attempts to steal a task from the front of the victim's deque (FIFO). The steal operation uses a compare-and-swap (CAS) to atomically dequeue from the front, minimizing synchronization overhead.
-
Victim Selection: To avoid thundering herd behavior where all idle workers target the same victim, the thief selects a random starting worker and probes neighbors in circular fashion. This distributes theft attempts uniformly across the pool.
-
Backoff: If a steal attempt fails (victim's deque is empty), the thief moves to the next victim. If all workers are idle, the worker blocks on a condition variable until new work arrives.
Work stealing provides excellent load balancing without explicit global state. Consider a scenario where Worker A receives 1000 tasks while Worker B is idle:
- Worker A executes tasks from its deque at full speed.
- Worker B, idle, repeatedly attempts to steal from Worker A's front.
- The steal succeeds as long as Worker A has pending tasks.
- As work redistributes, contention decreases naturally.
This dynamic load balancing emerges from local decisions by idle workers, avoiding the overhead of a centralized load balancer.
Because owners operate on the back (LIFO), they execute recently-enqueued tasks while the task's data is likely still in cache. Thieves steal from the front (FIFO), obtaining older tasks that may have less temporal locality, but this is acceptable because stealing happens only when the owner is idle—the performance cost of a cache miss on an old task is outweighed by the benefit of keeping both workers productive.
In a traditional global queue, idle threads spin or block on a central lock. Work stealing enables idle threads to directly search for work, avoiding context switches and lock contention. Furthermore, work redistribution happens automatically as thieves pull tasks toward underutilized workers, improving throughput on non-uniform workloads. Empirically, work-stealing pools achieve higher utilization than global-queue pools, especially as worker count increases.
Correctness is verified through a comprehensive test suite executed via CTest. All tests pass successfully on supported platforms.
Basic Enqueue Test — Verifies that tasks submitted via enqueue() execute in the correct order and complete successfully.
Future Test — Validates that submit() correctly returns futures, tasks execute asynchronously, and future.get() blocks until completion.
Void Future Test — Ensures that tasks with void return types work correctly via std::future<void>.
Exception Propagation Test — Verifies that exceptions thrown in tasks are propagated correctly when calling future.get(), preserving the exception's type and message.
Stress Test — Submits thousands of tasks concurrently and validates that all execute exactly once without data races or task loss.
Graceful Shutdown Test — Ensures that shutdown() waits for all pending tasks to complete before returning, and that the pool rejects new submissions after shutdown.
Work-Stealing Test — Verifies that idle workers successfully steal work from busy workers, improving utilization on skewed workloads.
# Configure and build
cmake -B build
cmake --build build
# Run all tests
cd build
ctest
# Run a specific test
ctest -R test_futures
# Run with verbose output
ctest -VAll tests pass successfully, confirming that the implementation is thread-safe and correct.
Performance is quantified using Google Benchmark. The following benchmarks measure throughput and latency under various conditions.
Enqueue Throughput — Measures the raw scheduling throughput of enqueue() by submitting tasks that immediately complete. This isolates the scheduler's overhead without the cost of std::packaged_task wrapping.
Submit Throughput — Measures the public API overhead of submit(). Each task completes immediately, so the benchmark measures both scheduling and future/packaged_task overhead.
CPU-Bound Benchmark — Submits tasks that perform 1000 iterations of CPU work. This evaluates scaling characteristics with realistic, non-trivial workloads where work-stealing and load balancing become significant.
std::async Comparison — Compares the thread pool against std::async, which creates a new thread (or uses an internal pool) for each task. This illustrates the benefit of reusing a pre-allocated pool.
| Workers | Throughput |
|---|---|
| 1 | 2.93M |
| 2 | 4.23M |
| 4 | 2.10M |
| 8 | 1.61M |
| Workers | Throughput |
|---|---|
| 1 | 1.37M |
| 2 | 1.29M |
| 4 | 980K |
| 8 | 933K |
| Method | Throughput |
|---|---|
| std::async | 77K |
| Thread Pool (8 workers) | 933K |
The enqueue throughput peaks at 2 threads (4.23M tasks/sec) and decreases with more workers because each additional thread increases contention on the scheduler's atomic counter. However, even at 8 threads, the pool sustains 1.61M tasks/sec—far exceeding a serial baseline and demonstrating the value of parallelism for workloads with task parallelism.
The submit throughput is approximately 2.7× lower than enqueue throughput because std::packaged_task and std::future incur additional allocation and synchronization overhead. This is expected and acceptable: when result retrieval is required, the overhead is a small price for correctness and integration with standard C++ patterns.
The comparison against std::async is striking: the thread pool is 12× faster than std::async on the same platform. This illustrates a fundamental advantage of pre-allocated thread pools: std::async must create a new thread (or allocate from an internal pool) for each task, incurring significant overhead. The fixed-size pool reuses threads, amortizing allocation costs across many tasks.
Why synchronization overhead dominates tiny tasks: When tasks complete in microseconds, the scheduler's synchronization overhead (atomic operations, potential lock acquisitions during work-stealing) becomes a larger fraction of total execution time. For larger tasks, this overhead is invisible. This is why the pool's throughput degrades gracefully as worker count increases—the bottleneck shifts from work execution to synchronization, a natural consequence of Amdahl's Law.
Why scaling decreases for extremely small tasks: At 8 workers competing for a single atomic counter, contention on the scheduler's round-robin counter becomes non-negligible. Additionally, work-stealing synchronization between multiple workers can introduce cache-line bouncing. These effects are inherent to the design and are acceptable trade-offs: the pool excels at larger, more realistic workloads where work-stealing truly improves utilization.
- C++20 compatible compiler (GCC 11+, Clang 13+, MSVC 2019+)
- CMake 3.15 or later
- Google Benchmark (optional, for benchmarks)
# Create and configure the build directory
cmake -B build
# Build the project
cmake --build build
# Run an example (if included)
./build/example_thread_pool# Build with optimizations enabled
cmake -B build-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-releaseRelease builds enable aggressive compiler optimizations (O3, LTO) that are essential for fair benchmarking. Debug builds include sanitizer flags and assertions that significantly impact performance.
Benchmarks require a Release build to produce meaningful results. Google Benchmark automatically runs each benchmark multiple times to account for variance.
# Build in Release mode
cmake -B build-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release
# Run individual benchmarks
./build-release/enqueue_bench
./build-release/submit_bench
./build-release/cpu_bench
./build-release/async_bench
# Or run all benchmarks
cmake --build build-release --target benchmarkEach benchmark produces detailed output including mean, standard deviation, and throughput metrics. Results are sensitive to system load, so run benchmarks on a quiet machine for reproducibility.
The current implementation uses mutexes for thread-safe deque access. A lock-free Chase-Lev deque would replace these locks with atomic operations and compare-and-swap, reducing synchronization overhead and improving performance on high-contention workloads. This would require careful handling of ABA problems and memory reclamation.
Impact: Reduced latency and improved throughput on highly concurrent workloads.
Currently, each submit() call enqueues a single task. Batching multiple tasks in a single operation could reduce per-task overhead and improve cache locality. A submit_batch() method could accept a range of tasks and enqueue them efficiently.
Impact: Improved throughput for bulk task submission patterns.
Worker threads could be pinned to specific CPU cores, improving cache coherency and reducing context-switch overhead. This would require platform-specific code but would benefit applications sensitive to NUMA locality.
Impact: Improved performance on multi-socket systems; more predictable latency.
On systems with multiple NUMA nodes, the scheduler could prefer stealing from workers on the same node, reducing cross-node memory traffic. This requires careful integration with thread affinity.
Impact: Superior performance on large multi-socket systems.
Integration with C++20 coroutines would enable stackless task execution, reducing memory overhead and supporting millions of concurrent tasks. The submit() API could accept coroutine tasks alongside traditional callables.
Impact: Dramatically increased scalability for I/O-bound applications.
Building this project deepens understanding of modern concurrent systems:
Thread Synchronization — Mutexes, condition variables, and atomics are used to coordinate worker threads safely. Understanding synchronization primitives is essential for avoiding race conditions and deadlocks.
Mutexes & Condition Variables — The implementation uses these standard synchronization tools to protect shared data structures and signal workers when tasks arrive or shutdown is requested.
Work-Stealing Algorithms — The core insight that idle workers can help busy workers by stealing their work is powerful and elegant. This concept extends far beyond thread pools (e.g., parallel task schedulers in Cilk, Swift's concurrency model).
Futures and Packaged Tasks — std::future and std::packaged_task enable asynchronous computation with type-safe result retrieval and exception propagation. Mastering these APIs is critical for modern C++ concurrent applications.
Modern CMake — Writing idiomatic, maintainable build configurations that scale to complex projects.
Automated Testing — CTest and comprehensive test coverage validate correctness and prevent regressions. Testing concurrent code is especially important due to subtle race conditions.
Performance Benchmarking — Google Benchmark provides reproducible performance measurement, enabling data-driven optimization decisions.
Concurrent Systems Design — Understanding trade-offs between throughput, latency, fairness, and complexity when building scalable systems.
MIT License
Copyright (c) 2026
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.