2024/12/03/introducing-outboxkit/

[giscus[bot]]

2024/12/03/introducing-outboxkit/

After talking about the outbox pattern so much, and implementing it from scratch a few times in a few different projects, finally figured it was high time for me to build something I could reuse.

https://blog.codingmilitia.com/2024/12/03/introducing-outboxkit/

[mivano]

Nice article and implementation, thanks!

[hramza]

Nice article, thanks.
How can horizontal scaling work with a distributed lock approach for a worker that processes millions of messages per day, and what happens if the instance holding the lock fails?

[joaofbantunes]

Glad it was interesting.

By design, OutboxKit only allows one worker to produce at a time, to ensure that message ordering can be maintained. If the instance holding the lock fails, particularly in the relational database scenario described in the post, the lock will be released, and another instance will be able to pick up where the other left off.

Even with this simple single producer at a time approach, a few thousand messages can be produced per second, easily reaching the millions you mention. Note that this is based on some simple tests, don't really know the ceiling for it, as I don't have that kind of scale at work.

This good performance is not really because OutboxKit is doing anything special, quite the opposite, it's trying to be as simple as possible. "All" it does, is query a batch of messages from the database and make them available for production. With the right batching approaches, you'll probably be able to produce thousands of messages in a few milliseconds.

More advanced techniques could be implemented to allow worker parallelism (e.g. partitioning), but as it isn't a need for me at the moment, I kept things as simple as possible.