Note: This is an unofficial fork of Netflix/dynomite, maintained by @zcalusic. It is not affiliated with or endorsed by Netflix. Upstream is no longer actively maintained, so this fork collects fixes needed to keep Dynomite building and running on current systems (newer GCC and Debian releases, memory leak and crash fixes). Releases are tagged
vX.Y.Z, starting withv0.9.0. A Docker image, zcalusic/dynomite, is built from zcalusic/dockerfiles. Please report issues for this fork here, not upstream.
Dynomite, inspired by the Dynamo whitepaper, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Valkey, Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.
The ultimate goal with Dynomite is to be able to implement high availability and cross-datacenter replication on storage engines that do not inherently provide that functionality. The implementation is efficient, not complex (few moving parts), and highly performant.
Valkey is now our recommended and default-tested backend in place of Redis. Valkey is the Linux Foundation fork of Redis 7.2.4 and keeps the permissive BSD-3-Clause license.
Since 2024, Redis is no longer BSD-licensed. Version 8 is offered under RSALv2, SSPLv1 or AGPLv3. As an Apache-2.0 project, we want the backends we recommend, test against and use in our examples to be permissively licensed, so users can adopt the full stack without copyleft or source-available terms to evaluate.
Redis remains fully supported: Valkey and Redis speak the same protocol (RESP), selected with data_store: 0, and the configurations in conf/ work with either. CI runs the functional tests against the latest Valkey, Redis and memcached releases. Dynomite supports the commands listed in notes/redis.md, on either server, including ones that only newer releases have, such as hash field expiry (HSETEX, HGETEX, HEXPIRE, ...) or Valkey's DELIFEQ: a server without such a command refuses it as usual, so upgrade all of a cluster's servers before using it.
Memcached is supported with a smaller command set and fewer guarantees; see data_store under Configuration.
All development happens on the master branch, the only branch in this fork. Releases are signed tags named vX.Y.Z (v0.9.0, v0.9.1, ...; before v0.9.0 they were v0.6.22-N), published as GitHub releases; the Docker image is built from the latest release. Every push to master is built and tested by CI.
Install the build dependencies (Debian package names):
$ sudo apt-get install automake gcc git libc6-dev libssl-dev libyaml-dev make
Build from a git clone: the version is taken from git describe, which needs the history and tags.
To build and install Dynomite, as the Docker image does:
$ git clone https://github.com/zcalusic/dynomite.git
$ cd dynomite
$ autoreconf -fvi
$ ./configure
$ make
$ sudo make install
For a debug build, pass --enable-debug to configure:
--enable-debug=log: debug logs enabled, assertions disabled.--enable-debug=yes: debug logs enabled, failed assertions are logged.--enable-debug=full: debug logs enabled, failed assertions abort Dynomite.
For example, to build in debug mode for use with a debugger:
$ autoreconf -fvi
$ CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full
$ make
$ src/dynomite -h
The unit tests (src/dyn_test.c) need no extra dependencies; run them after building:
$ make check
The functional tests start a local cluster of Dynomite and Valkey nodes and compare the results of the same commands against Dynomite and a standalone Valkey. Install their dependencies (Debian package names):
$ sudo apt-get install openssl valkey-server python3 python3-plumbum python3-redis python3-requests python3-yaml
They run valkey-server, or redis-server when Valkey isn't installed; to test another server, such as a build from source, set DYN_TEST_SERVER to its binary.
Then run them from the repository; this builds Dynomite with debug logs first (-n skips the build). Each cluster runs plaintext and with mutual TLS between its nodes (certificates are made on the fly with openssl), and test/tls_test.py checks what the TLS peer port accepts and rejects:
$ test/run_tests.sh
Usage: dynomite [-?hVdDt] [-v verbosity level] [-o output file]
[-c conf file] [-p pid file] [-x admin operation]
[-m mbuf size (deprecated)]
[-M max alloc messages (deprecated)]
Options:
-h, --help : this help
-V, --version : show version and exit
-t, --test-conf : test configuration for syntax errors and exit
-d, --daemonize : run as a daemon
-D, --describe-stats : print stats description and exit
-v, --verbosity=N : set logging level (default: 5, min: 0, max: 11)
-o, --output=S : set logging file (default: stderr)
-c, --conf-file=S : set configuration file (default: /usr/local/etc/dynomite.yml)
-p, --pid-file=S : set pid file (default: off)
-x, --admin-operation=N : set admin mode, 1 sends deletes to the local datastore only (default: 0, off)
-m, --mbuf-size=N : set size of mbuf chunk in bytes (default: 16384 bytes, deprecated)
-M, --max-msgs=N : set max number of messages to allocate (default: 200000, deprecated)
Signals: SIGTERM (docker stop, kubectl delete pod, systemd) and SIGINT make Dynomite exit cleanly at once, with exit status 0 and the pid file removed. SIGHUP reopens the log file, SIGTTIN and SIGTTOU raise and lower the logging level.
Dynomite is configured through a YAML 1.1 file (YAML 1.1 is not JSON compatible), given with the -c or --conf-file command-line argument on process start. The keys below go under a single top-level pool name, such as dyn_o_mite:. Dynomite understands the following keys:
- env: The environment of the node: aws or network (for a physical datacenter). Defaults to aws.
- datacenter: The name of the datacenter. Please refer to the architecture document.
- rack: The name of the rack. Please refer to the architecture document.
- dyn_listen: The port that dynomite nodes use to inter-communicate and gossip.
- enable_gossip: enable gossip instead of static tokens (default: false). Gossip is experimental.
- gos_interval: The sleeping time in milliseconds at the end of a gossip round. Defaults to 30000; must be at least 1.
- tokens: The token(s) owned by a node. Vnodes are not supported, so this works with one token only. A token is a number from 0 to 4294967295; anything else is a configuration error.
- dyn_seed_provider: A seed provider implementation to provide a list of seed nodes. One of simple_provider (uses dyn_seeds), florida_provider or dns_provider. Defaults to simple_provider.
- dyn_seeds: A list of seed nodes in the format: address:port:rack:dc:tokens (vnodes are not supported). The address can be a name that doesn't resolve yet, e.g. a container that isn't up: the node starts with that seed down and resolves the name again before each connection attempt, so it connects once the name resolves and follows the name to a new address after a reconnect. The lookup blocks the node while it runs, so a slow or unreachable DNS server delays it for up to the resolver's timeout (see resolv.conf) once per reconnect backoff.
- listen: The listening address and port (name:port or ip:port) for this server pool.
- backlog: The TCP listen backlog for the listening sockets. Defaults to 512; must be at least 1.
- hash: The hash function used to map keys to tokens. One of one_at_a_time, md5, crc16, crc32, crc32a, fnv1_64, fnv1a_64, fnv1_32, fnv1a_32, hsieh, murmur or jenkins. Defaults to murmur. Keep the default: Dyno's token-aware routing finds a key's node with a Murmur-based partitioner, so it only works with
murmur. Choose the hash once, before the cluster stores data: changing it on a running cluster maps nearly every key to another token. The other functions come from twemproxy, which offered libmemcached's list so that setups sharding keys in the client could move behind the proxy without remapping them; that doesn't apply to Dynomite's token ring. Avoidcrc32in particular: it is libmemcached's variant and only gives values from 0 to 32767, so all keys land on a single node (crc32ais the standard CRC-32). - hash_tag: A two-character string, such as
{}, that selects the part of a key to hash: when a key contains both characters, only the part between them is hashed, so related keys can be kept on the same node. Not set by default. - timeout: The timeout in msec for establishing a connection to the server or receiving a response from it. Defaults to 5000 msec; must be at least 1.
- data_store: An integer value that controls whether a server pool speaks RESP, the protocol of Valkey and Redis (0), or memcached (1). Defaults to 0. Memcached support covers the ASCII text protocol: get and gets (with several keys, split between the nodes that own them), set, add, replace, append, prepend, cas, incr, decr, delete, touch and quit; the binary and meta protocols and other commands, such as gat, stats or flush_all, are not supported, and get ERROR. cas works with one rack only: each memcached numbers its items itself, so replicas have different cas uniques, and with more than one rack Dynomite refuses cas with SERVER_ERROR rather than change only the replicas whose unique matched (gets still works, but under DC_SAFE_QUORUM and DC_EACH_SAFE_QUORUM fails when the replicas' uniques differ). Keys are sharded and replicated as with Valkey and Redis, but replies from replicas are not reconciled and there are no read repairs: when the replies disagree, DC_QUORUM returns the first one and the stricter levels return an error. Replicas can disagree byte for byte after an incr or decr that makes the number shorter, since memcached pads the result with spaces in some cases and not in others. The functional tests compare replies through Dynomite with a standalone memcached's, on the same clusters as for Valkey and Redis, but run fewer tests: the peer TLS and seed checks run against Valkey and Redis only.
- server_retry_timeout: The longest wait in msec between attempts to reconnect to the server: the wait starts at 1 second and doubles after each failure, up to this limit in whole seconds (rounded down), so values below 2000, including 0, reconnect every second. Defaults to 10000 msec.
- servers: A list with the local datastore's address, port and weight (name:port:weight or ip:port:weight) for this server pool. Exactly one is supported; a second entry is a configuration error.
- read_consistency, write_consistency: The consistency level for reads and writes: DC_ONE, DC_QUORUM, DC_SAFE_QUORUM or DC_EACH_SAFE_QUORUM (case-insensitive). Both default to DC_ONE. Under the quorum levels, a quorum is more than half of a datacenter's racks sending the same reply; a datacenter can have up to 8 racks, and a node with more in its seeds doesn't start. Please refer to the consistency document. Both can also be changed at runtime through the stats_listen REST endpoint (
/set_consistency/read/<level>and/set_consistency/write/<level>); the new level applies to client connections opened after the change. - read_repairs_enabled: A boolean value that enables read repairs, which update replicas found with stale data during a read; they don't run with DC_ONE. Defaults to false. Please refer to the read repair document. Can also be changed at runtime through the stats_listen REST endpoint (
/read_repairs/enableand/read_repairs/disable). - stats_listen: The address and port number for the REST endpoint and for accessing statistics.
GET /helplists the commands. Paths must match exactly; an unknown path gets 404 Not Found, an invalid argument (log level, timeout factor, consistency level, missing peer name) 400 Bad Request, an unknown peer 404, and a method other than GET 405 Method Not Allowed.GET /pingis the health check: OK while the node is connected to its datastore, and 503 Service Unavailable with "Datastore is not connected" while it isn't; the node reconnects on its own, waiting at mostserver_retry_timeoutbetween attempts. - stats_interval: The stats aggregation interval in msec (default: 30000 msec; at least 1).
- mbuf_size: The size of an mbuf chunk in bytes (default: 16384 bytes; 512 to 512000, a multiple of 16).
- max_msgs: The maximum number of messages to allocate (default: 200000; 100000 to 1000000). Under load, Dynomite applies limits in three places, without dropping requests already accepted. First, it stops reading a client, or a peer sending requests, that has 20000 requests waiting for their replies, and resumes at 10000, so TCP pushes back on the sender. Second, a new client request that would go over max_msgs closes that client, with a warning in the log; replies and requests from peers are always accepted. Third, a connection to a peer with 20000 requests queued refuses new ones until fewer than 20000 are queued: a client waiting on that peer gets "-ERR Peer: Peer Node has too many requests queued", and a copy for another rack or DC is dropped. The stats counters peer_queue_full_requests and remote_peer_queue_full_requests count the refused requests.
- datastore_connections: Maximum number of connections to the local datastore. Defaults to 1; 1 to 255.
- local_peer_connections: Maximum number of connections to a local DC peer. Defaults to 1; 1 to 255.
- remote_peer_connections: Maximum number of connections to a remote DC peer. Defaults to 1; 1 to 255.
- conn_msg_rate: The maximum number of messages per second sent on each connection to a node in a remote datacenter. Defaults to 50000; must be at least 1.
- peer_tls: A boolean value that turns on mutual TLS 1.3 for the traffic between nodes, in both directions; see Securing traffic between nodes. Defaults to false.
- peer_tls_cert: The node's certificate in PEM format, followed by any intermediate CA certificates. Required with
peer_tls: true. - peer_tls_key: The node's private key in PEM format, unencrypted. Required with
peer_tls: true. - peer_tls_ca: The cluster CA certificate(s) in PEM format: only peers with certificates issued by them are trusted. Required with
peer_tls: true.
The keys client_connections, distribution, server_connections, dyn_port, dyn_connections, dyn_read_timeout, dyn_write_timeout, recon_key_file, recon_iv_file, auto_eject_hosts, server_failure_limit, preconnect and pem_key_file are still accepted for compatibility, but are ignored. The inter-node port comes from dyn_listen. Dynomite always connects to the datastore and to its peers at startup.
The built-in encryption of traffic between nodes was removed: it crashed with RSA keys over 1024 bits, lost writes with all, still accepted unencrypted messages, and gave no integrity or forward secrecy. Mutual TLS replaces it, see the next section. secure_server_option is still accepted with the value none and ignored; any other value is a configuration error, so that a node doesn't quietly run unencrypted when its configuration asks for encryption. All nodes of a cluster that used encryption have to switch to none together: a node closes peer connections that send encrypted messages.
See conf/dynomite.yml for an example; conf/ has more, such as conf/redis_single.yml.
Finally, to make writing syntactically correct configuration files easier, dynomite provides a command-line argument -t or --test-conf that can be used to test the YAML configuration file for syntax errors. With peer_tls: true it also loads the certificate, key and CA files and reports what is wrong with them.
With peer_tls: true, every connection between nodes (the dyn_listen port, gossip included) is mutual TLS 1.3: each node presents its certificate when connecting and when accepting, and both sides accept only certificates issued by the CA(s) in peer_tls_ca. A connection without a valid client certificate is closed before any message is read, so nothing but the cluster's own nodes can talk to the peer port, whatever network it is on. Nodes are verified by the CA only: the names and addresses in the certificates are not checked, so one certificate can be reused on another node, and a node can be reached under any address. OpenSSL's defaults apply otherwise: the TLS 1.3 ciphersuites, and the hybrid post-quantum key exchange (X25519MLKEM768) on OpenSSL 3.5 and later. Session resumption is off: every connection does a full handshake. A handshake that does not finish within 10 seconds is closed.
What stays unencrypted: the client port (listen), the stats port (stats_listen) and the connection to the datastore. Keep those on a trusted network.
Use one private CA per cluster, never a shared company CA and never a public one: anything that CA signs can join the cluster. OpenSSL is enough to run it; this makes a CA and a certificate for one node, both Ed25519 (ECDSA P-256 or RSA 2048+ work the same way), the certificate valid for a year:
$ openssl req -x509 -newkey ed25519 -nodes -keyout ca.key -out ca.pem -days 3650 -subj /CN=dynomite-ca
$ openssl req -newkey ed25519 -nodes -keyout node1.key -out node1.csr -subj /CN=node1
$ printf 'subjectAltName=DNS:node1,IP:10.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\n' > node1.ext
$ openssl x509 -req -in node1.csr -CA ca.pem -CAkey ca.key -CAcreateserial -days 365 -extfile node1.ext -out node1.pem
Each certificate needs the serverAuth and clientAuth extended key usages (or none at all), since the same certificate serves both directions. Keep ca.key off the nodes; give each node its node1.pem, node1.key (mode 0600) and ca.pem:
peer_tls: true
peer_tls_cert: /etc/dynomite/node1.pem
peer_tls_key: /etc/dynomite/node1.key
peer_tls_ca: /etc/dynomite/ca.pem
In the Docker image, mount the files read-only and point the configuration at them, for example -v /etc/dynomite/tls:/tls:ro with peer_tls_cert: /tls/node1.pem and so on.
All nodes of a cluster switch together: a node with peer_tls: true closes plaintext peers, and a node with it off closes TLS peers, so a mixed cluster doesn't replicate until the last node is switched. When new certificates are needed, replace the files and restart the node (a rolling restart, one node at a time, keeps the cluster serving); there is no reload. To rotate the CA, put the old and the new CA certificate into peer_tls_ca on every node and restart, then reissue the node certificates from the new CA, and finally drop the old CA from the file.
At startup, a node checks its own certificate the way its peers will and logs a warning when it doesn't verify with peer_tls_ca, has expired or is not valid yet; it starts anyway, and the handshake failures show up on the peers. It warns daily when the certificate expires within 14 days. Missing or unreadable files, a key that doesn't match the certificate or a CA file without a certificate are configuration errors and the node doesn't start. The stats REST endpoint (/info) has peer_tls_cert_days_left, and the counters peer_tls_handshakes, peer_tls_handshake_errors and peer_tls_plaintext_rejected; every failed handshake is logged with the peer's address and the reason. To check a node from outside, connect with a node certificate (it must fail without one):
$ openssl s_client -connect node1:8101 -CAfile ca.pem -cert node2.pem -key node2.key
Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0. See LICENSE and NOTICE.
