Skip to content

Commit 1dc0507

Browse files
jbeemsterquetzalliwritesremotesynth
authored
DOC-313: Kinesis: add option for a rust based mock (#731)
Co-authored-by: Quetzalli <hola@quetzalliwrites.com> Co-authored-by: Brian Rinaldi <brian.rinaldi@gmail.com>
1 parent 2bb2dd6 commit 1dc0507

2 files changed

Lines changed: 34 additions & 9 deletions

File tree

src/content/docs/aws/configuration/config/configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,7 @@ Also see [OpenSearch configuration variables](#opensearch) which are used to man
246246
| `KINESIS_SHARD_LIMIT` | `100` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default shard limit. |
247247
| `KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT` | `10` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default on demand stream count limit. |
248248
| `KINESIS_LATENCY` | `500` (default), `0` (to disable)| Integer value of milliseconds, causing the Kinesis API to delay returning a response in order to mimic latency from a live AWS call. |
249-
| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. |
249+
| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` \| `rust` | String value of `node` (default), `scala`, or `rust` that determines the underlying build of Kinesis Mock. See [Performance Tuning](/aws/services/kinesis#performance-tuning) for guidance on choosing an engine. |
250250
| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. |
251251
| `KINESIS_MOCK_INITIAL_HEAP_SIZE` | `256m` (default) | JVM memory format string that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. |
252252

src/content/docs/aws/services/kinesis.mdx

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -170,21 +170,44 @@ You can fetch the CloudWatch logs for your Lambda function reading records from
170170
171171
### Performance Tuning
172172
173-
For high-volume workloads or large payloads, we recommend switching to the Scala engine via the `KINESIS_MOCK_PROVIDER_ENGINE=scala` flag, delivering up to 10x better performance compared to the default Node.js engine.
173+
By default, Kinesis Mock runs on the Node.js engine (`KINESIS_MOCK_PROVIDER_ENGINE=node`).
174+
For high-volume workloads, large payloads, or long-running setups, you can switch to a different engine using the `KINESIS_MOCK_PROVIDER_ENGINE` flag.
174175
175-
Additionally, the following parameters can be tuned:
176+
#### Scala engine
177+
178+
The Scala engine (`KINESIS_MOCK_PROVIDER_ENGINE=scala`) delivers up to 10x better performance compared to the default Node.js engine.
179+
The following parameters can be tuned for the Scala engine:
176180
177181
- Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure.
178182
- Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory.
179-
- Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`).
180-
181-
Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters.
182183
183184
:::note
184185
`KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine.
185-
Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use.
186186
:::
187187
188+
#### Rust engine
189+
190+
The Rust engine (`KINESIS_MOCK_PROVIDER_ENGINE=rust`), introduced in LocalStack 2026.07, is a LocalStack-native rebuild of Kinesis Mock and is the recommended choice for any demanding or long-lived workload.
191+
It provides:
192+
193+
- **Native multi-account and multi-region support** from a single binary, regardless of how many accounts you use — this removes the per-account instance overhead described under [Limitations](#limitations).
194+
- **Faithful enforcement of all AWS API limits**, including per-shard rate ingress/egress limits.
195+
- **An improved persistence model** that replaces the legacy unbounded in-memory queue, allowing the mock to run under sustained load for long periods without the risk of running out of memory.
196+
197+
:::caution
198+
This is the first release of the Rust engine and it should be considered experimental for anyone opting in.
199+
200+
Persistence is **not compatible** between the Rust engine and the Node.js or Scala engines, in either direction.
201+
Any existing Kinesis persistence (including Cloud Pods and state created with the `node` or `scala` engines) will not carry over when you switch to `rust` — and vice versa.
202+
Switching engines effectively starts from a fresh state.
203+
:::
204+
205+
#### All engines
206+
207+
Regardless of the selected engine, you can reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable them entirely with `0`).
208+
209+
Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/aws/configuration/config/configuration/#kinesis) for more details on these parameters.
210+
188211
## Resource Browser
189212
190213
The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters.
@@ -209,8 +232,10 @@ The following code snippets and sample applications provide practical examples o
209232
210233
## Limitations
211234
212-
In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used.
213-
[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature.
235+
When using the default Node.js or the Scala engine, each AWS account launches a separate instance of Kinesis Mock in multi-account setups, which is very resource intensive when a large number of AWS accounts are used.
236+
[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature for the upstream engines.
237+
238+
The [Rust engine](#performance-tuning) (`KINESIS_MOCK_PROVIDER_ENGINE=rust`) is not affected by this limitation, as it natively supports any number of accounts and regions from a single binary.
214239
215240
## API Coverage
216241

0 commit comments

Comments
 (0)