From f248fbcd139e77a98287b8c23180752afb5d54cc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 19:31:06 +0000 Subject: [PATCH] docs: add guide to optimize hydration requirements Moves the hydration-strategies content out of the Hydration concept page (explanation) into a new task-oriented guide under Clusters, and adds strategies sourced from internal engineering notes: limiting hydration concurrency, preferring delta joins, avoiding RETAIN HISTORY on indexes, and validating full-cluster hydration before relying on slim deployments. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013P8uyNjTsVmjGBgUQgU8GS --- doc/user/content/clusters/_index.md | 1 + .../clusters/operational-guidelines/_index.md | 6 +- .../optimize-hydration-requirements.md | 159 ++++++++++++++++++ .../content/fundamentals/concepts/clusters.md | 7 +- .../fundamentals/concepts/hydration.md | 96 +---------- 5 files changed, 174 insertions(+), 95 deletions(-) create mode 100644 doc/user/content/clusters/optimize-hydration-requirements.md diff --git a/doc/user/content/clusters/_index.md b/doc/user/content/clusters/_index.md index ed91451c25d4b..8dd22ceef8ab4 100644 --- a/doc/user/content/clusters/_index.md +++ b/doc/user/content/clusters/_index.md @@ -12,4 +12,5 @@ Clusters provide the compute resources for running dataflows in Materialize. - Learn about [clusters](/fundamentals/concepts/clusters/). - Follow the [operational guidelines](/clusters/operational-guidelines/). +- [Optimize hydration requirements](/clusters/optimize-hydration-requirements/). - Understand [system clusters](/clusters/system-clusters/). diff --git a/doc/user/content/clusters/operational-guidelines/_index.md b/doc/user/content/clusters/operational-guidelines/_index.md index d514f6c95f296..57a21a57a82f6 100644 --- a/doc/user/content/clusters/operational-guidelines/_index.md +++ b/doc/user/content/clusters/operational-guidelines/_index.md @@ -67,8 +67,10 @@ For upsert sources, snapshotting is a resource-intensive operation that can requ When sizing a cluster, budget for hydration memory on top of the steady-state cost. The table below summarizes, per object type, when each object hydrates and -the memory it uses. For more on hydration, including strategies to reduce its -impact, see [Hydration](/fundamentals/concepts/hydration/). +the memory it uses. For more on hydration, see +[Hydration](/fundamentals/concepts/hydration/). For strategies to reduce +hydration memory, see [Optimize hydration +requirements](/clusters/optimize-hydration-requirements/). {{% yaml-table data="hydration-objects-table" %}} diff --git a/doc/user/content/clusters/optimize-hydration-requirements.md b/doc/user/content/clusters/optimize-hydration-requirements.md new file mode 100644 index 0000000000000..e05a9b286f796 --- /dev/null +++ b/doc/user/content/clusters/optimize-hydration-requirements.md @@ -0,0 +1,159 @@ +--- +title: "Optimize hydration requirements" +description: "Strategies to reduce the memory Materialize needs to hydrate objects, and to speed up or avoid hydration altogether." +menu: + main: + parent: "clusters" + weight: 6 + identifier: "optimize-hydration-requirements" +--- + +[Hydration](/fundamentals/concepts/hydration/) primarily impacts memory usage, +and its speed scales with cluster size. This guide covers strategies to reduce +the memory a cluster needs during hydration, speed hydration up, or avoid +triggering it in the first place. + +## Isolate hydration-heavy objects + +- Use a dedicated cluster for [sources](/fundamentals/concepts/sources/). + +- In addition, use a dedicated cluster for upsert sources; i.e., do not + co-locate with append-only Kafka sources or CDC database sources. + + - Keeping append-only Kafka sources and CDC database sources (PostgreSQL, + MySQL, and SQL Server sources) on a separate cluster isolates ingestion + from possible OOM loops caused by memory-heavy objects such as Kafka + upsert sources. + + - Note: PostgreSQL, MySQL, and SQL Server sources run on a single replica, + the oldest, and remain there until that replica is removed. As such, the + use of a burst replica (through [`AUTO SCALING STRATEGY (ON + HYDRATION)`](#provision-extra-capacity-while-hydrating)) has no impact on + these single-replica sources. + +- Distribute materialized views and indexes across multiple clusters. Each + cluster's replicas hydrate their objects independently, which distributes + the memory required for hydration, lets objects on different clusters + hydrate in parallel, and limits how much must re-hydrate when any one + replica restarts. + +## Limit concurrent hydration + +By default, a replica hydrates up to 4 dataflows at a time. Lowering this +limit spreads out the memory spikes from hydrating many objects at once, at +the cost of a longer total hydration time; raising it can shorten total +hydration time at the cost of a higher peak. [Contact our team](/support/) if +you'd like to tune this for your cluster. + +## Reuse arrangements across consumers + +If multiple objects in the **same** cluster consume the same view, add an +[index](/fundamentals/concepts/indexes/) to that view **before** creating the +consumers. Consumers in that cluster can reuse the indexed arrangement instead +of each building equivalent in-memory state, which can reduce both memory +usage during hydration and steady-state memory. + +- Index reuse is limited to the cluster the index is on, and the index must + exist **before** its consumers are created for the optimizer to reuse it. +- For a view with only one consumer, an index generally adds memory instead of + saving it. + +## Split large materialized views + +For a very large materialized view, consider splitting it into several +smaller materialized views, for example by a partition key such as customer, +region, or date range. Smaller materialized views can hydrate as separate +dataflows, which can bound peak memory compared with hydrating one very large +materialized view. + +- This helps most when a cluster runs only a few large materialized views, + where a single view's hydration spike can dictate the cluster size. A + cluster with many materialized views already hydrates them as separate + dataflows and gets this benefit naturally. +- A re-plan or replacement of one split view affects only that portion of the + data. A replica restart still re-hydrates all views on the replica, though + in smaller units. +- If the split views share expensive computation, put that computation in a + [common indexed view first](#reuse-arrangements-across-consumers), creating + the index **before** creating the split views. Otherwise, each split view + may rebuild its own copy of the shared work, increasing total memory. +- Queries must target or combine the split views. + +## Reduce arrangement memory + +- For multi-way joins (more than two inputs), a [delta + join](/transform-data/optimization/#optimize-multi-way-joins-with-delta-joins) + keeps no intermediate results in memory, unlike a differential join. Index + all the join keys so the optimizer can choose a delta join. + +- If your workload has columns with a small set of often-repeated values (for + example, status strings or enum-like labels), [dictionary + compression](/transform-data/dictionary-compression/) can reduce the memory + those arrangements use, at the cost of CPU and slower hydration. + +- Avoid setting [`RETAIN HISTORY`](/serve-results/durable-subscriptions/) on + indexes: an index's arrangement holds its full retained history in memory, + not just the storage layer. Configure history retention on a materialized + view instead. + +## Provision extra capacity while hydrating + +Add an [`AUTO SCALING STRATEGY (ON +HYDRATION)`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) +to your cluster with memory-heavy objects. With this strategy, Materialize +automatically provisions an extra, larger replica (a burst replica) while the +cluster has unhydrated objects, then removes it once a steady-size replica +catches up. You pay for the burst replica while it is provisioned, but not at +steady state. + +If a steady-size replica runs out of memory during hydration, resize the +cluster. During the resize, the cluster continues to serve from the burst +replica. + +## Avoid unnecessary full-cluster hydration + +When changing a materialized view or index, or forcing dependents to re-plan +(for example, after dropping an index and recreating the dependents), build +the new version to the side to avoid downtime: + +- A [blue/green deployment](/developer-tools/dbt/blue-green-deployments/) + hydrates the new version alongside the old and cuts over when hydrated, with + no serving gap. Note that blue/green requires sources and sinks to live on + dedicated clusters that are excluded from the swap. For more information, + see [blue/green deployment](/developer-tools/dbt/blue-green-deployments/). + +- For a single materialized view, creating and hydrating a [replacement + materialized view (public preview) and replacing the existing view in + place](/transform-data/updating-materialized-views/replace-materialized-view/) + may be simpler, but briefly reduces freshness. The replacement materialized + view can be either on the same or different cluster. + +{{< note >}} +A [slim deployment](/developer-tools/dbt/slim-deployments/) that redeploys +only changed objects onto a copy of a cluster only exercises those objects' +hydration spike, not the full cluster's combined spike. A cluster that +passes a slim deployment can still fail to hydrate on the next full restart +or upgrade. Use a [blue/green +deployment](/developer-tools/dbt/blue-green-deployments/) in production to +validate hydration for the whole cluster. +{{< /note >}} + +{{< note >}} + +The burst-replica and blue/green strategies run extra replicas alongside the +existing ones, as do a resize or a zero-downtime upgrade. During the overlap, +the cluster temporarily uses additional resources. Account for the additional +cost and, on self-managed deployments, the additional capacity required. + +{{< /note >}} + +## Related pages + +- [Hydration](/fundamentals/concepts/hydration/) +- [Operational guidelines](/clusters/operational-guidelines/) +- [Query optimization](/transform-data/optimization/) +- [Dictionary compression](/transform-data/dictionary-compression/) +- [Durable subscriptions](/serve-results/durable-subscriptions/) +- [Snapshotting](/fundamentals/concepts/snapshotting/) +- [Clusters](/fundamentals/concepts/clusters/) +- [Troubleshooting](/serve-results/troubleshooting/#hydrating-objects) diff --git a/doc/user/content/fundamentals/concepts/clusters.md b/doc/user/content/fundamentals/concepts/clusters.md index 762f4ef56e0cc..46286851ff315 100644 --- a/doc/user/content/fundamentals/concepts/clusters.md +++ b/doc/user/content/fundamentals/concepts/clusters.md @@ -282,8 +282,10 @@ that provisions an extra burst replica at a larger size while the cluster has un-hydrated objects. {{< /tip >}} -For more information, including hydration strategies and the memory usage of -hydrating objects, see [Hydration](/fundamentals/concepts/hydration/). +For more information, including the memory usage of hydrating objects, see +[Hydration](/fundamentals/concepts/hydration/). For strategies to reduce +hydration memory, see [Optimize hydration +requirements](/clusters/optimize-hydration-requirements/). ## Best practices @@ -315,6 +317,7 @@ production cluster(s) to run development workloads or non-production tasks. - [`CREATE CLUSTER`](/sql/create-cluster) - [`ALTER CLUSTER`](/sql/alter-cluster) - [Hydration](/fundamentals/concepts/hydration/) +- [Optimize hydration requirements](/clusters/optimize-hydration-requirements/) - [System clusters](/sql/system-clusters) - [Usage & billing](/materialize-cloud/billing/) - [Operational guidelines](/clusters/operational-guidelines/) diff --git a/doc/user/content/fundamentals/concepts/hydration.md b/doc/user/content/fundamentals/concepts/hydration.md index 8d73ee4e489cf..116adfe1894d7 100644 --- a/doc/user/content/fundamentals/concepts/hydration.md +++ b/doc/user/content/fundamentals/concepts/hydration.md @@ -28,102 +28,16 @@ table. {{% yaml-table data="hydration-objects-table" %}} -## Hydration strategies +## Reducing hydration memory Hydration primarily impacts memory usage, and its speed scales with cluster -size. Some hydration-related strategies you may want to consider: - -- Use a dedicated cluster for [sources](/fundamentals/concepts/sources/). - -- In addition, use a dedicated cluster for upsert sources; i.e., do not - co-locate with append-only Kafka sources or CDC database sources. - - - Keeping append-only Kafka sources and CDC database sources (PostgreSQL, - MySQL, and SQL Server sources) on a separate cluster isolates ingestion from - possible OOM loops caused by memory-heavy objects such as Kafka upsert - sources. - - - Note: PostgreSQL, MySQL, and SQL Server sources run on a single - replica, the oldest, and remain there until that replica is removed. As - such, the use of a burst replica (through `AUTO SCALING STRATEGY (ON - HYDRATION)`) has no impact on these single-replica sources. - -- Add an [`AUTO SCALING STRATEGY (ON HYDRATION)`](/sql/alter-cluster/) to your - cluster with memory-heavy objects. With this strategy, Materialize - automatically provisions an extra, larger replica (a burst replica) while the - cluster has unhydrated objects, then removes it once a steady-size replica - catches up. You pay for the burst replica while it is provisioned, but not at - steady state. - - - If a steady-size replica runs out of memory during hydration, resize the - cluster. During the resize, the cluster continues to serve from the burst - replica. - -- Distribute materialized views and indexes across multiple clusters. Each - cluster's replicas hydrate their objects independently, which distributes the - memory required for hydration, lets objects on different clusters hydrate in - parallel, and limits how much must re-hydrate when any one replica restarts. - -- When changing a materialized view or index, or forcing dependents to re-plan - (for example, after dropping an index and recreating the dependents), build - the new version to the side to avoid downtime: - - - A [blue/green deployment](/developer-tools/dbt/blue-green-deployments/) hydrates the - new version alongside the old and cuts over when hydrated, with no serving - gap. Note that blue/green requires sources and sinks to live on dedicated - clusters that are excluded from the swap. For more information, see - [blue/green deployment](/developer-tools/dbt/blue-green-deployments/). - - - For a single materialized view, creating and hydrating a [replacement - materialized view (public preview) and replacing the existing view in - place](/transform-data/updating-materialized-views/replace-materialized-view/) - may be simpler, but briefly reduces freshness. The replacement materialized - view can be either on the same or different cluster. - -{{< note >}} - -The burst-replica and blue/green strategies run extra replicas alongside the -existing ones, as do a resize or a zero-downtime upgrade. During the overlap, -the cluster temporarily uses additional resources. Account for the additional -cost and, on self-managed deployments, the additional capacity required. - -{{< /note >}} - -In addition, consider the following strategies. These strategies trade off peak -hydration memory against added operational complexity, extra objects, and -potentially longer total hydration time. You can use them when peak hydration -memory is the bottleneck rather than as a default modeling pattern. - -- If multiple objects in the **same** cluster consume - the same view, add an [index](/fundamentals/concepts/indexes/) to that view **before** - creating the consumers. Consumers in that cluster can reuse the indexed - arrangement instead of each building equivalent in-memory state, which can - reduce both memory usage during hydration and steady-state memory. Note: - - Index reuse is limited to the cluster the index is on, and the index must - exist **before** its consumers are created for the optimizer to reuse it. - - For a view with only one consumer, an index generally adds memory instead of - saving it. - -- For a very large materialized view, consider splitting it into several smaller - materialized views, for example by a partition key such as customer, region, - or date range. Smaller materialized views can hydrate as separate dataflows, - which can bound peak memory compared with hydrating one very large - materialized view. - - This helps most when a cluster runs only a few large materialized views, - where a single view's hydration spike can dictate the cluster size. A - cluster with many materialized views already hydrates them as separate - dataflows and gets this benefit naturally. - - A re-plan or replacement of one split view affects only that portion of - the data. A replica restart still re-hydrates all views on the replica, - though in smaller units. - - If the split views share expensive computation, put that computation in a - [common indexed view first](#index-order), creating the index **before** - creating the split views. Otherwise, each split view may rebuild its own - copy of the shared work, increasing total memory. - - Queries must target or combine the split views. +size. For strategies to reduce hydration memory, speed hydration up, or avoid +triggering it, see [Optimize hydration +requirements](/clusters/optimize-hydration-requirements/). ## Related pages +- [Optimize hydration requirements](/clusters/optimize-hydration-requirements/) - [Snapshotting](/fundamentals/concepts/snapshotting/) - [Clusters](/fundamentals/concepts/clusters/) - [Sources](/fundamentals/concepts/sources/)