From 24c54cd890b335a62f12493447f612a30d11ffb1 Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Wed, 13 May 2026 18:34:45 +0200 Subject: [PATCH 1/7] nethsm: document cluster failure recovery --- source/components/nethsm/administration.rst | 5 +- source/components/nethsm/clustering.rst | 66 +- source/components/nethsm/images/states.svg | 820 +++----------------- 3 files changed, 190 insertions(+), 701 deletions(-) diff --git a/source/components/nethsm/administration.rst b/source/components/nethsm/administration.rst index 4b4f9af4df..da37cb0024 100644 --- a/source/components/nethsm/administration.rst +++ b/source/components/nethsm/administration.rst @@ -99,7 +99,7 @@ The boot mode can be changed as follows. At next boot, the NetHSM will behave ac State ~~~~~ -The NetHSM software has four states: *Unprovisioned*, *Provisioned*, *Locked*, and *Operational*. +The NetHSM software has five states: *Unprovisioned*, *Provisioned*, *Locked*, *Operational* and *Failed*. +-----------------+-------------------------------------------------------------------------+ | State | Description | @@ -116,6 +116,9 @@ The NetHSM software has four states: *Unprovisioned*, *Provisioned*, *Locked*, a | | Typically, the next step is to unlock the system. The *Locked* state | | | implies the *Provisioned* state. | +-----------------+-------------------------------------------------------------------------+ +| *Failed* | NetHSM with a currently unavailable database | +| | (see :ref:`recovering-a-failed-node`). | ++-----------------+-------------------------------------------------------------------------+ .. figure:: ./images/states.svg :scale: 100 diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 5c77f22bb1..3d8385bec5 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -4,7 +4,6 @@ Clustering .. note:: This feature is currently a technical preview with the following temporary limitations: - - If a cluster is lost (quorum is lost), the only means of recovery is factory-reset + restore. Make sure to back up often. Future releases will include means to recover from on-disk data. - Active/passive setup to support two-node clusters, either by utilizing etcd Learner or Mirror, is not yet available. - System time between nodes must be manually synchronized for now. A future release will include automatic clock sync. @@ -25,11 +24,11 @@ One Node Goes Down and Quorum is Still Reached In a 3-node cluster, if one node fails (crashes or becomes unreachable due to network conditions), the two other nodes will continue to work and serve requests. -If the failed node is still healthy (e.g. it was just a network problem), it will be inoperable while isolated (not even read-only). +If the failed node is still healthy (e.g. it was just a network problem), it will be in the _Failed_ state while isolated, refusing normal operations (not even read-only). -However if the node recovers, it will cleanly resynchronize with the rest of the cluster and becomes operable again, without losing data. +However if the node recovers, it will cleanly resynchronize with the rest of the cluster and become operable again, without losing data. -If it never recovers, it has to be removed from the cluster (see next section), factory reset, and go through the join process again from scratch. +If it never recovers, it has to be removed from the cluster (see :ref:`removing-a-node-cleanly`) and either undergo recovery (see :ref:`recovering-a-failed-node`) to access its data (but it will not be part of the cluster anymore), or be factory reset and go through the join process again from scratch. A Network Partition Happens and Quorum is Still Reached ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -37,13 +36,15 @@ A Network Partition Happens and Quorum is Still Reached This is just a generalization of the previous scenario. In a 5-node cluster where e.g. 3 nodes are in one physical location A and 2 nodes are in another location B, a network problem isolating A and B would mean the following: - The 3 nodes in location A are meeting the quorum (3 in this case), so they continue to operate. -- The 2 nodes in location B are **not** meeting the quorum (still 3), so they will stop operating (even read-only). +- The 2 nodes in location B are **not** meeting the quorum (still 3), so they will enter the _Failed_ state and stop operating (even read-only). - If the network issue is resolved, the 2 nodes will cleanly join back the 3 others. +.. _lost-quorum: + The Quorum is Durably Lost ~~~~~~~~~~~~~~~~~~~~~~~~~~ -A failure causing all subsets of the cluster to lose quorum will render the cluster and its data completely lost, unless the failure is resolved. In this case, nodes must be factory-reset and a backup must be restored. +A failure causing all subsets of the cluster to lose quorum will render the cluster completely inoperable (all remaining nodes will be in the _Failed_ state), unless the failure is resolved. In this case, manual recovery must be performed (see :ref:`recovering-a-failed-node`). This can happen for example if a single node fails in a 2-node cluster (where the quorum is 2). In this situation, the failed node cannot be cleanly removed from the cluster after the fact, because the remaining healthy node is already inoperable since it has lost quorum. @@ -249,7 +250,7 @@ Depending on the networking and cluster conditions, this operation may take a fe If this join is successful, the node will end up in a ``Locked`` state, and has to be unlocked with the unlock passphrase of the node that was used for registration. Afterwards the unlock passphrase can be changed (unlock passphrases remain node-specific and are not shared across nodes). .. note:: - Even after the join has succeeded, if the cluster's database is large or if the cluster is busy, it may take some time for the new joiner to synchronize its state fully. During that time, all nodes (including in particular the new joiner) may be less responsive or unresponsive. The new joiner in particular may initially return errors when trying to unlock it for example. In that case, give it some time and try again. + Even after the join has succeeded, if the cluster's database is large or if the cluster is busy, it may take some time for the new joiner to synchronize its state fully. During that time, all nodes (including in particular the new joiner) may be less responsive or unresponsive (in the _Failed_ state). The new joiner in particular may initially return errors when trying to unlock it for example. In that case, give it some time and try again. You can use the `/health/diagnose` endpoint to understand the current status of the database. Adding a Witness Node --------------------- @@ -387,6 +388,8 @@ This operation remains compatible with backups made on previous versions of the In other words, only perform a restore in a cluster with backups done in the same cluster (though again nodes may have been removed or added since). If you want to restore a foreign backup on a node, first safely remove it from its cluster, then factory reset it and restore the backup. +.. _removing-a-node-cleanly: + Removing a Node Cleanly ~~~~~~~~~~~~~~~~~~~~~~~ @@ -394,7 +397,54 @@ As long as some part of the cluster is still meeting quorum, any of its members You first have to know the ID of the node you want to remove, by listing all nodes through ``GET /cluster/members`` and looking for the right one. -Then it can be removed by calling ``DELETE /cluster/members/``. If the node in question was still healthy, this will isolate it from the rest of the cluster and render it inoperable. +Then it can be removed by calling ``DELETE /cluster/members/``. If the node in question was still healthy, this will isolate it from the rest of the cluster and transition it to the _Failed_ state. + +.. _recovering-a-failed-node: + +Recovering a Failed Node +~~~~~~~~~~~~~~~~~~~~~~~~ + +A node reporting a _Failed_ state will refuse to answer most normal operations. +It can still be shut-down, rebooted, reset, *diagnosed* or *isolated*. + +.. warning:: + The existence of the _Failed_ state, ``diagnose`` and ``force-new`` operations + only applies to version 4.1 and onward. In version 4.0, a node with a + lost quorum will stop responding to *all* requests. It *must* be factory-reset. + +Common causes for a node to be in the _Failed_ state include: +- a durably lost quorum (see :ref:`lost-quorum`) +- a temporarily lost quorum (e.g. when adding a second node to the cluster, and + the second node has not joined yet) +- ``etcd`` is currently restarting (e.g. because the certificates have changed, or + the network has been re-configured) +- the cluster is under a very high load (e.g. during the restoration of a very + large backup) + +To help you understand which case your node is in, the ``GET /health/diagnose`` +endpoint remains available and returns information about the current status of +``etcd`` and its database, including logs (refer to the API documentation). + +.. note:: + If and when ``etcd`` becomes available again (e.g. the quorum is restored + because the network issue has been solved), the NetHSM will automatically + transition out of the _Failed_ state to the state it was in before (or resume + the normal boot sequence if it was booting), without any manual action + needed. + +If you conclude that the failure is durable (e.g. lost quorum with no hope of +resolving the underlying condition), you can either: +- *factory-reset* the node, which will erase all data, and restore a backup. +- *isolate* the node with the ``POST /cluster/force-new`` endpoint, which will + irreversibly forget all other cluster members and restart ``etcd`` with the + data present on disk. If the underlying failure was cluster-related, the node + will transition out of the _Failed_ state. + +.. note:: + If a node is isolated with ``force-new``, it will now be desynchronized with + the cluster: any new writes on it or the cluster cannot be reconciled. The + node can still re-join the cluster, but will lose all its local + modifications. Software Updates in Clusters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/components/nethsm/images/states.svg b/source/components/nethsm/images/states.svg index e554ef0a90..a9975a01c3 100644 --- a/source/components/nethsm/images/states.svg +++ b/source/components/nethsm/images/states.svg @@ -1,693 +1,129 @@ - - -UnprovisionedOperationalLockedProvisionedFactory Reseta) Provision: Enter Unlock Passphrase and Administrator Passphrase -b) Restorea) Attended Boot: Enter Unlock Passphrase -b) Unattended Boota) Lock -b) Reboot -c) Shutdown -d) Commit update -e) Reset -f) Restore + + + + + + + + + + + + + Unprovisioned + + + + + + Operational + + + + a) Set Unlock Passphrase + and R-Administrator passphrase + b) restore backup + + + + + Locked + + + + a) Attended Boot: Enter Unlock Passphrase + b) Unattended Boot: - + + + + + + + Reset + + + + + + Provisioned + + + + + + + + + + + a) Lock + b) Reboot + c) Shutdown + d) commit update + e) Reset + f) Restore + + + + + + + + + Failed + + + + + + + + a) Reset + + + + + + + a) Quorum loss + + + + + + + a) Automatic recovery if + quorum reached again + and previous state was + "Operational" + + + + + + + a) Automatic recovery if quorum reached again + and previous state was "Locked" + b) Automatic recovery if quorum reached again + and there was no previous state (boot) + c) Reboot if quorum reached again + d) Hard recover (/cluster/force-new) + + + + + + + a) Reboot if quorum + (still) lost + + + + + + + a) Quorum loss + + + From 83cd4c9f2388f0ddaa2bb6fc58d55be8aaef8d2a Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Wed, 13 May 2026 20:32:16 +0200 Subject: [PATCH 2/7] nethsm: address recovery comments --- source/components/nethsm/clustering.rst | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 3d8385bec5..2ea019ba47 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -421,6 +421,11 @@ Common causes for a node to be in the _Failed_ state include: - the cluster is under a very high load (e.g. during the restoration of a very large backup) +No matter the cause, the NetHSM will only transition to the _Failed_ state after +a number of unsuccessful attempts to interact with ``etcd``, which can take a +few dozens of seconds. This is to avoid spurious transitions caused by very +short instabilities. + To help you understand which case your node is in, the ``GET /health/diagnose`` endpoint remains available and returns information about the current status of ``etcd`` and its database, including logs (refer to the API documentation). @@ -430,15 +435,17 @@ endpoint remains available and returns information about the current status of because the network issue has been solved), the NetHSM will automatically transition out of the _Failed_ state to the state it was in before (or resume the normal boot sequence if it was booting), without any manual action - needed. + needed. It may take a dozen of seconds after the issue has been resolved to + detect the resolution and change state. If you conclude that the failure is durable (e.g. lost quorum with no hope of resolving the underlying condition), you can either: - *factory-reset* the node, which will erase all data, and restore a backup. - *isolate* the node with the ``POST /cluster/force-new`` endpoint, which will - irreversibly forget all other cluster members and restart ``etcd`` with the - data present on disk. If the underlying failure was cluster-related, the node - will transition out of the _Failed_ state. + irreversibly forget all other cluster members, recover the ``etcd`` data + present on disk and reboot. If the underlying failure was cluster-related, the + node will follow the normal boot sequence and end up in either _Locked_ or + _Operational_ depending on the unattended boot setting. .. note:: If a node is isolated with ``force-new``, it will now be desynchronized with From d0fb4262af9401a6002166d950d978e0be418d62 Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Mon, 18 May 2026 17:53:33 +0200 Subject: [PATCH 3/7] rewordings, shorter URLs, durations --- source/components/nethsm/clustering.rst | 34 ++++++++++++------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 2ea019ba47..5d4989ef20 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -26,9 +26,9 @@ In a 3-node cluster, if one node fails (crashes or becomes unreachable due to ne If the failed node is still healthy (e.g. it was just a network problem), it will be in the _Failed_ state while isolated, refusing normal operations (not even read-only). -However if the node recovers, it will cleanly resynchronize with the rest of the cluster and become operable again, without losing data. +However if the node recovers, it will cleanly resynchronize with the rest of the cluster and exit the _Failed_ state, resuming normal operation without losing data. -If it never recovers, it has to be removed from the cluster (see :ref:`removing-a-node-cleanly`) and either undergo recovery (see :ref:`recovering-a-failed-node`) to access its data (but it will not be part of the cluster anymore), or be factory reset and go through the join process again from scratch. +If it never recovers, it has to be removed from the cluster (see :ref:`removing`) and either undergo recovery (see :ref:`recovering`) to access its data (but it will not be part of the cluster anymore), or be factory reset and go through the join process again from scratch. A Network Partition Happens and Quorum is Still Reached ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -44,7 +44,7 @@ This is just a generalization of the previous scenario. In a 5-node cluster wher The Quorum is Durably Lost ~~~~~~~~~~~~~~~~~~~~~~~~~~ -A failure causing all subsets of the cluster to lose quorum will render the cluster completely inoperable (all remaining nodes will be in the _Failed_ state), unless the failure is resolved. In this case, manual recovery must be performed (see :ref:`recovering-a-failed-node`). +A failure causing all subsets of the cluster to lose quorum will render the cluster completely inoperable (all remaining nodes will be in the _Failed_ state), unless the failure is resolved. In this case, manual recovery must be performed (see :ref:`recovering`). This can happen for example if a single node fails in a 2-node cluster (where the quorum is 2). In this situation, the failed node cannot be cleanly removed from the cluster after the fact, because the remaining healthy node is already inoperable since it has lost quorum. @@ -388,7 +388,7 @@ This operation remains compatible with backups made on previous versions of the In other words, only perform a restore in a cluster with backups done in the same cluster (though again nodes may have been removed or added since). If you want to restore a foreign backup on a node, first safely remove it from its cluster, then factory reset it and restore the backup. -.. _removing-a-node-cleanly: +.. _removing: Removing a Node Cleanly ~~~~~~~~~~~~~~~~~~~~~~~ @@ -399,7 +399,7 @@ You first have to know the ID of the node you want to remove, by listing all nod Then it can be removed by calling ``DELETE /cluster/members/``. If the node in question was still healthy, this will isolate it from the rest of the cluster and transition it to the _Failed_ state. -.. _recovering-a-failed-node: +.. _recovering: Recovering a Failed Node ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -413,30 +413,30 @@ It can still be shut-down, rebooted, reset, *diagnosed* or *isolated*. lost quorum will stop responding to *all* requests. It *must* be factory-reset. Common causes for a node to be in the _Failed_ state include: -- a durably lost quorum (see :ref:`lost-quorum`) -- a temporarily lost quorum (e.g. when adding a second node to the cluster, and - the second node has not joined yet) +- A durably lost quorum (see :ref:`lost-quorum`). +- A temporarily lost quorum (e.g. when adding a second node to the cluster, and + the second node has not joined yet). - ``etcd`` is currently restarting (e.g. because the certificates have changed, or - the network has been re-configured) -- the cluster is under a very high load (e.g. during the restoration of a very - large backup) + the network has been re-configured). +- The cluster is under a very high load (e.g. during the restoration of a very + large backup). No matter the cause, the NetHSM will only transition to the _Failed_ state after -a number of unsuccessful attempts to interact with ``etcd``, which can take a -few dozens of seconds. This is to avoid spurious transitions caused by very -short instabilities. +no less than a full minute of unsuccessful attempts to interact with its +database. This is to avoid spurious transitions caused by very short +instabilities. To help you understand which case your node is in, the ``GET /health/diagnose`` endpoint remains available and returns information about the current status of ``etcd`` and its database, including logs (refer to the API documentation). .. note:: - If and when ``etcd`` becomes available again (e.g. the quorum is restored + If and when the database becomes available again (e.g. the quorum is restored because the network issue has been solved), the NetHSM will automatically transition out of the _Failed_ state to the state it was in before (or resume the normal boot sequence if it was booting), without any manual action - needed. It may take a dozen of seconds after the issue has been resolved to - detect the resolution and change state. + needed. It will take up to a minute after the issue has been resolved for the + cluster to stabilize, the HSM to detect the resolution and to change state. If you conclude that the failure is durable (e.g. lost quorum with no hope of resolving the underlying condition), you can either: From 24a5c5a527e59e08d2fc74695f4b5cce3d346181 Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Mon, 18 May 2026 19:02:55 +0200 Subject: [PATCH 4/7] use inline links rather than 'see X' --- source/components/nethsm/clustering.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 5d4989ef20..13892d42f2 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -28,7 +28,7 @@ If the failed node is still healthy (e.g. it was just a network problem), it wil However if the node recovers, it will cleanly resynchronize with the rest of the cluster and exit the _Failed_ state, resuming normal operation without losing data. -If it never recovers, it has to be removed from the cluster (see :ref:`removing`) and either undergo recovery (see :ref:`recovering`) to access its data (but it will not be part of the cluster anymore), or be factory reset and go through the join process again from scratch. +If it never recovers, it has to be :ref:`removed ` from the cluster and either undergo :ref:`recovery ` to access its data (but it will not be part of the cluster anymore), or be factory reset and go through the join process again from scratch. A Network Partition Happens and Quorum is Still Reached ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -44,7 +44,7 @@ This is just a generalization of the previous scenario. In a 5-node cluster wher The Quorum is Durably Lost ~~~~~~~~~~~~~~~~~~~~~~~~~~ -A failure causing all subsets of the cluster to lose quorum will render the cluster completely inoperable (all remaining nodes will be in the _Failed_ state), unless the failure is resolved. In this case, manual recovery must be performed (see :ref:`recovering`). +A failure causing all subsets of the cluster to lose quorum will render the cluster completely inoperable (all remaining nodes will be in the _Failed_ state), unless the failure is resolved. In this case, manual :ref:`recovery ` must be performed. This can happen for example if a single node fails in a 2-node cluster (where the quorum is 2). In this situation, the failed node cannot be cleanly removed from the cluster after the fact, because the remaining healthy node is already inoperable since it has lost quorum. @@ -413,7 +413,7 @@ It can still be shut-down, rebooted, reset, *diagnosed* or *isolated*. lost quorum will stop responding to *all* requests. It *must* be factory-reset. Common causes for a node to be in the _Failed_ state include: -- A durably lost quorum (see :ref:`lost-quorum`). +- A durably :ref:`lost quorum `. - A temporarily lost quorum (e.g. when adding a second node to the cluster, and the second node has not joined yet). - ``etcd`` is currently restarting (e.g. because the certificates have changed, or From 8d1b0968a6f6c90c102b7d98401a953e4cfe11c2 Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Mon, 18 May 2026 20:49:23 +0200 Subject: [PATCH 5/7] remove labels on single-case transitions --- source/components/nethsm/images/states.svg | 60 +++++++++++----------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/source/components/nethsm/images/states.svg b/source/components/nethsm/images/states.svg index a9975a01c3..0add79d4a2 100644 --- a/source/components/nethsm/images/states.svg +++ b/source/components/nethsm/images/states.svg @@ -1,6 +1,6 @@ - + @@ -20,10 +20,10 @@ Operational - - a) Set Unlock Passphrase - and R-Administrator passphrase - b) restore backup + + a) Set Unlock Passphrase + and R-Administrator passphrase + b) restore backup @@ -78,52 +78,52 @@ - - a) Reset + + Reset - - + + - - a) Quorum loss + + Quorum loss - - a) Automatic recovery if - quorum reached again - and previous state was - "Operational" + + a) Automatic recovery if + quorum reached again + and previous state was + "Operational" - - a) Automatic recovery if quorum reached again - and previous state was "Locked" - b) Automatic recovery if quorum reached again - and there was no previous state (boot) - c) Reboot if quorum reached again - d) Hard recover (/cluster/force-new) + + a) Automatic recovery if quorum reached again + and previous state was "Locked" + b) Automatic recovery if quorum reached again + and there was no previous state (boot) + c) Reboot if quorum reached again + d) Hard recover (/cluster/force-new) - - a) Reboot if quorum - (still) lost + + Reboot if quorum + (still) lost - - + + - - a) Quorum loss + + Quorum loss From 455a68d273885c2c61a529896ac072c4488b0d86 Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Mon, 18 May 2026 21:00:26 +0200 Subject: [PATCH 6/7] explain consequence of losing quorum --- source/components/nethsm/clustering.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 13892d42f2..7898ed0009 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -17,7 +17,9 @@ Operational Redundancy We will call "node" a NetHSM that is expected to be part of a cluster. **A cluster of** ``N`` **nodes will continue to operate as long as at least** ``(N/2)+1`` **nodes are healthy and reachable.** That minimal amount of healthy, reachable nodes is called the **quorum**. -This implies the following scenarios. +On a cluster that goes below this threshold (e.g. because of a network issue), +no leader can be elected and the local instance of ``etcd`` on each node becomes +unable to perform reads and writes. This implies the following scenarios. One Node Goes Down and Quorum is Still Reached ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From c778343551fb078b8ed6cba321e52f57eb5043ce Mon Sep 17 00:00:00 2001 From: Virgile Robles Date: Tue, 19 May 2026 20:43:32 +0200 Subject: [PATCH 7/7] latest round of review --- source/components/nethsm/administration.rst | 2 +- source/components/nethsm/clustering.rst | 14 +++++++++----- source/components/nethsm/images/states.svg | 10 +++++----- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/source/components/nethsm/administration.rst b/source/components/nethsm/administration.rst index da37cb0024..4eb907eb80 100644 --- a/source/components/nethsm/administration.rst +++ b/source/components/nethsm/administration.rst @@ -117,7 +117,7 @@ The NetHSM software has five states: *Unprovisioned*, *Provisioned*, *Locked*, * | | implies the *Provisioned* state. | +-----------------+-------------------------------------------------------------------------+ | *Failed* | NetHSM with a currently unavailable database | -| | (see :ref:`recovering-a-failed-node`). | +| | (see :ref:`recovering`). | +-----------------+-------------------------------------------------------------------------+ .. figure:: ./images/states.svg diff --git a/source/components/nethsm/clustering.rst b/source/components/nethsm/clustering.rst index 7898ed0009..c99d594713 100644 --- a/source/components/nethsm/clustering.rst +++ b/source/components/nethsm/clustering.rst @@ -41,6 +41,10 @@ This is just a generalization of the previous scenario. In a 5-node cluster wher - The 2 nodes in location B are **not** meeting the quorum (still 3), so they will enter the _Failed_ state and stop operating (even read-only). - If the network issue is resolved, the 2 nodes will cleanly join back the 3 others. +In other words, a worst-case network partition (in a cluster with an odd number of +nodes) will leave the bigger half of the cluster healthy, and the smaller half +inoperable until the partition is resolved. + .. _lost-quorum: The Quorum is Durably Lost @@ -411,7 +415,7 @@ It can still be shut-down, rebooted, reset, *diagnosed* or *isolated*. .. warning:: The existence of the _Failed_ state, ``diagnose`` and ``force-new`` operations - only applies to version 4.1 and onward. In version 4.0, a node with a + only applies to version 5.0 and onward. In version 4.0, a node with a lost quorum will stop responding to *all* requests. It *must* be factory-reset. Common causes for a node to be in the _Failed_ state include: @@ -442,12 +446,12 @@ endpoint remains available and returns information about the current status of If you conclude that the failure is durable (e.g. lost quorum with no hope of resolving the underlying condition), you can either: -- *factory-reset* the node, which will erase all data, and restore a backup. -- *isolate* the node with the ``POST /cluster/force-new`` endpoint, which will +- *Factory-reset* the node, which will erase all data, and restore a backup. +- *Isolate* the node with the ``POST /cluster/force-new`` endpoint, which will irreversibly forget all other cluster members, recover the ``etcd`` data present on disk and reboot. If the underlying failure was cluster-related, the - node will follow the normal boot sequence and end up in either _Locked_ or - _Operational_ depending on the unattended boot setting. + node will follow the normal boot sequence and end up in either the _Locked_ or + _Operational_ state depending on the unattended boot setting. .. note:: If a node is isolated with ``force-new``, it will now be desynchronized with diff --git a/source/components/nethsm/images/states.svg b/source/components/nethsm/images/states.svg index 0add79d4a2..c202776eaa 100644 --- a/source/components/nethsm/images/states.svg +++ b/source/components/nethsm/images/states.svg @@ -92,11 +92,11 @@ - - a) Automatic recovery if - quorum reached again - and previous state was - "Operational" + + Automatic recovery if + quorum reached again + and previous state was + "Operational"