fix: default exitOnStopLeading to true in LeaderElectionConfigurationBuilder (#3619)

LeaderElectionConfigurationBuilder.build() defaulted exitOnStopLeading to
false, contradicting the deprecated LeaderElectionConfiguration constructors
and the LeaderElectionManager javadoc, which both treat true as the default.
Operators configured through the builder, including those configured via the
josdk.leader-election.* properties handled by ConfigLoader, therefore kept
running after losing the lead, risking two instances reconciling in parallel.

Introduce EXIT_ON_STOP_LEADING_DEFAULT_VALUE, use it from build() and the
deprecated constructors, and route build() and buildForTest(boolean) through
a common private build(boolean) instead of build() delegating to
buildForTest(false).

Author: csviri

docs/content/en/docs/documentation/operations/leader-election.md

@@ -44,7 +44,16 @@ See details under [configurations](configuration.md) page.
    the lease.
 2. Once leadership is acquired, event processing begins normally.
 3. If leadership is lost (e.g. the leader pod becomes unresponsive), another instance acquires the lease
-   and takes over reconciliation. The instance that lost the lead is terminated (`System.exit()`)
+   and takes over reconciliation. The instance that lost the lead is terminated (`System.exit(1)`), so
+   that it is restarted by Kubernetes and no two instances reconcile the same resources in parallel.
+   This does not happen on a graceful shutdown (`Operator.stop()`), only when the lead is lost while
+   the operator is running.
+
+{{% alert title="Note" color="primary" %}}
+Exiting on lost leadership is always on in production. `LeaderElectionConfigurationBuilder` exposes
+`buildForTest(boolean exitOnStopLeading)` to turn it off, but as the name says this is only meant for
+tests, where terminating the JVM would kill the test run.
+{{% /alert %}}
 
 ### Identity and Namespace Inference
 

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/LeaderElectionConfiguration.java

@@ -25,6 +25,7 @@ public class LeaderElectionConfiguration {
   public static final Duration LEASE_DURATION_DEFAULT_VALUE = Duration.ofSeconds(15);
   public static final Duration RENEW_DEADLINE_DEFAULT_VALUE = Duration.ofSeconds(10);
   public static final Duration RETRY_PERIOD_DEFAULT_VALUE = Duration.ofSeconds(2);
+  public static final boolean EXIT_ON_STOP_LEADING_DEFAULT_VALUE = true;
 
   private final String leaseName;
   private final String leaseNamespace;
@@ -50,7 +51,7 @@ public LeaderElectionConfiguration(String leaseName, String leaseNamespace, Stri
         RETRY_PERIOD_DEFAULT_VALUE,
         identity,
         null,
-        true);
+        EXIT_ON_STOP_LEADING_DEFAULT_VALUE);
   }
 
   /**
@@ -79,7 +80,15 @@ public LeaderElectionConfiguration(
       Duration leaseDuration,
       Duration renewDeadline,
       Duration retryPeriod) {
-    this(leaseName, leaseNamespace, leaseDuration, renewDeadline, retryPeriod, null, null, true);
+    this(
+        leaseName,
+        leaseNamespace,
+        leaseDuration,
+        renewDeadline,
+        retryPeriod,
+        null,
+        null,
+        EXIT_ON_STOP_LEADING_DEFAULT_VALUE);
   }
 
   /**
@@ -133,6 +142,12 @@ public Optional<LeaderCallbacks> getLeaderCallbacks() {
     return Optional.ofNullable(leaderCallbacks);
   }
 
+  /**
+   * Whether the process should exit (via {@code System.exit(1)}) when this instance stops leading
+   * outside of a graceful shutdown. Defaults to {@value #EXIT_ON_STOP_LEADING_DEFAULT_VALUE};
+   * {@code false} is only meant for testing purposes, see {@link
+   * LeaderElectionConfigurationBuilder#buildForTest(boolean)}.
+   */
   public boolean isExitOnStopLeading() {
     return exitOnStopLeading;
   }

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/LeaderElectionConfigurationBuilder.java

@@ -81,11 +81,26 @@ public LeaderElectionConfigurationBuilder withExitOnStopLeading(boolean exitOnSt
             + " instead");
   }
 
+  /**
+   * Builds the configuration with {@code exitOnStopLeading} set to {@value
+   * LeaderElectionConfiguration#EXIT_ON_STOP_LEADING_DEFAULT_VALUE}, meaning that the process exits
+   * when this instance stops leading outside of a graceful shutdown, so that another replica can
+   * take over without two instances reconciling in parallel.
+   */
   public LeaderElectionConfiguration build() {
-    return buildForTest(false);
+    return build(EXIT_ON_STOP_LEADING_DEFAULT_VALUE);
   }
 
+  /**
+   * Same as {@link #build()}, but allows turning off the exit on stop leading behavior. This should
+   * only be used for testing purposes, since without exiting, two instances might reconcile the
+   * same resources in parallel.
+   */
   public LeaderElectionConfiguration buildForTest(boolean exitOnStopLeading) {
+    return build(exitOnStopLeading);
+  }
+
+  private LeaderElectionConfiguration build(boolean exitOnStopLeading) {
     return new LeaderElectionConfiguration(
         leaseName,
         leaseNamespace,