Offboarding Nodes - runtime cluster/Cloud Elasticity


Node Offboarding

Offboarding of runtime cluster/Cloud nodes is intended to ensure that nodes being removed from a cluster are cleanly shut down, terminating running processes and execution workers, and moving any node specific data that needs to be retained.  The process of offboarding runs from one of the other active nodes in the cluster, meaning that it will never run if there is only one node remaining.  A single node cannot offboard itself.  The general flow of offboarding is a set of steps that are either run synchronously (meaning the step blocks waiting for the node to finish it) or asynchronously (meaning it initiates the step on the node and moves on, checking periodically to see if the node has completed it).  Each step in the offboarding process must complete with a status of success in order for the offboarding to complete and the node to shut down cleanly.

The offboarding process is triggered by the creation of a <Node ID>.offboard file in the nodes-deleted directory of the Boomi installation folder, when the node is online.  If the node is offline then a <Node ID>.cleanup file will be created in the same nodes-deleted directory.  The creation of these files can be triggered either through:

  1. A POST request to the Node Offboarding object API endpoint.
  2. Version 4 runtimes create a <Node ID>.offboard file in response to a SIGTERM request sent to a Pod.
  3. Version 3 runtimes create a <Node ID>.offboard file when the scaledown.sh script is run by the Kubernetes preStop hook.

The current steps (and concurrency) involved in offboarding a target node in a cluster are:

USES_basic runtime_QUEUES (synchronous)
Determine if any of the accounts on the target node uses basic runtime queues which will require moving messages to the current node during the offboarding.

ALREADY_OFFBOARDING_CHECK (synchronous)
Checks if the target node is already offboarding another node, if it is then we won't continue and the offboarding will stop at this step.

SET_NODE_OFFBOARD_STATUS (synchronous)
Sets the status of the target node to offboarding, which will be used by the node when abandoning running executions and by its execution workers.

FIRE_WORKERS (asynchronous)
Tell the target node to spin down any execution workers.  This step periodically checks the status by asking the target for its active workers, and waits until none are active.

FIRE_NODE (asynchronous)
Request the target node to pause for stop.  Periodically checks the status by asking the target nodes for its status, and waits until it reports a PAUSED_FOR_STOP state.
Inform the target node to decommission shutdown to shut it all the way off.  Checks the nodes status by examining the cluster view and waits until the target is gone.
The decommission shutdown procedure follows the same process as documented in our help guide (https://help.boomi.com/bundle/integration/page/t-atm-Shutting_down_an_basic runtime_or_runtime cluster_node_gracefully.html)  With the exception that the Force Restart After X Minutes property (com.boomi.runtime.forceRestart) is not disabled, but is instead replaced by the Force Restart Override for runtime cluster Scale Down property (com.boomi.runtime.elasticity.forceRestartOverride), which defaults to 10 minutes if not set.

QUEUE_MESSAGE_OFFBOARD (synchronous)
Move basic runtime queue messages, if any basic runtime queues are used, from the target node to the current node performing the offboarding.

MESSAGE_OFFBOARD (synchronous)
Takes all of the target nodes platform messages (related to platform communication) and moves them to the current node, so they still get processed.

REMOVE_NODE_FROM_PLATFORM (synchronous)
Send a message to the Boomi platform telling it the target node has now been offboarded and remove it from the cluster view panel.

There are two advanced properties that should be set to control the timing of the offboarding process.

Async Message Poller Timeout for runtime cluster Scale Down

This Advanced Property controls the amount of time that the node performing the offboarding process will wait for each of the asynchronous steps above to complete.  If any of the asynchronous steps take longer than this time, then an "Async operation timed out" error will occur and the offboarding will either attempt to rollback (depending on the status) or fail and stop offboarding.  This value must be greater than the Force Restart Override for runtime cluster Scale Down (see below).
 

Force Restart Override for runtime cluster Scale Down

This Advanced Property determines the amount of time that the offboarding process will wait for any running processes to complete.  If a process does not complete within this time then the node will attempt to terminate the process.  Processes terminated in this way will report an error in Process Reporting, but the error will indicate the reason the process was terminated.  This is usually because the basic runtime is stopping.  This property is only used to terminate running processes as part of the offboarding process.  You should still set the 'Force Restart After X Minutes' property to terminate processes as part of a normal basic runtime restart (such as when applying the monthly updates or initiating a restart from the Boomi Platform).
 

Kubernetes terminationGracePeriodSeconds

You need to consider the values of both the above properties in co-ordination with the terminationGracePeriodSeconds when deploying on Kubernetes. The 'Async Message Poller Timeout for runtime cluster Scale Down' and 'Force Restart Override for runtime cluster Scale Down' both default to 600,000 milliseconds (10 minutes) when not set. The Kubernetes terminationGracePeriodSeconds for a Pod is 60 seconds (1 minutes), meaning if all the values are left at the defaults then node offboarding will always fail. You should determine if these values are appropriate for your implementation, based on the time you wish to wait for running processes to successfully complete during node offboarding, then set all the parameters accordingly.
 

What if Offboarding does not work?

If the offboarding fails for some reason then you need to review the runtime logs to find the node that was controlling the offboarding.  This will always be a different node to the one that failed because the offboarding always runs from one of the other active nodes in the cluster. Look for runtime log messages associated with the 'ClusterOffboardingService' and 'BaseClusterOffboardListener'. In the example below runtime cluster_node_0 is attempting to offboard moelcule_node_1 but the FIRE_NODE asynchronous step fails after a few checks. In this case it fails because the 'Force Restart Override for runtime cluster Scale Down' was set to greater than the Kubernetes terminationGracePeriodSeconds, and Kubernetes has killed the Pod before the node offboarding could complete due to waiting on a running process. This is probably the most common reason for offboarding failure when scaling nodes in a cluster.

May 23, 2023 10:31:16 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.ClusterOffboardingService runOffboardOperation] Attempting operation EXECUTE on node [runtime cluster_node_1] for step FIRE_NODE
May 23, 2023 10:31:16 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logOperationAndStep] Executing EXECUTE operation for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
May 23, 2023 10:31:16 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logStepMessage] EXECUTE operation for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0,  step action: Sent pauseForStop message to offboarded node.
May 23, 2023 10:31:16 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logResult] Result of EXECUTE operation for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0 was: false
May 23, 2023 10:31:16 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logAsyncOperationCheckAndStep] Checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
May 23, 2023 10:31:21 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logAsyncOperationCheckAndStep] Checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
May 23, 2023 10:31:26 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logAsyncOperationCheckAndStep] Checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
May 23, 2023 10:31:31 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logAsyncOperationCheckAndStep] Checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
May 23, 2023 10:31:36 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logAsyncOperationCheckAndStep] Checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0
---
May 23, 2023 10:31:58 AM GMT INFO    [com.boomi.runtime.cloudlet.cluster.offboard.BaseClusterOffboardListener logErrorAsyncOperationCheckAndStep] Error checking async EXECUTE operation result for step FIRE_NODE on node runtime cluster_node_1 using host: runtime cluster_node_0


You also often see 'Connection error' reported for offline nodes in the Cluster Status page, which can also be a sign that offboarding did not successfully complete and the node was not removed from the Platform (which is the final step in offboarding).

offboarding_error.png