#Manage Nodes on Bare Metal

This document explains how to scale worker nodes up and down, replace specific machines, and recover failed nodes on the bare-metal provider. Node management uses Cluster API Machine resources orchestrated through MachineDeployment; the provider binds each Machine to one MachineInventory from a pool and drives the host through clean / reprovision plans.

#Prerequisites

#Overview

The four resources that compose a worker node group:

1. MachineInventoryPool (<cluster-name>-worker-pool) — the allowed set of MachineInventory names. Already created in Create Cluster → Step 2.
2. BaremetalMachineTemplate (<cluster-name>-worker-template) — references the worker pool. To move the workers to a different pool, see Moving Workers to a Different Pool.
3. KubeadmConfigTemplate (<cluster-name>-worker-bootstrap) — cloud-init user-data for kubeadm join. The bare-metal provider normalizes this user-data at reprovision time (hostname, provider-id, criSocket); operators should not pre-fill those fields.
4. MachineDeployment — controls replica count, version, rollout strategy.

Managed data disks are not part of BaremetalMachineTemplate. Their declaration and retained data belong to each long-lived MachineInventory, so two members of the same worker pool may expose different host-local datasets unless operators standardize and verify their storage declarations. Managed storage is changed separately on an unallocated, inactive MachineInventory; see Manage Data Disks on Bare-Metal Hosts.

To add another worker node pool with YAML, follow Step 4: Deploy Worker Nodes on the cluster creation page.

#Node Management Operations

#Scaling Worker Nodes

#Adding Worker Nodes

Use Case: Increase cluster capacity.

Prerequisites:

• The worker pool has enough Available inventories. If not, register the additional hosts (boot the SeedImage ISO and confirm the new MachineInventory objects), prepare any managed storage, and append them to MachineInventoryPool.spec.machineInventories[].

Procedure:

1. Check current state

   kubectl -n cpaas-system get machines.cluster.x-k8s.io \
     -l cluster.x-k8s.io/deployment-name=<cluster-name>-workers
   kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io \
     <cluster-name>-worker-pool

   Confirm that MachineInventoryPool.status.available is high enough for the planned increment.

2. Extend the worker pool when more inventories are needed

   kubectl -n cpaas-system edit machineinventorypool <cluster-name>-worker-pool

   Append the new MachineInventory names — they must already be registered and Ready. Inventories with managed volumes must also be Prepared; otherwise they increase status.total but remain unavailable for allocation. Save and exit, then confirm the expected increase in status.available.

3. Scale up the MachineDeployment

   kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
     --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":<new-replica-count>}]'

4. Monitor

   kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w
   kubectl get nodes                # workload cluster

   New BaremetalMachine objects advance Pending → Allocated → Reprovisioning → Running; the pool's available counter decreases as new nodes are bound.

#Removing Worker Nodes

Two strategies are supported, identical in shape to the upstream Cluster API contract:

#Random Removal

kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
  --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":<new-replica-count>}]'

CAPI selects machines for deletion in its standard order. Each selected BaremetalMachine moves to Preparing, the provider writes a clean plan, and once the plan reports Applied, the inventory returns to Available.

#Targeted Removal

1. Identify the machines

   kubectl -n cpaas-system get machines.cluster.x-k8s.io \
     -l cluster.x-k8s.io/deployment-name=<cluster-name>-workers

2. Annotate the target machines

   kubectl -n cpaas-system patch machine <machine-name> \
     --type='merge' -p='{"metadata":{"annotations":{"cluster.x-k8s.io/delete-machine":"true"}}}'

   Repeat for every machine you want to remove.

3. Scale down by exactly the number of annotated machines

   kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
     --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":<new-replica-count>}]'

   Reducing by fewer leaves annotated machines in place; reducing by more sends random machines through the clean plan as well.

4. Verify cleanup

   kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
   kubectl -n cpaas-system get machineinventories.elemental.cattle.io <inventory-name> -o yaml

   The released inventory must show baremetal.alauda.io/allocation-state=Available, the baremetal.alauda.io/owner-* annotations must be cleared, and the pool annotation must remain. MachineInventory itself is not deleted. If it declares managed volumes, it must return to status.storage.phase=Prepared with StorageActive=False/Inactive; its business mount paths must be inactive while filesystem UUIDs and data remain unchanged.

#Replacing a Single Failed Node

If a BaremetalMachine lands in Failed, the safest recovery is to delete the failed Machine. CAPI immediately creates a replacement Machine (because replicas did not change), and the bare-metal provider picks an Available inventory from the same pool. Most often the replacement is a different MachineInventory; the provider does not guarantee that the freshly released inventory will be re-picked. Host-local managed data does not follow the replacement Machine. If the application must reuse a specific inventory's data, use a dedicated or deliberately ordered pool and verify BaremetalMachine.status.machineInventoryRef before relying on that data.

kubectl -n cpaas-system delete machine <machine-name>
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w

Investigate the original failure separately: read BaremetalMachine.status.conditions, MachineInventory.status.plan.state, and the failing plan secret's failed-output key on the host that ran the plan. Common root causes are documented in the Common Failure Modes section below.

#Moving Workers to a Different Pool

BaremetalMachineTemplate carries only the pool reference — there is no CPU, memory, or disk declaration on the template.

Use Case: New hosts have been registered and placed in a new MachineInventoryPool, and the nodes should move onto them. Create a new BaremetalMachineTemplate that points to the new pool, then point the MachineDeployment at it. Nodes are replaced one by one.

Before you start:

• The target pool is in the same namespace as the template.

• The target pool has at least as many Available inventories as the MachineDeployment has replicas. Each moved node uses one host of the target pool; hosts released by the old nodes return to the old pool and do not join the target pool. Read the count from status.available:

  kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io <target-pool-name>

• The new hosts meet the same prerequisites as the current ones: registered and ready, managed data disks prepared if you use them, the target Kubernetes version and CPU architecture in the image catalog, and the CNI network device present. See Confirm the Worker Pool Capacity.

• Keep the old template and the old pool until the move is finished.

Procedure:

1. Create a new BaremetalMachineTemplate with a new metadata.name referencing the target pool. Use the format in Configure the Worker BaremetalMachineTemplate.

2. Apply the new template.

3. Patch the MachineDeployment:

   kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
     --type='merge' \
     -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":"<new-template-name>"}}}}}'

4. Watch the rolling replacement (maxSurge: 0, one node at a time) and the target pool's status.available:

   kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w
   kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io <target-pool-name>

What to expect:

• The old node is deleted before its replacement is created. If the target pool runs out of hosts, the node pool stays one node short until you add hosts.
• Node names and IP addresses change.
• Node-local data is not carried over. Data on managed data disks stays on the old host, bound to its MachineInventory, and does not appear on the new host. Declare storage on the new hosts separately; see Manage Data Disks on Bare-Metal Hosts.

If the move stalls with no free host (BaremetalMachine shows InventoryAllocated=False / Reason=PoolExhausted), add Available hosts to the target pool; they are picked up automatically. Alternatively, roll back by patching infrastructureRef.name back to the old template. Nodes that were already moved are replaced again one by one, so the old pool must still have enough Available hosts.

To move control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead. Nodes are replaced one at a time. In addition:

• maxSurge: 0 requires replicas of 3 or more.
• During each replacement only two etcd members run. Start only when the control plane is healthy, and keep 3 Available hosts ready in the target pool.
• The new hosts must be in the same layer-2 network as the control plane VIP.

After the move is complete, you can delete the old template and the old pool. Hosts released by the old nodes stay in the old pool.

#Updating Bootstrap Templates

Modifying an existing KubeadmConfigTemplate in place does not roll out existing machines; only newly created machines pick up the change.

To roll out a bootstrap change:

1. Export the existing template:

   kubectl -n cpaas-system get kubeadmconfigtemplate <cluster-name>-worker-bootstrap -o yaml \
     > new-worker-bootstrap.yaml

2. Change metadata.name, remove server-generated fields (resourceVersion, uid, creationTimestamp, managedFields, kubectl.kubernetes.io/last-applied-configuration) and the entire status, and edit the desired fields. For Kubernetes 1.35 or later, add the required kubelet patch settings to /etc/kubernetes/patches/kubeletconfiguration0+strategic.json.

3. Apply the new template:

   kubectl apply -f new-worker-bootstrap.yaml

4. Patch the MachineDeployment to reference the new template:

   kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
     --type='merge' \
     -p='{"spec":{"template":{"spec":{"bootstrap":{"configRef":{"name":"<new-template-name>"}}}}}}'

   This triggers a rolling replacement.

#Pool and Inventory Observability

The annotations the provider maintains on each MachineInventory are the authoritative source for "what is this host being used for right now":

Each inventory's default plan secret also carries baremetal.alauda.io/plan.type. Normal node lifecycle uses clean or reprovision; inventory storage reconciliation additionally uses storage-prepare and storage-release. Inspect MachineInventory.status.storage.conditions and .operation together with the plan type when diagnosing storage.

#Common Failure Modes

For the full operator-side state machine reference, see Provider Overview → BaremetalMachine.

#Next Steps

• Create Clusters
• Manage Data Disks
• Upgrade Clusters