#Managing Nodes on VMware vSphere

This document explains how to manage worker nodes on VMware vSphere after the baseline cluster is running. Node lifecycle operations are managed through VSphereMachineConfigPool, VSphereMachineTemplate, KubeadmConfigTemplate, and MachineDeployment resources.

#Prerequisites

Before you begin, ensure the following conditions are met:

• The workload cluster was created successfully. See Creating Clusters on VMware vSphere.
• The worker machine config pool has enough available slots.
• The control plane is healthy and reachable.
• You know which manifest files currently define the worker nodes.

#Steps

- hostname: "<worker_node_name_2>"
  datacenter: "<worker_02_datacenter>"
  network:
    primary:
      networkName: "<nic1_network_name>"
      ip: "<worker_02_nic1_ip>/<nic1_prefix>"
      gateway: "<nic1_gateway>"
      dns:
      - "<nic1_dns_1>"
  persistentDisks:
  - name: var-cpaas
    sizeGiB: <worker_var_cpaas_size_gib>
    mountPath: /var/cpaas
    fsFormat: ext4
  - name: var-lib-containerd
    sizeGiB: <worker_var_lib_containerd_size_gib>
    mountPath: /var/lib/containerd
    fsFormat: ext4

  persistentDisks:
  # ...required disks...
  - name: app-data
    sizeGiB: 50

replicas: <worker_replicas>

kubectl apply -f 17-vspheremachineconfigpool-worker.yaml
kubectl apply -f 30-workers-md-0.yaml

kubectl -n <namespace> get machinedeployment,machine,vspheremachine,vspherevm
kubectl --kubeconfig=/tmp/<cluster_name>.kubeconfig get nodes -o wide

#Updating Bootstrap Templates

KubeadmConfigTemplate resources are also immutable. Changes to an existing template do not trigger rollouts of existing machines. To update bootstrap configuration, create a new template and update the MachineDeployment reference.

1. Export the existing template

   kubectl get kubeadmconfigtemplate <cluster_name>-worker-bootstrap -n <namespace> -o yaml > new-bootstrap-template.yaml

2. Modify the template

   Edit new-bootstrap-template.yaml:

   • Set metadata.name to a new unique name (for example, <cluster_name>-worker-bootstrap-v2)
   • Update the desired bootstrap configuration fields
   • For Kubernetes 1.35 or later, add the required kubelet patch settings to /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
   • Remove the same server-generated fields listed in Roll out updated worker node configuration step 2

3. Apply the new template

   kubectl apply -f new-bootstrap-template.yaml

4. Update the MachineDeployment reference

   kubectl patch machinedeployment <cluster_name>-md-0 -n <namespace> \
     --type='merge' -p='{
       "spec": {
         "template": {
           "spec": {
             "bootstrap": {
               "configRef": {
                 "name": "<new-bootstrap-template-name>"
               }
             }
           }
         }
       }
     }'

   The Cluster API controller triggers a rolling update. Existing machines continue using the old bootstrap configuration until they are replaced.

#Runtime Topology Changes

Networking, placement, and disk topology are part of the machine templates and machine config pool. Treat a change to these fields as an immutable rollout, not as an in-place VM edit:

1. Add or update the required slot definitions in the existing VSphereMachineConfigPool before increasing replicas or starting replacement.
2. Create a new VSphereMachineTemplate when its immutable VM, network, placement, or system-disk fields change.
3. Create a new KubeadmConfigTemplate when bootstrap files or node configuration change.
4. Update the MachineDeployment references only after the new pool capacity and templates exist.
5. Monitor replacement Machines and confirm that every node becomes Ready before removing old templates.

For the exact second-NIC, failure-domain, and persistent-disk fields, use the creation-time variants in Creating Clusters on VMware vSphere. Preserve the same cross-resource relationships during Day-2 changes:

• Keep VSphereMachineConfigPool.spec.configs[].network.additional, VSphereMachineTemplate.spec.template.spec.network.devices, and VSphereFailureDomain.spec.topology.networks consistent when NIC topology changes.
• Configure VSphereCluster.spec.failureDomainSelector and the CPI [Labels] block together when enabling failure domains.
• Do not remove required var-cpaas, var-lib-containerd, or control-plane var-lib-etcd disks.
• Do not treat machineNamingStrategy, wipeFilesystem, or releaseDelayHours as a switch that moves disks between pools or accelerates datastore cleanup.

When you expand from one NIC to two NICs, update all of the following fields together:

1. Append the second NIC entry to VSphereMachineConfigPool.spec.configs[].network.additional; keep network.primary unchanged.
2. Add the second device to VSphereMachineTemplate.spec.template.spec.network.devices in the new immutable template.
3. Add the second network to VSphereFailureDomain.spec.topology.networks when failure domains are enabled.

When you revert from two NICs to one NIC, remove the corresponding second entry from all three fields. Leave network.additional empty or remove the key entirely, and roll out a new VSphereMachineTemplate; do not edit existing VMs directly.

#Troubleshooting

Use the following checks first when worker node management fails:

• Check VSphereMachine conditions for MachineConfigPoolReady. If False, the reason indicates why slot allocation failed:
  • PoolBoundToOtherConsumer: the pool is already bound to a different KubeadmControlPlane or MachineDeployment.
  • NoAvailableSlots: no slots match the required datacenter or failure domain.
• Verify that the worker machine config pool still has free slots.
• Verify that the worker IP addresses, gateway, and DNS settings are correct.
• Verify that the worker VM template still matches the required Kubernetes version and guest-tools requirements.
• Check VSphereVM.status.addresses when a node is waiting for IP allocation.

#Next Steps

For infrastructure preparation and persistent-disk diagnostics, see VMware vSphere Infrastructure Preparation. For cluster creation manifest fields, see Creating Clusters on VMware vSphere.