#Manage Nodes on Huawei DCS

This document explains how to manage worker nodes using Cluster API Machine resources.

#Prerequisites

#Overview

Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. A worker node pool is built from four resources:

1. IP-Hostname Pool Configuration - Network settings for worker nodes
2. Machine Template Setup - VM specifications
3. Bootstrap Configuration - Node initialization and join settings
4. Machine Deployment - Orchestration of node creation and management

To add another worker node pool with YAML, follow Deploy Worker Nodes on the cluster creation page.

#Node Management Operations

This section covers common operational tasks for managing worker nodes, including scaling, machine infrastructure changes, and bootstrap template updates.

#Scaling Worker Nodes

Worker node scaling allows you to adjust cluster capacity based on workload demands. The Cluster API manages the node lifecycle automatically through the MachineDeployment resource.

#Adding Worker Nodes

Increase the number of worker nodes to handle increased workload or add new capacity.

Use Case: Scale up cluster to add more compute resources

Prerequisites:

• Verify the IP pool has sufficient available IP addresses for new nodes
• Ensure the DCS platform has adequate resources to provision new VMs

Procedure:

1. Check Current Node Status

   View the current machines in the cluster:

   # List all machines in the cluster
   kubectl get machines -n cpaas-system
   
   # List machines for a specific MachineDeployment
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<cluster-name>-md-0

2. Extend IP Pool

   Before scaling up, add new IP configurations to the pool for the additional nodes.

   INFO

   IP Pool Expansion The IP pool must contain at least as many entries as the desired replica count. Add new IP entries for each additional worker node you plan to deploy.

   Add IP entries to the pool:

   First, export the current pool configuration to preserve existing entries:

   kubectl get dcsiphostnamepool <cluster-name>-worker-pool -n cpaas-system -o yaml

   Then use the following command to add new IP configurations. The pool array must include all existing entries plus the new entries:

   kubectl patch dcsiphostnamepool <cluster-name>-worker-pool -n cpaas-system \
     --type='merge' -p='
   {
     "spec": {
       "pool": [
         {
           "ip": "<existing-ip-1>",
           "mask": "<worker-cidr-prefix-length>",
           "gateway": "<worker-gateway>",
           "dns": "<worker-dns>",
           "hostname": "<existing-hostname-1>",
           "machineName": "<existing-machine-name-1>",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "<datastore-cluster-name>",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "<existing-ip-2>",
           "mask": "<worker-cidr-prefix-length>",
           "gateway": "<worker-gateway>",
           "dns": "<worker-dns>",
           "hostname": "<existing-hostname-2>",
           "machineName": "<existing-machine-name-2>",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "<datastore-cluster-name>",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "<new-worker-ip-1>",
           "mask": "<worker-cidr-prefix-length>",
           "gateway": "<worker-gateway>",
           "dns": "<worker-dns>",
           "hostname": "<new-worker-hostname-1>",
           "machineName": "<new-worker-machine-name-1>",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "<datastore-cluster-name>",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "<new-worker-ip-2>",
           "mask": "<worker-cidr-prefix-length>",
           "gateway": "<worker-gateway>",
           "dns": "<worker-dns>",
           "hostname": "<new-worker-hostname-2>",
           "machineName": "<new-worker-machine-name-2>",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "<datastore-cluster-name>",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         }
       ]
     }'
   WARNING

   Important Notes

   • The pool array must include all existing entries plus the new entries you want to add
   • Copy the existing entries from the exported YAML to avoid data loss
   • kubectl patch --type='merge' replaces the entire spec.pool array, so copy every existing persistentDisk block unchanged, including isThin if it is present, unless you are intentionally adding new disks
   • Ensure each new entry has unique ip, hostname, and machineName values
   • If new worker nodes also need the platform-required /var/cpaas disk, declare it in each new entry's persistentDisk
   • Network parameters (mask, gateway, dns) typically match existing entries

   Example: Adding 2 new nodes to an existing pool of 3 nodes

   # Current pool has 3 entries (10.0.1.11, 10.0.1.12, 10.0.1.13)
   # Adding 2 more entries for nodes 4 and 5
   kubectl patch dcsiphostnamepool worker-pool-1-ippool -n cpaas-system \
     --type='merge' -p='
   {
     "spec": {
       "pool": [
         {
           "ip": "10.0.1.11",
           "mask": "24",
           "gateway": "10.0.1.1",
           "dns": "10.0.0.2",
           "hostname": "worker-node-1",
           "machineName": "worker-vm-1",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "shared-datastore-cluster",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "10.0.1.12",
           "mask": "24",
           "gateway": "10.0.1.1",
           "dns": "10.0.0.2",
           "hostname": "worker-node-2",
           "machineName": "worker-vm-2",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "shared-datastore-cluster",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "10.0.1.13",
           "mask": "24",
           "gateway": "10.0.1.1",
           "dns": "10.0.0.2",
           "hostname": "worker-node-3",
           "machineName": "worker-vm-3",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "shared-datastore-cluster",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "10.0.1.14",
           "mask": "24",
           "gateway": "10.0.1.1",
           "dns": "10.0.0.2",
           "hostname": "worker-node-4",
           "machineName": "worker-vm-4",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "shared-datastore-cluster",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         },
         {
           "ip": "10.0.1.15",
           "mask": "24",
           "gateway": "10.0.1.1",
           "dns": "10.0.0.2",
           "hostname": "worker-node-5",
           "machineName": "worker-vm-5",
           "persistentDisk": [
             {
               "slot": 0,
               "quantityGB": 40,
               "datastoreClusterName": "shared-datastore-cluster",
               "path": "/var/cpaas",
               "format": "xfs"
             }
           ]
         }
       ]
     }'

3. Verify IP Pool Capacity

   After extending the IP pool, verify it has sufficient entries for the desired replica count:

   kubectl get dcsiphostnamepool -n cpaas-system <cluster-name>-worker-pool -o yaml

   Check that the pool contains at least as many entries as the desired replica count.

4. Scale Up the MachineDeployment

   Update the replicas field to the desired number of nodes:

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

   Example: Scale from 3 to 5 nodes

   kubectl patch machinedeployment worker-pool-1 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 5}]'

5. Monitor the Scaling Progress

   Watch the machine creation process:

   # Watch machines being created
   kubectl get machines -n cpaas-system -w
   
   # Check MachineDeployment status
   kubectl get machinedeployment <cluster-name>-md-0 -n cpaas-system

   The Cluster API controller will automatically create new machines based on the MachineDeployment template.

6. Verify Nodes Joined the Cluster

   Switch to the target cluster context and verify the new nodes:

   # Switch to target cluster context
   kubectl config use-context <target-cluster-context>
   
   # Check all nodes are Ready
   kubectl get nodes

   The new nodes should appear in the list and transition to Ready status.

#Removing Worker Nodes

Decrease the number of worker nodes to reduce cluster capacity or remove underutilized resources. The Cluster API supports two removal strategies:

1. Random removal: Reduce replicas, the platform randomly selects and deletes machines
2. Targeted removal: Mark specific machines for deletion, then reduce replicas (recommended for IP recovery)

#Random Removal

Use Case: Scale down cluster where any node can be removed (no specific IP requirements)

Procedure:

1. Identify Current Machine Status

   View the current machines in the MachineDeployment:

   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<cluster-name>-md-0

2. Scale Down the MachineDeployment

   Update the replicas field to reduce the node count:

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

   Example: Scale from 5 to 3 nodes

   kubectl patch machinedeployment worker-pool-1 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 3}]'

   The Cluster API controller will randomly select and delete machines to match the desired replica count.

3. Monitor the Removal Progress

   Watch the machine deletion process:

   kubectl get machines -n cpaas-system -w

   The Cluster API controller will:

   • Drain the selected nodes (evict pods if possible)
   • Delete the underlying VMs from the DCS platform
   • Remove the machine resources

4. Verify Nodes Removed

   Switch to the target cluster context:

   kubectl config use-context <target-cluster-context>
   kubectl get nodes

   The removed nodes should no longer appear in the list.

#Targeted Removal

Use Case: Remove specific machines (e.g., for IP recovery, replace unhealthy nodes)

Procedure:

1. Identify Machines to Remove

   View the current machines:

   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<cluster-name>-md-0

   Note the <machine-name> of the machines you want to remove.

2. Annotate Machines for Deletion

   Mark the specific machines for deletion:

   kubectl patch machine <machine-name> -n cpaas-system \
     --type='merge' -p='{"metadata": {"annotations": {"cluster.x-k8s.io/delete-machine": "true"}}}'

   Repeat for each machine you want to remove.

   Example: Remove two specific machines

   kubectl patch machine worker-pool-1-abc123 -n cpaas-system \
     --type='merge' -p='{"metadata": {"annotations": {"cluster.x-k8s.io/delete-machine": "true"}}}'
   
   kubectl patch machine worker-pool-1-def456 -n cpaas-system \
     --type='merge' -p='{"metadata": {"annotations": {"cluster.x-k8s.io/delete-machine": "true"}}}'

3. Scale Down the MachineDeployment

   After annotating the machines, reduce the replica count:

   INFO

   Replica Count Must Match Annotated Machines Reduce replicas by exactly the number of annotated machines.

   • If you reduce by fewer, not all annotated machines will be removed
   • If you reduce by more, additional machines will be randomly selected for deletion

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

   Example: If you annotated 2 machines, reduce replicas by exactly 2 (e.g., from 5 to 3)

   The platform will delete the annotated machines, not randomly selected ones.

4. Monitor the Removal Progress

   Watch the machine deletion process:

   kubectl get machines -n cpaas-system -w

5. Verify Nodes Removed

   Switch to the target cluster context:

   kubectl config use-context <target-cluster-context>
   kubectl get nodes

   The removed nodes should no longer appear in the list.

#Changing Machine Infrastructure

To change the machine specification of worker nodes (CPU, memory, disk, VM template), create a new machine template, then point the MachineDeployment at it. The worker nodes are replaced one by one.

1. Create a New Machine Template

   Export the DCSMachineTemplate referenced by your MachineDeployment to a new file:

   kubectl get dcsmachinetemplate <current-template-name> -n cpaas-system -o yaml > new-template.yaml

   In the new file:

   • Set metadata.name to a unique name
   • Modify the required values (spec.template.spec.vmTemplateName, spec.template.spec.vmConfig.dcsMachineCpuSpec.quantity, spec.template.spec.vmConfig.dcsMachineMemorySpec.quantity, spec.template.spec.vmConfig.dcsMachineDiskSpec)
   • Remove the server-generated metadata (resourceVersion, uid, generation, creationTimestamp, managedFields, the kubectl.kubernetes.io/last-applied-configuration annotation) and the entire status field
   • Leave spec.template.spec.providerID unset

   Keep pool-managed persistent disks, including /var/cpaas, in DCSIpHostnamePool.spec.pool[].persistentDisk and additional NICs in DCSIpHostnamePool.spec.pool[].additionNic[]. Use dcsMachineDiskSpec only for the system disk and template-local disks.

   Apply the new template:

   kubectl apply -f new-template.yaml -n cpaas-system

2. Update Machine Deployment

   Point the MachineDeployment at the new template:

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='merge' -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":"<new-template-name>"}}}}}'

3. Monitor the Rolling Update

   • Worker nodes are replaced one by one with the new specification

   • Any disks declared in DCSIpHostnamePool.spec.pool[].persistentDisk are detached from the old VM and reattached to the replacement VM. If the pool uses persistent disks, keep spec.strategy.rollingUpdate.maxSurge at 0 on the MachineDeployment

   • Monitor the progress:

     kubectl get machinedeployment <cluster-name>-md-0 -n cpaas-system
     kubectl get machines -n cpaas-system -w

To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.

To move the nodes to the Alauda OS image of a new Kubernetes version, follow Upgrade Kubernetes on Huawei DCS instead.

If you are migrating an existing cluster from the old template-disk layout to pool-managed persistent disks, follow Migrate Existing Huawei DCS Clusters to Pool-Managed Persistent Disks before you rely on upgrade-time data preservation.

#Updating Bootstrap Templates

Bootstrap templates (KubeadmConfigTemplate) are used by MachineDeployment and MachineSet resources. Changes to existing templates do not automatically trigger rollouts of existing machines; only new machines use the updated template.

Update Process:

1. Export Existing Template

   kubectl get KubeadmConfigTemplate <template-name> -o yaml > new-template.yaml

2. Modify Configuration

   • Update the desired fields in the exported YAML
   • For Kubernetes 1.35 or later, add the required kubelet patch settings to /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
   • Change the metadata.name to a new unique name
   • Remove extraneous metadata fields (resourceVersion, uid, creationTimestamp, etc.)

3. Create New Template

   kubectl apply -f new-template.yaml

4. Update MachineDeployment

   • Modify the MachineDeployment resource
   • Update spec.template.spec.bootstrap.configRef.name to reference the new template
   • Apply the changes to trigger a rolling update

---

#Managing Node Pools Using the Web UI

Adding, deleting, and viewing node pools through Fleet Essentials are not affected by the upgrade limitations described in Fleet Essentials Boundary.

Node pools provide a declarative way to manage groups of nodes with identical configurations. You can view, add, and delete worker node pools through the web UI.

Requirement: This workflow requires Fleet Essentials. The persistentDisk declaration on DCSIpHostnamePool remains YAML-only and is not exposed in the node-pool UI.

If the node pool relies on pool-managed persistent disks, prepare or update the corresponding DCSIpHostnamePool entry with YAML before you use the web UI workflow here.

#Viewing Node Pools

The Node Pools Tab displays all node pools in the cluster:

Control Plane Node Pool:

• Fixed at 3 replicas for high availability
• Displays Kubernetes version with upgrade indicator if available
• Shows Conditions link for detailed status

Worker Node Pools:

• Customizable replica counts
• Individual Kubernetes version management
• Scale and upgrade operations

Node Pool Card Information:

#Adding a Worker Node Pool

Navigation: Node Pools Tab → Click Add Worker Node Pool

Form Fields:

Validation:

• Pool name must be unique within the cluster
• IP Pool must have sufficient available IP addresses (≥ Replicas)
• maxSurge/maxUnavailable constraints must be satisfied
• If the node pool relies on persistent disks, keep maxSurge = 0 so Machines are replaced one by one

Tip: Prefix the pool name with the cluster name followed by a hyphen (e.g., mycluster-worker-1) to avoid naming conflicts.

After creation, new nodes appear in the Nodes Tab. The number of nodes equals the configured Replicas value.

#Deleting a Worker Node Pool

Steps:

1. Click the delete icon on the Worker Node Pool card
2. Confirm deletion in the dialog

#Viewing Conditions (Control Plane Only)

Click the Conditions link on the Control Plane Node Pool card to view detailed status information.

Conditions List:

---

#Troubleshooting

#A new worker Machine stays in Provisioning and the DCS VM stays in creating

The diagnostic flow is identical for control-plane and worker Machine objects — when a DCSMachine reaches Provisioning with status.cdRomFile already set (DCS API auth and ISO upload succeeded), but the underlying DCS VM never schedules to a host, the failure is on the DCS platform side and not in the controller.

See Infrastructure → Troubleshooting: VM stuck in creating for the full diagnostic ladder, which covers:

• Confirming the symptom is a placement stall (not a slow boot).
• Checking datastore free space.
• Checking host CPU / memory utilisation and overcommit ratios.
• Coordinating with the DCS platform team to rebalance load when a single DCS Host is saturated.
• Escalation path when the deploy task is genuinely deadlocked.

For control-plane high availability across multiple physical hosts, see Infrastructure → Cross-Host High Availability for Control Plane.

The same diagnostic applies whether the stuck node is the first control-plane node of a fresh cluster, a new worker added through MachineDeployment scaling, or a replacement worker created during a rolling update.

---

#Next Steps

• Create Clusters
• Upgrade Clusters