#Manage Nodes on Huawei Cloud Stack

This document explains how to manage worker nodes using Cluster API Machine resources on the Huawei Cloud Stack platform.

#Prerequisites

When using the YAML examples in this document, replace only values enclosed in <> with environment-specific values. Preserve the remaining fields unless your cluster policy requires a different value.

#Overview

Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. A worker node pool is built from four resources:

1. Machine Configuration Pool - Network settings for worker nodes
2. Machine Template - VM specifications
3. Bootstrap Configuration - Node initialization settings
4. Machine Deployment - Orchestration of node creation and management

You can manage these resources through the web UI or with YAML manifests. The web UI provides a guided experience and requires Fleet Essentials; YAML works without it.

#Using the Web UI

Requirement: This workflow requires Fleet Essentials 1.0.2 or later. Without it, manage node pools with the YAML workflows in the Using YAML section.

The Node Pools tab lets you view the control plane and worker node pools, add or delete worker node pools, and edit the reserved IP addresses of an existing pool.

#Viewing Node Pools

The Node Pools tab shows the Control Plane Node Pool and each Worker Node Pool as a card. Both card types share the same fields:

#Adding a Worker Node Pool

Click Add Worker Node Pool and complete the dialog. The fields match Step 5 of the creation wizard: pool name (prefixed with <cluster-name>-), Flavor, Availability Zone, Machine Configs (hostname, networks, persistent disks), System Disk, Data Volumes, Replicas, Rollout Strategy, and SSH Authorized Keys. New nodes appear in the Nodes tab after the pool is created.

The selected Availability Zone applies to every node in the new worker pool. To place worker nodes in another zone, create a separate worker pool and select that zone; one pool does not automatically spread its replicas across zones. The dropdown disables zones that HCS reports as unavailable. See Availability Zone Placement for automatic selection, persistent disk, and control plane placement constraints.

#Managing Node IP Addresses

Each node in an HCS pool uses a reserved static IP address from its subnet. To change which addresses a pool reserves — for example, to add capacity before scaling the pool up — open the pool card's action menu and select Update Config Pool to open the Manage IPs dialog.

The dialog edits the machine configurations of the pool's HCSMachineConfigPool. For each node you can set the hostname, the networks (each an IP address chosen from its subnet, plus the subnet), and the persistent disks. IP selection follows the same rules as the creation wizard: choose the subnet first, then pick an assignable address from the list; addresses already in use are not offered.

#Deleting a Worker Node Pool

Open the Worker Node Pool card's action menu, select Delete, and confirm.

#Using YAML

Manage worker nodes with Cluster API manifests. To add another worker node pool with YAML, follow Deploy Worker Nodes on the cluster creation page.

#Scaling Worker Nodes

Worker node scaling allows you to adjust cluster capacity based on workload demands.

#Adding Worker Nodes

Increase the number of worker nodes to handle increased workload.

Procedure:

1. Check Current Node Status

   # List all machines in the cluster
   kubectl get machines -n cpaas-system
   
   # List machines for a specific MachineDeployment
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<cluster-name>-md-0

2. Extend Configuration Pool

   Add new machine configurations to the pool for the additional nodes. If the new workers need preserved node-local state such as /var/cpaas, include the matching persistentDisks[] entries in each new configuration.

   kubectl get hcsmachineconfigpool <cluster-name>-worker-pool -n cpaas-system -o yaml

   Modify the pool to include new IP entries, then apply:

   kubectl apply -f <updated-pool-config.yaml>

   When you edit the pool, keep all existing configs[] entries and their accepted persistentDisks[] entries unchanged unless you are intentionally appending a new disk slot.

3. Scale Up the MachineDeployment

   Update the replicas field to the desired number of nodes:

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

4. Monitor the Scaling Progress

   # Watch machines being created
   kubectl get machines -n cpaas-system -w
   
   # Check MachineDeployment status
   kubectl get machinedeployment <cluster-name>-md-0 -n cpaas-system

#Removing Worker Nodes

Decrease the number of worker nodes to reduce cluster capacity.

Procedure:

1. Scale Down the MachineDeployment

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

2. Monitor the Removal Progress

   kubectl get machines -n cpaas-system -w

   The Cluster API controller will:

   • Drain the selected nodes (evict pods if possible)
   • Delete the underlying VMs from the HCS platform
   • Remove the machine resources

#Changing Machine Infrastructure

To change worker machine specifications (CPU, memory, disk, VM image), create a new machine template and point the MachineDeployment at it. The worker nodes are then replaced one by one. Follow these steps:

Note: HCS dataVolumes[] are not preserved during replacement. To preserve node-local state such as /var/cpaas, declare it in HCSMachineConfigPool.spec.configs[].persistentDisks[] before the rollout and keep MachineDeployment.spec.strategy.rollingUpdate.maxSurge: 0. If your cluster still keeps these disks in HCSMachineTemplate.spec.template.spec.dataVolumes[], first complete Migrate Existing Huawei Cloud Stack Clusters to Pool-Managed Persistent Disks.

1. Create New Machine Template

   Copy the existing HCSMachineTemplate and modify the required values:

   • imageName - VM image
   • flavorName - Instance type
   • rootVolume.size - System disk size
   • dataVolumes - Temporary data disk configurations

   If you need to add a new pool-managed persistent disk, append it to the worker HCSMachineConfigPool first. The provider creates or claims the EVS disk, but the running ECS does not mount it until this rolling replacement creates a replacement worker.

   kubectl get hcsmachinetemplate <current-template> -n cpaas-system -o yaml > new-template.yaml

   Then edit new-template.yaml before applying:

   • Change metadata.name to <new-template>
   • Leave runtime identity fields unset, including spec.template.spec.providerID and spec.template.spec.serverId
   • Remove server-generated fields such as:
     • metadata.resourceVersion
     • metadata.uid
     • metadata.creationTimestamp
     • metadata.managedFields
     • status

2. Deploy New Template

   kubectl apply -f new-template.yaml -n cpaas-system

3. Update Machine Deployment

   Modify the MachineDeployment to reference the new template:

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='merge' -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":"<new-template>"}}}}}'

4. Monitor Rolling Update

   kubectl get machines -n cpaas-system -w

To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.

#Updating Bootstrap Templates

To change the bootstrap configuration of worker nodes (for example kubelet settings or the commands that run when a node joins), create a new KubeadmConfigTemplate and point the MachineDeployment at it. Modifying an existing template in place does not roll out existing machines; only newly created machines pick up the change.

1. Export the Existing Template

   kubectl get kubeadmconfigtemplate <cluster-name>-worker-kct -n cpaas-system -o yaml > new-bootstrap-template.yaml

2. Modify the Template

   Edit new-bootstrap-template.yaml:

   • Set metadata.name to a new unique name, <new-bootstrap-template>
   • Remove server-generated metadata (resourceVersion, uid, creationTimestamp, managedFields, the kubectl.kubernetes.io/last-applied-configuration annotation) and the entire status field
   • Edit the fields you want to change. Keep the existing entries of the kubelet patch file /etc/kubernetes/patches/kubeletconfiguration0+strategic.json, including the certificate paths, and change only what you need
   • For Kubernetes 1.35 or later, add the required kubelet patch settings to that file

3. Apply the New Template

   kubectl apply -f new-bootstrap-template.yaml -n cpaas-system

4. Update the MachineDeployment

   kubectl patch machinedeployment <cluster-name>-md-0 -n cpaas-system \
     --type='merge' -p='{"spec":{"template":{"spec":{"bootstrap":{"configRef":{"name":"<new-bootstrap-template>"}}}}}}'

   The rolling replacement of the worker nodes then starts:

   kubectl get machines -n cpaas-system -w

#Verification

After deploying worker nodes, verify the deployment:

# Check machine status
kubectl get machines -n cpaas-system

# Verify nodes are Ready
kubectl get nodes

# Check MachineDeployment status
kubectl get machinedeployment -n cpaas-system

#Troubleshooting

#Viewing Controller Logs

# View HCS controller logs
kubectl logs -n cpaas-system deployment/hcs-controller-manager

# View machine details
kubectl describe hcsmachine <machine-name> -n cpaas-system

#Common Issues

Node fails to join cluster

• Verify the VM template matches the Kubernetes version
• Check network connectivity between nodes
• Ensure the configuration pool has available entries

Machine stuck in provisioning

• Check HCS platform for resource availability
• Verify credentials and permissions
• Review controller logs for error messages