#VMware vSphere Infrastructure Preparation

This document helps you prepare the infrastructure and collect the values required to create a VMware vSphere workload cluster. Complete this checklist before you apply the manifests in Creating Clusters on VMware vSphere. The provider behavior and fields on this page are validated against VMware vSphere Provider v1.0.17.

#Scenarios

Use this checklist in the following scenarios:

• You are preparing a new VMware vSphere cluster deployment.
• You want to validate external dependencies before you start the deployment.
• You plan to enable optional creation-time topology variants such as multiple datacenters, multiple NICs, or extra worker nodes.

#Prerequisites

Before you begin, ensure the following conditions are met:

• You can access the global cluster with kubectl.
• The workload cluster objects must be stored in the cpaas-system namespace.
• You have access to the target vCenter inventory, networks, datastores, and templates.

#How to Use This Checklist

Use this checklist in the following order:

1. Collect the deployment parameters listed in this document.
2. Replace every placeholder in the manifest templates with the actual values collected here.
3. Reuse the same value everywhere a placeholder appears in multiple manifests.
4. If an optional feature is not enabled, omit the corresponding YAML block exactly as described in Creation-Time Topology Variants.
5. If an optional field (such as deviceName) is not needed, remove the entire line from the YAML manifest.

#Parameter Sources and Safe Collection

Separate parameter collection into the following three sources. A value that can be discovered from vCenter is not necessarily a product-supported version, and a product-derived value is not a substitute for a network or capacity decision.

#Optional vCenter discovery with govc

If govc is already authenticated through your protected local environment, use read-only inventory commands such as:

govc find / -type d
govc find / -type c
govc find / -type p
govc find / -type n
govc find / -type s
govc find / -type m

Interpret the results as candidates, not automatic selections:

• d: datacenters
• c: compute clusters
• p: resource pools
• n: networks or distributed port groups
• s: datastores
• m: VMs and templates; confirm the selected object is a template

Use the openssl command in Thumbprint to retrieve the certificate fingerprint without printing vCenter credentials.

#Terminology

The following terms are used consistently throughout the VMware vSphere cluster-creation documents.

#machine config pool

A machine config pool is the VSphereMachineConfigPool custom resource. It predefines node slots. Each slot can include:

• A node hostname
• A target datacenter
• Static IP configuration for each NIC
• Persistent disk definitions

#Node slot

A node slot is an entry under VSphereMachineConfigPool.spec.configs[]. A single slot usually maps to one node, such as cp-01 or worker-01. The slot hostname drives the Kubernetes node name, the kubelet serving certificate DNS SAN, and (combined with the resolved primary NIC addresses) the kubelet node-ip; it must be a valid DNS-1123 subdomain.

#Slot network layout

Each slot declares its NIC layout under network.primary and network.additional:

• network.primary is required. Its networkName must be set and is used as the node-ip source for the kubelet.
• network.additional is an optional list of extra NICs merged after the primary NIC in the order listed.

The dns values in VSphereMachineConfigPool.spec.configs[].network describe the static network metadata for each slot. In affected VMware deployments, these values might not update the guest operating system's /etc/resolv.conf reliably. The deployment manifests therefore also write /etc/resolv.conf through both KubeadmControlPlane.spec.kubeadmConfigSpec.files and KubeadmConfigTemplate.spec.template.spec.files.

#deviceName

deviceName is an optional field in the VSphereMachineConfigPool network configuration. It is used to control the NIC name seen inside the guest operating system, such as eth0 or eth1.

Use the following distinctions when you fill the values:

• networkName is the vCenter network or port group name.
• deviceName is the NIC name inside the guest operating system.
• If deviceName is omitted, CAPV typically assigns names such as eth0, eth1, and eth2 by NIC order.

#vCenter resource pool

A vCenter resource pool is the native vCenter inventory object, for example:

/Datacenter1/host/cluster1/Resources

When the failure-domain creation variant is enabled, this path is used by VSphereDeploymentZone.spec.placementConstraint.resourcePool.

#Compute cluster

The compute cluster is the target vCenter compute-cluster name. In these documents, it is primarily used when a VSphereFailureDomain is mapped to a specific deployment target.

#Datastore

The datastore is the vSphere storage location that stores VM disks. Both system disks and data disks must be placed on concrete datastores.

#VM template

The VM template is the source template used to create node virtual machines. When you enable multiple datacenters, the same template must already exist in every target datacenter and must be resolvable by the same template name.

#Thumbprint

The thumbprint is the SHA-1 fingerprint of the vCenter server certificate. CAPV uses it to validate the target vCenter server.

Use the following command to retrieve it:

openssl s_client -connect <vsphere_server>:443 -servername <vsphere_server> </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha1

#Management Cluster Prerequisites

Use the following table to record the required values and validation results.

#vCenter and Template Prerequisites

#vCenter connection information

Note: These documents assume the default vCenter HTTPS port 443.

#VM template requirements

The template should also meet the following requirements:

• It uses an operating system supported by your platform image policy.
• It includes cloud-init.
• It includes VMware Tools or open-vm-tools.
• It includes containerd.
• It includes the baseline components required by kubeadm bootstrap.
• It includes pre-exported container image tar files under /root/images/. These files are imported into containerd by capv-load-local-images.sh before kubeadm runs, so that node bootstrap does not depend on pulling images from a remote registry.
• The tar files include the kubeadm control-plane images supplied with the OS template, including etcd and kube-apiserver, under the exact image references generated from clusterConfiguration.imageRepository, the Kubernetes version, and the component image tags. Although etcd and kube-apiserver run as static Pods, they still require container images. A missing or mismatched local reference causes the runtime to try the configured Registry instead.
• The /root/images/*.tar files must include the sandbox (pause) image whose reference exactly matches the sandbox_image value (containerd v1) or sandbox value (containerd v2) configured in /etc/containerd/config.toml. For example, if containerd is configured with sandbox_image = "registry.example.com/tkestack/pause:3.10", one of the tar files must contain that exact image reference. A mismatch causes containerd to pull the sandbox image from the network, which defeats the purpose of local preloading and fails in air-gapped environments.

#Load Balancer Prerequisites

This section covers the external LoadBalancer endpoint, which the baseline cluster uses. Provision it before cluster creation and follow Plan the Control Plane Endpoint. A cluster can instead be created with the provider-managed Self-built VIP, which needs no external LoadBalancer and requires ACP v4.4 or later; see Control Plane Endpoint Modes.

#Cluster Baseline Parameters

#Registry tag validation

First read the permanent Registry address from the global cluster, or confirm the equivalent address with the platform administrator:

kubectl -n cpaas-system get cluster global \
  -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}{"\n"}'
kubectl -n cpaas-system get secret public-registry-credential -o name

The Secret command verifies existence only and does not print credential data. Export the confirmed Cluster Registry address and the image Registry used by the CPI and kubeadm manifests. If both placeholders identify the same Registry, use the same value for both variables.

export CLUSTER_REGISTRY_ADDRESS="<registry_address>"
export IMAGE_REGISTRY_ADDRESS="<image_registry>"
# Repository names depend on the target version. The values below are the
# ACP v4.4 and later ones; for an earlier target, change them to:
#   Kube-OVN chart -> acp/chart-cpaas-kube-ovn
#   CoreDNS        -> tkestack/coredns
export KUBE_OVN_CHART_REPO="acp/chart-kube-ovn"
export COREDNS_REPO="acp/coredns"

curl -sk "https://${CLUSTER_REGISTRY_ADDRESS}/v2/${KUBE_OVN_CHART_REPO}/tags/list"
curl -sk "https://${IMAGE_REGISTRY_ADDRESS}/v2/${COREDNS_REPO}/tags/list"
curl -sk "https://${IMAGE_REGISTRY_ADDRESS}/v2/ait/cloud-provider-vsphere/tags/list"

A NAME_UNKNOWN response means the repository name does not match the target version; correct KUBE_OVN_CHART_REPO or COREDNS_REPO from the mapping above and retry, rather than reading it as a missing tag.

The authoritative chart-name boundary is in Upgrade Kube-OVN Before the Control Plane.

The /v2/ segment belongs only to these Registry HTTP API URLs. Image references use <registry>/<repository>:<tag>, and the Cluster annotation uses <host>:<port>.

Confirm that the responses include the values you plan to use for:

• <kube_ovn_version>
• <dns_image_tag>
• <cpi_image_tag>

The Registry does not need to serve the etcd or kube-apiserver images when the VM template contains their exact image references and capv-load-local-images.sh imports them before kubeadm runs. Static-Pod deployment alone is not sufficient: verify the local images whenever the OS template, Kubernetes version, imageRepository, or component tags change.

#Validate the locally preloaded images

Create and boot a disposable VM from the exact candidate template. Run the following commands inside that VM to exercise the same import path used before kubeadm and list the image references that are actually available to containerd:

sudo bash -euo pipefail <<'EOF'
shopt -s nullglob
image_files=(/root/images/*.tar)
if (( ${#image_files[@]} == 0 )); then
  echo "ERROR: no tar files found in /root/images" >&2
  exit 1
fi

systemctl restart containerd
for image_file in "${image_files[@]}"; do
  ctr -n k8s.io images import "${image_file}"
done
ctr -n k8s.io images list -q | sort -u
EOF

sudo grep -nE '^[[:space:]]*(sandbox_image|sandbox)[[:space:]]*=' \
  /etc/containerd/config.toml

Compare the first command's output with the complete references required by the selected Kubernetes version and manifest values. Confirm that every required kubeadm image is present, including the exact etcd and kube-apiserver references generated from <image_registry>/tkestack, <k8s_version>, and the component tags. Compare the second command's sandbox reference with the imported pause-image reference; they must be identical.

Perform this validation again whenever the template, Kubernetes version, imageRepository, etcd tag, or containerd sandbox setting changes. Discard the validation VM afterward; do not convert it back into the source template.

Set cpaas.io/registry-address in the workload Cluster annotation to <registry_address>.

#Minimum Single-Datacenter Parameters

#Datacenter and resource placement

#Primary NIC parameters

#Control plane machine config pool

#Worker machine config pool

#Compute sizing

#Disk sizing model

VSphereMachineTemplate.spec.template.spec.diskGiB and the two disk lists under VSphereMachineConfigPool.spec.configs[] define different disks.

Do not add the data disk sizes to diskGiB. For example, if the OS image has a 100 GiB system disk and the node also needs a 100 GiB persistent disk plus three ephemeral disks, use a system disk value that is at least 100 GiB and declare the data disks in persistentDisks and ephemeralDisks. Setting diskGiB to the combined size would create an oversized system disk in addition to the data disks.

#Standard data disks

Each node role requires a set of dedicated data disks. The disks listed below are the minimum required disks. You can append additional disks to either list if your workload requires them. An allocated slot still accepts new entries in both lists — they take effect when the VM is next recreated — but it never accepts removing or reshaping a disk, so plan sizes and mount paths before the first Machine takes the slot.

The split follows the same model as the Huawei Cloud Stack and Huawei DCS providers: only platform state that must survive VM replacement (/var/cpaas) is a persistent disk, while the rebuildable runtime directories (/var/lib/kubelet, /var/lib/containerd, /var/lib/etcd) are recreated with every VM.

#Persistent disk fields

#Ephemeral disk fields

ephemeralDisks[] mirrors the guest-facing fields of a persistent disk — name, sizeGiB, mountPath, mountOptions, fsFormat, datastore, and storagePolicy — and drops the persistence fields. There is no wipeFilesystem, because the backing VMDK is always newly created and formatted from scratch, and no unitNumber, because the controller assigns the SCSI unit at clone time and reports it in status.ephemeralDiskStatuses[].

Disk names and mount paths share one namespace across both lists within a slot. For the field-by-field comparison and the guidance on which list to use, see Choose persistent or ephemeral.

#Persistent disk lifecycle

Persistent disks are owned by a slot in one specific VSphereMachineConfigPool, not by the short-lived VM. A slot in another pool is a different allocation even when it declares the same hostname, IP address, or disk names. The expected replacement lifecycle is:

1. While a Machine uses the slot, status.configStatuses[].state is InUse and the VMDKs are attached to its VM.
2. When the Machine and VM are deleted, the slot becomes Released. The VMDKs remain available for replacement and are not permanent orphaned disks.
3. A replacement Machine that references the same existing pool can reuse the Released slot immediately. The provider selects a released slot before an unused slot and reattaches the declared persistent VMDKs to the replacement VM. releaseDelayHours is not a wait time for reuse.
4. If the slot remains unused until releaseDelayHours expires, the controller can make it generally available and reclaim persistent VMDKs only after it confirms they are no longer attached. Asynchronous progress appears in status.persistentDiskStatuses[].phase.

wipeFilesystem controls data handling after a persistent disk is attached to a new VM; it does not control slot selection or datastore reclamation. false preserves the existing filesystem contents. true deletes the mounted contents during the new VM's first boot; it does not make VMDK reclamation faster and must not be treated as a datastore-capacity setting.

#Redeployment and teardown capacity

Creating a new VSphereMachineConfigPool with a different metadata.name creates a separate set of slots. Those slots do not reuse VMDKs owned by an earlier pool. Repeated teardown and redeployment with different pool names can therefore temporarily consume multiple complete sets of persistent disks until every earlier pool finishes provider-managed reclaim.

Use these rules when retrying or tearing down a deployment:

• For Machine replacement or rolling updates within the same cluster, keep each VSphereMachineTemplate.spec.template.spec.machineConfigPoolRef.name pointed at the same existing pool. Changing KubeadmControlPlane.spec.machineNamingStrategy does not transfer disks between pools and is not a datastore cleanup control.
• If a new deployment intentionally uses a new pool name, include the new pool's complete persistent-disk allocation in datastore capacity planning. Do not assume matching hostnames or disk names enable reuse across pools.
• Before starting another deployment against a capacity-constrained datastore, verify that earlier pool objects have completed deletion and that their VMDKs have been reclaimed. Pool deletion is asynchronous because the finalizer waits for Machines to be removed and for persistent-disk reclaim to finish.
• Do not remove the pool finalizer or manually delete backing VMDKs to accelerate teardown. If reclaim does not progress, collect the pool status, Events, provider-controller logs, and vCenter attachment state before treating the condition as a provider defect.
• Size the datastore for both lists. Ephemeral VMDKs occupy datastore space for as long as their VM exists, but they are deleted with it, so they never accumulate across pools and never hold a slot release. Only persistent VMDKs survive a teardown and wait for provider-managed reclaim.

Use the following fields when you investigate a released slot or a datastore capacity alert:

Read per-disk state from status.persistentDiskStatuses[]. Its phase field carries Creating, Attached, Available, Reclaiming, Reclaimed, or Error, and each entry also reports lastError, retryAfter, taskRef, and the owning machine. The status.configStatuses[].reclaimStatus subtree and the spec.configs[].persistentDisks[].volumePath and .diskUUID fields are deprecated, frozen, and scheduled for removal. Existing pools are migrated onto persistentDiskStatuses[] by the provider itself; see Upgrading the Provider.

Inspect the pool and related Events without reading credentials:

kubectl -n <namespace> get vspheremachineconfigpool <pool_name> -o yaml
kubectl -n <namespace> describe vspheremachineconfigpool <pool_name>
kubectl -n <namespace> get events \
  --field-selector involvedObject.kind=VSphereMachineConfigPool,involvedObject.name=<pool_name>

#Control plane nodes (4 disks per node)

#Worker nodes (3 disks per node)

#Size parameters

#vSphere CPI parameters

#Optional Creation-Time Topology Parameters

#Multiple datacenters and multiple failure domains

#Second NIC parameters

#Final Readiness Check

Before you start the deployment, confirm all of the following items:

1. The global cluster is reachable.
2. The two cluster plugins are installed: Alauda Container Platform Kubeadm Provider and Alauda Container Platform VMware vSphere Infrastructure Provider.
3. ClusterResourceSet=true is enabled.
4. The vCenter server, username, password, and thumbprint are collected.
5. The external LoadBalancer satisfies the Layer 4 listener, backend, health-check, ownership, and reachability contract. If the cluster uses the Self-built VIP instead, the Self-built VIP prerequisites are met and <vrid> and <vip_interface> are decided.
6. The Pod CIDR, Service CIDR, and kube-ovn-join-cidr do not overlap with existing networks.
7. The VM template is available in every required datacenter.
8. The required datastores and vCenter resource pool paths are confirmed.
9. The permanent platform Registry contains the required Kube-OVN, CoreDNS, and vSphere CPI tags.
10. The VM template contains the exact locally preloaded etcd and kube-apiserver image references required by the selected Kubernetes and component versions.
11. The cpaas.io/registry-address value is planned as <registry_address> without a scheme or /v2/ path.
12. The control plane and worker bootstrap manifests both write /etc/resolv.conf through kubeadm files.
13. The machine config pool values for the minimum single-datacenter topology are complete.
14. The baseline system disk and data disk sizing is confirmed, each required disk is declared in the correct list (/var/cpaas persistent; /var/lib/kubelet, /var/lib/containerd, and /var/lib/etcd ephemeral), and data disk sizes are not added into diskGiB.
15. releaseDelayHours, same-pool persistent-disk reuse, new-pool capacity allocation, and provider-managed teardown reclaim are understood before datastore capacity is allocated.
16. Every required parameter has a real value.

#Next Steps

After you complete this checklist, continue with Creating Clusters on VMware vSphere.