Alauda Container Platform
  • English

    • Using Velero

    Velero is an open-source Kubernetes cluster backup and migration tool by VMware. KubeVirt provides a Velero plugin to support virtual machine backup and restoration.

    TOC

    PrerequisitesOperation StepsPreparationBackupRestoreCross-Cluster RestoreRestore to a Different NamespaceRestore to a Different Storage Class

    Prerequisites

    • Kubernetes Version: 1.20 or higher (Velero requirement)
    • S3 Storage: For Velero BackupStorageLocation, use platform-provided Ceph or MinIO object storage
    • Block Storage: Platform-provided Ceph RBD

    Operation Steps

    Preparation

    1. Deploy Velero on the ACP platform via Platform Management → Cluster Management → Backup and Recovery → Backup Repository. Use object storage details to create the backup repository.

    2. Enable the CSI feature and add the KubeVirt plugin:

      kubectl edit deploy -n cpaas-system velero

      Add the following to the Velero deployment:

      containers:
- args:
  - server
  - --uploader-type=restic
  - --namespace=cpaas-system
  - --features=EnableCSI
  command:
  - /velero

      Add to the initContainers section:

      initContainers:
- image: registry.example.org/3rdparty/kubevirt/kubevirt-velero-plugin:v0.7.0
  imagePullPolicy: IfNotPresent
  name: velero-plugin-kubevirt
  resources: {}
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  - mountPath: /target
    name: plugins

    3. Verify the KubeVirt plugin installation:

      POD=$(kubectl -n cpaas-system get pod -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
kubectl -n cpaas-system exec -ti "$POD" -- /velero get plugins | grep kubevirt

      Example output:

      kubevirt-velero-plugin/backup-datavolume-action               BackupItemAction
kubevirt-velero-plugin/backup-datavolume-action               BackupItemAction
kubevirt-velero-plugin/backup-virtualmachine-action           BackupItemAction
kubevirt-velero-plugin/backup-virtualmachine-action           BackupItemAction
kubevirt-velero-plugin/backup-virtualmachineinstance-action   BackupItemAction
kubevirt-velero-plugin/backup-virtualmachineinstance-action   BackupItemAction
kubevirt-velero-plugin/restore-pod-action                     RestoreItemAction
kubevirt-velero-plugin/restore-pod-action                     RestoreItemAction
kubevirt-velero-plugin/restore-pvc-action                     RestoreItemAction
kubevirt-velero-plugin/restore-pvc-action                     RestoreItemAction
kubevirt-velero-plugin/restore-vm-action                      RestoreItemAction
kubevirt-velero-plugin/restore-vm-action                      RestoreItemAction
kubevirt-velero-plugin/restore-vmi-action                     RestoreItemAction
kubevirt-velero-plugin/restore-vmi-action                     RestoreItemAction

    4. Adjust node-agent to mount the host kubelet directory and enable privileged mode (required for data movement via /var/lib/kubelet):

      kubectl edit ds -n cpaas-system node-agent

      Update with:

      securityContext:
  privileged: true
volumeMounts:
- mountPath: /host_pods
  mountPropagation: HostToContainer
  name: host-pods
- mountPath: /var/lib/kubelet/plugins
  mountPropagation: HostToContainer
  name: host-plugins
- mountPath: /scratch
  name: scratch
securityContext:
  runAsUser: 0
volumes:
- hostPath:
    path: /var/lib/kubelet/pods
    type: ""
  name: host-pods
- hostPath:
    path: /var/lib/kubelet/plugins
    type: ""
  name: host-plugins
- emptyDir: {}
  name: scratch

    Backup

    1. Create a virtual machine with multiple disks as needed.

    2. Create a backup resource:

      apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/resource-timeout: 10m0s
    velero.io/source-cluster-k8s-gitversion: v1.30.4
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "30"
  labels:
    velero.io/storage-location: default
  name: example-backup
  namespace: cpaas-system
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  hooks: {}
  includedNamespaces:
  - example-namespace
  itemOperationTimeout: 4h0m0s
  metadata: {}
  snapshotMoveData: true
  storageLocation: default
  ttl: 720h0m0s

    3. Wait for the backup to complete and verify:

      POD=$(kubectl -n cpaas-system get pod -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
kubectl -n cpaas-system exec -ti "$POD" -- /velero backup describe example-backup --details
kubectl -n cpaas-system exec -ti "$POD" -- /velero backup get example-backup

      Example output:

      NAME             STATUS                       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
example-backup   WaitingForPluginOperations   0        0          2025-01-25 14:14:08 +0000 UTC   29d       default            <none>

    4. Check dataupload status for data movement:

      kubectl get dataupload -n cpaas-system

      Example output:

      NAME                   STATUS      STARTED   BYTES DONE    TOTAL BYTES   STORAGE LOCATION   AGE   NODE
example-backup-fhc6b   Completed   11h       21474836480   21474836480   default            11h   192.168.254.66
example-backup-qwwzh   Completed   11h       21474836480   21474836480   default            11h   192.168.254.15

    5. Verify backup completion:

      POD=$(kubectl -n cpaas-system get pod -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
kubectl -n cpaas-system exec -ti "$POD" -- /velero backup get example-backup

      Example output:

      NAME             STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
example-backup   Completed   0        0          2025-01-25 14:14:08 +0000 UTC   29d       default            <none>

    6. Check the backup repository (e.g., MinIO) for directories:

      mc ls <backup-repository>

      Example output:

      [2025-01-26 09:23:08 CST]     0B backups/
[2025-01-26 09:23:08 CST]     0B kopia/
[2025-01-26 09:23:08 CST]     0B restic/

    Restore

    1. Delete the original virtual machine.

    2. Create a restore resource via Platform Management → Cluster Management → Backup and Recovery → Restore Management or manually:

      apiVersion: velero.io/v1
kind: Restore
metadata:
  annotations:
    cpaas.io/description: ""
  finalizers:
  - restores.velero.io/external-resources-finalizer
  name: example-restore
  namespace: cpaas-system
spec:
  backupName: example-backup
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  hooks: {}
  includedNamespaces:
  - example-namespace
  itemOperationTimeout: 4h0m0s
  namespaceMapping: {}

    3. Verify the restore:

      kubectl exec -ti -n cpaas-system velero-5df7bb7598-ljjrn -- /velero restore get

      Check datadownload status:

      kubectl get datadownload -n cpaas-system

      Example output:

      NAME       STATUS      STARTED   BYTES DONE    TOTAL BYTES   STORAGE LOCATION   AGE   NODE
example-restore-7lfcm   Completed   11m       21474836480   21474836480   default            15m   192.168.254.66
example-restore-cjcd8   Completed   15m       21474836480   21474836480   default            15m   192.168.254.66

      Restore status:

      NAME   BACKUP           STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
aa     example-backup   Completed   2025-01-26 01:53:14 +0000 UTC   2025-01-26 02:01:24 +0000 UTC   0        2          2025-01-26 01:53:14 +0000 UTC   <none>

    4. Confirm the virtual machine is restored and operational.

    Cross-Cluster Restore

    1. Deploy Velero on the new cluster and complete the preparation steps.
    2. Configure the same backup repository (same bucket and directory) as the original cluster. The backup resource will automatically appear.
    3. Follow the restore steps above.

    Restore to a Different Namespace

    Add a namespaceMapping field to the restore spec:

    spec:
  namespaceMapping:
    example-namespace: test-namespace

    Restore to a Different Storage Class

    1. Create a ConfigMap in the cpaas-system namespace before initiating the restore: 
      apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: cpaas-system
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  vm-cephrbd: vm-topolvm-a
    2. Ensure storage capabilities (e.g., cross-node access, RWX) are consistent.
    3. Note: RWX mode modification is not currently supported.