Upgrade from Log Storage for Elasticsearch

Upgrade an existing ACP deployment from the legacy Elasticsearch log storage plugin to ClickHouse or OpenSearch.

WARNING

Do not uninstall the legacy plugin, stop the legacy data path, or delete or unprotect its PVCs and PVs before this procedure requires it. Do not stop or scale the legacy Elasticsearch, Kafka, ZooKeeper, lanaya, or Razor workloads.

Before you start

This flow upgrades a cluster running Alauda Container Platform Log Storage for Elasticsearch to the ACP 4.4 ClickHouse or OpenSearch data path. Supported source Logging plugin versions are 4.2.x and 4.3.x; earlier versions are not supported yet.

Run these checks on the workload cluster before installing the new data path:

kubectl -n cpaas-system get deploy log-storage-operator-controller-manager
kubectl -n cpaas-system get apprelease logcenter
kubectl -n cpaas-system get statefulset cpaas-elasticsearch cpaas-elasticsearch-master --ignore-not-found

AppRelease/logcenter is the trigger for the legacy upgrade flow. If it is missing, stop: the annotation alone does not start this flow.

Install the new data path

Install the new data path by creating the connection Secrets and the PlatformLogForward for ClickHouse or OpenSearch.

The only upgrade-specific change is this annotation on PlatformLogForward:

metadata:
  annotations:
    log.alauda.io/legacy-es-upgrade: "true"

Keep installMode: Fresh; Adopt is not part of this flow.

Wait until PlatformLogForward/platform-default is Ready and the LegacyESUpgrade condition has reason LegacyESUpgradeCompleted:

kubectl get platformlogforward platform-default -w

For details or troubleshooting:

kubectl get platformlogforward platform-default \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

Wait until the queued data in the legacy Kafka is consumed; the platform records LegacyKafkaDrained when the old consumer lag reaches zero. Before continuing, confirm that new log, event, and audit records are queryable from the new storage.

Check the drain status (True/LagZero means drained):

kubectl -n cpaas-system get platformlogforward platform-default \
  -o jsonpath='{range .status.conditions[?(@.type=="LegacyKafkaDrained")]}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

Decide on historical data

Uninstalling the legacy plugin is the point of no return for the pre-cutover history. The new data path already carries everything written after cutover, but the older data still lives only on the legacy Elasticsearch volumes.

If you need that history in the new storage, set it up now: follow Historical Data Migration to create the migration resource and wait until it reports PrecaptureReady, which captures the source. Then return here to uninstall.
If you do not need it, continue.

Uninstall the legacy Elasticsearch plugin

WARNING

Before this step, verify all of the following:

PlatformLogForward/platform-default is Ready and the LegacyESUpgrade condition has reason LegacyESUpgradeCompleted.
The legacy Elasticsearch PVC/PV protection is complete and has not been modified.
The PlatformLogForward condition LegacyKafkaDrained is present.

PVC/PV protection must pass. Any failure must stop the procedure; contact Alauda support. Do not modify or delete logagent or its dependencies.

Run the following two deletes in order without interruption; do not wait for ModuleInfo to disappear, check AppRelease, or perform any other check between them.

On the global cluster

Set CLUSTER to the workload cluster name as registered in the global cluster, then run:

set -euo pipefail

CLUSTER="<workload-cluster-name>"   # the workload cluster name as registered in the global cluster

# 1. Resolve the ModuleInfo to operate on. An ambiguous result is an error.
MODULE_INFOS="$(kubectl get moduleplugin logcenter \
  -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')"
MODULE_COUNT="$(printf '%s\n' "$MODULE_INFOS" | sed '/^$/d' | wc -l | tr -d ' ')"
if [ "$MODULE_COUNT" -gt 1 ]; then
  echo "More than one logcenter ModuleInfo exists for cluster $CLUSTER; stop and contact support." >&2
  exit 1
fi

if [ "$MODULE_COUNT" -eq 1 ]; then
  MODULE_INFO="$MODULE_INFOS"
  MODULE_VERSION="$(kubectl get moduleinfo "$MODULE_INFO" -o jsonpath='{.spec.version}')"
  MODULE_CONFIG="logcenter-${MODULE_VERSION}"

  # 2. Back up the current management objects before changing them.
  kubectl get moduleinfo "$MODULE_INFO" -o yaml > moduleinfo-logcenter.backup.yaml
  kubectl get moduleplugin logcenter -o yaml > moduleplugin-logcenter.backup.yaml
  kubectl get moduleconfig "$MODULE_CONFIG" -o yaml > moduleconfig-logcenter.backup.yaml

  # 3. Clear the discovery flags to bypass the logagent dependency check.
  kubectl patch moduleplugin logcenter --type=merge -p '{"spec":{"labelCluster":""}}'
  kubectl patch moduleconfig "$MODULE_CONFIG" --type=merge -p '{"spec":{"labelCluster":""}}'

  # 4. Return as soon as the delete request is accepted. Do not wait here.
  kubectl delete moduleinfo "$MODULE_INFO" --ignore-not-found --wait=false
else
  echo "No logcenter ModuleInfo found for cluster $CLUSTER; continuing to verify the install record."
fi

On the workload cluster

Switch the kubectl context immediately. Do not run any wait or verification first.

# 5. Delete the per-cluster install record before it can recreate ModuleInfo.
kubectl delete clusterplugininstance logcenter --ignore-not-found

Wait for the legacy data path to be removed

Switch back to the global cluster:

CLUSTER="<workload-cluster-name>"   # the workload cluster name as registered in the global cluster
MODULE_INFOS="$(kubectl get moduleplugin logcenter \
  -o jsonpath="{range .status.installed[?(@.cluster==\"$CLUSTER\")]}{.name}{'\n'}{end}" | sed '/^$/d')"
MODULE_INFO="$(printf '%s\n' "$MODULE_INFOS" | sed -n '1p')"

if [ -n "$MODULE_INFO" ]; then
  kubectl wait --for=delete "moduleinfo/$MODULE_INFO" --timeout=10m
else
  echo "No ModuleInfo remains for cluster $CLUSTER."
fi

Switch to the workload cluster and wait for the legacy AppRelease to be removed:

kubectl -n cpaas-system wait --for=delete "apprelease/logcenter" --timeout=10m

Wait 60 seconds, then verify that the resources stayed gone.

On the global cluster

kubectl get moduleinfo -l 'cpaas.io/module-name=logcenter,cpaas.io/cluster-name=<workload-cluster-name>' --ignore-not-found

On the workload cluster

kubectl get clusterplugininstance logcenter --ignore-not-found
kubectl -n cpaas-system get apprelease logcenter --ignore-not-found
kubectl -n cpaas-system get statefulset cpaas-elasticsearch --ignore-not-found

Also confirm that the following stable entry-point resources still exist:

kubectl -n cpaas-system get service razor
kubectl -n cpaas-system get ingress razor

TIP

Resource reappearance handling:

For ClusterPluginInstance/logcenter, delete it first, then delete any new ModuleInfo, and repeat the 60-second check.
Do not restore the cleared discovery fields manually.
AppRelease/logcenter removal must complete. Do not remove finalizers; contact Alauda support.
A global-cluster sequence failure before ModuleInfo deletion must be fixed and the global sequence rerun. Do not run the workload-cluster sequence for that attempt.

Retain the source volumes

Keep the protected legacy Elasticsearch PVCs and PVs until new-storage verification is complete and cleanup is approved. Do not remove their protection, and do not delete the PVCs or PVs.

Only the Elasticsearch PVCs and PVs are retained by this upgrade. Legacy Kafka and ZooKeeper volumes are not retained.

If you set up historical data migration, the worker starts copying now that the plugin is gone. Return to Historical Data Migration to watch it reach Succeeded and verify the data before these volumes are released.

#Upgrade from Log Storage for Elasticsearch

#TOC

#Before you start

#Install the new data path

#Decide on historical data

#Uninstall the legacy Elasticsearch plugin

#Retain the source volumes