Logging Docs
  • English

    • Historical Data Migration

    Migrate historical log, event, and audit data from the legacy Elasticsearch PVCs to the ClickHouse or OpenSearch storage used by the new data path.

    TOC

    Do you need this?PrerequisitesCreate the migration resourceWait for PrecaptureReadyWait for the migration to succeedVerify the migration

    Do you need this?

    Historical data migration is optional. Once the new data path is live (see Upgrade), it already carries every log, event, and audit record written from cutover onward. This migration only backfills the older data that still lives in the legacy Elasticsearch volumes.

    If you skip it, the data written before cutover stays only in the legacy Elasticsearch PVCs and is not queryable from ClickHouse or OpenSearch. Once those source PVCs and PVs are released (see Upgrade, "Retain the source volumes"), that historical data is gone for good. If your site does not need to query the pre-cutover history from the new storage, skip this chapter and release the source volumes once new-storage verification is complete.

    Prerequisites

    • The source is a Logging 4.3 Elasticsearch deployment. Migrating historical data to 4.4 is supported only from 4.3; other source versions are not supported yet.
    • The new data path is Ready (see Upgrade).
    • The source Elasticsearch PVCs and PVs still exist and their protection is intact.
    • The migration worker image for this release. It is delivered separately from the Logging plugin package — contact Alauda technical support to obtain the image that matches your release. An image that does not match the installed Logging plugin version can fail to exit after data verification. Set it as spec.image below.
    • The migrated time range is within the retention of the target tables.

    source.indexScope supports exact index names and globs and must match at least one index.

    Create LegacyESMigration while the legacy plugin is still installed, before you uninstall it. The operator then captures the source Elasticsearch volumes automatically, so you do not list them by hand.

    WARNING

    The source PVCs and PVs must still exist. Once the source PVCs are deleted, or their PVs are released and cleaned up, the historical data cannot be migrated.

    Create the migration resource

    Choose the tab for your target storage. The two definitions differ only in target.type, the connection Secret, and the options defaults.

    ClickHouse
    OpenSearch
    apiVersion: log.alauda.io/v1alpha1
kind: LegacyESMigration
metadata:
  name: platform-es-history                               # Stable migration name, used by the commands below
  namespace: cpaas-system
spec:
  image: <migration-worker-image>                         # Required; release-matched image from Alauda technical support, with registry and tag or digest
  source:
    indexScope:                                           # Required, selects the historical indices to migrate
      - "log-workload-*"
      - "log-platform-*"
      - "log-system-*"
      - "log-kubernetes-*"
      - "event-*"
      - "audit-*"
  target:
    type: clickhouse                                      # Must match the PlatformLogForward
    secretRef:
      name: platform-default-ch-conn                      # The same connection Secret as PlatformLogForward
      namespace: cpaas-system
  options:
    batchSize: 250                                        # Documents written per batch, 1~100000; default 250
    syncIntervalSeconds: 5                                # Minimum wait between batches; omit to use the default (1 second for ClickHouse)
    maxConcurrentJobs: 1                                  # Concurrent jobs; ClickHouse is always serial (1)

    Save the YAML as legacy-es-migration.yaml and apply it:

    kubectl apply -f legacy-es-migration.yaml
    TIP

    Fallback for an already-uninstalled plugin: if the legacy plugin was uninstalled before this resource was created, the operator can no longer capture the source volumes for you. Add source.pvcRefs listing every retained PVC that backed the legacy Elasticsearch StatefulSets (cpaas-elasticsearch, cpaas-elasticsearch-master); find them with kubectl -n cpaas-system get pvc before they are released. Prefer creating the resource before uninstalling, which avoids this.

    Wait for PrecaptureReady

    Watch the resource:

    kubectl -n cpaas-system get legacyesmigration platform-es-history -w

    Wait for the resource to reach PrecaptureReady. Data copy has not started at this phase.

    PrecaptureReady means the source is captured and the volumes are protected. Now uninstall the legacy plugin: follow Upgrade, "Uninstall the legacy Elasticsearch plugin", then return here. (If it was already uninstalled, the copy proceeds on its own.)

    WARNING

    If the phase is Blocked or Failed, do not delete or recreate the migration resource, Jobs, source volumes, or new-storage data. Read the condition message and contact Alauda support.

    Wait for the migration to succeed

    After the legacy plugin is uninstalled, the worker starts copying data. Watch the same resource until the phase is Succeeded:

    kubectl -n cpaas-system get legacyesmigration platform-es-history \
  -o jsonpath='{.status.phase}{"\n"}'

    Do not change the PlatformLogForward or its connection Secret while migration is running.

    Verify the migration

    Run the production log, event, and audit queries against ClickHouse or OpenSearch and confirm that the historical range is present.

    Keep the protected source Elasticsearch PVCs and PVs until verification is complete and cleanup is approved. Phase=Succeeded is not approval to release them.