Migrating the observability stack

Part of Phase 6. Mesh v1 shipped its own tracing pipeline — an OpenTelemetry v1 collector and a Jaeger 1.x backend — and both were removed with it in Phase 5. This document installs the current stack, keeps the traces that were collected before the migration queryable, and restores the telemetry behavior of your applications.

Tracing is interrupted until this document is finished

Spans are dropped between the moment the Mesh v1 collector is deleted in Phase 5 and the moment the mesh is repointed at the new collector here. Metrics and access logs are not affected. Plan this part for a window where a gap in traces is acceptable.

The OpenTelemetry v1 and v2 Operators own the same custom resource definitions, so v2 can only be installed after Mesh v1 — which brought v1 — is gone.

Before you begin

Load the variables of the migration, source "${BACKUP_DIR}/vars.env", and collect what Phase 0 recorded:

${BACKUP_DIR}/v1-jaeger.txt — the Elasticsearch index prefix and the retention of the Mesh v1 tracing backend.

${BACKUP_DIR}/v1-jaeger-es-credentials.yaml — the secret the Mesh v1 backend authenticated to Elasticsearch with. The new backend is deployed against the same Elasticsearch cluster and needs the same credentials:

ES_SECRET=$(kubectl create -f "${BACKUP_DIR}/v1-jaeger-es-credentials.yaml" --dry-run=client -o json)
export ES_ENDPOINT=$(sed -n 's/.*url=\([^ ]*\).*/\1/p' "${BACKUP_DIR}/v1-jaeger.txt")
export ES_USER=$(jq -r '.data.ES_USERNAME' <<<"${ES_SECRET}" | base64 -d)
export ES_PASS=$(jq -r '.data.ES_PASSWORD' <<<"${ES_SECRET}" | base64 -d)
echo "${ES_ENDPOINT} user=${ES_USER}"

These are the ES_ENDPOINT, ES_USER and ES_PASS that the installation steps below ask for. The keys in the Mesh v1 secret are named ES_USERNAME and ES_PASSWORD.

${BACKUP_DIR}/instrumentations.yaml — the Mesh v1 Instrumentation resource, which holds the exporter endpoint, the sampler and the service name pattern your applications report.
${BACKUP_DIR}/v1-instrumented-pods.txt — the workloads that use Java auto-instrumentation, one line each: namespace, workload, application container, microservice name and reported service name.
${BACKUP_DIR}/v1-injected-env.txt — the variables the Mesh v1 webhooks added to the application containers.

Keep one trace from before the migration at hand — its trace ID, its service name and its span count. It is the reference for the verification at the end.

Take over the retention of the existing indices

The Mesh v1 tracing backend ran a cron job that deleted its indices once they were older than the configured retention. That job was owned by the Mesh v1 tracing resource and was deleted with it, so nothing expires those indices now.

Apply a delete-only index lifecycle policy to the existing index family, following Take over retention of the legacy indices, with the values from ${BACKUP_DIR}/v1-jaeger.txt:

LEGACY_ES_INDEX_PREFIX — the es.index-prefix of the Mesh v1 tracing backend, asm-mesh-<meshID> by default.
The age at which indices are deleted — the numberOfDays of its index cleaner, 7 by default.

Install the new tracing stack

Follow the Prerequisites of Configuring distributed tracing platform with Service Mesh to install the Alauda Container Platform Distributed Tracing (previously Alauda Build of Jaeger v2) cluster plugin, the Alauda Container Platform Application Observability (previously Alauda Build of OpenTelemetry v2) Operator, a Jaeger v2 instance and the v2 collector in the jaeger-system namespace.

Three choices matter for a migrated mesh:

Use the same Elasticsearch cluster as the Mesh v1 backend, otherwise the existing traces cannot be served by the new one.
Choose the index prefix per mesh, not per cluster. A mesh that spans several clusters writes into one index family so that a trace which crosses clusters is assembled from one place. The steps that act on Elasticsearch — the lifecycle policy and the index initialization — then belong to the mesh, not to a cluster: run them once, with the prefix of the mesh, and skip them on every other cluster. The Jaeger instance of any cluster serves the traces of the whole mesh afterwards. Keep the default prefix and never reuse the Mesh v1 one: the steps that take over the retention of the Mesh v1 indices and later retire them select indices by <prefix>-jaeger-*, so a shared prefix applies them to the new index family as well.
Give every cluster its own collector. The collectors of all clusters share one service name, and the step that repoints the mesh declares that service cluster-local, so each proxy exports to the collector of its own cluster, see Keep trace reporting inside each cluster.

Stop after the prerequisites. The mesh-side configuration already exists from the migration, and the next step updates it.

Attach the existing indices to the new read aliases

Attaching the Mesh v1 indices to the read aliases of the new index family makes traces from before the migration queryable in the new Jaeger UI. Nothing is copied or reindexed, and the old indices stay read-only.

Follow Attach the legacy indices to the new read aliases, with LEGACY_ES_INDEX_PREFIX set to the Mesh v1 prefix and JAEGER_ES_INDEX_PREFIX to the prefix of the new instance. In a multi-cluster mesh, attach them once for the whole mesh.

NOTE

The Mesh v1 backend writes only span and service indices. The attach step also covers dependency and sampling indices, and reports index_not_found_exception for those two. That is expected and does not affect the result.

Repoint the mesh at the new collector

The mesh sends spans to a provider defined in the Istio resource. Replace the Mesh v1 provider with one that points at the new collector, and update the mesh-wide Telemetry resource to use it.

SETTINGS=$(kubectl get istio default -o json \
  | jq -c '[(.spec.values.meshConfig.serviceSettings // [])[]
            | select((.hosts // []) | index("otel-collector.jaeger-system.svc.cluster.local") | not)]
           + [{settings: {clusterLocal: true}, hosts: ["otel-collector.jaeger-system.svc.cluster.local"]}]')
kubectl patch istio default --type=merge -p "$(cat <<EOF
{"spec":{"values":{"meshConfig":{
  "extensionProviders":[
    {"name":"otel","opentelemetry":{"port":4317,"service":"otel-collector.jaeger-system.svc.cluster.local"}}
  ],
  "serviceSettings":${SETTINGS}}}}}
EOF
)"
kubectl wait --for condition=Ready istio/default --timeout=5m

kubectl -n istio-system patch telemetry asm-default --type=merge \
  -p "{\"spec\":{\"tracing\":[{\"providers\":[{\"name\":\"otel\"}],\"randomSamplingPercentage\":${V1_TRACE_SAMPLING}}]}}"

Run the two commands one after the other. Between them the mesh-wide Telemetry resource still names zipkin, a provider the Istio resource no longer defines. Spans are dropped for those few seconds — which is the state the mesh has been in since Phase 5 anyway — and nothing else is affected.
serviceSettings declares the collector service cluster-local, which keeps every proxy exporting to the collector of its own cluster, see Configuring distributed tracing platform with Service Mesh. The setting changes nothing in a single-cluster mesh. A merge patch replaces both lists as a whole, so the command reads the service settings the mesh already has — Phase 2 copied them from Mesh v1 — and writes them back with the collector entry appended. The extensionProviders list is replaced: if your mesh defines other providers, edit the resource with kubectl edit istio default instead.
V1_TRACE_SAMPLING is the sampling percentage the mesh used before the migration, recorded in ${BACKUP_DIR}/v1-mesh-settings.json. Only the provider changes here; how much the mesh traces stays what it was.

If the Istio resource restricts discovery with discoverySelectors, label the jaeger-system namespace so that the control plane can resolve the collector service.

If a Sidecar resource restricts egress.hosts, add jaeger-system/* to it, otherwise its proxies drop every span without an error, see Using Sidecar import.

Mesh spans now reach the new backend. Check the Jaeger UI before you continue.

Migrate Java auto-instrumentation

Applications that are instrumented by the OpenTelemetry Operator need three things restored: the injection annotations, an Instrumentation resource, and the environment they reported before the migration.

In Mesh v1, a platform webhook added the injection annotations and the identity variables to each pod, and the Instrumentation resource in cpaas-system referenced them. Neither exists now, so both the annotations and the values must be written where they survive on their own.

Migrate one workload at a time

Every step writes values that belong to one workload into that workload's template. Set MESH_NAMESPACE, WORKLOAD, APP_CONTAINER, MS_NAME and SERVICE_NAME for one workload, run the steps for it — service name, variables, annotations, rollout — and check the result before you set them for the next one. The agent image is chosen once for the cluster and the Instrumentation resource once per namespace; everything after them is per workload.

Set the variables of the workload

Every step below acts on the workload these variables name. Take their values from the line of that workload in ${BACKUP_DIR}/v1-instrumented-pods.txt, recorded in Phase 0 as <namespace> <workload> <application-container> <microservice> <service-name>. Replace <namespace> and <workload> with your own values:

export MESH_NAMESPACE=<namespace> WORKLOAD=<workload>
read -r APP_CONTAINER MS_NAME SERVICE_NAME <<<"$(awk -v ns="${MESH_NAMESPACE}" -v w="${WORKLOAD}" \
  '$1 == ns && $2 == w {print $3, $4, $5}' "${BACKUP_DIR}/v1-instrumented-pods.txt")"
export APP_CONTAINER MS_NAME SERVICE_NAME
echo "container=${APP_CONTAINER} msname=${MS_NAME} service=${SERVICE_NAME}"

No value may be empty or read -. APP_CONTAINER is the container the agent is injected into and the container the variables belong to, MS_NAME the microservice the workload belonged to in Mesh v1 — which is not always the name of the workload — and SERVICE_NAME the name the application reported before the migration. A wrong or empty value is accepted and reported like any other: the identity your applications report changes without anything failing.

Prepare the Java agent image

Mesh v1 shipped its own Java agent image. Mesh v2 does not, so choose an image and make it reachable from the cluster, as described in Preparing the Java agent image. Replace <java-auto-instrumentation-image> with your own value.

export JAVA_AGENT_IMAGE=<java-auto-instrumentation-image>

Create the `Instrumentation` resource

Recreate the Mesh v1 configuration with the values from ${BACKUP_DIR}/instrumentations.yaml, translated for the v2 agent. One resource per namespace serves every instrumented workload in it:

cat <<EOF | kubectl apply -f -
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java
  namespace: ${MESH_NAMESPACE}
spec:
  exporter:
    endpoint: http://otel-collector.jaeger-system.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  env:
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    - name: OTEL_METRICS_EXPORTER
      value: otlp
    - name: OTEL_LOGS_EXPORTER
      value: none
  java:
    image: ${JAVA_AGENT_IMAGE}
EOF

The Java agent 2.x exports with http/protobuf by default, where 1.x used gRPC. Either point the endpoint at the HTTP port 4318, as above, or keep port 4317 and add OTEL_EXPORTER_OTLP_PROTOCOL: grpc to spec.env.
Mesh v1 used a sampler of its own that sampled every root span and followed the decision of the caller for the rest. parentbased_traceidratio with argument: "1.0" is the equivalent built into the agent. Lower the argument if you sampled less than 100 percent.
Mesh v1 also exported metrics in the Prometheus format from the agent. The v2 agent reports them through OTLP; scrape them from the collector instead of from the application pods.
The 2.x agent exports logs by default, which Mesh v1 did not. A collector without a logs pipeline answers those requests with 404 and the agent logs a warning on every export. OTEL_LOGS_EXPORTER: none keeps the behavior your applications had. Set it to otlp only after you add a logs pipeline to the collector.

Keep the service names of your applications

Mesh v1 built the reported service name from the microservice name and the namespace, <microservice>.<namespace>, using variables that a Mesh v1 webhook injected. Without that webhook, the OpenTelemetry Operator falls back to the name of the workload, which changes how your applications appear in Jaeger, in dashboards and in alerts.

Set the name explicitly on the application container, with the SERVICE_NAME of the workload:

kubectl -n "${MESH_NAMESPACE}" patch deploy "${WORKLOAD}" -p "$(cat <<EOF
{"spec":{"template":{"spec":{"containers":[
  {"name":"${APP_CONTAINER}","env":[{"name":"OTEL_SERVICE_NAME","value":"${SERVICE_NAME}"}]}
]}}}}
EOF
)"

This is a strategic merge patch: it matches the container by its name and the variable by its name. Only that one variable is written, the env list is created if the container has none, and running the step again updates the value instead of appending a second definition of the same variable. Setting the variable on the application container keeps the name stable no matter which component injects the agent.

Restore the variables your applications read

OTEL_SERVICE_NAME is not the only variable the Mesh v1 webhook added to the application container. ${BACKUP_DIR}/v1-injected-env.txt from Phase 0 lists the rest — ASM_MS_NAME, ASM_POD_NAMESPACE, ASM_CLUSTER_NAME and ASM_CONTAINER_NAME. An application that reads one of them keeps running after the migration, reads an empty value, and reports nothing: a field that used to hold a name such as frontend-ms.app-all holds null.null instead.

Add the ones your applications depend on to the workload template, as literal values, with the MS_NAME of the workload:

kubectl -n "${MESH_NAMESPACE}" patch deploy "${WORKLOAD}" -p "$(cat <<EOF
{"spec":{"template":{"spec":{"containers":[
  {"name":"${APP_CONTAINER}","env":[
    {"name":"ASM_MS_NAME","value":"${MS_NAME}"},
    {"name":"ASM_POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}
  ]}
]}}}}
EOF
)"

Then check what the application container is left with:

kubectl -n "${MESH_NAMESPACE}" get deploy "${WORKLOAD}" -o json \
  | jq -r --arg c "${APP_CONTAINER}" '[.spec.template.spec.containers[] | select(.name == $c) | (.env // [])[]
      | select(.name | test("^(OTEL_|ASM_)"))]
    | group_by(.name)[]
    | "\(.[0].name) \(length)x \([.[] | .value // .valueFrom.fieldRef.fieldPath // ""] | join(" | "))"'

Example output

ASM_MS_NAME 1x frontend-ms
ASM_POD_NAMESPACE 1x metadata.namespace
OTEL_SERVICE_NAME 1x frontend-ms.app-all

Every variable is listed once and with a value. The patches above update a variable in place, so an empty value only means the step ran before its variables were set: set them and run the step again. A variable listed 2x was already defined twice in the workload template before this step — the container reads the last definition, the API server warns hides previous definition of "..." on every later change to the workload, and a later apply may drop it. Remove the extra entries with kubectl -n "${MESH_NAMESPACE}" edit deploy "${WORKLOAD}", keeping the definition that holds the value you want.

Skipping this step is safe as long as no application reads these variables. Confirm that with the application owners rather than assuming it: nothing in the cluster reports a variable that resolves to an empty string.

Turn injection back on

Mesh v1 added the injection annotations to the pod. Write them into the workload template instead:

kubectl -n "${MESH_NAMESPACE}" patch deploy "${WORKLOAD}" --type=merge -p "$(cat <<EOF
{"spec":{"template":{"metadata":{"annotations":{
  "instrumentation.opentelemetry.io/inject-java": "java",
  "instrumentation.opentelemetry.io/java-container-names": "${APP_CONTAINER}"
}}}}}
EOF
)"
kubectl -n "${MESH_NAMESPACE}" rollout status "deploy/${WORKLOAD}" --timeout=300s

For a namespace where every workload is a Java application, annotate the namespace instead of each workload:

kubectl annotate ns "${MESH_NAMESPACE}" instrumentation.opentelemetry.io/inject-java=java --overwrite
kubectl -n "${MESH_NAMESPACE}" rollout restart deploy

Name the container that is instrumented

Without java-container-names, the agent is injected into the first container of the pod — which in a mesh workload can be the proxy rather than your application. Name the container explicitly on every instrumented workload, with instrumentation.opentelemetry.io/java-container-names as above, or with instrumentation.opentelemetry.io/container-names when the pod runs applications in more than one language. An agent injected into the proxy container instruments nothing and the workload disappears from the traces.

Connect Kiali to Jaeger v2

Set the tracing configuration of the Kiali resource to the new Jaeger query service, as described in Integrating distributed tracing platform with Alauda Build of Kiali. Phase 1 installed Kiali without one.

In a multi-cluster mesh, point Kiali at the Jaeger instance of its own cluster and set the query scope of the tracing section to the mesh, istio.mesh_id: <mesh-id>, as the monitoring query scope already is. One index family holds the whole mesh, so that instance answers for every cluster.

Verification

kubectl -n jaeger-system get pod
kubectl -n "${MESH_NAMESPACE}" get pod -o json | jq -r '.items[]
  | select(any(.spec.initContainers[]?; .name == "opentelemetry-auto-instrumentation-java"))
  | .metadata.name as $pod
  | .spec.containers[]
  | select(any(.env[]?; .name == "JAVA_TOOL_OPTIONS"))
  | "\($pod) \(.name) \([.env[]? | select(.name=="OTEL_SERVICE_NAME") | .value] | join(","))"'

The second command lists, for every pod the agent was injected into, the container it landed in and the name that container reports.

Traces produced after the migration appear in the Jaeger v2 UI, for the mesh proxies and for the instrumented applications.
The trace you kept from before the migration is still found by its trace ID, with the same span count and the same service name. That trace is stored, not regenerated, so it is unchanged. A new trace of the same request can hold fewer spans than it did before the migration — the 2.x Java agent instruments a different set of libraries than the 1.x agent Mesh v1 shipped. Compare an old trace with an old trace, not with a new one.
The service names of the instrumented applications are the ones they reported before the migration, and the container the agent landed in is the application container, never istio-proxy.
Kiali shows traces on the service detail pages.
Application metrics that were collected from the Java agent are available again, now through the collector.

Return to Finalizing the migration for the final acceptance of the migration.

#Migrating the observability stack

#TOC

#Before you begin

#Take over the retention of the existing indices

#Install the new tracing stack

#Attach the existing indices to the new read aliases

#Repoint the mesh at the new collector

#Migrate Java auto-instrumentation

#Set the variables of the workload

#Prepare the Java agent image

#Create the Instrumentation resource

#Keep the service names of your applications

#Restore the variables your applications read

#Turn injection back on

#Connect Kiali to Jaeger v2

#Verification