Alauda Service Mesh v2 Docs
  • English

    • Installing Kiali in a multi-cluster mesh

    Deploy the Kiali server on one cluster of the mesh, grant it read access to every other cluster, and create the resources that it authenticates with on those clusters.

    NOTE

    In this procedure, CLUSTER1 is the East cluster and hosts the Kiali server, and CLUSTER2 is the West cluster and contributes only remote cluster resources.

    You can adapt these instructions for a mesh spanning more than two clusters by repeating the West steps on each additional cluster.

    TOC

    PrerequisitesProcedureVerificationRemoving a cluster from KialiCleaning up KialiAdditional resources

    Prerequisites

    • You have deployed a multi-cluster mesh by following either Installing a multi-primary multi-network mesh or Installing a primary-remote multi-network mesh.

    • You have installed the Alauda Build of Kiali Operator on every cluster in the mesh, including the clusters where no Kiali server is deployed. See Installing Alauda Build of Kiali.

    • Every cluster in the mesh collects metrics with ACP Monitoring with VictoriaMetrics, and all of them report to the same VictoriaMetrics Center. See ACP Monitoring with VictoriaMetrics.

      Kiali reads the metrics of the whole mesh from a single endpoint. Read the address of the shared endpoint by running the following command:

      kubectl get feature monitoring -o jsonpath='{.spec.accessInfo.database.address}'

      Every cluster that reports to the same VictoriaMetrics Center returns this address. It is the value you set in spec.external_services.prometheus.url of the Kiali resource.

    • You have configured metrics collection by following Configuring Monitoring with Service Mesh. Kiali distinguishes clusters by the source_cluster and destination_cluster labels that Istio adds to these metrics, and it filters them by the mesh_id label, so where each resource goes matters:

      • Create the ServiceMonitor and PodMonitor objects on every cluster in the mesh, because monitoring scrapes each cluster separately.
      • Create the Telemetry resource, including its mesh_id tag override, on every cluster that runs a control plane. In a primary-remote mesh, the primary control plane also configures the sidecars of the remote clusters, and applying the resource on a remote cluster is rejected by the validation.istio.io webhook, since the istiod service there has no endpoints.

    • The Kiali deployment namespace and instance name are identical on all clusters. If you change either default, apply the same value to spec.deployment.namespace and spec.deployment.instance_name in the Kiali resource on every cluster. This procedure uses the defaults, istio-system and kiali.

    • You know the meshID of the mesh and the name that Istio uses for each cluster. Later steps pass the cluster name to the --remote-cluster-name option and use it as the key of the remote cluster secret, so it must match exactly. Where to read it depends on the topology:

      • In a multi-primary mesh, every cluster has its own Istio resource carrying spec.values.global.meshID and spec.values.global.multiCluster.clusterName.
      • In a primary-remote mesh, only the primary cluster's Istio resource carries those fields. The name of a remote cluster appears in that cluster's own Istio resource as part of spec.values.istiodRemote.injectionPath, which has the form /inject/cluster/<cluster-name>/net/<network>.

      This procedure uses mesh1, cluster1, and cluster2.

    • You have kubectl configured with a context for each cluster. The following environment variables are used throughout:

      export CTX_CLUSTER1=<your cluster1 context>
export CTX_CLUSTER2=<your cluster2 context>

    Procedure

    Deploy the Kiali server on the East cluster

    Follow Configuring Monitoring with Kiali to create the Kiali resource on the East cluster, then add the multi-cluster specific setting shown below.

    Because the monitoring store also holds metrics produced outside the mesh, restrict Kiali to this mesh by setting a query scope:

    kiali-query-scope.yaml
    spec:
  external_services:
    prometheus:
      query_scope:
        mesh_id: mesh1
    1. Must match .spec.values.global.meshID in the Istio resource of every cluster in the mesh. Kiali appends this label to every metrics query, so series that do not carry it are ignored.

    Apply the setting by running the following command:

    kubectl --context "${CTX_CLUSTER1}" -n istio-system patch kiali kiali \
  --type merge -p "$(cat kiali-query-scope.yaml)"

    Create a Kiali resource on the West cluster that produces only remote cluster resources

    Create a YAML file named kiali-remote.yaml with the following content:

    kiali-remote.yaml
    apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  deployment:
    instance_name: kiali
    remote_cluster_resources_only: true
    1. Must be the same namespace that holds the Kiali server on the East cluster.
    2. Must be the same instance name that the Kiali server uses on the East cluster. The Operator derives the service account name from it, so a mismatch makes the server unable to authenticate.
    3. Instructs the Operator to create only the resources that the remote Kiali server needs — a kiali-service-account service account and a kiali-viewer cluster role with its binding. No Kiali server, service, or ingress is created on this cluster.

    Apply the YAML file on the West cluster by running the following command:

    kubectl --context "${CTX_CLUSTER2}" apply -f kiali-remote.yaml

    Wait for the resource to reconcile:

    kubectl --context "${CTX_CLUSTER2}" wait --for=condition=Successful \
  kialis/kiali -n istio-system --timeout=3m

    Create a long-lived service account token on the West cluster

    The Kiali server authenticates to remote clusters with a service account token that does not expire. Kubernetes no longer generates such a token automatically, so create it explicitly.

    Create a YAML file named kiali-svc-account-token.yaml with the following content:

    kiali-svc-account-token.yaml
    apiVersion: v1
kind: Secret
metadata:
  name: kiali-service-account
  namespace: istio-system
  annotations:
    kubernetes.io/service-account.name: kiali-service-account
type: kubernetes.io/service-account-token
    1. Binds the token to the service account that step 2 created. For a non-default instance name, use <instance_name>-service-account.
    2. Makes the token controller populate the token key of this secret.

    Apply the YAML file on the West cluster by running the following command:

    kubectl --context "${CTX_CLUSTER2}" apply -f kiali-svc-account-token.yaml

    Obtain a kubeconfig for the West cluster that uses the platform API endpoint

    Download the kubeconfig of the West cluster from the Alauda Container Platform web console, or retrieve it from the platform API:

    curl -sS -k -H "Authorization: Bearer <your-platform-api-token>" \
  "<platform-url>/auth/v1/clusters/<west-cluster-name>/kubeconfig" \
  -o west-kubeconfig.json
    WARNING

    The platform API returns the kubeconfig as JSON, and it carries certificate-authority-data as a PEM block instead of base64. Saving the response unchanged and pointing kubectl at it fails with illegal base64 data at input byte 0. Encode that field before use, or download the kubeconfig from the web console instead.

    The following commands encode the field, convert the file to YAML, and inline every credential so that the result is easy to merge later. The first command is safe to run more than once, because it only touches values that are still PEM:

    jq '.clusters[].cluster."certificate-authority-data" |=
  (if startswith("-----BEGIN") then @base64 else . end)' \
  west-kubeconfig.json > west-kubeconfig-fixed.json

KUBECONFIG=west-kubeconfig-fixed.json \
  kubectl config view --flatten --raw > west-kubeconfig.yaml

    The downloaded kubeconfig contains two contexts. Use the proxy-connect context, which is the default, and whose server address has the following form:

    <platform-url>/kubernetes/<west-cluster-name>
    DANGER

    The remote cluster secret must address the remote cluster through this platform endpoint, not through the cluster API server address used by the direct-connect context.

    The platform endpoint accepts both the Kiali service account token and the platform tokens of logged-in users, whereas a cluster API server accepts only the former. With a direct-connect address, the Kiali server still reports the cluster as accessible, but every user-scoped request fails and the remote cluster contributes no namespaces to the Kiali console. For the symptoms, see The remote cluster is reported as accessible, but its namespaces are missing from the console.

    Add the proxy-connect context to the kubeconfig that also holds your East cluster context, so that a single kubeconfig can reach both clusters, then export its name:

    export CTX_CLUSTER2_PROXY=<your cluster2 proxy-connect context>
    NOTE

    Every cluster's downloaded kubeconfig names its contexts proxy-connect and direct-connect. When you merge the files of several clusters, rename the contexts first so that they do not collide.

    Create a remote cluster secret on the East cluster

    The remote cluster secret holds a kubeconfig that lets the Kiali server on the East cluster read the West cluster. Generate it with the kiali-prepare-remote-cluster.sh script.

    Download the script and make it executable by running the following commands:

    curl -L -o kiali-prepare-remote-cluster.sh \
  https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh
chmod +x kiali-prepare-remote-cluster.sh

    Run the script by using the following command:

    ./kiali-prepare-remote-cluster.sh \
  --kiali-cluster-context "${CTX_CLUSTER1}" \
  --remote-cluster-context "${CTX_CLUSTER2_PROXY}" \
  --remote-cluster-name cluster2 \
  --remote-cluster-namespace istio-system \
  --kiali-resource-name kiali-service-account \
  --process-remote-resources false \
  --process-kiali-secret true \
  --view-only false

    The options have the following effects:

    • --remote-cluster-name: the name that Istio uses for the remote cluster, taken from .spec.values.global.multiCluster.clusterName. Always set it explicitly, because the name the script would otherwise derive from the context may contain characters that are invalid in a secret key.
    • --remote-cluster-namespace: the namespace on the remote cluster that holds the service account and its token.
    • --kiali-resource-name: the service account that the Operator created in step 2.
    • --process-remote-resources false: skips creating the service account and roles, because the Kiali resource already manages them.
    • --process-kiali-secret true: creates the secret in the Kiali deployment namespace on the East cluster.
    • --view-only false: grants Kiali write access to the remote cluster. Set it to true to restrict Kiali to read-only operations on that cluster.

    On success the script reports the server address it read from the context and the name of the secret it created:

    INFO: remote_cluster_server_url=<platform-url>/kubernetes/<west-cluster-name>
secret/kiali-remote-cluster-secret-cluster2 created
    TIP

    Use the --help option to display all the options that the script supports, including --delete true for removing what it created.

    The Operator finds the generated secret by its label. For the other ways to register a remote cluster secret, see Registering remote cluster secrets.

    Trigger reconciliation of the Kiali server

    The Operator mounts remote cluster secrets into the Kiali server pod when it reconciles the Kiali resource. Annotate the resource to start a reconciliation:

    kubectl --context "${CTX_CLUSTER1}" annotate kiali kiali -n istio-system \
  --overwrite kiali.io/reconcile="$(date)"

    Wait for the resource and the server to become ready by running the following commands:

    kubectl --context "${CTX_CLUSTER1}" wait --for=condition=Successful \
  kialis/kiali -n istio-system --timeout=3m
kubectl --context "${CTX_CLUSTER1}" rollout status deployment/kiali -n istio-system

    Log in to the Kiali console

    Open the Kiali console of the East cluster at <platform-url>/clusters/<east-cluster-name>/kiali.

    Verification

    1. Confirm that the Kiali server discovered every cluster by running the following command:

      kubectl --context "${CTX_CLUSTER1}" logs -n istio-system deployment/kiali | grep "Discovered cluster"

      Each cluster in the mesh appears with Accessible=true, and exactly one of them with IsKialiHome=true:

      INF Discovered cluster: Name=[cluster1], Accessible=true, IsKialiHome=true, ApiEndpoint=[https://100.4.0.1:443], SecretName=[]
INF Discovered cluster: Name=[cluster2], Accessible=true, IsKialiHome=false, ApiEndpoint=[<platform-url>/kubernetes/<west-cluster-name>], SecretName=[kiali-remote-cluster-secret-cluster2]

    2. In the Kiali console, navigate to Namespaces and confirm that namespaces from every cluster are listed. The cluster name is shown on each namespace card.

    3. Navigate to Mesh and confirm that every cluster in the mesh appears. How the control plane is drawn depends on the topology:

      • In a multi-primary mesh, each cluster shows its own control plane node next to its data plane node.
      • In a primary-remote mesh, only the primary cluster shows a control plane node. Each remote cluster shows a data plane node that the primary's control plane connects to.

    4. Navigate to Traffic Graph, select a namespace that exists on more than one cluster, and confirm that the graph draws the clusters separately with edges between them. Enable Display > Cluster Boxes to group the nodes by cluster.

    Removing a cluster from Kiali

    Procedure

    1. Delete the remote cluster secret from the Kiali deployment namespace on the cluster that hosts the Kiali server:

      kubectl --context "${CTX_CLUSTER1}" -n istio-system \
  delete secret kiali-remote-cluster-secret-cluster2
      NOTE

      If you created the secret with kiali-prepare-remote-cluster.sh, run the script again with the same options plus --delete true.

    2. Remove the service account and roles from the remote cluster by deleting its Kiali resource:

      kubectl --context "${CTX_CLUSTER2}" -n istio-system delete kiali kiali
      NOTE

      The long-lived token secret disappears with the service account that it belongs to, so there is nothing else to delete.

    3. Make the Kiali server stop looking for the deleted secret:

      • With label based auto-detection, trigger a reconciliation:

        kubectl --context "${CTX_CLUSTER1}" annotate kiali kiali -n istio-system \
  --overwrite kiali.io/reconcile="$(date)"

      • With explicit declaration, remove the corresponding entry from spec.clustering.clusters.

      • With one combined secret, no action is needed. The Operator notices the change and redeploys the Kiali server.

    4. If the removed cluster had a remote cluster secret of its own, recreate the Kiali server deployment so that it stops mounting the deleted secret:

      kubectl --context "${CTX_CLUSTER1}" -n istio-system delete deployment kiali
kubectl --context "${CTX_CLUSTER1}" annotate kiali kiali -n istio-system \
  --overwrite kiali.io/reconcile="$(date)"
kubectl --context "${CTX_CLUSTER1}" wait --for=condition=Successful \
  kialis/kiali -n istio-system --timeout=3m
kubectl --context "${CTX_CLUSTER1}" rollout status deployment/kiali -n istio-system

      Confirm that the removed cluster no longer has a volume:

      kubectl --context "${CTX_CLUSTER1}" get deployment kiali -n istio-system \
  -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{"\n"}{end}'
      WARNING

      Do not skip this step. Reconciliation on its own does not drop the volume of a deleted remote cluster secret from the Kiali server deployment. The running pod keeps serving, but the next restart leaves it stuck in ContainerCreating. This step is unnecessary only when all remote clusters share one combined secret, because the volume then refers to a secret that still exists.

    Cleaning up Kiali

    Remove Kiali from the whole mesh by unregistering the remote clusters first and deleting the Kiali server last. In the other order, the Kiali server is gone before the remote clusters are cleaned up, and their unused service accounts and roles stay behind.

    Procedure

    1. Remove every remote cluster from Kiali by following Removing a cluster from Kiali for each of them.

      NOTE

      The Kiali server is deleted in the next step, so you can skip step 4, which recreates its deployment.

    2. Delete the Kiali resource on the East cluster:

      kubectl --context "${CTX_CLUSTER1}" -n istio-system delete kiali kiali

      Confirm that the Operator removed the server:

      kubectl --context "${CTX_CLUSTER1}" -n istio-system get deployment kiali

      Example output

      Error from server (NotFound): deployments.apps "kiali" not found
    NOTE

    The Alauda Build of Kiali Operator stays installed on every cluster. To remove it and the Kiali custom resource definitions as well, see Uninstalling Alauda Build of Kiali.

    Additional resources