Deploy as ACP logging storage

Alauda Container Platform can store platform logs in ClickHouse. The logging installation chapter assumes that a ClickHouse cluster already exists and asks you for its address, cluster name, database and account. This page builds exactly that cluster, in the recommended three-node high-availability layout, and ends with the values you type into the logging chapter.

The layout is one shard with three replicas. A shard is a slice of the data; a replica is a copy of a shard. One shard with three replicas means every log row is stored three times, on three different infra nodes, so the logs survive the loss of a node.

Do this page before section 1.2 of the logging installation chapter. For the other profiles the chapter describes (single-node evaluation, six or nine node scale-out), change shardsCount and replicasCount in the manifest and use the same two numbers in the PlatformLogForward.

Steps at a glance

  1. Check the prerequisites: operator, Keeper quorum, infra nodes, storage, free capacity.
  2. Choose two passwords and turn them into digests.
  3. Create the cluster secret.
  4. Fill in the manifest and apply it.
  5. Create the observability database.
  6. Verify, then hand the values over to the logging installation.

Names the logging stack expects

ItemValue used on this pageWhere the logging chapter reads it
Namespacecpaas-systemall connection Secrets live here
Instance namecpaas-clickhouseService clickhouse-cpaas-clickhouse
Endpointhttp://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123endpoint in platform-default-ch-conn
Cluster namereplicatedcluster in the Secret; used by ON CLUSTER
Databaseobservabilitydatabase in the Secret; must exist beforehand
Accountplatform-loggingusername / password in the Secret
Topology1 shard x 3 replicasshards / replicas in the PlatformLogForward

These are the ACP baseline defaults. If you change one of them, change it in the logging chapter too.

Prerequisites

You needDetails
The operator installedThe instance below is created through the operator.
A Keeper quorumA three-member ClickHouseKeeperInstallation named keeper in cpaas-system, created as described in Deploy a Keeper quorum. Replication cannot start without it. A quorum you built by hand with an earlier release of that guide also works; see the note under the manifest.
Three infra nodesLabelled and tainted node-role.kubernetes.io/infra. Each node holds one Keeper member and one ClickHouse replica.
A StorageClass for the data volumesThe manifest uses your cluster's default StorageClass. It does not have to be the one the Keeper quorum uses; the two workloads have separate volumes, so choose whatever suits each. Size the data volumes from the logging capacity planning guide, not from the example value below.
Free capacity on the three nodesKubernetes places a pod by comparing the pod's resource requests with what is still unreserved on a node, not with what the node is currently using. Check the "Allocated resources" section of kubectl describe node <node> for each infra node, and make sure the CPU and memory that are still free cover the requests in the manifest. kubectl top does not tell you this. On ACP the storage components (for example rook-ceph) reserve a large share of a small worker while using little of it; a pod that does not fit stays Pending with Insufficient memory.

Step 1: choose the passwords

The manifest creates two ClickHouse accounts:

  • admin: for you, used once in step 5 to create the database and later for inspection;
  • platform-logging: for the logging service, which connects with it on every write.

Choose a password for each and keep both. You type the admin password in step 5, and the platform-logging password into the logging chapter's connection Secret.

The manifest never contains a password itself. It contains the SHA-256 digest of each password (a 64-character fingerprint) in the lines admin/password_sha256_hex and platform-logging/password_sha256_hex. Produce each digest with this command, once per password:

  printf '%s' '<password>' | sha256sum | cut -d' ' -f1

The output is 64 hexadecimal characters. Copy it into the matching line of the manifest in step 3.

Step 2: create the cluster secret

The three replicas talk to each other to copy data, and that traffic is protected by a shared password called the cluster secret. On ACP the operator is not allowed to create Secrets, so create it yourself before the instance. The value is random and you never need to know it:

kubectl -n cpaas-system create secret generic cpaas-clickhouse-cluster-secret \
  --from-literal=secret="$(openssl rand -hex 24)"

Do not use the operator's secret: auto option for this; see the explanation under the manifest.

Step 3: fill in and apply the manifest

Copy the manifest below into a file named cpaas-clickhouse.yaml and replace the placeholders:

PlaceholderReplace withRequired
<admin-sha256-hex>The digest of the admin password from step 1.Yes
<platform-logging-sha256-hex>The digest of the platform-logging password from step 1.Yes
<storage-class>Only if you do not want the default StorageClass: remove the # in front of storageClassName and put the StorageClass name here.No

Also review the storage: 200Gi line and the CPU and memory values; the capacity planning guide tells you what your log volume needs.

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: cpaas-clickhouse
  namespace: cpaas-system
spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: data-volume
      podTemplate: ch-pod
      serviceTemplate: internal
  configuration:
    zookeeper:
      keeper:
        name: keeper
        namespace: cpaas-system
    clusters:
      - name: replicated
        secret:
          valueFrom:
            secretKeyRef:
              name: cpaas-clickhouse-cluster-secret
              key: secret
        layout:
          shardsCount: 1
          replicasCount: 3
    users:
      # One-time administration: creating the database, inspection.
      admin/password_sha256_hex: <admin-sha256-hex>
      admin/networks/ip:
        - "::/0"
      admin/access_management: "1"
      # The account the logging data path uses. Full rights on the target
      # database, plus the right to run ON CLUSTER statements.
      platform-logging/password_sha256_hex: <platform-logging-sha256-hex>
      platform-logging/networks/ip:
        - "::/0"
      platform-logging/grants/query:
        - "GRANT ALL ON observability.*"
        - "GRANT CLUSTER ON *.*"
  templates:
    serviceTemplates:
      - name: internal
        spec:
          type: ClusterIP
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
    podTemplates:
      - name: ch-pod
        podDistribution:
          - type: ShardAntiAffinity
        spec:
          tolerations:
            - key: node-role.kubernetes.io/infra
              operator: Exists
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: node-role.kubernetes.io/infra
                        operator: Exists
          securityContext:
            runAsUser: 101
            runAsGroup: 101
            fsGroup: 101
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: clickhouse
              resources:
                requests:
                  cpu: "2"
                  memory: 8Gi
                limits:
                  cpu: "4"
                  memory: 16Gi
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop:
                    - ALL
    volumeClaimTemplates:
      - name: data-volume
        spec:
          accessModes:
            - ReadWriteOnce
          # storageClassName: <storage-class>   # uncomment to use a StorageClass other than the default
          resources:
            requests:
              storage: 200Gi

What the pieces do:

  • zookeeper.keeper names the Keeper quorum from the prerequisites. The operator resolves it to the three members' addresses; replication and ON CLUSTER statements go through them. If you run a quorum you built by hand instead, replace the keeper block with nodes: and one - host: keeper-<n>.keeper-headless.cpaas-system entry per member; do not set both.
  • secret.valueFrom.secretKeyRef points at the Secret you created in step 2, so that the replicas authenticate each other. Do not replace it with secret: auto: on ACP the operator may only read Secrets, so the Secret that auto is supposed to generate never appears and every pod stays in CreateContainerConfigError.
  • users creates the two accounts. access_management lets admin manage users and grants. grants/query gives platform-logging its privileges on every replica without you running a GRANT statement; CLUSTER is needed because the logging schema migrations use ON CLUSTER. Note that ALL ON observability.* is broad: it also covers user, role and row-policy management inside that database. Narrow it to SELECT, INSERT, CREATE, ALTER, DROP ON observability.* if least privilege matters more than simplicity.
  • networks/ip: "::/0" lets other pods reach the two accounts. Without it, an account may only connect from inside its own pod, so every connection from the logging service would be refused. Narrow it to your pod network range if your policy requires it.
  • serviceTemplate: internal makes the instance's Service a ClusterIP. The default is a LoadBalancer, which never gets an address on clusters without a load-balancer provider, and the logging service only ever connects from inside the cluster.
  • podDistribution: ShardAntiAffinity together with the node affinity places the three replicas on three different infra nodes, one each.
  • No container image is given. The operator supplies the server image it was installed with.

Apply the file and wait for the three replicas:

kubectl -n cpaas-system apply -f cpaas-clickhouse.yaml
kubectl -n cpaas-system get statefulset -l clickhouse.altinity.com/chi=cpaas-clickhouse \
  -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas

Each replica is its own StatefulSet, so the second command lists three StatefulSets; wait until each shows READY 1. The operator starts them one at a time, so this takes a few minutes. kubectl -n cpaas-system get chi cpaas-clickhouse shows the instance status, which ends at Completed.

Step 4: create the database

The logging chapter requires the observability database to exist before the logging storage operator starts. Create it once, as admin, with the plain admin password from step 1. The command runs clickhouse-client inside the first replica's pod; ON CLUSTER 'replicated' creates the database on all three replicas:

kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- \
  clickhouse-client -u admin --password '<admin-password>' -q \
  "CREATE DATABASE IF NOT EXISTS observability ON CLUSTER 'replicated'"

The command prints nothing on success.

If it fails instead with Code: 999 ... Coordination::Exception: Cannot use any of provided ZooKeeper nodes. (KEEPER_EXCEPTION), the instance is fine but it could not open a session to any of the three Keeper members. This is a Keeper problem, not a database one. The usual reasons, in order:

  • The quorum is not where the manifest says. The manifest names the ClickHouseKeeperInstallation keeper in cpaas-system. Check kubectl -n cpaas-system get chk keeper: it must exist and report Completed.
  • The pods exist but never formed a quorum. The Keeper manifest needs three nodes with the node-role.kubernetes.io/infra label, one pod per node. On a cluster with fewer such nodes the pods stay Pending. Step 2 of Deploy a Keeper quorum shows how to check the leader and followers.
  • Port 2181 is blocked by a NetworkPolicy in cpaas-system.

Run step 2 of Deploy a Keeper quorum to tell these apart, then look at the server log:

kubectl -n cpaas-system get pods -l clickhouse-keeper.altinity.com/chk=keeper -o wide
kubectl -n cpaas-system logs chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse | grep -iE 'keeper|zookeeper' | tail -20

The log shows which of the three it is: a name that does not resolve, a connection refused, or a session rejected. Once one member reports zk_server_state leader, run the CREATE DATABASE command again; nothing needs to change on the instance.

Step 5: verify

Check the cluster membership and the connection to Keeper. Run each query with the same clickhouse-client command as in step 4, passing the query after -q:

SELECT cluster, shard_num, replica_num, host_name
FROM system.clusters WHERE cluster = 'replicated' ORDER BY replica_num;

SELECT name FROM system.zookeeper WHERE path = '/';

The first query returns three rows, one shard with replica numbers 1 to 3. The second returns a short list of names. If the second one fails, fix the Keeper quorum before continuing; see ON CLUSTER DDL fails with NO_ELEMENTS_IN_CONFIG.

Then prove that the platform-logging account can do what the logging migrations will do. Run the following from a shell in any pod in cpaas-system, with the plain platform-logging password. The first command creates a small replicated table on the cluster, the second drops it again:

curl -s "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123/" \
  -u "platform-logging:<platform-logging-password>" --data-binary \
  "CREATE TABLE observability.probe ON CLUSTER 'replicated' (ts DateTime) \
   ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/observability/probe', '{replica}') ORDER BY ts"
curl -s "http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123/" \
  -u "platform-logging:<platform-logging-password>" --data-binary \
  "DROP TABLE observability.probe ON CLUSTER 'replicated'"

Both commands print nothing on success. An error containing Not enough privileges names the missing privilege; add it to grants/query in the manifest and apply the manifest again.

Step 6: hand over to the logging installation

Continue with section 1.2 of the logging installation chapter using these values:

FieldValue
platform-default-ch-conn endpointhttp://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123
platform-default-ch-conn clusterreplicated
platform-default-ch-conn databaseobservability
platform-default-ch-conn username / passwordplatform-logging / the plain password you chose in step 1
platform-default-ch-conn tls.caomit; the endpoint is plain HTTP inside the cluster
PlatformLogForward externalStorage.shards / replicas1 / 3

The logging chapter's helper command reads the database name from spec.configuration.settings.default_database. This instance does not set that field, so the helper prints an empty database name; use observability.

Notes

  • The built-in default user is not for the logging service. It may only connect from inside its own pod, on purpose. Always give the logging service its own account, as above.
  • Placement settings belong in this manifest. The logging documentation says not to patch generated ClickHouseInstallation resources for placement; that applies to resources the logging operator creates itself. This instance is yours, and its pod template is the right place for tolerations and affinity.
  • Keeper and the database share the infra nodes. Size each node for one Keeper member plus one replica: the capacity planning guide for the replica, and a small fixed allowance for Keeper.
  • Backup and retention are handled by the logging documentation (TTL in the PlatformLogForward, backup how-tos); the operator provides neither.
  • Embedded Keeper is an alternative. The knowledge base article How to Deploy a Replicated ClickHouse Cluster with Embedded ClickHouse Keeper (KB260600102) runs Keeper inside the three replicas instead of as a separate quorum, and includes log-storage tuning such as TTLs on system tables. If you take that route, carry the names from the table at the top of this page, the platform-logging account and the placement settings into its manifest; the hand-over values stay the same.