Installation
This chapter installs the logging stack on a fresh platform. If you are moving an existing ACP 4.3 install to 4.4 instead, see Upgrade from ACP 4.3 to 4.4.
The stack has three parts and installs in this order:
- Log storage — deploy either ClickHouse or OpenSearch, plus Kafka, then install the Log Storage operator and hand the endpoints to it. This runs on every storage cluster.
- Log Collector — install the collector plugin on every cluster whose logs, events, or audit records must be collected, and point it at the storage cluster.
- Log Essentials — install once on the
globalcluster; publishes the query APIs used by the console.
At the end, the ACP console can query the collected data in Operations Center > Logs.
TOC
What you install1. Install log storage1.1 Prepare the external components1.1.1 Kafka1.1.2 ClickHouse or OpenSearch1.2 Install the Log Storage operatorConsoleVerify1.3 Create the connection Secrets1.3.1 Kafka Secret1.3.2 ClickHouse Secret1.3.3 OpenSearch Secret1.4 Create the PlatformLogForward2. Install Log CollectorConsoleYAMLVerify3. Install Log EssentialsConsoleYAMLVerify4. Verify end-to-endNext stepsWhat you install
The stack ships as three cluster components — two plugins (Log Essentials and Log Collector) and one operator (Log Storage Management). Install only the components that match your topology.
A single storage cluster picks ClickHouse or OpenSearch as its backend and serves many collector clusters; a single collector cluster ships to exactly one storage cluster.
For a side-by-side comparison of ClickHouse and OpenSearch, see Log Component Selection Guide.
- Upgrading from ACP 4.2 or 4.3 that uses Log Storage for Elasticsearch? Do not follow this chapter. Follow Upgrade from Log Storage for Elasticsearch and Historical Data Migration instead. Applying a fresh
PlatformLogForwardon top of a chart-managed installation collides with the operator's auto-adoption path. - Alauda Container Platform Log Storage for Elasticsearch is not offered on ACP 4.4. Fresh installs pick ClickHouse or OpenSearch only; there is no Elasticsearch install form to fall back to.
1. Install log storage
Do this on every cluster that will store logs.
1.1 Prepare the external components
The storage side depends on two external systems: a Kafka cluster (mandatory) and a ClickHouse or OpenSearch cluster (choose one). Prepare them before installing the Log Storage operator.
1.1.1 Kafka
Kafka buffers records between the collector and the storage backend. Both ClickHouse and OpenSearch backends require it.
If you already run a Kafka cluster, provision the account, topics, and ACLs on it and continue to Section 1.1.2.
Using Alauda Container Platform Kafka. ACP ships its own Kafka product (Alauda Kafka Operator, delivered as a separate cluster plugin). For the full install, tuning, and operations guide, see the Alauda Container Platform Kafka documentation.
A minimal set of CRs to bring a Kafka cluster up for logging looks like the following. The kinds shown use the Alauda strimzi fork (middleware.alauda.io/v1, RdsKafka / RdsKafkaUser / RdsTopic), which is the recommended path on ACP. Upstream strimzi (kafka.strimzi.io/v1beta2, Kafka / KafkaUser / KafkaTopic) also works — the operator does not care which fork you use, only that the mq-conn Secret and the broker-side ACL / topics line up.
The authoritative reference for this section is log-storage-operator/config/samples/quickstart_external_ch_kafka.yaml; the snippets below are trimmed for the doc.
The manifest below carries placeholders and example values that you MUST customise for your cluster. Do not kubectl apply it as-is.
- Password (
Secret platform-logging-password.stringData.password): replace<generate a strong password of at least 32 characters>with a real secret generated by one of theopensslcommands shown. On FIPS hosts this is enforced: < 32 chars is silently rejected by the broker. - StorageClass (
RdsKafka.spec.storage.classANDRdsKafka.spec.controller.storage.class): fill in your cluster's StorageClass. Leaving it as""only works when your cluster has a default StorageClass; otherwise brokers Pending on PVC. - Broker storage size (
RdsKafka.spec.storage.size): default100Gifits an evaluation cluster. For production, size it byretention.ms × daily volume × replication factor. - Replicas (
RdsKafka.spec.replicas,RdsKafka.spec.controller.replicas,RdsTopic.spec.replicas): the manifest shows production values (3/3/3). For a single-node evaluation cluster, set every one of them to1. Mixing production topic replicas with a 1-broker cluster blocks topic creation. - Topic partitions (
RdsTopic.spec.partitions):30is a production starting point. Lower for small clusters; must match (or divide into) the aggregate-vector parallelism you expect downstream. - Replication-factor config (
RdsKafka.spec.config.default.replication.factor/min.insync.replicas/offsets.topic.replication.factor/transaction.state.log.replication.factor/transaction.state.log.min.isr): manifest ships3 / 2 / 3 / 3 / 2for a 3-broker production cluster. For a single-broker evaluation cluster set every3to1and every2to1; a 3-broker config on a 1-broker cluster leaves__consumer_offsetsunder-replicated and no consumer can commit. - Resources (
RdsKafka.spec.resources,controller.resources,entityOperator.topicOperator.resources/userOperator.resources,kafkaExporter.resources): the example limits/requests are a reasonable per-broker baseline. Tune for your traffic — brokers scale with throughput, entity-operator stays small. - Tolerations (
controller.template.pod.tolerations,kafka.template.pod.tolerations,entityOperator.template.pod.tolerations,kafkaExporter.template.pod.tolerations): the manifest tolerates everything so single-node evaluation clusters (where control-plane tainted nodes host workloads) can schedule. Tighten or remove for production clusters where you want Kafka only on worker nodes. - upgradeOption.autoUpgrade (
RdsKafka.spec.upgradeOption.autoUpgrade):falsepins the version — SRE drives upgrades. Settingtruelets a CSV bump roll your brokers unattended.
Apply only after every placeholder above is substituted.
The CR's metadata.name is a DNS-safe kebab-case name (alauda-log-topic) and spec.topicName is the actual name the broker exposes (ALAUDA_LOG_TOPIC, uppercase with underscores). Alauda Container Platform Log Collector and the storage operator always read the broker-side name, so the values you paste into the Kafka connection Secret in Section 1.3 must match spec.topicName, not metadata.name.
1.1.2 ClickHouse or OpenSearch
Pick one backend per storage cluster. See Log Component Selection Guide if you have not decided.
ClickHouse
Cluster topology. Choose one of the profiles below and use the same shards / replicas values when you fill in PlatformLogForward in Section 1.4. Setting the PlatformLogForward values lower silently leaves shards or replicas unused; setting them higher makes the data path try to reach nodes that do not exist.
For component sizing at each profile, see Log Component Capacity Planning.
If you already run a ClickHouse cluster, ensure the database and the logging account are provisioned on it. To build a cluster from scratch, see the Alauda Container Platform ClickHouse documentation.
OpenSearch
If you already run an OpenSearch cluster, ensure analysis-ik is installed on every node and the logging account exists. To build a cluster from scratch, follow the Alauda Container Platform OpenSearch product documentation.
1.2 Install the Log Storage operator
Alauda Container Platform Log Storage Management installs log-storage-operator on the storage cluster. One operator covers both ClickHouse and OpenSearch backends; the backend choice is made in the PlatformLogForward you create in Section 1.4, not in the operator install form.
Console
- Navigate to Marketplace > Cluster Plugins and select the storage cluster.
- Find Alauda Container Platform Log Storage Management and click Install in its row.
- Accept the defaults. Optionally set the node selector and tolerations to pin
log-storage-operatorto infra nodes. - Click Install.
Verify
The READY column must show 1/1.
1.3 Create the connection Secrets
Two Secrets in cpaas-system carry the endpoint and credentials for the external stack. log-storage-operator reads them by their default names (platform-default-*); do not rename them.
Every command in this section runs on the storage cluster.
1.3.1 Kafka Secret
Required for both backends.
bootstrap: the FQDN:port of your Kafka's bootstrap Service. The examplecpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092matches the plain SASL listener on the RdsKafka from Section 1.1.1; change the hostname if your RdsKafka has a different name, and change the port to:9093only if you additionally enabled atls:listener in the RdsKafka.kafkaClusterName: themetadata.nameof yourRdsKafkaCR.username: the SASL account you created withRdsKafkaUser(Section 1.1.1 usesplatform-logging).password: resolve fromKafkaUser.spec.authentication.password.valueFrom(plaintext — operator reads this Secret, not the K8s SASL Secret the broker reads). The retrieval command is below the manifest.- TLS is off by default in this example (plain SASL on
:9092). If you enabled thetls:listener in the RdsKafka, add atls.ca:key with the broker CA PEM (readable from Secret<RdsKafka name>-cluster-ca-cert, keyca.crt) and pointbootstrapat:9093.
Apply only after every placeholder above is substituted.
Read the values you do not have at hand:
Save the manifest as platform-default-mq-conn.yaml and apply it:
Do not add tls.insecure_skip_verify: "true". When the listener uses TLS, provide tls.ca; when it uses plain text, omit tls.ca and connect on the plain port.
1.3.2 ClickHouse Secret
Skip this section if you use OpenSearch.
endpoint: HTTPS or HTTP URL of your ClickHouse — the native HTTP interface (default port 8123 / TLS 8443), not the native TCP port 9000.username/password: the ClickHouse user the operator will connect as. The example<clickhouse-password>is a placeholder — put the real password here. The user must haveCREATE DATABASE,CREATE TABLE,INSERT,SELECTon the logging database (razor'sschema-migrationsjob creates it on first run). Write the plain-text password here, not a hash. Some ClickHouse deployment guides (for example the ecosystem team'sdeploy_as_logging_storagehow-to) ask you to put asha256_hexdigest into theClickHouseInstallationmanifest; that digest is for the ClickHouse server itself. Theplatform-default-ch-connSecret is read byrazor-api/aggregate-vectorto actually connect to ClickHouse, so it needs the matching plain-text value. Mixing the two produces a PLF that goes Ready but aschema-migrationsJob that fails everyCREATE TABLE ... ON CLUSTERwith516 Authentication failed.tls.ca: only when ClickHouse runs HTTPS with a certificate your K8s pods do not already trust. Replace<clickhouse-ca-certificate>with the PEM block, or remove it for HTTP / system-trusted CA.
Apply only after every placeholder above is substituted.
Read values:
Apply:
Do not add tls.insecure_skip_verify: "true". Provide tls.ca for HTTPS endpoints; omit it for plain HTTP endpoints.
1.3.3 OpenSearch Secret
Skip this section if you use ClickHouse.
endpoints: HTTPS or HTTP URL of your OpenSearch (default port 9200). Comma-separated for multi-node coordination.username/password: the OpenSearch account the operator will connect as. The example<opensearch-password>is a placeholder — put the real password here. The user must be able to manage index templates and ILM policies on the logging index pattern.tls.ca: only when OpenSearch runs HTTPS with a certificate your K8s pods do not already trust. Remove the wholetls.cakey for HTTP / system-trusted CA.
Apply only after every placeholder above is substituted.
Apply:
The OpenSearch Secret does not carry a tls.ca field. The platform connects to OpenSearch with certificate verification disabled, so a private-CA endpoint needs no CA material here. Use https:// in endpoints and the connection succeeds regardless of the issuer.
1.4 Create the PlatformLogForward
PlatformLogForward (PLF) is the cluster-scoped CR that log-storage-operator watches to build the data path. It is always named platform-default, and there is exactly one per storage cluster.
Three sections are environment-specific — pick values that match your cluster size before kubectl apply:
externalStorage.shards/.replicasmust match the ClickHouse profile you picked in Section 1.1.2. Setting these lower than the ClickHouse topology silently leaves nodes unused; setting them higher makes the data path reach for nodes that do not exist. Single-node evaluation =1 / 1; three-node HA =1 / 3; six-node scale-out =2 / 3; nine-node scale-out =3 / 3.aggregateVector.replicasscales the Kafka→ClickHouse ingest tier. Evaluation =1; production =3(operator default) and up for high ingest throughput.razor.replicasandrazor.resourcesscale the query-plane serving razor-api. Example values below are a reasonable per-pod baseline; production sites with heavy query load should raise replicas (3+) and tune the limits for the expected concurrent-query budget.
Apply only after the three sizing choices match your target topology.
For OpenSearch:
externalStorage.shards / .replicas are still accepted on the OpenSearch branch for consistency with the ClickHouse schema, even though OpenSearch manages its own shard/replica allocation on each index. Setting them to 1 / 1 is safe for every topology — the fields are informational for the operator, not propagated to the OpenSearch cluster. Scale aggregateVector.replicas and razor.replicas the same way as the ClickHouse branch above.
Save the file as platform-log-forward.yaml and apply:
Watch the resource until Phase becomes Ready and Ready becomes True, then stop with Ctrl-C:
If it does not go Ready within about ten minutes, read the status conditions:
Optional fields worth setting only when needed:
When log-storage-operator starts on a cluster that already contains chart-managed ClickHouse or Kafka resources from a legacy ACP 4.2 or 4.3 installation, it creates PlatformLogForward/platform-default on its own with installMode: Adopt so that migration can proceed non-destructively.
On a fresh install this should not happen. If kubectl get platformlogforward platform-default returns a resource before you apply your manifest and its spec.installMode is Adopt, something on this cluster was mistaken for a legacy installation. Recover by deleting the auto-created object and re-applying yours in one step:
For OpenSearch, replace platform-default-ch-conn with platform-default-os-conn.
If the cluster genuinely holds legacy logging data (AppRelease/logclickhouse exists, or PVCs from a previous ACP release are still bound), stop here and follow Upgrade from Log Storage for Elasticsearch and Historical Data Migration instead.
2. Install Log Collector
Install this plugin on every cluster whose logs, events, or audit records must be collected. That includes the global cluster if its own data must be collected.
Console
- Navigate to Marketplace > Cluster Plugins and select the target cluster.
- Find Alauda Container Platform Log Collector and click Install in its row.
- Configure the parameters:
- Optional. Expand the advanced form to pin the collector to specific nodes with a node selector and tolerations.
- Click Install.
YAML
Every <...> is a placeholder and the version string is example-only. Apply as-is will fail — ACP's cluster-transformer rejects the ModuleInfo if the name/label does not resolve or the version is not registered.
- Replace both
<collector-cluster>occurrences (inmetadata.labels.cpaas.io/cluster-nameand inmetadata.name) with the name of the cluster where the collector runs. Both must match, and the K8s name constraint rejects</>/ uppercase. - Replace
<storage-cluster>(inspec.config.crossClusterDependency.logstorage) with the cluster wherePlatformLogForward/platform-defaultis Ready. - Replace
spec.version: v4.4.0with the actual version your cluster has:kubectl get moduleconfigs | grep logagentand copy alogagent-v4.4.xname. The ModuleConfig CR must exist for the version you set here, otherwise the webhook rejects withmoduleconfig <name> not found. - Review
spec.config.dataSource.*andspec.config.components.nevermore.storageSpec.dataDir: the defaults are fine for a stock Alauda host, but Alauda OS uses/var/cpaas/data/nevermoreand air-gapped sites may need a different volume.
Apply only after every placeholder above is substituted.
Key fields:
Verify
Run from the global cluster:
Every collector cluster's row must show STATUS=Running.
3. Install Log Essentials
Run on the global cluster. Install once per platform. The install flow is unchanged from earlier releases.
Console
- Navigate to Marketplace > Cluster Plugins and select the
globalcluster. - Find Alauda Container Platform Log Essentials and click Install in its row.
- Accept the defaults and click Install.
YAML
Replace spec.version: v4.4.0 with the actual version registered in your cluster: kubectl get moduleconfigs | grep log-api and copy a log-api-v4.4.x name. If the version is not registered, cluster-transformer rejects the ModuleInfo with moduleconfig <name> not found. The metadata.name (global-log-api) and metadata.labels.cpaas.io/cluster-name (global) are fixed — the log-api plugin is a global-cluster-only component.
Verify
The STATUS column must be Running.
4. Verify end-to-end
After every plugin is Running, verify that new records actually flow from a collector cluster into the storage backend.
-
On every storage cluster:
Must read
Ready/True. -
On every storage cluster:
aggregate-vector-*andrazor-*pods must beRunning; theschema-migrations-*Job must haveSTATUS=Completed. -
Log in to the ACP console, open Operations Center > Logs > Log Query Analysis, pick the storage cluster in the top-of-page cluster picker, and query for Workload Logs from a namespace that is actively producing logs. Recent records must appear.
If no records arrive within ten minutes, before changing configuration, work through the layers in order: collector ModuleInfo status → PlatformLogForward conditions → Kafka topic activity → backend ingest.
Next steps
Once every plugin above reports installed:
- Walk the seven-step How to Verify a Fresh Install of the Logging Stack checklist to confirm the write path and the read path are both working end to end.
- If a step fails, jump straight to the matching entry in Troubleshooting the Logging Stack.
- When you eventually retire this install, follow How to Uninstall the Logging Stack. The external ClickHouse (or OpenSearch) cluster, the external Kafka cluster, and the two SRE-authored connection Secrets are outside the plugin lifecycle — that page names the exact commands to reclaim each.