Planning Infra Nodes for Logging Storage

This guide explains the planning considerations for running Logging storage components on dedicated Kubernetes infra nodes.

Objectives

  • Isolate resources: Prevent contention with business workloads.
  • Enforce stability: Reduce evictions and scheduling conflicts.
  • Simplify management: Centralize infra components with consistent scheduling rules.

Where to Configure Placement

The storage-side workloads (Razor, aggregate-vector, and the one-shot schema-migrations Job) are derived by log-storage-operator from PlatformLogForward. Set their placement through the plugin install form or through the underlying ModuleInfo; do not patch the generated Deployments directly.

  • For Alauda Container Platform Log Storage Management (ClickHouse or OpenSearch backend), use the Advanced Configuration section of the operator install form to set the node selector and tolerations. See Installation.
  • For Alauda Container Platform Log Collector (a DaemonSet), use the plugin install form on the collector cluster; the same Advanced Configuration section applies.
  • For Alauda Container Platform Log Storage for Elasticsearch (legacy), configure placement through spec.valuesOverride.ait/chart-alauda-log-center.global.nodeSelector and spec.valuesOverride.ait/chart-alauda-log-center.global.tolerations in the ACP 4.3 documentation for that plugin.

Do not patch the generated StatefulSets, Deployments, or ClickHouseInstallation resources as the standard way to place Logging storage workloads on infra nodes.

Before You Configure Placement

  1. Plan the infra nodes according to Cluster Node Planning.
  2. Confirm whether your storage uses LocalVolume or other PVs with spec.nodeAffinity.
  3. Make sure the selected infra nodes can satisfy both the scheduling rules and the storage placement constraints.
INFO

On Alauda OS nodes, local storage paths are under /var/cpaas, for example /var/cpaas/data/clickhouse or /var/cpaas/data/nevermore. These nodes are immutable and /cpaas, the layout of the traditional operating system, is not writable there. Plan the infra nodes so that the /var/cpaas path is backed by a persistent disk that survives node re-provisioning, and expect pv.spec.local.path on those nodes to reference the /var/cpaas path instead of the /cpaas paths shown in the examples below. For the per-plugin parameters, see Installation.

Check Local PVs and nodeAffinity

If your components use local storage (for example TopoLVM, local PV), confirm whether PVs have spec.nodeAffinity. If so, either:

  1. Add all nodes referenced by pv.spec.nodeAffinity to the infra node group, or
  2. Redeploy components using a storage class without node affinity (for example Ceph/RBD).

Example (Elasticsearch):

# 1) Get ES PVCs
kubectl get pvc -n cpaas-system | grep elastic

# 2) Inspect one PV
kubectl get pv elasticsearch-log-node-pv-192.168.135.243 -o yaml

If the PV shows:

spec:
  local:
    path: /cpaas/data/elasticsearch/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 192.168.135.243

Then Elasticsearch data is pinned to node 192.168.135.243. Ensure that node is part of the infra node group, or migrate storage.

The same principle applies to any Logging storage component that uses node-bound local storage.

Historical Kafka and Zookeeper Nodes

Kafka runs in KRaft mode and does not deploy Zookeeper. On clusters upgraded from an earlier release, however, historical Kafka and Zookeeper node labels may still be in use. Ensure any historically labeled Kafka or Zookeeper nodes are also included in the infra node group:

kubectl get nodes -l kafka=true
kubectl get nodes -l zk=true
# Add the listed nodes into infra nodes as above.

Newly provisioned clusters only need the kafka label; the zk label is not used.

Troubleshooting

Common issues and fixes:

IssueDiagnosisSolution
Pods stuck in Pendingkubectl describe pod <pod> | grep EventsAdd tolerations or adjust selectors
Taint/toleration mismatchkubectl describe node <node> | grep TaintsAdd matching tolerations to the workloads
Resource starvationkubectl top nodes -l node-role.kubernetes.io/infraScale infra nodes or tune resource requests

Example error:

Events:
  Warning  FailedScheduling  2m  default-scheduler  0/3 nodes are available:
  3 node(s) had untolerated taint {node-role.kubernetes.io/infra: true}

Fix: add matching tolerations to the plugin configuration and make sure the selected infra nodes also satisfy the required storage placement constraints.