Log Component Capacity Planning

The log storage components hold logs, events, and audit records collected from one or more clusters. Estimate your log scale before install and size the storage cluster using the profiles below.

WARNING
  • The tables in this chapter are laboratory reference numbers. Provision more resources than the reference numbers, and do not push the collected volume above the profile's throughput.
  • Disk baseline for the tables: 6000 iops, 250 MB/s sequential read and write, SSD, independently mounted. Weaker disks require a larger profile than the table suggests and additional CPU + memory headroom.
  • Kafka runs in KRaft mode. No Zookeeper is deployed; the tables in earlier releases had a separate Zookeeper row that no longer applies.
  • Older ACP releases ran the Elasticsearch backend with a separate Lanaya consumer between Kafka and Elasticsearch. The recommended backends (ClickHouse and OpenSearch) do not use Lanaya; the ingest role belongs to aggregate-vector.
  • In ACP 4.4 the storage backend (ClickHouse or OpenSearch) and the Kafka cluster are user-owned external endpoints; the sizes below assume the storage cluster and the Kafka cluster are sized against the same profile. If your Kafka is shared with other workloads, size the shared cluster to the sum of every consumer's Kafka broker row.

Common components

All backends share these components:

ComponentWhere it runs
nevermoreDaemonSet on every collector node. CPU / memory sized by the number of active containers on the node, not by cluster throughput.
RazorDeployment on the storage cluster.
aggregate-vectorDeployment on the storage cluster (ClickHouse and OpenSearch paths).
KafkaBroker cluster on or near the storage cluster.

ClickHouse profiles

The ClickHouse backend is the recommended choice for high throughput and cost-sensitive retention.

Single node — evaluation only, up to ~18,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
ClickHouse12C4G1 shard × 1 replica
Kafka broker12C4GSingle broker, no HA
Razor11C1G
aggregate-vector12C4G

Three nodes — up to ~20,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
ClickHouse32C4G1 shard × 3 replicas
Kafka broker32C4G
Razor21C1G
aggregate-vector22C4G

Six nodes — up to ~40,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
ClickHouse64C8G2 shards × 3 replicas
Kafka broker32C4G
Razor21C1G
aggregate-vector24C8G

Nine nodes — up to ~69,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
ClickHouse94C8G3 shards × 3 replicas
Kafka broker32C4G
Razor21C1G
aggregate-vector24C8G

OpenSearch profiles

The OpenSearch backend is the recommended choice when Lucene-style search experience matters more than raw ingest throughput.

Small — three data nodes, up to ~9,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
OpenSearch data32C4G
Kafka broker32C4G
Razor21C2G
aggregate-vector22C4G

Medium — three master, five data nodes, up to ~25,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
OpenSearch master32C4GDedicated master
OpenSearch data58C16GHeap capped at 30 GB on any node ≥ 60 GB memory (compressed-oops boundary)
Kafka broker32C4G
Razor21C2G
aggregate-vector22C4G

Large — three master, seven data nodes, up to ~30,000 logs/s

ComponentReplicasCPU limitMemory limitNotes
OpenSearch master32C4GDedicated master
OpenSearch data78C16GSee heap note above
Kafka broker32C4G
Razor21C2G
aggregate-vector22C4G

Elasticsearch profiles (legacy)

The Elasticsearch backend remains available for existing customers. The classic ACP 4.3 pipeline (Filebeat → Kafka → Lanaya → Elasticsearch) is unchanged; the sizing tables below are reproduced for reference.

Small — three nodes, up to ~6,300 logs/s

ComponentReplicasCPU limitMemory limit
Elasticsearch32C4G
Kafka32C4G
Lanaya22C4G
Razor21C2G

Small — five nodes, up to ~9,900 logs/s

ComponentReplicasCPU limitMemory limit
Elasticsearch52C4G
Kafka32C4G
Lanaya22C4G
Razor21C2G

Large — three master, five data nodes, up to ~25,000 logs/s

ComponentReplicasCPU limitMemory limit
Elasticsearch master32C4G
Elasticsearch data58C16G
Kafka32C4G
Lanaya22C4G
Razor21C2G

Large — three master, seven data nodes, up to ~30,000 logs/s

ComponentReplicasCPU limitMemory limit
Elasticsearch master32C4G
Elasticsearch data78C16G
Kafka32C4G
Lanaya22C4G
Razor21C2G

Sizing notes

  • Set PlatformLogForward.spec.aggregateVector.replicas and spec.razor.replicas to match the profile you deploy. The CRD defaults (3 and 2) suit the medium and large profiles; drop them to 1 on a single-node evaluation cluster.
  • Raise spec.razor.resources.limits.cpu above the profile default when the collector-side buffer grows during traffic bursts and Razor CPU sits at its limit. See How to size Razor for burst traffic once that page is published.
  • Kafka broker sizing is separate from the storage backend sizing. Follow the Alauda Container Platform Kafka capacity guide for broker-side numbers.