#Installation

This chapter introduces how to install the following logging plugins: Alauda Container Platform Log Essentials, Alauda Container Platform Log Collector, Alauda Container Platform Log Storage for ClickHouse, and Alauda Container Platform Log Storage for Elasticsearch.

#Installation Planning

Alauda Container Platform Log Storage plugins can be installed in any cluster, and any cluster's log storage component can be selected for log collection to interface with the storage data.

So , before installing the log storage plugin , you need to plan the cluster and node where the log storage component will be installed.

• Avoid deploying log storage plugins in the global cluster. Instead, deploy them in workload clusters to ensure management cluster failures do not disrupt log-based issue resolution.

• Prioritize centralizing logs to a single log storage cluster. If log volume exceeds maximum capacity thresholds, distribute logs across multiple storage clusters.

• Deploy at least one log storage instance per network zone to aggregate logs locally, minimizing cross-data-center public network traffic (which incurs high costs and latency).

• Dedicate exclusive nodes for log storage, avoiding colocating log storage with other applications or platform components. Log storage requires high I/O throughput and may be affected by interference.

• Mount dedicated SSD disks for log storage to significantly enhance performance.

#Install Alauda Container Platform Log Essentials

#Console

1. Navigate to Marketplace > Cluster Plugins and select the global cluster.

2. Click the action button to the right of Alauda Container Platform Log Essentials and select Install.

3. Click Install.

#YAML

1. Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources, in the globalcluster:

# kubectl get moduleplugin | grep log-api
log-api                       30h
# kubectl get moduleconfig | grep log-api
log-api-v4.2.0                30h

This indicates that the ModulePlugin log-api exists in the cluster and version v4.2.0 is published.

2. Create a ModuleInfo

Create a ModuleInfo resource to install the plugin without any configuration parameters:

apiVersion: cluster.alauda.io/v1alpha1
kind: ModuleInfo
metadata:
  annotations:
    cpaas.io/display-name: log-api
    cpaas.io/module-name: '{"en": "Alauda Container Platform Log Essentials", "zh":
      "Alauda Container Platform Log Essentials"}'
  labels:
    cpaas.io/cluster-name: global
    cpaas.io/module-name: log-api
    cpaas.io/module-type: plugin
    cpaas.io/product: Platform-Center
  name: global-b4a67e6adb67bad47544354e9bf2bdff
spec:
  version: v4.2.0
  valuesOverride:
    ait/chart-alauda-log-api:
      global:
        nodeSelector:
          node-role.kubernetes.io/infra: "true"
        tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          value: reserved
          operator: Equal

YAML field reference:

3. Verify installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=log-api
NAME                                             CLUSTER         MODULE      DISPLAY_NAME   STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          log-api   log-api      Running   v4.2.0          v4.2.0           v4.2.0

Field explanations:

• NAME: ModuleInfo resource name
• CLUSTER: Cluster where the plugin is installed
• MODULE: Plugin name
• DISPLAY_NAME: Display name of the plugin
• STATUS: Installation status; Running means successfully installed and running
• TARGET_VERSION: Intended installation version
• CURRENT_VERSION: Version before installation
• NEW_VERSION: Latest available version for installation

#Install Alauda Container Platform Log Storage for Elasticsearch

#Console

1. Navigate to Marketplace > Cluster Plugins and select the target cluster.

2. Click the action button to the right of Alauda Container Platform Log Storage for Elasticsearch and select Install.

3. Refer to the following instructions to configure relevant parameters.

   Parameter	Description
   External Elasticsearch	Keep this switch off to install the platform-managed Elasticsearch storage plugin.
   size	Choose Single Node, Small Cluster, or Big Cluster based on the target scale.
   Storage Method	Choose LocalVolume or StorageClass for the storage backend.
   Log Node	Displayed when LocalVolume is used. Select the Elasticsearch data nodes.
   Storage path on host	Displayed when LocalVolume is used. Enter the absolute host path for Elasticsearch data.
   Capacity	Reserve storage capacity in Gi for the current data section. The page uses this label for both Elasticsearch data and Kafka data sections.
   Log data nodes resources	Configure CPU and memory requests and limits for Elasticsearch data nodes.
   Kafka Nodes	Select the Kafka nodes used by the plugin.
   LogPlatform / LogWorkload / LogSystem / LogKubernetes / Kubernetes Event / Audit	Configure retention days by log type in the Log reserve config, in days section.

4. Click Install.

#YAML

1. Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources, in the globalcluster:

# kubectl get moduleplugin | grep logcenter
logcenter                       30h
# kubectl get moduleconfig | grep logcenter
logcenter-v4.1.0                30h

This indicates that the ModulePlugin logcenter exists in the cluster and version v4.1.0 is published.

2. Create a ModuleInfo

Create a ModuleInfo resource to install the plugin without any configuration parameters:

apiVersion: cluster.alauda.io/v1alpha1
kind: ModuleInfo
metadata:
  annotations:
    cpaas.io/display-name: logcenter
    cpaas.io/module-name: '{"en": "Alauda Container Platform Log Storage for Elasticsearch", "zh": "Alauda Container Platform Log Storage for Elasticsearch"}'
  labels:
    cpaas.io/cluster-name: go
    cpaas.io/module-name: logcenter
    cpaas.io/module-type: plugin
    cpaas.io/product: Platform-Center
  name: <cluster>-log-center
spec:
  config:
    self:
      storage:
        disabled: false
    components:
      elasticsearch:
        address: ""
        basicAuthSecretName: ""
        hostpath: /cpaas/data/elasticsearch
        httpPort: 9200
        install: true
        k8sNodes:
        - 192.168.139.75
        masterK8sNodes: []
        masterReplicas: 0
        masterResources:
          limits:
            cpu: "2"
            memory: 4Gi
          requests:
            cpu: 200m
            memory: 256Mi
        masterStorageSize: 5
        nodeReplicas: 1
        nodeStorageSize: 200
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 1Gi
        tcpPort: 9300
        type: single
      kafka:
        address: ""
        auth: true
        basicAuthSecretName: ""
        exporterPort: 9308
        install: true
        k8sNodes:
        - 192.168.139.75
        port: 9092
        storageSize: 10
        tls: true
        zkElectPort: 3888
        zkExporterPort: 9141
        zkLeaderPort: 2888
        zkPort: 2181
      kibana:
        install: false
      storageClassConfig:
        name: elasticsearch-local-log-sc
        type: LocalVolume
      zookeeper:
        storageSize: 1
    ttl:
      audit: 180
      event: 180
      logKubernetes: 7
      logPlatform: 7
      logSystem: 7
      logWorkload: 7
  version: v4.1.0
  valuesOverride:
    ait/chart-alauda-log-center:
      global:
        nodeSelector:
          node-role.kubernetes.io/infra: "true"
        tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          value: reserved
          operator: Equal

YAML field reference:

3. Verify installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=logcenter
NAME                                             CLUSTER         MODULE      DISPLAY_NAME   STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          logcenter   logcenter      Running   v4.0.12          v4.0.12           v4.0.12

Field explanations:

• NAME: ModuleInfo resource name
• CLUSTER: Cluster where the plugin is installed
• MODULE: Plugin name
• DISPLAY_NAME: Display name of the plugin
• STATUS: Installation status; Running means successfully installed and running
• TARGET_VERSION: Intended installation version
• CURRENT_VERSION: Version before installation
• NEW_VERSION: Latest available version for installation

#Placement on Infra Nodes

If you want the Elasticsearch storage plugin workloads to run on dedicated infra nodes, use the plugin YAML configuration instead of patching the generated workloads after installation.

Configure the following fields in spec.valuesOverride.ait/chart-alauda-log-center.global:

valuesOverride:
  ait/chart-alauda-log-center:
    global:
      nodeSelector:
        node-role.kubernetes.io/infra: "true"
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          value: reserved
          operator: Equal

Before applying these scheduling rules, make sure your infra node planning and local storage placement are compatible. For the node planning considerations, see Planning Infra Nodes for Logging Storage.

#Install Alauda Container Platform Log Storage for ClickHouse

#Console

If you plan to use S3 for ClickHouse log data storage or cold data storage, prepare the required S3 Secret before installation. For the complete procedure, including the Secret format, HTTPS CA certificate Secret, and usage modes, see How to Use S3 Storage with ClickHouse.

1. Navigate to Marketplace > Cluster Plugins and select the target cluster.

2. Click the action button to the right of Alauda Container Platform Log Storage for ClickHouse and select Install.

3. Refer to the following instructions to configure relevant parameters.

   Parameter	Description
   Size	Choose Single Node, Small Cluster, or Big Cluster based on the target scale.
   Storage Policy	Controls how ClickHouse stores metadata and log data. Available values are Default and Separation of storage and compute. This setting cannot be changed after installation.
   Storage Method	Displayed when Storage Policy is Default. Available values are LocalVolume and StorageClass.
   Metadata Storage Type	Displayed when Storage Policy is Separation of storage and compute. Available values are LocalVolume and StorageClass.
   Log Node	Displayed when local storage is used. Under Default, select the nodes that store ClickHouse log data locally. Under Separation of storage and compute, select the nodes that store ClickHouse metadata locally.
   Storage Path on Host	Displayed when Storage Policy is Default and Storage Method is LocalVolume. Default value: /cpaas/data/clickhouse.
   Metadata Storage Path	Displayed when Metadata Storage Type is LocalVolume. Default value: /cpaas/data/clickhouse.
   Log Replicas	Displayed when the selected storage mode uses StorageClass. Configure the number of ClickHouse replicas created for the active storage section.
   StorageClass	Displayed when Storage Policy is Default and Storage Method is StorageClass. Select the StorageClass used for ClickHouse log data.
   Metadata StorageClass	Displayed when Metadata Storage Type is StorageClass. Select the StorageClass used for ClickHouse metadata.
   Capacity	Displayed when Storage Policy is Default. Reserve capacity in Gi for ClickHouse log data.
   Metadata Capacity	Reserve capacity in Gi for ClickHouse metadata. The page recommends 20 Gi.
   Advanced Configuration	Collapsible section for plugin-level scheduling parameters.
   Node Selectors	Displayed in Advanced Configuration. Configure plugin-level node selector rules for the ClickHouse storage plugin workloads.
   Node Tolerations	Displayed in Advanced Configuration. Configure plugin-level toleration rules for the ClickHouse storage plugin workloads.
   S3 for Log Data	Displayed when Storage Policy is Separation of storage and compute. Select the prepared S3 Secret for log data storage. This setting cannot be changed after installation.
   Log Platform / Log Workload / Log System / Log Kubernetes / Kubernetes Event / Audit	Configure hot-data retention days by log type in the Log Retention Settings section.
   Cold and Hot Separate	Enables tiered storage for ClickHouse. After it is enabled, you can configure an S3 location for cold data and set cold-data retention days by log type. This switch cannot be changed after installation.
   S3 for Log Cold Data	Displayed when Cold and Hot Separate is enabled. Select the prepared S3 Secret for cold data storage. This setting cannot be changed after installation.
   Log Platform / Log Workload / Log System / Log Kubernetes / Kubernetes Event / Audit	When Cold and Hot Separate is enabled, the cold-storage settings section uses these same field names for cold-data retention. A value of 0 means that the log type is not transferred to cold storage.

4. Click Install.

#YAML

1. Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources, in the globalcluster:

# kubectl get moduleplugin | grep logclickhouse
logclickhouse                       30h
# kubectl get moduleconfig | grep logclickhouse
logclickhouse-v4.1.0                30h

This indicates that the ModulePlugin logclickhouse exists in the cluster and version v4.1.0 is published.

2. Create a ModuleInfo

Create a ModuleInfo resource to install the plugin. The following example uses Separation of storage and compute together with S3-backed cold storage:

apiVersion: cluster.alauda.io/v1alpha1
kind: ModuleInfo
metadata:
  name: global-logclickhouse
  labels:
    cpaas.io/cluster-name: global
    cpaas.io/module-name: logclickhouse
    cpaas.io/module-type: plugin
spec:
  version: v4.1.0
  config:
    components:
      nodeSelector:
        - key: kubernetes.io/os
          value: linux
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          operator: Exists
      clickhouse:
        storagePolicy:
          type: SSC
          sscSpec:
            metaStorage:
              type: LocalVolume
              size: 20
              localVolumeSpec:
                nodes:
                  - xxx.xxx.xxx.xx
                path: /cpaas/data/clickhouse
            dataStorage:
              type: S3
              s3Spec:
                credentials: clickhouse-s3-config
        tieredPolicy:
          enabled: true
          coldStorage:
            type: S3
            s3Spec:
              credentials: clickhouse-cold-s3-config
          coldTTL:
            audit: 30
            event: 30
            logKubernetes: 30
            logPlatform: 30
            logSystem: 30
            logWorkload: 30
        resources:
          limits:
            cpu: "2"
            memory: 4Gi
          requests:
            cpu: 200m
            memory: 256Mi
        ttl:
          audit: 180
          event: 180
          logKubernetes: 7
          logPlatform: 7
          logSystem: 7
          logWorkload: 7
        type: normal
      razor:
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: 10m
            memory: 256Mi
      vector:
        resources:
          limits:
            cpu: "4"
            memory: 1Gi
          requests:
            cpu: 10m
            memory: 256Mi

YAML field reference (ClickHouse):

Upgrade notes:

• Upgrading the ClickHouse storage plugin does not allow you to switch the Storage Policy type or toggle Cold and Hot Separate on or off.
• After installation, only the effective retention settings, such as hot or cold retention days, should be adjusted.
• To use this capability, upgrade Alauda Container Platform Log Essentials and Alauda Container Platform Log Storage for ClickHouse to compatible versions together.
• In disaster recovery scenarios, use different S3 buckets for the primary and standby ClickHouse deployments.

3. Verify installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=logclickhouse
NAME                                             CLUSTER         MODULE      DISPLAY_NAME   STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          logclickhouse   logclickhouse      Running   v4.0.12          v4.0.12           v4.0.12

Field explanations:

• NAME: ModuleInfo resource name
• CLUSTER: Cluster where the plugin is installed
• MODULE: Plugin name
• DISPLAY_NAME: Display name of the plugin
• STATUS: Installation status; Running means successfully installed and running
• TARGET_VERSION: Intended installation version
• CURRENT_VERSION: Version before installation
• NEW_VERSION: Latest available version for installation

#Placement on Infra Nodes

If you want the ClickHouse storage plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade.

• In the console, use Advanced Configuration to set Node Selectors and Node Tolerations.
• In YAML, set spec.config.components.nodeSelector and spec.config.components.tolerations.

Example:

config:
  components:
    nodeSelector:
      - key: kubernetes.io/os
        value: linux
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists

Before applying these scheduling rules, make sure your infra node planning and local storage placement are compatible. For the node planning considerations, see Planning Infra Nodes for Logging Storage.

#Install Alauda Container Platform Log Collector Plugin

#Console

1. Navigate to Marketplace > Cluster Plugins and select the target cluster.

2. Click the action button to the right of Alauda Container Platform Log Collector and select Install.

3. Refer to the following instructions to configure relevant parameters.

   Parameter	Description
   Storage Cluster	Select the cluster where Alauda Container Platform Log Storage has been installed.
   Log types	Click Select/Deselect to set the scope of log collection in the cluster.
   Storage Path	Configure the local working-data path for Log Collector. Use an absolute path. Default value: /cpaas/data/nevermore. This setting cannot be changed after installation.

4. Click Install.

#YAML

1. Check available versions

Ensure the plugin has been published by checking for ModulePlugin and ModuleConfig resources, in the globalcluster:

# kubectl get moduleplugin | grep logagent
logagent                       30h
# kubectl get moduleconfig | grep logagent
logagent-v4.1.0                30h

This indicates that the ModulePlugin logagent exists in the cluster and version v4.1.0 is published.

2. Create a ModuleInfo

Create a ModuleInfo resource to install the plugin without any configuration parameters:

apiVersion: cluster.alauda.io/v1alpha1
kind: ModuleInfo
metadata:
  annotations:
    cpaas.io/display-name: logagent
    cpaas.io/module-name: '{"en": "Alauda Container Platform Log Collector", "zh": "Alauda Container Platform Log Collector"}'
  labels:
    cpaas.io/cluster-name: go
    cpaas.io/module-name: logagent
    cpaas.io/module-type: plugin
    cpaas.io/product: Platform-Center
    logcenter.plugins.cpaas.io/cluster: go
  name: <cluster>-log-agent
spec:
  config:
    crossClusterDependency:
      logcenter: go
      logclickhouse: null
    dataSource:
      audit: true
      event: true
      kubernetes: true
      platform: false
      system: false
      workload: true
    components:
      nevermore:
        storageSpec:
          dataDir: /cpaas/data/nevermore
    storage:
      type: ElasticSearch
  version: v4.1.0
  valuesOverride:
    ait/chart-alauda-log-agent:
      global:
        nodeSelector:
          node-role.kubernetes.io/infra: "true"
        tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          value: reserved
          operator: Equal

YAML field reference (Log Collector):

3. Verify installation

Since the ModuleInfo name changes upon creation, locate the resource via label to check the plugin status and version:

kubectl get moduleinfo -l cpaas.io/module-name=logagent
NAME                                             CLUSTER         MODULE      DISPLAY_NAME   STATUS    TARGET_VERSION   CURRENT_VERSION   NEW_VERSION
global-e671599464a5b1717732c5ba36079795          global          logagent   logagent      Running   v4.0.12          v4.0.12           v4.0.12