Logging Docs
  • English

    • How to Interface with External ES Storage Clusters

    Applies to earlier releases only

    This guide describes the Elasticsearch storage plugin's elasticsearch.address / basicAuthSecretName install-form fields, retained here as reference material for ACP 4.2 / 4.3 installations that are still running the chart-installed Elasticsearch stack.

    ACP 4.4 does not ship the Elasticsearch backend at all. Fresh 4.4 installs reach an external ClickHouse or OpenSearch and an external Kafka through the two platform-default-* connection Secrets — external interfacing is the default install path, not an opt-in. Follow Installation and skip this guide. Customers upgrading from ACP 4.3 on Elasticsearch migrate to ClickHouse or OpenSearch during the upgrade; see the Upgrade guide.

    You can interface with external Elasticsearch or Kafka clusters by writing YAML configurations. Depending on your business requirements, you can choose to interface with only the external Elasticsearch cluster (while installing Kafka in the current cluster), or you can interface with both the external Elasticsearch and Kafka clusters simultaneously.

    TIP

    The supported versions for interfacing with external Elasticsearch are as follows:

    • Elasticsearch 6.x supports versions 6.6 - 6.8;
    • Elasticsearch 7.x supports versions 7.0 - 7.10.2, with a recommendation to use 7.10.2.

    TOC

    Resource PreparationOperating Procedures

    Resource Preparation

    Before interfacing, you need to prepare the required credential information.

    1. In the left navigation bar, click on Cluster Management > Resource Management, then switch to the cluster that needs the plugin installation.

    2. Click on Create Resource Object, and fill in the code box after modifying the parameters according to the code comments.

    • Credentials required for interfacing with external Elasticsearch:

      apiVersion: v1
type: Opaque
data:
  password: dEdWQVduSX5kUW1mc21acg==  # Must be base64 encoded. Reference command: echo -n <password_value>| base64
  username: YWRtaW4=                  # Must be base64 encoded. Reference command: echo -n <username_value>| base64
kind: Secret
metadata:
  name: elasticsearch-basic-auth      # Credential name. Ensure that the value of elasticsearch.basicAuthSecretName in the log storage YAML matches this parameter.
  namespace: cpaas-system             # The namespace where the Elasticsearch component is located, generally cpaas-system.

    • If you need to use an external Kafka cluster, you will also need to create credentials for interfacing with the external Kafka cluster:

      apiVersion: v1
type: Opaque
data:
  password: dEdWQVduSX5kUW1mc21acg==  # Must be base64 encoded. Reference command: echo -n <password_value>| base64
  username: YWRtaW4=                  # Must be base64 encoded. Reference command: echo -n <username_value>| base64
kind: Secret
metadata:
  name: kafka-basic-auth              # Credential name. Ensure that the value of kafka.basicAuthSecretName in the log storage YAML matches this parameter.
  namespace: cpaas-system             # The namespace where the Kafka component is located, generally cpaas-system.
    1. Click on Create.

    Operating Procedures

    1. In the left navigation bar, click Marketplace > Cluster Plugins.

    2. In the top navigation, select the Cluster Name where you want to install Alauda Container Platform Log Storage for Elasticsearch.

    3. Click the action button on the right side of Alauda Container Platform Log Storage for Elasticsearch and select Install.

    4. Enable the External Elasticsearch switch, then configure the YAML file using the following examples and parameter descriptions:

    • Interfacing with the external Elasticsearch cluster while installing Kafka in the current cluster:

      elasticsearch:
  install: false
  address: http://fake:9200                # External ES access address, e.g., http://192.168.143.252:11780/es_proxy
  basicAuthSecretName: elasticsearch-basic-auth # Credentials required for interfacing with external Elasticsearch created in the prerequisites.
storageClassConfig:
  type: "LocalVolume"   # Default is LocalVolume. Options are "LocalVolume" or "StorageClass".
kafka:
  auth: true                               # Whether to enable authentication.
  k8sNodes:
  - log1                                   # Node name, obtained from kubectl get nodes.
  - log2
  - log3
  storageSize: 10                          # Storage size, in Gi, default is 10 Gi.

    • Interfacing with both the external Elasticsearch cluster and the external Kafka cluster:

      elasticsearch:
  install: false
  address: http://fake:9200                # External ES access address, e.g., http://192.168.143.252:11780/es_proxy
  basicAuthSecretName: elasticsearch-basic-auth # Credentials required for interfacing with external Elasticsearch created in the prerequisites.
kafka:
  auth: true                               # Whether to enable authentication.
  install: false
  basicAuthSecretName: kafka-basic-auth  # Credentials required for interfacing with external Kafka created in the prerequisites.
  address: 192.168.130.169:9092,192.168.130.187:9092,192.168.130.193:9092     # Kafka access addresses, separated by commas.