How to Archive Logs to Third-Party Storage

Currently, the logs generated by the platform will be stored in the log storage component; however, the retention period for these logs is relatively short. For enterprises with high compliance requirements, logs typically require longer retention times to meet audit demands. Additionally, the economic aspect of storage is also one of the key concerns for enterprises.

Based on the above scenarios, the platform offers a log archiving solution, allowing users to transfer logs to external NFS or object storage.

INFO

This document describes the log export or archive solution that transfers logs out of the platform to external storage.

If you want ClickHouse to use S3 as its native log data storage or cold data storage, see How to Use S3 Storage with ClickHouse and Installation.

ACP 4.4 caveat

The archive workflow below launches a standalone razor consumer --database-type=file Deployment that reads the platform's Kafka topics and writes to NFS or S3. The example manifests were captured against ACP 4.3 and still describe the intended pattern, but several details drift on ACP 4.4:

The example Razor image tag (v3.16.0-beta.3.g3df8e987) is stale. Read the current image tag from the running deployment/razor on the storage cluster before you copy the example.
The example uses serviceAccount: lanaya. On ACP 4.4 ClickHouse or OpenSearch clusters there is no lanaya ServiceAccount; use the current archive ServiceAccount that ships with the storage operator, or create your own SA with pods/exec, secrets/get, and access to the archive hostPath.
The example mounts kafka-basic-auth from the ACP 4.3 chart layout. In ACP 4.4 the Kafka credentials live in platform-default-mq-conn (see the Kafka connection Secret section of Installation). Either mount platform-default-mq-conn and adjust the razor consumer flags to read SASL from those keys, or copy the credentials into a Secret with the layout the example expects.

For a v4.4-verified archive workflow, please contact support. The pattern below still works on the Elasticsearch (legacy) backend as documented.

INFO

On Alauda OS nodes, the host directory that the log export Deployment mounts must be under /var/cpaas, for example /var/cpaas/data/logarchive. These nodes are immutable and /cpaas, the layout of the traditional operating system, is not writable there. Change only spec.template.spec.volumes[].hostPath.path in the examples below; the mountPath values and the output.path value in the log-exporter-config ConfigMap are paths inside the container and stay unchanged.

Transfer to External NFS

Prerequisites

Resource	Description
NFS	Set up the NFS service in advance and determine the NFS path to be mounted.
Kafka	Obtain the Kafka service address in advance.
Image Address	You must use the CLI tool in the `global` cluster to execute the following commands to get the image addresses: - Get alpine image address: `kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'` - Get razor image address: `kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'`

Create Log Synchronization Resources

Click on Cluster Management > Clusters in the left navigation bar.
Click the action button on the right side of the cluster where the logs will be transferred > CLI Tool.

Modify the YAML based on the following parameter descriptions; after modifying, paste the code into the open CLI Tool command line and hit enter to execute.

Resource Type	Field Path	Description
ConfigMap	`data.export.yml.output.compression`	Compress log text; supported options are none (no compression), zlib, gzip.
ConfigMap	`data.export.yml.output.file_type`	The type of exported log file; supports txt, csv, json.
ConfigMap	`data.export.yml.output.max_size`	Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
ConfigMap	`data.export.yml.scopes`	The scope of log transfer; currently supported logs include: system logs, application logs, Kubernetes logs, product logs.
Deployment	`spec.template.spec.containers[0].command[7]`	Kafka service address.
Deployment	`spec.template.spec.volumes[3].hostPath.path`	NFS path to be mounted.
Deployment	`spec.template.spec.initContainers[0].image`	Alpine image address.
Deployment	`spec.template.spec.containers[0].image`	Razor image address.

cat << "EOF" |kubectl apply -f -
apiVersion: v1
data:
  export.yml: |
    scopes: # The scope of log transfer; by default, only application logs are collected
      system: false  # System logs
      workload: true # Application logs
      kubernetes: false # Kubernetes logs
      platform: false # Product logs
    output:
      type: local
      path: /cpaas/data/logarchive
      layout: TimePrefixed
      # Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
      max_size: 200
      compression: zlib    # Optional: none (no compression) / zlib / gzip
      file_type: txt   # Optional: txt csv json
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      automountServiceAccountToken: true
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
              chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 # Alpine image address
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
          - /razor
          - consumer
          - --v=1
          - --kafka-group-log=log-nfs
          - --kafka-auth-enabled=true
          - --kafka-tls-enabled=true
          - --kafka-endpoint=192.168.143.120:9092  # Fill in based on actual environment
          - --database-type=file
          - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # Razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # NFS path to be mounted
            type: DirectoryOrCreate
          name: data
EOF

Once the container status changes to Running, you can view the continuously archived logs in the NFS path; the log file directory structure is as follows:
/cpaas/data/logarchive/$date/$project/$namespace-$cluster/logfile

Transfer to External S3 Storage

Prerequisites

Resource	Description
S3 Storage	Prepare the S3 storage service address in advance, and obtain the values for `access_key_id` and `secret_access_key`; create the bucket where the logs will be stored.
Kafka	Obtain the Kafka service address in advance.
Image Address	You must use the CLI tool in the `global` cluster to execute the following commands to get the image addresses: - Get alpine image address: `kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'` - Get razor image address: `kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'`

Create Log Synchronization Resources

Click on Cluster Management > Clusters in the left navigation bar.
Click the action button on the right side of the cluster where the logs will be transferred > CLI Tool.

Modify the YAML based on the following parameter descriptions; after modifying, paste the code into the open CLI Tool command line and hit enter to execute.

Resource Type	Field Path	Description
Secret	`data.access_key_id`	Base64 encode the obtained access_key_id.
Secret	`data.secret_access_key`	Base64 encode the obtained secret_access_key.
ConfigMap	`data.export.yml.output.compression`	Compress log text; supported options are none (no compression), zlib, gzip.
ConfigMap	`data.export.yml.output.file_type`	The type of exported log file; supports txt, csv, json.
ConfigMap	`data.export.yml.output.max_size`	Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
ConfigMap	`data.export.yml.scopes`	The scope of log transfer; currently supported logs include: system logs, application logs, Kubernetes logs, product logs.
ConfigMap	`data.export.yml.output.s3.bucket_name`	Bucket name.
ConfigMap	`data.export.yml.output.s3.endpoint`	S3 storage service address.
ConfigMap	`data.export.yml.output.s3.region`	Region information for the S3 storage service.
Deployment	`spec.template.spec.containers[0].command[7]`	Kafka service address.
Deployment	`spec.template.spec.volumes[3].hostPath.path`	Local path to be mounted, used for temporarily storing log information. Log files will be automatically deleted after synchronization to S3 storage.
Deployment	`spec.template.spec.initContainers[0].image`	Alpine image address.
Deployment	`spec.template.spec.containers[0].image`	Razor image address.

cat << "EOF" |kubectl apply -f -
apiVersion: v1
type: Opaque
data:
  # Must include the following two keys
  access_key_id: bWluaW9hZG1pbg==  # Base64 encode the obtained access_key_id
  secret_access_key: bWluaW9hZG1pbg==  # Base64 encode the obtained secret_access_key
kind: Secret
metadata:
  name: log-export-s3-secret
  namespace: cpaas-system

---
apiVersion: v1
data:
  export.yml: |
    scopes: # The scope of log transfer; by default, only application logs are collected
      system: false  # System logs
      workload: true # Application logs
      kubernetes: false # Kubernetes logs
      platform: false # Product logs
    output:
      type: s3
      path: /cpaas/data/logarchive

      s3:
        s3forcepathstyle: true
        bucket_name: baucket_name_s3           # Fill in the prepared bucket name
        endpoint: http://192.168.179.86:9000   # Fill in the prepared S3 storage service address
        region: "dummy"                        # Region information
        access_secret: log-export-s3-secret
        insecure: true

      layout: TimePrefixed
      # Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
      max_size: 200
      compression: zlib                        # Optional: none (no compression) / zlib / gzip
      file_type: txt                           # Optional: txt, csv, json
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
              chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 # Alpine image address
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
            - /razor
            - consumer
            - --v=1
            - --kafka-group-log=log-s3
            - --kafka-auth-enabled=true
            - --kafka-tls-enabled=true
            - --kafka-endpoint=192.168.179.86:9092  # Fill in the Kafka service address based on actual environment
            - --database-type=file
            - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # Razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # Local temporary storage address for logs
            type: DirectoryOrCreate
          name: data
EOF

Once the container status changes to Running, you can view the continuously archived logs in the bucket.

#How to Archive Logs to Third-Party Storage

#TOC

#Transfer to External NFS

#Prerequisites

#Create Log Synchronization Resources

#Transfer to External S3 Storage

#Prerequisites

#Create Log Synchronization Resources

How to Archive Logs to Third-Party Storage

TOC

Transfer to External NFS

Prerequisites

Create Log Synchronization Resources

Transfer to External S3 Storage

Prerequisites

Create Log Synchronization Resources