如何将日志归档到第三方存储

当前，平台生成的日志会存储在日志存储组件中；但是，这些日志的保留周期相对较短。对于合规要求较高的企业，日志通常需要更长的保留时间以满足审计需求。此外，存储的经济性也是企业关注的重点之一。

基于上述场景，平台提供了日志归档解决方案，允许用户将日志传输到外部 NFS 或对象存储。

INFO

本文档描述了将日志从平台导出或归档到外部存储的解决方案。

如果你希望 ClickHouse 将 S3 用作其原生日志数据存储或冷数据存储，请参见 How to Use S3 Storage with ClickHouse 和 Installation。

INFO

在 Alauda OS 节点上，日志导出 Deployment 挂载的主机目录必须位于 /var/cpaas 下，例如 /var/cpaas/data/logarchive。这些节点是只读不可变的，因此传统操作系统中的 /cpaas 目录布局在此处不可写。下面示例中仅修改 spec.template.spec.volumes[].hostPath.path；mountPath 值以及 log-exporter-config ConfigMap 中的 output.path 值都属于容器内路径，保持不变。

传输到外部 NFS

前提条件

Resource	Description
NFS	预先搭建 NFS 服务，并确定需要挂载的 NFS 路径。
Kafka	预先获取 Kafka 服务地址。
Image Address	需要使用 `global` 集群中的 CLI 工具执行以下命令以获取镜像地址： - 获取 alpine 镜像地址：`kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'` - 获取 razor 镜像地址：`kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'`

创建日志同步资源

在左侧导航栏中点击 集群管理 > 集群。
点击要传输日志的集群右侧的操作按钮 > CLI Tool。

根据以下参数说明修改 YAML；修改后，将代码粘贴到打开的 CLI Tool 命令行中并按回车执行。

Resource Type	Field Path	Description
ConfigMap	`data.export.yml.output.compression`	压缩日志文本；支持 none（不压缩）、zlib、gzip。
ConfigMap	`data.export.yml.output.file_type`	导出的日志文件类型；支持 txt、csv、json。
ConfigMap	`data.export.yml.output.max_size`	单个归档文件的大小；单位为 MB。如果超过该值，系统将根据 compression 字段的配置自动压缩并归档日志。
ConfigMap	`data.export.yml.scopes`	日志传输的范围；当前支持的日志包括：系统日志、应用日志、Kubernetes 日志、产品日志。
Deployment	`spec.template.spec.containers[0].command[7]`	Kafka 服务地址。
Deployment	`spec.template.spec.volumes[3].hostPath.path`	要挂载的 NFS 路径。
Deployment	`spec.template.spec.initContainers[0].image`	Alpine 镜像地址。
Deployment	`spec.template.spec.containers[0].image`	Razor 镜像地址。

cat << "EOF" |kubectl apply -f -
apiVersion: v1
data:
  export.yml: |
    scopes: # The scope of log transfer; by default, only application logs are collected
      system: false  # System logs
      workload: true # Application logs
      kubernetes: false # Kubernetes logs
      platform: false # Product logs
    output:
      type: local
      path: /cpaas/data/logarchive
      layout: TimePrefixed
      # Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
      max_size: 200
      compression: zlib    # Optional: none (no compression) / zlib / gzip
      file_type: txt   # Optional: txt csv json
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      automountServiceAccountToken: true
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
              chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 # Alpine image address
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
          - /razor
          - consumer
          - --v=1
          - --kafka-group-log=log-nfs
          - --kafka-auth-enabled=true
          - --kafka-tls-enabled=true
          - --kafka-endpoint=192.168.143.120:9092  # Fill in based on actual environment
          - --database-type=file
          - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # Razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # NFS path to be mounted
            type: DirectoryOrCreate
          name: data
EOF

当容器状态变为 Running 后，你可以在 NFS 路径中查看持续归档的日志；日志文件目录结构如下：
/cpaas/data/logarchive/$date/$project/$namespace-$cluster/logfile

传输到外部 S3 存储

前提条件

Resource	Description
S3 Storage	预先准备好 S3 存储服务地址，并获取 `access_key_id` 和 `secret_access_key` 的值；创建用于存储日志的 bucket。
Kafka	预先获取 Kafka 服务地址。
Image Address	需要使用 `global` 集群中的 CLI 工具执行以下命令以获取镜像地址： - 获取 alpine 镜像地址：`kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}'` - 获取 razor 镜像地址：`kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}'`

创建日志同步资源

在左侧导航栏中点击 集群管理 > 集群。
点击要传输日志的集群右侧的操作按钮 > CLI Tool。

根据以下参数说明修改 YAML；修改后，将代码粘贴到打开的 CLI Tool 命令行中并按回车执行。

Resource Type	Field Path	Description
Secret	`data.access_key_id`	对获取到的 access_key_id 进行 Base64 编码。
Secret	`data.secret_access_key`	对获取到的 secret_access_key 进行 Base64 编码。
ConfigMap	`data.export.yml.output.compression`	压缩日志文本；支持 none（不压缩）、zlib、gzip。
ConfigMap	`data.export.yml.output.file_type`	导出的日志文件类型；支持 txt、csv、json。
ConfigMap	`data.export.yml.output.max_size`	单个归档文件的大小；单位为 MB。如果超过该值，系统将根据 compression 字段的配置自动压缩并归档日志。
ConfigMap	`data.export.yml.scopes`	日志传输的范围；当前支持的日志包括：系统日志、应用日志、Kubernetes 日志、产品日志。
ConfigMap	`data.export.yml.output.s3.bucket_name`	bucket 名称。
ConfigMap	`data.export.yml.output.s3.endpoint`	S3 存储服务地址。
ConfigMap	`data.export.yml.output.s3.region`	S3 存储服务的地域信息。
Deployment	`spec.template.spec.containers[0].command[7]`	Kafka 服务地址。
Deployment	`spec.template.spec.volumes[3].hostPath.path`	要挂载的本地路径，用于临时存储日志信息。同步到 S3 存储后，日志文件将自动删除。
Deployment	`spec.template.spec.initContainers[0].image`	Alpine 镜像地址。
Deployment	`spec.template.spec.containers[0].image`	Razor 镜像地址。

cat << "EOF" |kubectl apply -f -
apiVersion: v1
type: Opaque
data:
  # Must include the following two keys
  access_key_id: bWluaW9hZG1pbg==  # Base64 encode the obtained access_key_id
  secret_access_key: bWluaW9hZG1pbg==  # Base64 encode the obtained secret_access_key
kind: Secret
metadata:
  name: log-export-s3-secret
  namespace: cpaas-system

---
apiVersion: v1
data:
  export.yml: |
    scopes: # The scope of log transfer; by default, only application logs are collected
      system: false  # System logs
      workload: true # Application logs
      kubernetes: false # Kubernetes logs
      platform: false # Product logs
    output:
      type: s3
      path: /cpaas/data/logarchive

      s3:
        s3forcepathstyle: true
        bucket_name: baucket_name_s3           # Fill in the prepared bucket name
        endpoint: http://192.168.179.86:9000   # Fill in the prepared S3 storage service address
        region: "dummy"                        # Region information
        access_secret: log-export-s3-secret
        insecure: true

      layout: TimePrefixed
      # Size of a single archived file; unit is MB. If it exceeds this value, logs will be automatically compressed and archived based on the compression field's configuration.
      max_size: 200
      compression: zlib                        # Optional: none (no compression) / zlib / gzip
      file_type: txt                           # Optional: txt, csv, json
kind: ConfigMap
metadata:
  name: log-exporter-config
  namespace: cpaas-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service_name: log-exporter
  name: log-exporter
  namespace: cpaas-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      service_name: log-exporter
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: lanaya
        cpaas.io/product: Platform-Center
        service_name: log-exporter
        version: v1
      namespace: cpaas-system
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: service_name
                      operator: In
                      values:
                        - log-exporter
                topologyKey: kubernetes.io/hostname
              weight: 50
      initContainers:
        - args:
            - -ecx
            - |
              chown -R 697:697 /cpaas/data/logarchive
          command:
            - /bin/sh
          image: registry.example.cn:60080/ops/alpine:3.16 # Alpine image address
          imagePullPolicy: IfNotPresent
          name: chown
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /cpaas/data/logarchive
              name: data
      containers:
        - command:
            - /razor
            - consumer
            - --v=1
            - --kafka-group-log=log-s3
            - --kafka-auth-enabled=true
            - --kafka-tls-enabled=true
            - --kafka-endpoint=192.168.179.86:9092  # Fill in the Kafka service address based on actual environment
            - --database-type=file
            - --export-config=/etc/log-export/export.yml
          image: registry.example.cn:60080/ait/razor:v3.16.0-beta.3.g3df8e987  # Razor image
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: log-export
          ports:
            - containerPort: 80
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /metrics
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: 440m
              memory: 1280Mi
          securityContext:
            runAsGroup: 697
            runAsUser: 697
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/secrets/kafka
              name: kafka-basic-auth
              readOnly: true
            - mountPath: /etc/log-export
              name: config
              readOnly: true
            - mountPath: /cpaas/data/logarchive
              name: data
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 697
      serviceAccount: lanaya
      serviceAccountName: lanaya
      terminationGracePeriodSeconds: 10
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/cpaas-system
          operator: Exists
      volumes:
        - name: kafka-basic-auth
          secret:
            defaultMode: 420
            secretName: kafka-basic-auth
        - name: elasticsearch-basic-auth
          secret:
            defaultMode: 420
            secretName: elasticsearch-basic-auth
        - configMap:
            defaultMode: 420
            name: log-exporter-config
          name: config
        - hostPath:
            path: /cpaas/data/logarchive    # Local temporary storage address for logs
            type: DirectoryOrCreate
          name: data
EOF

当容器状态变为 Running 后，你可以在 bucket 中查看持续归档的日志。

#如何将日志归档到第三方存储

#目录

#传输到外部 NFS

#前提条件

#创建日志同步资源

#传输到外部 S3 存储

#前提条件

#创建日志同步资源

如何将日志归档到第三方存储

目录

传输到外部 NFS

前提条件

创建日志同步资源

传输到外部 S3 存储

前提条件

创建日志同步资源