#How to Forward Logs to an External System with LogForward

The LogForward custom resource lets tenants forward the platform's log stream to an external system (Elasticsearch, OpenSearch, Loki, syslog, HTTP, ClickHouse, Kafka, console, or an advanced-sink escape hatch), independently of the platform's built-in storage operator. Each CR reads the platform's Kafka topics and produces one Vector Deployment.

LogForward is separate from and does not replace the platform's storage pipeline. Records still land in the storage cluster's ClickHouse or OpenSearch as usual; LogForward produces an additional copy.

The authoritative CRD source is api/v1alpha1/logforward_types.go in log-storage-operator. The samples below are trimmed from config/samples/logforward_*.yaml.

#Prerequisites

• The storage cluster the tenant lives on has Alauda Container Platform Log Storage Management installed and PlatformLogForward/platform-default is Ready. See Installation.
• The external destination is reachable from the storage cluster's cpaas-system namespace.
• Any Secret referenced by spec.output.authSecretRef or spec.output.tls.caSecretRef lives in cpaas-system. The webhook rejects a LogForward that references Secrets in any other namespace.

#Spec shape

Every LogForward has four sections:

• spec.inputs[] — one or more logical topics the CR consumes. Each input picks a type (log, audit, or event) that maps to the platform's Kafka topic, and an optional selector that filters by namespaces or podLabels.
• spec.filters[] — optional list of typed transforms (parse, filter, redact, prune, remapFields). Raw VRL is not allowed.
• spec.pipelines[] — required list that wires each pipeline's inputRefs and optional filterRefs into the single output.
• spec.output — exactly one sink. type picks the sink and the matching type-specific block carries its parameters; the shared endpoint, authSecretRef, and tls fields sit at the output's top level.

#Common Recipes

#Mirror the platform's log and audit streams to a user Kafka

Two inputs, no filters, one pipeline per input, all fanned into the same Kafka mirror sink.

apiVersion: log.alauda.io/v1alpha1
kind: LogForward
metadata:
  name: audit-mirror
  namespace: cpaas-system
spec:
  replicas: 2
  inputs:
    - name: audit-in
      type: audit
      initialOffset: Latest
    - name: log-in
      type: log
      initialOffset: Latest
      selector:
        namespaces:
          include: [default, kube-system]
  pipelines:
    - name: audit-pipe
      inputRefs: [audit-in]
    - name: log-pipe
      inputRefs: [log-in]
  output:
    type: kafka
    # Secret keys: username / password. Mounted as OUTPUT_KAFKA_USERNAME
    # and OUTPUT_KAFKA_PASSWORD, rendered into [sinks.output.sasl].
    authSecretRef:
      name: user-kafka-mirror-auth
    kafka:
      bootstrapServers: user-broker-1.example.com:9092,user-broker-2.example.com:9092
      topic: platform-mirror-logs
      saslMechanism: SCRAM-SHA-512      # PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512 | ""
      compression: gzip                 # none | gzip | snappy | lz4 | zstd

#Forward the platform's log stream to a user OpenSearch, with prune and redact

apiVersion: log.alauda.io/v1alpha1
kind: LogForward
metadata:
  name: user-opensearch
  namespace: cpaas-system
spec:
  replicas: 2
  inputs:
    - name: platform-log
      type: log
      initialOffset: Latest
  filters:
    - name: strip-noise
      type: prune
      prune:
        deny: [pod_labels, node_name]
    - name: redact-token
      type: redact
      redact:
        fields: [access_token, password]
        replacement: "[REDACTED]"
  pipelines:
    - name: to-os
      inputRefs: [platform-log]
      filterRefs: [strip-noise, redact-token]
  output:
    type: opensearch                     # elasticsearch driver, OS-compatible
    endpoint: https://user-os.example.com:9200
    authSecretRef:
      name: user-os-basic-auth           # Secret keys: username / password
    tls:
      verify: true
      caSecretRef:
        name: user-os-tls-ca             # Secret key: ca.crt
    opensearch:
      index: acp-forward-%Y%m%d          # strftime template
      bulkAction: create                 # create | index | update

#Forward audit records to an HTTP intake

apiVersion: log.alauda.io/v1alpha1
kind: LogForward
metadata:
  name: compliance-audit-http
  namespace: cpaas-system
spec:
  inputs:
    - name: audit-in
      type: audit
  pipelines:
    - name: to-intake
      inputRefs: [audit-in]
  output:
    type: http
    endpoint: https://intake.example.com/logs
    authSecretRef:
      name: audit-intake-basic           # Secret keys: username / password
    http:
      method: post                       # post | put
      encodingCodec: json                # json | text
      framingMethod: newline_delimited   # newline_delimited | character_delimited

#Ship a namespace's log stream to a syslog collector

apiVersion: log.alauda.io/v1alpha1
kind: LogForward
metadata:
  name: security-workload-syslog
  namespace: cpaas-system
spec:
  inputs:
    - name: security-ns
      type: log
      selector:
        namespaces:
          include: [security]
  pipelines:
    - name: to-syslog
      inputRefs: [security-ns]
  output:
    type: syslog
    endpoint: syslog.example.com:514
    syslog:
      mode: tcp                          # tcp | udp | unix | tls

#Verify

After you apply a LogForward, log-storage-operator builds a Vector Deployment for it in cpaas-system. Watch the resource until it reports Ready=True:

kubectl -n cpaas-system get logforward <name> -w

Read the conditions if it does not go Ready:

kubectl -n cpaas-system get logforward <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

The LogForward status is deliberately minimal — observedGeneration, Phase, and readiness conditions. It does not report consumer group, topic, lag, partition, offset, replica detail, output name, endpoint, or Secret name. For runtime data-path issues, read the generated Vector Deployment's pod logs.

#Limits and Notes

• One LogForward produces exactly one Vector Deployment. Each replica joins the same Kafka consumer group.
• The consumer group name is logforward-<lf-name>. The platform's KafkaUser must include a logforward- group prefix ACL rule; without it, vector silently fails with GroupAuthorizationFailed and the LF reports Ready=True while reading zero records. See Installation Section 1.1.1 step 3 for the required ACL. (Bug 168d v2.)
• One LogForward has exactly one output. To fan out to multiple destinations, create one LogForward per destination.
• Every filter of type filter must set at least one non-empty predicate (equals, notEquals, contains, or matches) per condition. An all-empty condition compiles to false and aborts every event silently — the webhook rejects it at admission. (Bug 173 pt1.)
• Uninstalling Alauda Container Platform Log Storage Management stops the operator; any active LogForward Deployments are removed as part of the storage operator uninstall.
• LogForward is not intended as a replacement for the platform's storage pipeline. Removing the storage operator removes log query on the console; LogForward alone does not restore that.

#Related Topics

• Installation — how to install the storage operator that publishes the LogForward CRD, and how to set up the logforward- KafkaUser prefix ACL rule.
• How to Archive Logs to Third-Party Storage — the older razor consumer --database-type=file archive workflow, intended for NFS / S3 archival rather than real-time forwarding.