Alauda Container Platform
  • 简体中文

    • 软数据中心 LB 解决方案（Alpha）

    通过在集群外创建一个高可用负载均衡器，部署纯软件的数据中心负载均衡器（LB），为多个 ALB 提供负载均衡能力，以确保业务稳定运行。它支持仅配置 IPv4、仅配置 IPv6，或同时配置 IPv4 和 IPv6 双栈。

    目录

    前提条件操作步骤验证

    前提条件

    1. 准备两个或更多主机节点作为 LB。建议在 LB 节点上安装 Ubuntu 22.04 操作系统，以减少 LB 将流量转发到异常后端节点所需的时间。

    2. 在外部 LB 的所有主机节点上预先安装以下软件（本章以两个外部 LB 主机节点为例）：

      • ipvsadm

      • container-runtime such as containerd

    3. 确保每个主机的 container-runtime 在开机时启动。

    4. 确保每个主机节点的时钟已同步。

    5. 准备用于启动外部 LB 服务的 Keepalived 镜像；平台已包含该镜像。镜像地址格式如下：<image repository address>/tkestack/keepalived:<version suffix>。不同版本的版本后缀可能略有差异。可按以下方式获取镜像仓库地址和版本后缀。本文档以 build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923 为例。

      • 在 global 集群中执行 kubectl get prdb base -o json | jq .spec.registry.address，获取 镜像仓库地址 参数。

      • 在解压安装包的目录中执行 cat ./installer/res/artifacts.json |grep keepalived -C 2|grep tag|awk '{print $2}'|awk -F '"' '{print $2}'，获取 版本后缀。

    操作步骤

    注意：以下操作必须在每个外部 LB 主机节点上执行一次，且主机节点的 hostname 不能重复。

    1. 将以下配置信息添加到文件 /etc/modules-load.d/alive.kmod.conf 中。

      ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
nf_conntrack
ip6t_MASQUERADE
nf_nat_masquerade_ipv6
ip6table_nat
nf_conntrack_ipv6
nf_defrag_ipv6
nf_nat_ipv6
ip6_tables

    2. 将以下配置信息添加到文件 /etc/sysctl.d/alive.sysctl.conf 中。

      net.ipv4.ip_forward = 1
net.ipv4.conf.all.arp_accept = 1
net.ipv4.vs.conntrack = 1
net.ipv4.vs.conn_reuse_mode = 0
net.ipv4.vs.expire_nodest_conn = 1
net.ipv4.vs.expire_quiescent_template = 1
net.ipv6.conf.all.forwarding=1

    3. 使用 reboot 命令重启。

    4. 为 Keepalived 配置文件创建一个文件夹。

      mkdir -p /etc/keepalived
mkdir -p /etc/keepalived/kubecfg

    5. 根据以下文件中的注释修改配置项，并将其保存到 /etc/keepalived/ 目录下，文件名为 alive.yaml。

      instances:
  - vip: # Multiple VIPs can be configured
      vip: 192.168.128.118 # VIPs must be different
      id: 20 # Each VIP's ID must be unique, optional
      interface: "eth0"
      check_interval: 1 # optional, default 1: interval to execute check script
      check_timeout: 3  # optional, default 3: check script timeout period
      name: "vip-1" # Identifier for this instance, can only contain alphanumeric characters and hyphens, cannot start with a hyphen
      peer: [ "192.168.128.116", "192.168.128.75" ] # Keepalived node IP, actual generated keepalived.conf will remove all IPs on the interface.
      kube_lock:
        kubecfgs: # The kube-config list used by kube-lock will sequentially attempt these kubecfgs for leader election in Keepalived
          - "/live/cfg/kubecfg/kubecfg01.conf"
          - "/live/cfg/kubecfg/kubecfg02.conf"
          - "/live/cfg/kubecfg/kubecfg03.conf"
    ipvs: # Configuration for option IPVS
      ips: [ "192.168.143.192", "192.168.138.100","192.168.129.100" ] # IPVS backend, change k8s master node IP to ALB node's node IP
      ports: # Configure health check logic for each port on the VIP
        - port: 80 # The port on the virtual server must match the real server's port
          virtual_server_config: |
            delay_loop 10  # Interval for performing health checks on the real server
            lb_algo rr
            lb_kind NAT
            protocol TCP
          raw_check: |
            TCP_CHECK {
                connect_timeout 10
                connect_port 1936
            }
  - vip:
      vip: 2004::192:168:128:118
      id: 102
      interface: "eth0"
      peer: [ "2004::192:168:128:75","2004::192:168:128:116" ]
      kube_lock:
        kubecfgs: # The kube-config list used by kube-lock will sequentially attempt these kubecfgs for leader election in Keepalived
          - "/live/cfg/kubecfg/kubecfg01.conf"
          - "/live/cfg/kubecfg/kubecfg02.conf"
          - "/live/cfg/kubecfg/kubecfg03.conf"
    ipvs:
      ips: [ "2004::192:168:143:192","2004::192:168:138:100","2004::192:168:129:100" ]
      ports:
        - port: 80
          virtual_server_config: |
            delay_loop 10
            lb_algo rr
            lb_kind NAT
            protocol TCP
          raw_check: |
            TCP_CHECK {
                connect_timeout 1
                connect_port 1936
            }

    6. 在业务集群中执行以下命令，检查配置文件中的证书过期时间，确保该证书仍然有效。证书过期后，LB 功能将不可用，需要联系平台管理员更新证书。

      openssl x509 -in <(cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $NF}' | base64 -d ) -noout -dates

    7. 将 Kubernetes 集群中三个 Master 节点的 /etc/kubernetes/admin.conf 文件复制到外部 LB 节点的 /etc/keepalived/kubecfg 目录下，并按索引命名，例如 kubecfg01.conf，同时将这三个文件中的 apiserver 节点地址修改为 Kubernetes 集群的实际节点地址。

      注意：平台证书更新后，需要重新执行此步骤，并覆盖原有文件。

    8. 检查证书有效性。

      1. 将业务集群 Master 节点上的 /usr/bin/kubectl 复制到 LB 节点。

      2. 执行 chmod +x /usr/bin/kubectl 授予执行权限。

      3. 执行以下命令确认证书是否有效。

        kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node

        如果返回以下结果，则证书有效。

        kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg01.conf get node
## Output
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg02.conf get node
## Output
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

kubectl --kubeconfig=/etc/keepalived/kubecfg/kubecfg03.conf get node
## Output
NAME              STATUS   ROLES                  AGE     VERSION
192.168.129.100   Ready    <none>                 7d22h   v1.25.6
192.168.134.167   Ready    control-plane,master   7d22h   v1.25.6
192.168.138.100   Ready    <none>                 7d22h   v1.25.6
192.168.143.116   Ready    control-plane,master   7d22h   v1.25.6
192.168.143.192   Ready    <none>                 7d22h   v1.25.6
192.168.143.79    Ready    control-plane,master   7d22h   v1.25.6

    9. 将 Keepalived 镜像上传到外部 LB 节点，并使用 nerdctl 运行 Keepalived。

      nerdctl run -dt --restart=always --privileged --network=host -v /etc/keepalived:/live/cfg build-harbor.alauda.cn/tkestack/keepalived:v3.16.0-beta.3.g598ce923

    10. 在访问 keepalived 的节点上运行以下命令：sysctl -w net.ipv4.conf.all.arp_accept=1。

    验证

    1. 运行 ipvsadm -ln 命令查看 IPVS 规则，可以看到适用于业务集群 ALB 的 IPv4 和 IPv6 规则。

      IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight        ActiveConn InActConn
TCP  192.168.128.118:80 rr
  -> 192.168.129.100:80           Masq    1      0          0
  -> 192.168.138.100:80           Masq    1      0          0
  -> 192.168.143.192:80           Masq    1      0          0
TCP  [2004::192:168:128:118]:80 rr
  -> [2004::192:168:129:100]:80   Masq    1      0          0
  -> [2004::192:168:138:100]:80   Masq    1      0          0
  -> [2004::192:168:143:192]:80   Masq    1      0          0

    2. 关闭 VIP 所在的 LB 节点，测试 IPv4 和 IPv6 的 VIP 是否都能成功迁移到另一节点，通常会在 20 秒内完成。

    3. 在非 LB 节点上使用 curl 命令测试与 VIP 的通信是否正常。

      curl 192.168.128.118

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>

<p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
      curl -6 [2004::192:168:128:118]:80 -g

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>

<p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>