使用 MLflow Tracking 和 TrustyAI Evaluation 的每日微调流水线

这是一个完整的 Kubeflow Pipeline:对 LLM 进行微调(此示例使用 Qwen3-0.6B),在 MLflow 中跟踪运行,将生成的模型注册到 MLflow Model Registry,将其部署为 KServe InferenceService,使用 TrustyAI LMEvalJob 对其进行评估,将评估结果写回同一个 MLflow experiment,并清理临时的服务资源。然后,流水线会连接到 KFP Recurring Run,每天运行一次,从而逐渐积累 train → eval 运行记录;MLflow 的比较视图会将这些记录转化为回归信号。

本指南将已分别介绍的各个部分——使用 Kubeflow Pipelines、Kubeflow Pipeline + MLflow Integration、Evaluate LLM——整合成一套可运行的方案。如果不熟悉其中任何步骤,请先阅读相关指南。

流水线的作用

                     ┌───────────────┐
   Qwen3-0.6B  ─────►│  fine-tune    │──► loss/eval_loss/perplexity ─► MLflow parent run
      base model     │  (SFTTrainer) │──► fine-tuned model artifacts ─► MLflow Model Registry
                     └──────┬────────┘        (registered_model_name="qwen3-0.6b-sft")
                            │
                            ▼
                     ┌───────────────┐
                     │ deploy-for-   │──► KServe InferenceService (temporary)
                     │ evaluation    │
                     └──────┬────────┘
                            │
                            ▼
                     ┌───────────────┐
                     │ evaluate      │──► TrustyAI LMEvalJob (arc_easy, mmlu, …)
                     │ (LM-Eval)     │──► eval metrics ─► MLflow nested "eval" run
                     └──────┬────────┘                    + tag on model version
                            │
                            ▼
                     ┌───────────────┐
                     │  cleanup      │──► delete InferenceService + LMEvalJob
                     └───────────────┘

每天运行一次,就会在 MLflow experiment 中新增一行。由于训练指标和评估指标都位于同一个父运行下,MLflow 的 Compare 和 Chart 视图无需额外的粘合代码,即可绘制每日损失和评估准确率的变化情况。

前提条件

要求详情
Alauda AI 2.5 或更高版本已安装 Kubeflow Pipelines、MLflow(已配置 Model Registry artifact store)、TrustyAI 和 KServe。
预构建的流水线运行时镜像以下四个 KFP 组件使用 docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0 (linux/amd64)。该镜像内置完整的 CUDA / HF / trl / MLflow / KServe / kubernetes 技术栈,因此 Pod 启动时无需运行 pip install。将镜像镜像到内部 registry 后,同样的方案也适用于隔离网络的集群。请参阅下文的构建自定义镜像,了解如何基于此镜像创建自己的镜像;如需查看源 Containerfile 目录,请参阅 Training Runtime Images。
MLflow artifact store在 MLflow plugin 中配置与 S3 兼容的对象存储。mlflow.transformers.log_model() 需要此存储来持久保存模型文件,供 KServe 稍后拉取。请参阅 MLflow install 的 High availability and storage 部分。
带有 MLflow 标签的命名空间 finetune命名空间上需设置 mlflow-enabled=true,使其显示为工作区。
共享 PVC finetune-shared (RWX)用于在组件之间缓存基础模型和微调后的检查点。任何支持 RWX 的 StorageClass(CephFS、NFS、JuiceFS)均可使用。
一个 GPU 节点一块显存 ≥ 16 GiB 的 NVIDIA GPU,即可在小型数据集上对 Qwen3-0.6B 执行全精度 SFT;进行小型 LoRA 运行时,8 GiB 即可。
MLflow token Secret一个名为 mlflow-token 的 Secret,其中包含键 token,其值为有权访问 finetune MLflow workspace 的服务账户的 Dex ID token。请参阅从命令行获取 token。
finetune 命名空间中的 RBAC流水线 ServiceAccount 可以对 inferenceservices.serving.kserve.io 和 lmevaljobs.trustyai.opendatahub.io 执行 get/create/delete。
训练数据集(可选)fine_tune 组件默认使用一个小型的进程内合成语料库,因此执行冒烟测试无需暂存数据集。对于实际工作负载,请预先将 JSONL 文件暂存到共享 PVC 或 seaweedfs / S3 中,并将其路径作为 dataset_path 传入;请参阅下文的使用真实数据集。
可访问 Hugging Face(或使用离线 PVC)默认情况下,有两个组件会从 Hugging Face 拉取内容:(a) fine_tune 通过 AutoModelForCausalLM.from_pretrained(...) 下载基础模型 Qwen/Qwen3-0.6B;(b) evaluate 组件的 LMEvalJob 会获取 tokenizer 和数据集——即使 tokenized_requests 为 False,AutoTokenizer.from_pretrained(...) 仍会运行。对于 (a),请将基础模型镜像到共享 PVC 中一次(请参阅使用真实数据集 → 基础模型说明)。对于 (b),请将评估作业切换到离线模式(请参阅 Evaluate LLM):预先在 PVC 中填充 tokenizer 和数据集缓存,设置 spec.offline.storage.pvcName,并将 tokenizer modelArgs 指向挂载路径。

步骤 1 — 创建 MLflow token Secret 和共享 PVC

NS=finetune

# 1. Mint a Dex id token (browser-free) — see the SDK guide for ID_TOKEN.
kubectl -n $NS create secret generic mlflow-token --from-literal=token="$ID_TOKEN"

# 2. Shared PVC used by fine-tune (write) and deploy (read).
kubectl -n $NS apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: finetune-shared
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 50Gi
  storageClassName: cephfs   # any RWX-capable StorageClass
EOF

以下流水线组件假定这两个对象都已存在于运行流水线的命名空间中。

步骤 2 — 流水线

将以下内容保存为 finetune_pipeline.py。流水线拆分为四个组件,以便将故障范围限定在局部,并降低重新运行的成本;组件通过 PVC 和模型版本上的 MLflow 标签共享状态。

# finetune_pipeline.py
from kfp import dsl, compiler, kubernetes

BASE_MODEL      = "Qwen/Qwen3-0.6B"
REGISTERED_NAME = "qwen3-0.6b-sft"
WORKSPACE       = "finetune"                       # MLflow workspace (= namespace)
EXPERIMENT      = "qwen3-0.6b-daily-sft"
PVC_NAME        = "finetune-shared"
PVC_MOUNT       = "/mnt/shared"

MLFLOW_URI      = "http://mlflow-tracking-server.kubeflow:5000"

# Pre-built runtime image with torch (CUDA 12.6) + HF + trl + mlflow + kserve
# + kubernetes baked in. No pip install runs at pod start, so the same tag
# works on air-gapped clusters once it's mirrored into an internal registry.
RUNTIME_IMAGE   = "docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0"


# ---------------------------------------------------------------------------
# 1. Fine-tune with the HF Trainer, autolog to MLflow, register the model.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def fine_tune(
    workspace: str,
    experiment: str,
    base_model: str,
    registered_name: str,
    pvc_mount: str,
    run_id: str,
    # Empty = generate a tiny synthetic instruction corpus in-process (no
    # network egress; smoke-shape only, not a meaningful fine-tune). Non-empty
    # = a local JSONL path (e.g. "/mnt/shared/datasets/alpaca.jsonl") or an
    # "s3://…" URI to a JSONL file with a `text` column. See the "Using a
    # real dataset" section for how to prepare the file.
    dataset_path: str = "",
    num_train_epochs: int = 1,
    learning_rate: float = 2e-4,
    per_device_train_batch_size: int = 2,
    max_samples: int = 512,
) -> str:
    """Full SFT of `base_model` on a small synthetic corpus (or a JSONL file
    at `dataset_path`). Logs metrics + registers the fine-tuned model.
    Returns the model version as a string."""
    import operator, os, random, mlflow, mlflow.transformers
    from datasets import Dataset, load_dataset
    from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
    from trl import SFTTrainer

    # 1) MLflow (auth via MLFLOW_TRACKING_TOKEN injected from the Secret).
    mlflow.set_tracking_uri(MLFLOW_URI)
    mlflow.set_workspace(workspace)
    mlflow.set_experiment(experiment)
    mlflow.transformers.autolog(log_models=False)          # models are logged manually below

    # 2) Data. Synthetic by default so the pipeline's smoke run needs no
    # network egress and no dataset pre-staging. Swap in a real dataset by
    # passing `dataset_path` (JSONL file with a `text` column) — either a
    # PVC path or an S3 URI. See "Using a real dataset" below.
    if dataset_path.startswith("s3://"):
        ds = load_dataset(
            "json", data_files=dataset_path, split=f"train[:{max_samples}]",
            # storage_options / credentials come from AWS_* env; see the docs.
        )
    elif dataset_path:
        ds = load_dataset(
            "json", data_files=dataset_path, split=f"train[:{max_samples}]",
        )
    else:
        rng = random.Random(42)
        ops = [("+", operator.add), ("-", operator.sub), ("*", operator.mul)]
        rows = []
        for _ in range(max_samples):
            a, b = rng.randint(0, 99), rng.randint(0, 99)
            sym, op = rng.choice(ops)
            rows.append({"text":
                f"### Instruction:\nWhat is {a} {sym} {b}?\n"
                f"### Response:\nThe answer is {op(a, b)}."
            })
        ds = Dataset.from_list(rows)

    # 3) Model & tokenizer.
    tokenizer = AutoTokenizer.from_pretrained(base_model)
    model     = AutoModelForCausalLM.from_pretrained(base_model)

    output_dir = f"{pvc_mount}/models/{registered_name}/{run_id}"

    # 4) Train. `report_to="mlflow"` streams loss / eval_loss / lr per step.
    args = TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=num_train_epochs,
        learning_rate=learning_rate,
        per_device_train_batch_size=per_device_train_batch_size,
        logging_steps=10,
        save_strategy="no",
        report_to="mlflow",
        run_name=f"train-{run_id}",
    )
    with mlflow.start_run(run_name=f"pipeline-{run_id}") as parent:
        mlflow.set_tag("pipeline_run_id", run_id)
        mlflow.set_tag("base_model", base_model)
        with mlflow.start_run(run_name=f"train-{run_id}", nested=True):
            # TRL 0.12+ renamed the `tokenizer` kwarg to `processing_class`.
            trainer = SFTTrainer(model=model, args=args, train_dataset=ds,
                                 processing_class=tokenizer)
            trainer.train()
            trainer.save_model(output_dir)

        # 5) Register the fine-tuned model.
        # The artifact upload uses the MLflow plugin's artifact store (S3).
        # KServe then pulls the model back from that S3 URI in the deploy step.
        # `task="text-generation"` is required because MLflow cannot infer the
        # task from a locally-loaded (non-Hub) model.
        mv = mlflow.transformers.log_model(
            transformers_model={"model": trainer.model, "tokenizer": tokenizer},
            artifact_path="model",
            registered_model_name=registered_name,
            task="text-generation",
        )

        # 6) Record the PVC path on the model version so downstream steps
        #    can serve it from the shared PVC without another S3 round-trip.
        client = mlflow.MlflowClient()
        version = client.get_latest_versions(registered_name, stages=["None"])[0].version
        client.set_model_version_tag(registered_name, version, "pvc_path", output_dir)
        client.set_model_version_tag(registered_name, version, "pipeline_run_id", run_id)

    print(f"registered {registered_name} version {version} @ {output_dir}")
    return version


# ---------------------------------------------------------------------------
# 2. Deploy the just-registered model as a temporary KServe InferenceService.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def deploy_for_evaluation(
    workspace: str,
    registered_name: str,
    model_version: str,
    pvc_name: str,
    pvc_mount: str,
    run_id: str,
) -> str:
    """Create an InferenceService that serves the model straight off the PVC.
    Returns the InferenceService name."""
    import time, mlflow
    from kubernetes import client, config
    from kserve import KServeClient, constants
    from kserve import (V1beta1InferenceService, V1beta1InferenceServiceSpec,
                        V1beta1PredictorSpec, V1beta1ModelSpec, V1beta1ModelFormat)

    # Resolve the PVC path from the tag set by fine_tune().
    mlflow.set_tracking_uri(MLFLOW_URI); mlflow.set_workspace(workspace)
    tags = mlflow.MlflowClient().get_model_version(registered_name, model_version).tags
    pvc_path = tags["pvc_path"]

    isvc_name = f"{registered_name}-eval-{run_id[-8:]}"
    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()

    isvc = V1beta1InferenceService(
        api_version=constants.KSERVE_GROUP + "/v1beta1", kind=constants.KSERVE_KIND,
        metadata=client.V1ObjectMeta(name=isvc_name, namespace=ns, labels={
            "app.kubernetes.io/part-of": "finetune-pipeline",
            "pipeline-run-id": run_id,
        }),
        spec=V1beta1InferenceServiceSpec(predictor=V1beta1PredictorSpec(
            model=V1beta1ModelSpec(
                model_format=V1beta1ModelFormat(name="huggingface"),
                runtime="kserve-huggingfaceserver",
                storage_uri=f"pvc://{pvc_name}{pvc_path.removeprefix(pvc_mount)}",
                resources=client.V1ResourceRequirements(
                    limits={"nvidia.com/gpu": "1", "memory": "16Gi"},
                    requests={"nvidia.com/gpu": "1", "memory": "8Gi"},
                ),
            ),
        )),
    )
    KServeClient().create(isvc)

    # Wait for READY.
    deadline = time.time() + 15 * 60
    while time.time() < deadline:
        got = KServeClient().get(isvc_name, namespace=ns)
        cond = {c["type"]: c["status"] for c in (got.get("status", {}).get("conditions") or [])}
        if cond.get("Ready") == "True":
            print(f"InferenceService {isvc_name} is Ready")
            return isvc_name
        time.sleep(10)
    raise TimeoutError(f"{isvc_name} did not become Ready in 15 min")


# ---------------------------------------------------------------------------
# 3. Evaluate the InferenceService with a TrustyAI LMEvalJob, log to MLflow.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def evaluate(
    workspace: str,
    experiment: str,
    registered_name: str,
    model_version: str,
    isvc_name: str,
    run_id: str,
    tasks: list = ["arc_easy", "hellaswag"],
    limit: str = "50",
) -> dict:
    """Create an LMEvalJob against the InferenceService, wait for it to
    Complete, log the metrics into the parent MLflow run + as a nested run,
    and tag the model version with the primary accuracy."""
    import json, time, mlflow
    from kubernetes import client, config

    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()
    api = client.CustomObjectsApi()

    job_name = f"eval-{isvc_name}"
    body = dict(
        apiVersion="trustyai.opendatahub.io/v1alpha1", kind="LMEvalJob",
        metadata=dict(name=job_name, labels={"pipeline-run-id": run_id}),
        spec=dict(
            model="local-completions",
            modelArgs=[
                dict(name="model", value=isvc_name),
                dict(name="base_url",
                     value=f"http://{isvc_name}-predictor.{ns}.svc/v1/completions"),
                dict(name="num_concurrent", value="1"),
                dict(name="max_retries", value="3"),
                dict(name="tokenized_requests", value="True"),
                dict(name="tokenizer", value="Qwen/Qwen3-0.6B"),
            ],
            taskList=dict(taskNames=list(tasks)),
            allowOnline=True, allowCodeExecution=False,
            batchSize="1", limit=limit, logSamples=False,
            chatTemplate=dict(enabled=False),
            outputs=dict(pvcManaged=dict(size="100Mi")),
            # The ta-lmes-job image runs as UID 65532; a fresh PVC is root-owned,
            # so the driver's `open(stdout.log)` fails with permission denied
            # unless fsGroup gives the pod's group write access to the mount.
            pod=dict(securityContext=dict(fsGroup=65532)),
        ),
    )
    api.create_namespaced_custom_object(
        "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", body)

    # Wait for a terminal state. `state` reaches `Complete` on both success
    # and failure — the reason field is what actually distinguishes them.
    deadline = time.time() + 60 * 60
    while time.time() < deadline:
        got = api.get_namespaced_custom_object(
            "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", job_name)
        status = got.get("status") or {}
        state, reason = status.get("state"), status.get("reason")
        if state == "Complete" and reason == "Succeeded":
            break
        if state in {"Cancelled", "Failed"} or (state == "Complete" and reason in {"Failed", "Cancelled"}):
            raise RuntimeError(
                f"LMEvalJob {job_name} ended: state={state} reason={reason} "
                f"message={status.get('message')}")
        time.sleep(20)
    else:
        raise TimeoutError(f"LMEvalJob {job_name} did not finish in 1 h")

    results = json.loads(got["status"]["results"])["results"]

    # Log each task's metrics back to MLflow.
    mlflow.set_tracking_uri(MLFLOW_URI)
    mlflow.set_workspace(workspace)
    mlflow.set_experiment(experiment)
    parent = mlflow.search_runs(filter_string=f"tags.pipeline_run_id = '{run_id}'",
                                order_by=["attributes.start_time DESC"], max_results=1)
    parent_run_id = parent.iloc[0]["run_id"]

    flat = {}
    with mlflow.start_run(run_id=parent_run_id):
        with mlflow.start_run(run_name=f"eval-{run_id}", nested=True):
            for task, metrics in results.items():
                for k, v in metrics.items():
                    if isinstance(v, (int, float)) and "stderr" not in k:
                        name = f"{task}/{k.replace(',', '_')}"
                        mlflow.log_metric(name, float(v))
                        flat[name] = float(v)
            mlflow.set_tag("model_version", model_version)
            mlflow.set_tag("isvc_name", isvc_name)

    # Tag the primary accuracy on the model version so it shows up on the
    # Model Registry page next to the version number.
    primary = next((k for k in flat if k.endswith("/acc_none")), None)
    if primary is not None:
        mlflow.MlflowClient().set_model_version_tag(
            registered_name, model_version, "eval_acc", f"{flat[primary]:.4f}")
    return flat


# ---------------------------------------------------------------------------
# 4. Cleanup: delete the temporary InferenceService and LMEvalJob.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def cleanup(isvc_name: str):
    from kubernetes import client, config
    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()
    api = client.CustomObjectsApi()
    for gv, plural, name in [
        ("serving.kserve.io/v1beta1", "inferenceservices", isvc_name),
        ("trustyai.opendatahub.io/v1alpha1", "lmevaljobs", f"eval-{isvc_name}"),
    ]:
        group, version = gv.split("/")
        try:
            api.delete_namespaced_custom_object(group, version, ns, plural, name)
        except Exception as exc:
            print(f"delete {plural}/{name} skipped: {exc}")


# ---------------------------------------------------------------------------
# Pipeline: wire the four components together.
# ---------------------------------------------------------------------------
@dsl.pipeline(name="qwen3-sft-mlflow-trustyai",
              description="Daily SFT + MLflow tracking + TrustyAI evaluation")
def sft_mlflow_trustyai(
    workspace: str = WORKSPACE,
    experiment: str = EXPERIMENT,
    base_model: str = BASE_MODEL,
    registered_name: str = REGISTERED_NAME,
    # Default = synthetic in-process data (offline smoke). Override with a
    # JSONL path on the shared PVC or an s3:// URI to fine-tune on a real
    # dataset — see the "Using a real dataset" section.
    dataset_path: str = "",
    num_train_epochs: int = 1,
    learning_rate: float = 2e-4,
    max_samples: int = 512,
    eval_limit: str = "50",
):
    train = fine_tune(
        workspace=workspace, experiment=experiment,
        base_model=base_model, registered_name=registered_name,
        pvc_mount=PVC_MOUNT, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
        dataset_path=dataset_path, num_train_epochs=num_train_epochs,
        learning_rate=learning_rate, max_samples=max_samples,
    ).set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1)
    train.set_memory_limit("32Gi").set_cpu_limit("8")

    deploy = deploy_for_evaluation(
        workspace=workspace, registered_name=registered_name,
        model_version=train.output, pvc_name=PVC_NAME, pvc_mount=PVC_MOUNT,
        run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
    )

    ev = evaluate(
        workspace=workspace, experiment=experiment,
        registered_name=registered_name, model_version=train.output,
        isvc_name=deploy.output, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
    )

    with dsl.ExitHandler(cleanup(isvc_name=deploy.output)):
        ev  # cleanup runs whether evaluate() succeeds or not

    # Mount the shared PVC on training + deploy.
    for task in (train, deploy):
        kubernetes.mount_pvc(task, pvc_name=PVC_NAME, mount_path=PVC_MOUNT)

    # Inject the Dex id token into every component that talks to MLflow.
    for task in (train, deploy, ev):
        kubernetes.use_secret_as_env(
            task, secret_name="mlflow-token",
            secret_key_to_env={"token": "MLFLOW_TRACKING_TOKEN"})


compiler.Compiler().compile(sft_mlflow_trustyai, "pipeline.yaml")

编译:

pip install "kfp>=2.7" "kfp-kubernetes>=1.3"
python finetune_pipeline.py
# → pipeline.yaml

在隔离网络的集群上使用真实数据集

fine_tune 组件默认使用进程内的合成指令语料库(数百组算术问答)。这样,冒烟测试就能完全离线运行——无需 HuggingFace、ModelScope 或 S3——但无法产生有意义的微调结果。对于实际工作负载,请先将数据集暂存到集群中,然后在每次运行时将其路径作为 dataset_path 传入。fine_tune 组件接受本地 JSONL 路径(最简单)或 s3://… URI(可供多个流水线共享)。

1. 在可访问外网的机器上下载数据集

任何可以访问 Hugging Face 的机器都可以——笔记本电脑、跳板机,或位于可访问外网的命名空间中的 Workbench:

pip install huggingface_hub datasets
huggingface-cli download tatsu-lab/alpaca --repo-type dataset --local-dir ./alpaca-raw

如果 Hugging Face 被屏蔽,ModelScope 镜像了大多数热门数据集,可在中国大陆网络环境下使用:

pip install modelscope
modelscope download --dataset AI-ModelScope/alpaca-gpt4-data-en --local_dir ./alpaca-raw

2. 转换为包含 text 列的 JSONL 文件

TRL 的 SFTTrainer 要求每行包含一个文本字段。将原始数据集转换为流水线所需的格式:

from datasets import load_dataset

raw = load_dataset("./alpaca-raw")["train"]

def format_alpaca(row):
    prompt = row["instruction"]
    if row.get("input"):
        prompt += "\n\n" + row["input"]
    return {"text": f"### Instruction:\n{prompt}\n### Response:\n{row['output']}"}

raw.map(format_alpaca).to_json("alpaca.jsonl", orient="records", lines=True)

3a. 上传到共享 PVC(最简单)

如果集群中有挂载了 finetune-shared PVC 的跳板 Pod,请将 JSONL 文件 kubectl cp 到该 PVC:

kubectl -n finetune apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: dataset-uploader, namespace: finetune }
spec:
  restartPolicy: Never
  containers:
  - name: sh
    image: busybox
    command: ["sh", "-c", "mkdir -p /mnt/shared/datasets && sleep 3600"]
    volumeMounts: [{ name: shared, mountPath: /mnt/shared }]
  volumes:
  - name: shared
    persistentVolumeClaim: { claimName: finetune-shared }
EOF
kubectl -n finetune wait --for=condition=Ready pod/dataset-uploader
kubectl -n finetune cp alpaca.jsonl dataset-uploader:/mnt/shared/datasets/alpaca.jsonl
kubectl -n finetune delete pod dataset-uploader

然后使用 dataset_path="/mnt/shared/datasets/alpaca.jsonl" 提交流水线(请参阅下文的步骤 3)。

3b. 上传到 S3 / seaweedfs(供多个流水线共享)

对于运行多个流水线的团队,可将数据集存放在与 MLflow plugin 用于 artifact store 的同一个 S3 兼容存储桶中——集群内部的 seaweedfs 端点可从每个流水线 Pod 访问,客户端只需设置 AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY 环境变量。

从可访问 S3 端点的机器上传数据——通常是在同一集群中的 Workbench。Alauda AI Workbench 镜像已预装 mc(MinIO Client)和 boto3,因此无需额外安装。

使用 mc(交互式 / 单行命令):

mc alias set seaweedfs http://seaweedfs.mlops-demo-e2e:8333 <ACCESS_KEY> <SECRET_KEY>
mc mb --ignore-existing seaweedfs/mlops-datasets
mc cp alpaca.jsonl seaweedfs/mlops-datasets/alpaca/alpaca.jsonl

使用 boto3(脚本化 / 从 notebook 执行):

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://seaweedfs.mlops-demo-e2e:8333",
    aws_access_key_id="<ACCESS_KEY>",
    aws_secret_access_key="<SECRET_KEY>",
)
# Idempotent bucket create.
try:
    s3.create_bucket(Bucket="mlops-datasets")
except s3.exceptions.BucketAlreadyOwnedByYou:
    pass
s3.upload_file("alpaca.jsonl", "mlops-datasets", "alpaca/alpaca.jsonl")

(请根据 seaweedfs / MinIO Service 调整端点。凭据可从 MLflow plugin 的 artifact-store Secret 或本地 S3 提供商处获取。)

然后创建一个包含 S3 凭据的 Secret,并将其与 mlflow-token 一同注入 fine_tune:

kubectl -n finetune create secret generic mlflow-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=<key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret> \
  --from-literal=MLFLOW_S3_ENDPOINT_URL=http://seaweedfs.mlops-demo-e2e:8333

在流水线函数中添加以下内容(紧邻现有的挂载 MLflow token 的 use_secret_as_env 块):

kubernetes.use_secret_as_env(train, secret_name="mlflow-s3", secret_key_to_env={
    "AWS_ACCESS_KEY_ID":       "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY":   "AWS_SECRET_ACCESS_KEY",
    "MLFLOW_S3_ENDPOINT_URL":  "AWS_ENDPOINT_URL",   # what s3fs / fsspec read
})

然后使用 dataset_path="s3://mlops-datasets/alpaca/alpaca.jsonl" 提交流水线。运行时镜像中已包含 s3fs,而 load_dataset("json", data_files="s3://…") 在底层使用 fsspec——系统会自动读取 AWS_* 环境变量。

4. 关于基础模型的说明

AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B") 首次运行时也会访问 Hugging Face。若要完全在隔离网络中进行微调,请按相同方式镜像一次模型(huggingface-cli download Qwen/Qwen3-0.6B --local-dir ./qwen3-0.6b,然后复制到 PVC 或 S3),并设置 BASE_MODEL = "/mnt/shared/models/qwen3-0.6b"。之后,MLflow model registry 仍会像之前一样,将微调后的模型注册到 qwen3-0.6b-sft 下。

步骤 3 — 提交一次性运行

在 Kubeflow Dashboard → Pipelines → Upload Pipeline 中上传 pipeline.yaml,然后点击 Create Run;或者在 Workbench 中运行:

from kfp.client import Client

client = Client(host="<MY-KFP-ENDPOINT>")
run = client.create_run_from_pipeline_package(
    "pipeline.yaml",
    arguments=dict(
        # Omit for the synthetic-smoke default; set to a JSONL path or an
        # s3:// URI to fine-tune on a real dataset (see the section above).
        # dataset_path="/mnt/shared/datasets/alpaca.jsonl",
        num_train_epochs=1, eval_limit="50",
    ),
)
print("Run:", run.run_id)

运行期间:

  • 打开 Alauda AI Platform → Tools → MLFlow,然后选择 finetune 工作区。
  • qwen3-0.6b-daily-sft 实验会显示一个名为 pipeline-<pipeline-run-id> 的父级运行,其中包含两个嵌套运行:train-<...>(由 HF Trainer callback 流式写入)和稍后出现的 eval-<...>(由 evaluate 步骤写入)。
  • 在 Models 下,已注册模型 qwen3-0.6b-sft 会新增一个版本。

步骤 4 — 在 MLflow 中可视化训练指标

  1. 在 MLflow UI 中,打开父级运行 → Metrics 选项卡。
  2. 选择 loss、eval_loss 以及你添加的任何自定义指标(perplexity、learning_rate、grad_norm)→ 点击 Chart。
  3. 若要实时查看损失曲线,请在图表面板中将 Refresh 间隔设为 10s — Trainer 写入新步骤时,MLflow 会重新获取指标流。

对于关注准确率的分类类目标,可在 TrainingArguments 中添加一个返回 {"accuracy": ...} 的 compute_metrics callback;report_to="mlflow" 会将 compute_metrics 返回的内容直接流式传输到同一图表中。

步骤 5 — 将其安排为每日 Recurring Run

KFP 原生支持 cron 调度。将同一个 pipeline.yaml 配置为 Recurring Run,使其每天运行。

在 UI 中

  1. Pipelines → 选择 qwen3-sft-mlflow-trustyai → Create Run。
  2. 选择 Recurring Run 作为运行类型。
  3. Trigger:Cron,表达式 0 2 * * *(每天 UTC 时间 02:00 )。
  4. Max concurrent runs:1 — 流水线会占用一个 GPU;重叠的运行将进入队列。
  5. Catchup:off — 跳过错过的时间窗口,而不是在周一早上补跑积压的任务。
  6. Parameters:为每日任务设置 num_train_epochs=1、eval_limit=200(比一次性运行的默认值更有意义)。
  7. Start。

使用 SDK

from kfp.client import Client

client = Client(host="<MY-KFP-ENDPOINT>")

# One-time: upload the pipeline as a versioned resource.
uploaded = client.upload_pipeline(
    "pipeline.yaml", pipeline_name="qwen3-sft-mlflow-trustyai")

# Attach a daily schedule.
client.create_recurring_run(
    experiment_id=client.create_experiment("qwen3-daily").experiment_id,
    job_name="qwen3-sft-daily-02utc",
    cron_expression="0 0 2 * * *",     # KFP uses 6-field cron (with seconds)
    max_concurrency=1,
    no_catchup=True,
    enabled=True,
    pipeline_id=uploaded.pipeline_id,
    version_id=uploaded.pipeline_version_id,
    params=dict(num_train_epochs=1, eval_limit="200"),
)
WARNING

Secret/mlflow-token 中的 MLflow token 会过期(Dex id token 默认有效期为 24 小时)。对于运行时间超过 token 有效期的调度任务,可以在 fine_tune 组件内部使用 service-account 凭据生成 token 并刷新它(请参阅 SDK token 流程),或者运行一个小型 CronJob,按略短于 token TTL 的周期轮换 mlflow-token Secret。

步骤 6 — 在 MLflow 中比较每日评估结果

运行几天后,实验中每天都会有一条记录。只需点击一次,即可在 MLflow 中进行比较:

  1. 在 MLflow UI 中,打开 qwen3-0.6b-daily-sft 实验。
  2. 在运行表中,筛选嵌套评估运行:tags.mlflow.runName LIKE 'eval-%'。
  3. 按 attributes.start_time DESC 排序。
  4. 勾选要比较的运行(最近 7 天是一个不错的起点)→ 点击 Compare。
  5. 在 Chart 选项卡中,绘制 arc_easy/acc_none 和 hellaswag/acc_none,并按 tags.mlflow.runName 分组。曲线平稳表示微调过程稳定;如果曲线骤降,通常意味着训练数据集或基础检查点发生了变化。
  6. Parallel Coordinates 视图会汇总训练超参数和评估指标,方便你直观判断哪些超参数与最大的评估增益相关。

如需以编程方式进行比较(例如,用于触发 Slack 告警,或设置 CI 门禁:当准确率日环比下降超过 2 % 时使流水线失败),请使用搜索 API:

import mlflow, pandas as pd

mlflow.set_tracking_uri("http://mlflow-tracking-server.kubeflow:5000")
mlflow.set_workspace("finetune")

df = mlflow.search_runs(
    experiment_names=["qwen3-0.6b-daily-sft"],
    filter_string="tags.mlflow.runName LIKE 'eval-%'",
    order_by=["attributes.start_time DESC"],
    max_results=14,
)[["run_id", "start_time", "metrics.arc_easy/acc_none",
   "metrics.hellaswag/acc_none"]]

df.sort_values("start_time")\
  .assign(delta=lambda d: d["metrics.arc_easy/acc_none"].diff())\
  .to_string(index=False)

如需将得分最高的每日版本提升到下游服务堆栈,请使用 Model Registry 别名 API — 流水线已为每个版本添加 eval_acc 标记:

client = mlflow.MlflowClient()
best = max(client.search_model_versions("name='qwen3-0.6b-sft'"),
           key=lambda v: float(v.tags.get("eval_acc", "0")))
client.set_registered_model_alias("qwen3-0.6b-sft", "champion", best.version)
# Downstream InferenceService references models:/qwen3-0.6b-sft@champion.

故障排除

症状检查
log_model 因 NoCredentialsError / Endpoint URL error 而失败MLflow artifact store 未配置为使用 S3。请参阅 MLflow install 中的 High availability and storage 部分,并在 MLflow 插件中设置 artifact bucket。然后重新运行。
deploy_for_evaluation 始终无法看到 Ready=True检查 InferenceService:kubectl -n finetune describe isvc <name>。最常见的原因是 pvc_path 标签中的 PVC 路径不存在,或者集群上没有 KServe HF runtime 镜像。
LMEvalJob 一直处于 Scheduled评估作业 Pod 处于 Pending 状态。检查 Pod 的节点选择器和 PVC 绑定:kubectl -n finetune get pods -l app=<job-name>,并检查处于 Pending 状态的 Pod kubectl describe。
LMEvalJob 无法连接到 InferenceServicemodelArgs 中的 base_url 必须是 -predictor Service,而不是顶层 InferenceService。示例使用 http://<isvc>-predictor.<ns>.svc/v1/completions — 对于 OpenAI 兼容的端点,-predictor 后缀和 /v1/completions 路径都不可缺少。
LMEvalJob 以 state=Complete, reason=Failed, message="open …/output/stdout.log: permission denied" 结束operator 管理的 outputs PVC 归 root 所有,但 ta-lmes-job 容器以 UID 65532 运行,因此 driver 无法创建 stdout.log。请设置 spec.pod.securityContext.fsGroup: 65532(如示例所示),使挂载目录对 Pod 的补充组可写。
state=Complete,但即使运行失败,流水线仍将其视为成功state 对 任何 终止结果都会转换为 Complete;结果本身记录在 status.reason 中(Succeeded / Failed / Cancelled)。请同时检查这两项,如上面的 evaluate 组件所示。
local-completions 因 OSError: We couldn't connect to 'https://huggingface.co' to load this file … it looks like <name> is not the path to a directory containing a file named config.json. 而失败即使 tokenized_requests 为 "False",lm-evaluation-harness 客户端仍会调用 AutoTokenizer.from_pretrained(...) — tokenizer modelArgs 条目必须是评估 Pod 可访问的 repo id,或离线 PVC 下的本地路径。在隔离网络的集群上,请遵循 Evaluate LLM 中的 Optional: offline storage and PVC 部分 — 在 spec.offline.storage.pvcName 挂载 PVC,用 tokenizer 和数据集缓存填充该 PVC,并将 tokenizer 设置为挂载路径。
父运行中缺少嵌套 MLflow 运行在 search_runs 解析出父运行之前,就调用了 set_experiment — 但始终没有记录任何内容,因为父标签是在另一个组件的进程中写入的。请在父运行中使用 set_tag("pipeline_run_id", run_id),并按该标签搜索,如 evaluate 组件所示。
Recurring Run 已触发,但运行立即因 401 UNAUTHENTICATED 而失败mlflow-token Secret 在上一次成功运行流水线之后、今天的运行之前已过期。请轮换该 Secret(或将令牌生成操作移至组件内部);请参阅步骤 5 中的警告。
KFP v2 Pod 因 container has runAsNonRoot and image will run as root 而失败(在 argoexec init init-container 和 kfp-launcher init-container 中)KFP-v2 Pod 模板在 Pod 级别设置了 runAsNonRoot: true,但 argoexec / kfp-launcher 镜像没有设置非 root 的 USER。对于会暴露此问题的后端(如 DSPO 基于 Argo 的流水线堆栈),请在运行时使用 spec.podSpecPatch 修补 Workflow,并在 Pod 级别及每个 init-container(init + kfp-launcher)上设置 runAsUser: 1001。
微调组件因 PermissionError: [Errno 13] Permission denied: '/mnt/shared/...' 而失败共享 PVC 的根目录归 root 所有。请在相同的 podSpecPatch 中添加 fsGroup: 1001,或使用以 root 身份运行的一次性 Pod 初始化 PVC,并让其执行 chown -R 1001:1001 /mnt。
微调组件因 modelscope_hub.errors.CacheError: [E1022] Failed to create SDK directories: [Errno 13] Permission denied: '/.modelscope' 而失败python:3.12-slim 容器未设置 HOME=/,而 ModelScope 和 Hugging Face 默认将缓存放在 ~/。非 root Pod 无法写入该路径。在微调组件中,在首次调用 from modelscope import … / AutoModel.from_pretrained 之前,将 MODELSCOPE_CACHE、HF_HOME 和 HOME 设置为共享 PVC 上的路径(或 /tmp)。
工作节点无法访问 runtime 镜像标签隔离网络或受限出口的集群无法直接从 docker.io 拉取镜像。请使用 skopeo copy 或 crane copy 将 docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0 镜像到内部 registry,然后将 RUNTIME_IMAGE 更改为镜像后的引用,并将相应的 imagePullSecrets 添加到流水线 ServiceAccount。
在 KFP API server 处提交流水线时因 unknown component implementation: comp-exit-handler-1 而失败后端(例如 DSPO 基于 Argo 的流水线堆栈)不接受 KFP v2 dsl.ExitHandler 子图。请将 with dsl.ExitHandler(cleanup(...)): ev 替换为普通的 cleanup(...).after(ev),并确保 cleanup 具有幂等性(按标签选择器删除,以便在 evaluate 已在成功后删除资源时也能正常工作)。
KServe Pod 因 FileNotFoundError: No such file or directory: /mnt/models/model.safetensors 崩溃,尽管文件确实存在于 PVC 上KServe 容器以 UID 1000 运行(在 ClusterServingRuntime 上定义),而 fine_tune 组件以 UID 1001 和默认 umask 写入文件,生成的模式为 0660 — 组可写,但其他用户不可读。请在 save_pretrained 后添加一个 chmod 步骤(如示例所示):将每个目录 os.chmod 为 0755,并将每个文件设置为 0644。仅在微调 Pod 上设置 fsGroup: 1001 并不足够 — 它会更改挂载根目录的组,但不会递归设置子文件的权限模式。
evaluate 组件中的 mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: No Experiment with id=0 existsstart_run(run_id=parent) 会附加到现有运行,但嵌套的 start_run(run_name=...) 需要实验上下文。请在组件顶部、打开任何运行之前调用一次 mlflow.set_experiment(experiment)。否则,MLflow 会默认使用实验 0,而多租户服务器会拒绝该实验。
MLflow create_model_version 因 INVALID_PARAMETER_VALUE: Invalid model version source: '/mnt/…'. To use a local path as a model version source, the run_id request parameter has to be specified and the local path has to be contained within the artifact directory of the run specified by the run_id. 而失败,切换到 mlflow.log_artifacts 后又因 PermissionError: [Errno 13] Permission denied: '/mlflow' 而失败Alauda MLflow 插件默认将 artifact root 设为 /mlflow/artifacts — 这是 tracking server Pod 的 emptyDir 上的本地路径。客户端的 log_artifacts / log_model 无法写入该路径,而且会拒绝运行的 artifact 目录之外的 file:// 版本源。有两种方案:(a) 将 MLflow 部署重新配置为使用 S3 artifact 后端(请参阅 MLflow install 中的 High availability and storage 部分);或 (b) 跳过模型注册表,通过父运行上的标签协调下游步骤 — 每次运行的流水线仍会在 MLflow 中生成单独一行,并且可通过 mlflow.get_run(...).data.tags 访问 pvc_path、final_loss 和 eval_acc。
TypeError: SFTTrainer.__init__() got an unexpected keyword argument 'tokenizer'TRL 0.12+ 移除了 tokenizer 关键字 — 请改为将 tokenizer 作为 processing_class=tokenizer 传入。参考 runtime 镜像附带的 TRL 版本高于 0.12(由 LLaMA-Factory 0.9.4 间接引入)。上面的示例使用 processing_class=;仍传入 tokenizer= 的旧版 fork 也需要进行相同的重命名。
mlflow.transformers.log_model 因 MlflowException: The task could not be inferred from the model. If you are saving a custom local model that is not available in the Hugging Face hub, please provide the 'task' argument to the log_model or save_model function. 而失败MLflow 可以从 Hub 模型推断流水线任务,但无法从本地加载(路径镜像)的模型推断 — 微调 Pod 的 trainer.model 在 AutoModelForCausalLM.from_pretrained("/mnt/shared/models/…") 之后属于后者。请向 log_model 传入 task="text-generation"(或其他模型类型对应的任务字符串)。
MLflow 客户端调用因 UNAUTHENTICATED: Authentication with the Kubernetes API failed. The provided token may be invalid or expired. 而失败mlflow-tracking-server 的 oauth2-proxy sidecar 未转发 bearer token — 插件配置界面未提供此选项,因此必须将该标志修补到 Deployment 上。将 OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true 添加到 oauth2-proxy 容器的环境变量中,等待滚动更新,然后重新运行。请注意,mlflow-plugin controller 会协调 Deployment,可能会还原此环境变量;每次 operator 升级或手动重新安装后都要重新应用。

构建自定义镜像

参考镜像基于 llamafactory0.9-cu126-amd64 构建,仅添加了 mlflow>=3.10、kserve>=0.13 和 kubernetes>=29。要自行构建,例如替换 CUDA 版本、添加专有数据处理库,或使基础镜像符合 FIPS 要求,请编写一个仅包含一层的 Containerfile:

# syntax=docker/dockerfile:1.7
ARG BASE_IMAGE=docker.io/alaudadockerhub/llamafactory0.9-cu126-amd64:v0.1.0
FROM ${BASE_IMAGE}

USER 0
RUN uv pip install --no-cache-dir \
        "mlflow>=3.10" \
        "kubernetes>=29" \
        "kserve>=0.13" \
        # add anything your pipeline imports:
        # "your-private-etl-lib==1.2.3" \
        ;
USER 1000
WORKDIR /workspace

使用 buildctl(或 docker buildx build)通过你的 buildkitd 进行构建,推送到内部 registry,并相应设置 RUNTIME_IMAGE。流水线的其余代码无需更改。

如果你的集群已经运行 Kubeflow Trainer v2,也可以复用已发布的 TrainingRuntime 镜像之一 — llamafactory0.9-cu126-amd64:v0.1.0 已包含 torch / HF / trl / mlflow;只需为 deploy_for_evaluation + evaluate + cleanup 组件添加 kserve 和 kubernetes。参考镜像正是这样构建的。

相关指南