使用 MLflow Tracking 和 TrustyAI 评估的 Daily Fine-Tuning Pipeline

一个完整的 Kubeflow Pipeline,用于 fine-tune 一个 LLM(本示例中为 Qwen3-0.6B),在 MLflow 中跟踪运行,将生成的模型注册到 MLflow Model Registry,将其部署为 KServe InferenceService,使用 TrustyAI LMEvalJob 对其进行评估,将评估结果写回同一个 MLflow experiment,并清理临时 serving 资源。随后,该 pipeline 会连接到 KFP Recurring Run,使其每天执行一次——为你持续积累 train → eval 运行历史,而 MLflow 的 compare view 会将其转化为回归信号。

本指南将已有的独立文档内容整合为一个可运行方案——Use Kubeflow Pipelines、Kubeflow Pipeline + MLflow Integration、Evaluate LLM——如果某一步不熟悉,请先阅读这些文档。

Pipeline 的功能

                     ┌───────────────┐
   Qwen3-0.6B  ─────►│  fine-tune    │──► loss/eval_loss/perplexity ─► MLflow parent run
      base model     │  (SFTTrainer) │──► fine-tuned model artifacts ─► MLflow Model Registry
                     └──────┬────────┘        (registered_model_name="qwen3-0.6b-sft")
                            │
                            ▼
                     ┌───────────────┐
                     │ deploy-for-   │──► KServe InferenceService (temporary)
                     │ evaluation    │
                     └──────┬────────┘
                            │
                            ▼
                     ┌───────────────┐
                     │ evaluate      │──► TrustyAI LMEvalJob (arc_easy, mmlu, …)
                     │ (LM-Eval)     │──► eval metrics ─► MLflow nested "eval" run
                     └──────┬────────┘                    + tag on model version
                            │
                            ▼
                     ┌───────────────┐
                     │  cleanup      │──► delete InferenceService + LMEvalJob
                     └───────────────┘

每次 daily run 都会在 MLflow experiment 中新增一行。由于 training 和 evaluation metrics 都位于同一个父运行下,MLflow 的 Compare 和 Chart 视图可以直接绘制逐日 loss 和 evaluation accuracy,而无需额外的 glue code。

前提条件

要求详情
Alauda AI 2.5 或更高版本已安装 Kubeflow Pipelines、MLflow(已配置 Model Registry artifact store)、TrustyAI、KServe。
预构建的 pipeline runtime image下方四个 KFP component 使用 docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0(linux/amd64)。它内置完整的 CUDA / HF / trl / MLflow / KServe / kubernetes 栈,因此 pod 启动时不会执行 pip install——在 air-gapped 集群中,只要将该镜像镜像到内部 registry,也可以采用相同方案。请参见下方的 Bake a custom image 以派生你自己的镜像,并参考 Training Runtime Images 获取源 Containerfile 目录。
MLflow artifact store在 MLflow plugin 中配置的 S3 兼容对象存储。mlflow.transformers.log_model() 需要它来持久化模型文件,供 KServe 后续拉取。请参见 MLflow install 中 High availability and storage 部分。
带有 MLflow label 的 finetune namespace在 namespace 上设置 mlflow-enabled=true,使其显示为 workspace。
共享 PVC finetune-shared(RWX)用于在 component 之间缓存 base model 和 fine-tuned checkpoint。任何支持 RWX 的 StorageClass(CephFS、NFS、JuiceFS)都可以。
一个 GPU 节点1 × NVIDIA GPU,且 VRAM ≥ 16 GiB,即可在小数据集上对 Qwen3-0.6B 进行全精度 SFT;对于较小的 LoRA 运行,8 GiB 即可。
MLflow token Secret名为 mlflow-token 的 Secret,其 key token 中保存一个 Dex id token,对应具有访问 finetune MLflow workspace 权限的 service account。参见 Get a token from the command line。
finetune namespace 中的 RBACPipeline ServiceAccount 可以 get/create/delete inferenceservices.serving.kserve.io 和 lmevaljobs.trustyai.opendatahub.io。
训练数据集(可选)fine_tune component 默认使用一个小型、synthetic 的进程内语料库——无需准备数据集即可进行 smoke run。对于真实工作负载,请先在共享 PVC 或 seaweedfs / S3 上预先放置一个 JSONL 文件,并将其路径作为 dataset_path 传入;请参见下方的 Using a real dataset。
到 Hugging Face 的出站访问(或离线 PVC)默认有两个 component 会从 Hugging Face 拉取内容:(a) fine_tune 通过 AutoModelForCausalLM.from_pretrained(...) 下载 base model Qwen/Qwen3-0.6B,(b) evaluate component 的 LMEvalJob 会获取 tokenizer 和 dataset——即使 tokenized_requests 为 False,AutoTokenizer.from_pretrained(...) 也会执行。对于 (a),请将 base model 一次性镜像到共享 PVC(见 Using a real dataset → base-model note)。对于 (b),请将 eval job 切换为 offline mode(见 Evaluate LLM):预先在 PVC 中放入 tokenizer + dataset cache,设置 spec.offline.storage.pvcName,并将 tokenizer modelArgs 指向挂载路径。

步骤 1 — 创建 MLflow token Secret 和共享 PVC

NS=finetune

# 1. Mint a Dex id token (browser-free) — see the SDK guide for ID_TOKEN.
kubectl -n $NS create secret generic mlflow-token --from-literal=token="$ID_TOKEN"

# 2. Shared PVC used by fine-tune (write) and deploy (read).
kubectl -n $NS apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: finetune-shared
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 50Gi
  storageClassName: cephfs   # any RWX-capable StorageClass
EOF

下面的 pipeline component 假定这两个对象已存在于 pipeline 运行所在的 namespace 中。

步骤 2 — Pipeline

将以下内容保存为 finetune_pipeline.py。该 pipeline 被拆分为四个 component,因此故障会局部化,重跑成本也较低;component 通过 PVC 以及 模型版本上的 MLflow tags 共享状态。

# finetune_pipeline.py
from kfp import dsl, compiler, kubernetes

BASE_MODEL      = "Qwen/Qwen3-0.6B"
REGISTERED_NAME = "qwen3-0.6b-sft"
WORKSPACE       = "finetune"                       # MLflow workspace (= namespace)
EXPERIMENT      = "qwen3-0.6b-daily-sft"
PVC_NAME        = "finetune-shared"
PVC_MOUNT       = "/mnt/shared"

MLFLOW_URI      = "http://mlflow-tracking-server.kubeflow:5000"

# Pre-built runtime image with torch (CUDA 12.6) + HF + trl + mlflow + kserve
# + kubernetes baked in. No pip install runs at pod start, so the same tag
# works on air-gapped clusters once it's mirrored into an internal registry.
RUNTIME_IMAGE   = "docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0"


# ---------------------------------------------------------------------------
# 1. Fine-tune with the HF Trainer, autolog to MLflow, register the model.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def fine_tune(
    workspace: str,
    experiment: str,
    base_model: str,
    registered_name: str,
    pvc_mount: str,
    run_id: str,
    # Empty = generate a tiny synthetic instruction corpus in-process (no
    # network egress; smoke-shape only, not a meaningful fine-tune). Non-empty
    # = a local JSONL path (e.g. "/mnt/shared/datasets/alpaca.jsonl") or an
    # "s3://…" URI to a JSONL file with a `text` column. See the "Using a
    # real dataset" section for how to prepare the file.
    dataset_path: str = "",
    num_train_epochs: int = 1,
    learning_rate: float = 2e-4,
    per_device_train_batch_size: int = 2,
    max_samples: int = 512,
) -> str:
    """Full SFT of `base_model` on a small synthetic corpus (or a JSONL file
    at `dataset_path`). Logs metrics + registers the fine-tuned model.
    Returns the model version as a string."""
    import operator, os, random, mlflow, mlflow.transformers
    from datasets import Dataset, load_dataset
    from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
    from trl import SFTTrainer

    # 1) MLflow (auth via MLFLOW_TRACKING_TOKEN injected from the Secret).
    mlflow.set_tracking_uri(MLFLOW_URI)
    mlflow.set_workspace(workspace)
    mlflow.set_experiment(experiment)
    mlflow.transformers.autolog(log_models=False)          # models are logged manually below

    # 2) Data. Synthetic by default so the pipeline's smoke run needs no
    # network egress and no dataset pre-staging. Swap in a real dataset by
    # passing `dataset_path` (JSONL file with a `text` column) — either a
    # PVC path or an S3 URI. See "Using a real dataset" below.
    if dataset_path.startswith("s3://"):
        ds = load_dataset(
            "json", data_files=dataset_path, split=f"train[:{max_samples}]",
            # storage_options / credentials come from AWS_* env; see the docs.
        )
    elif dataset_path:
        ds = load_dataset(
            "json", data_files=dataset_path, split=f"train[:{max_samples}]",
        )
    else:
        rng = random.Random(42)
        ops = [("+", operator.add), ("-", operator.sub), ("*", operator.mul)]
        rows = []
        for _ in range(max_samples):
            a, b = rng.randint(0, 99), rng.randint(0, 99)
            sym, op = rng.choice(ops)
            rows.append({"text":
                f"### Instruction:\nWhat is {a} {sym} {b}?\n"
                f"### Response:\nThe answer is {op(a, b)}."
            })
        ds = Dataset.from_list(rows)

    # 3) Model & tokenizer.
    tokenizer = AutoTokenizer.from_pretrained(base_model)
    model     = AutoModelForCausalLM.from_pretrained(base_model)

    output_dir = f"{pvc_mount}/models/{registered_name}/{run_id}"

    # 4) Train. `report_to="mlflow"` streams loss / eval_loss / lr per step.
    args = TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=num_train_epochs,
        learning_rate=learning_rate,
        per_device_train_batch_size=per_device_train_batch_size,
        logging_steps=10,
        save_strategy="no",
        report_to="mlflow",
        run_name=f"train-{run_id}",
    )
    with mlflow.start_run(run_name=f"pipeline-{run_id}") as parent:
        mlflow.set_tag("pipeline_run_id", run_id)
        mlflow.set_tag("base_model", base_model)
        with mlflow.start_run(run_name=f"train-{run_id}", nested=True):
            # TRL 0.12+ renamed the `tokenizer` kwarg to `processing_class`.
            trainer = SFTTrainer(model=model, args=args, train_dataset=ds,
                                 processing_class=tokenizer)
            trainer.train()
            trainer.save_model(output_dir)

        # 5) Register the fine-tuned model.
        # The artifact upload uses the MLflow plugin's artifact store (S3).
        # KServe then pulls the model back from that S3 URI in the deploy step.
        # `task="text-generation"` is required because MLflow cannot infer the
        # task from a locally-loaded (non-Hub) model.
        mv = mlflow.transformers.log_model(
            transformers_model={"model": trainer.model, "tokenizer": tokenizer},
            artifact_path="model",
            registered_model_name=registered_name,
            task="text-generation",
        )

        # 6) Record the PVC path on the model version so downstream steps
        #    can serve it from the shared PVC without another S3 round-trip.
        client = mlflow.MlflowClient()
        version = client.get_latest_versions(registered_name, stages=["None"])[0].version
        client.set_model_version_tag(registered_name, version, "pvc_path", output_dir)
        client.set_model_version_tag(registered_name, version, "pipeline_run_id", run_id)

    print(f"registered {registered_name} version {version} @ {output_dir}")
    return version


# ---------------------------------------------------------------------------
# 2. Deploy the just-registered model as a temporary KServe InferenceService.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def deploy_for_evaluation(
    workspace: str,
    registered_name: str,
    model_version: str,
    pvc_name: str,
    pvc_mount: str,
    run_id: str,
) -> str:
    """Create an InferenceService that serves the model straight off the PVC.
    Returns the InferenceService name."""
    import time, mlflow
    from kubernetes import client, config
    from kserve import KServeClient, constants
    from kserve import (V1beta1InferenceService, V1beta1InferenceServiceSpec,
                        V1beta1PredictorSpec, V1beta1ModelSpec, V1beta1ModelFormat)

    # Resolve the PVC path from the tag set by fine_tune().
    mlflow.set_tracking_uri(MLFLOW_URI); mlflow.set_workspace(workspace)
    tags = mlflow.MlflowClient().get_model_version(registered_name, model_version).tags
    pvc_path = tags["pvc_path"]

    isvc_name = f"{registered_name}-eval-{run_id[-8:]}"
    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()

    isvc = V1beta1InferenceService(
        api_version=constants.KSERVE_GROUP + "/v1beta1", kind=constants.KSERVE_KIND,
        metadata=client.V1ObjectMeta(name=isvc_name, namespace=ns, labels={
            "app.kubernetes.io/part-of": "finetune-pipeline",
            "pipeline-run-id": run_id,
        }),
        spec=V1beta1InferenceServiceSpec(predictor=V1beta1PredictorSpec(
            model=V1beta1ModelSpec(
                model_format=V1beta1ModelFormat(name="huggingface"),
                runtime="kserve-huggingfaceserver",
                storage_uri=f"pvc://{pvc_name}{pvc_path.removeprefix(pvc_mount)}",
                resources=client.V1ResourceRequirements(
                    limits={"nvidia.com/gpu": "1", "memory": "16Gi"},
                    requests={"nvidia.com/gpu": "1", "memory": "8Gi"},
                ),
            ),
        )),
    )
    KServeClient().create(isvc)

    # Wait for READY.
    deadline = time.time() + 15 * 60
    while time.time() < deadline:
        got = KServeClient().get(isvc_name, namespace=ns)
        cond = {c["type"]: c["status"] for c in (got.get("status", {}).get("conditions") or [])}
        if cond.get("Ready") == "True":
            print(f"InferenceService {isvc_name} is Ready")
            return isvc_name
        time.sleep(10)
    raise TimeoutError(f"{isvc_name} did not become Ready in 15 min")


# ---------------------------------------------------------------------------
# 3. Evaluate the InferenceService with a TrustyAI LMEvalJob, log to MLflow.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def evaluate(
    workspace: str,
    experiment: str,
    registered_name: str,
    model_version: str,
    isvc_name: str,
    run_id: str,
    tasks: list = ["arc_easy", "hellaswag"],
    limit: str = "50",
) -> dict:
    """Create an LMEvalJob against the InferenceService, wait for it to
    Complete, log the metrics into the parent MLflow run + as a nested run,
    and tag the model version with the primary accuracy."""
    import json, time, mlflow
    from kubernetes import client, config

    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()
    api = client.CustomObjectsApi()

    job_name = f"eval-{isvc_name}"
    body = dict(
        apiVersion="trustyai.opendatahub.io/v1alpha1", kind="LMEvalJob",
        metadata=dict(name=job_name, labels={"pipeline-run-id": run_id}),
        spec=dict(
            model="local-completions",
            modelArgs=[
                dict(name="model", value=isvc_name),
                dict(name="base_url",
                     value=f"http://{isvc_name}-predictor.{ns}.svc/v1/completions"),
                dict(name="num_concurrent", value="1"),
                dict(name="max_retries", value="3"),
                dict(name="tokenized_requests", value="True"),
                dict(name="tokenizer", value="Qwen/Qwen3-0.6B"),
            ],
            taskList=dict(taskNames=list(tasks)),
            allowOnline=True, allowCodeExecution=False,
            batchSize="1", limit=limit, logSamples=False,
            chatTemplate=dict(enabled=False),
            outputs=dict(pvcManaged=dict(size="100Mi")),
            # The ta-lmes-job image runs as UID 65532; a fresh PVC is root-owned,
            # so the driver's `open(stdout.log)` fails with permission denied
            # unless fsGroup gives the pod's group write access to the mount.
            pod=dict(securityContext=dict(fsGroup=65532)),
        ),
    )
    api.create_namespaced_custom_object(
        "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", body)

    # Wait for a terminal state. `state` reaches `Complete` on both success
    # and failure — the reason field is what actually distinguishes them.
    deadline = time.time() + 60 * 60
    while time.time() < deadline:
        got = api.get_namespaced_custom_object(
            "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", job_name)
        status = got.get("status") or {}
        state, reason = status.get("state"), status.get("reason")
        if state == "Complete" and reason == "Succeeded":
            break
        if state in {"Cancelled", "Failed"} or (state == "Complete" and reason in {"Failed", "Cancelled"}):
            raise RuntimeError(
                f"LMEvalJob {job_name} ended: state={state} reason={reason} "
                f"message={status.get('message')}")
        time.sleep(20)
    else:
        raise TimeoutError(f"LMEvalJob {job_name} did not finish in 1 h")

    results = json.loads(got["status"]["results"])["results"]

    # Log each task's metrics back to MLflow.
    mlflow.set_tracking_uri(MLFLOW_URI)
    mlflow.set_workspace(workspace)
    mlflow.set_experiment(experiment)
    parent = mlflow.search_runs(filter_string=f"tags.pipeline_run_id = '{run_id}'",
                                order_by=["attributes.start_time DESC"], max_results=1)
    parent_run_id = parent.iloc[0]["run_id"]

    flat = {}
    with mlflow.start_run(run_id=parent_run_id):
        with mlflow.start_run(run_name=f"eval-{run_id}", nested=True):
            for task, metrics in results.items():
                for k, v in metrics.items():
                    if isinstance(v, (int, float)) and "stderr" not in k:
                        name = f"{task}/{k.replace(',', '_')}"
                        mlflow.log_metric(name, float(v))
                        flat[name] = float(v)
            mlflow.set_tag("model_version", model_version)
            mlflow.set_tag("isvc_name", isvc_name)

    # Tag the primary accuracy on the model version so it shows up on the
    # Model Registry page next to the version number.
    primary = next((k for k in flat if k.endswith("/acc_none")), None)
    if primary is not None:
        mlflow.MlflowClient().set_model_version_tag(
            registered_name, model_version, "eval_acc", f"{flat[primary]:.4f}")
    return flat


# ---------------------------------------------------------------------------
# 4. Cleanup: delete the temporary InferenceService and LMEvalJob.
# ---------------------------------------------------------------------------
@dsl.component(base_image=RUNTIME_IMAGE)
def cleanup(isvc_name: str):
    from kubernetes import client, config
    config.load_incluster_config()
    ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip()
    api = client.CustomObjectsApi()
    for gv, plural, name in [
        ("serving.kserve.io/v1beta1", "inferenceservices", isvc_name),
        ("trustyai.opendatahub.io/v1alpha1", "lmevaljobs", f"eval-{isvc_name}"),
    ]:
        group, version = gv.split("/")
        try:
            api.delete_namespaced_custom_object(group, version, ns, plural, name)
        except Exception as exc:
            print(f"delete {plural}/{name} skipped: {exc}")


# ---------------------------------------------------------------------------
# Pipeline: wire the four components together.
# ---------------------------------------------------------------------------
@dsl.pipeline(name="qwen3-sft-mlflow-trustyai",
              description="Daily SFT + MLflow tracking + TrustyAI evaluation")
def sft_mlflow_trustyai(
    workspace: str = WORKSPACE,
    experiment: str = EXPERIMENT,
    base_model: str = BASE_MODEL,
    registered_name: str = REGISTERED_NAME,
    # Default = synthetic in-process data (offline smoke). Override with a
    # JSONL path on the shared PVC or an s3:// URI to fine-tune on a real
    # dataset — see the "Using a real dataset" section.
    dataset_path: str = "",
    num_train_epochs: int = 1,
    learning_rate: float = 2e-4,
    max_samples: int = 512,
    eval_limit: str = "50",
):
    train = fine_tune(
        workspace=workspace, experiment=experiment,
        base_model=base_model, registered_name=registered_name,
        pvc_mount=PVC_MOUNT, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
        dataset_path=dataset_path, num_train_epochs=num_train_epochs,
        learning_rate=learning_rate, max_samples=max_samples,
    ).set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1)
    train.set_memory_limit("32Gi").set_cpu_limit("8")

    deploy = deploy_for_evaluation(
        workspace=workspace, registered_name=registered_name,
        model_version=train.output, pvc_name=PVC_NAME, pvc_mount=PVC_MOUNT,
        run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
    )

    ev = evaluate(
        workspace=workspace, experiment=experiment,
        registered_name=registered_name, model_version=train.output,
        isvc_name=deploy.output, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
    )

    with dsl.ExitHandler(cleanup(isvc_name=deploy.output)):
        ev  # cleanup runs whether evaluate() succeeds or not

    # Mount the shared PVC on training + deploy.
    for task in (train, deploy):
        kubernetes.mount_pvc(task, pvc_name=PVC_NAME, mount_path=PVC_MOUNT)

    # Inject the Dex id token into every component that talks to MLflow.
    for task in (train, deploy, ev):
        kubernetes.use_secret_as_env(
            task, secret_name="mlflow-token",
            secret_key_to_env={"token": "MLFLOW_TRACKING_TOKEN"})


compiler.Compiler().compile(sft_mlflow_trustyai, "pipeline.yaml")

编译:

pip install "kfp>=2.7" "kfp-kubernetes>=1.3"
python finetune_pipeline.py
# → pipeline.yaml

在 air-gapped 集群上使用真实数据集

fine_tune component 默认使用一个synthetic 的进程内 instruction corpus(几百对算术 Q/A 对)。这使得 smoke run 可以完全离线——无需 HuggingFace、无需 ModelScope、无需 S3——但不会产生有意义的 fine-tune。对于真实工作负载,请先将数据集预先放到集群中,然后在每次运行时将其路径作为 dataset_path 传入。fine_tune component 既接受本地 JSONL 路径(最简单),也接受 s3://… URI(可供多个 pipeline 共享)。

1. 在可联网的机器上下载数据集

任何可以访问 Hugging Face 的机器都可以——笔记本电脑、跳板机、或者位于有出站访问的 namespace 中的 Workbench:

pip install huggingface_hub datasets
huggingface-cli download tatsu-lab/alpaca --repo-type dataset --local-dir ./alpaca-raw

如果 Hugging Face 被阻断,ModelScope 镜像了大多数流行数据集,并且可在中国大陆网络中使用:

pip install modelscope
modelscope download --dataset AI-ModelScope/alpaca-gpt4-data-en --local_dir ./alpaca-raw

2. 转换为带有 text 列的 JSONL 文件

TRL 的 SFTTrainer 期望每一行都有一个 text 字段。请将原始数据集转换为 pipeline 所需的格式:

from datasets import load_dataset

raw = load_dataset("./alpaca-raw")["train"]

def format_alpaca(row):
    prompt = row["instruction"]
    if row.get("input"):
        prompt += "\n\n" + row["input"]
    return {"text": f"### Instruction:\n{prompt}\n### Response:\n{row['output']}"}

raw.map(format_alpaca).to_json("alpaca.jsonl", orient="records", lines=True)

3a. 上传到共享 PVC(最简单)

如果你的集群中有一个挂载了 finetune-shared PVC 的跳板 pod,请使用 kubectl cp 将 JSONL 复制进去:

kubectl -n finetune apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: dataset-uploader, namespace: finetune }
spec:
  restartPolicy: Never
  containers:
  - name: sh
    image: busybox
    command: ["sh", "-c", "mkdir -p /mnt/shared/datasets && sleep 3600"]
    volumeMounts: [{ name: shared, mountPath: /mnt/shared }]
  volumes:
  - name: shared
    persistentVolumeClaim: { claimName: finetune-shared }
EOF
kubectl -n finetune wait --for=condition=Ready pod/dataset-uploader
kubectl -n finetune cp alpaca.jsonl dataset-uploader:/mnt/shared/datasets/alpaca.jsonl
kubectl -n finetune delete pod dataset-uploader

然后在提交 pipeline 时使用 dataset_path="/mnt/shared/datasets/alpaca.jsonl"(参见下方步骤 3)。

3b. 上传到 S3 / seaweedfs(可供多个 pipeline 共享)

对于运行大量 pipeline 的团队,请将数据集放在与 MLflow plugin artifact store 使用的相同 S3 兼容 bucket 中——集群内部的 seaweedfs endpoint 可被每个 pipeline pod 访问,而客户端只需要 AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY 环境变量。

请从能够访问 S3 endpoint 的机器上传——通常是同一集群中的 Workbench。Alauda AI Workbench 镜像已内置 mc(MinIO Client)和 boto3,因此无需额外安装。

使用 mc(交互式 / 单行命令):

mc alias set seaweedfs http://seaweedfs.mlops-demo-e2e:8333 <ACCESS_KEY> <SECRET_KEY>
mc mb --ignore-existing seaweedfs/mlops-datasets
mc cp alpaca.jsonl seaweedfs/mlops-datasets/alpaca/alpaca.jsonl

使用 boto3(脚本化 / 在 notebook 中执行):

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://seaweedfs.mlops-demo-e2e:8333",
    aws_access_key_id="<ACCESS_KEY>",
    aws_secret_access_key="<SECRET_KEY>",
)
# Idempotent bucket create.
try:
    s3.create_bucket(Bucket="mlops-datasets")
except s3.exceptions.BucketAlreadyOwnedByYou:
    pass
s3.upload_file("alpaca.jsonl", "mlops-datasets", "alpaca/alpaca.jsonl")

(请根据你的 seaweedfs / MinIO Service 调整 endpoint。凭据来自 MLflow plugin 的 artifact-store Secret,或来自你的本地 S3 提供商。)

然后创建一个包含 S3 凭据的 Secret,并将其与 mlflow-token 一起注入到 fine_tune 中:

kubectl -n finetune create secret generic mlflow-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=<key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret> \
  --from-literal=MLFLOW_S3_ENDPOINT_URL=http://seaweedfs.mlops-demo-e2e:8333

将以下内容添加到 pipeline function 中(紧挨着现有的 use_secret_as_env block,该 block 挂载了 MLflow token):

kubernetes.use_secret_as_env(train, secret_name="mlflow-s3", secret_key_to_env={
    "AWS_ACCESS_KEY_ID":       "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY":   "AWS_SECRET_ACCESS_KEY",
    "MLFLOW_S3_ENDPOINT_URL":  "AWS_ENDPOINT_URL",   # what s3fs / fsspec read
})

然后使用 dataset_path="s3://mlops-datasets/alpaca/alpaca.jsonl" 提交 pipeline。s3fs 已包含在 runtime image 中,而 load_dataset("json", data_files="s3://…") 会在底层使用 fsspec——AWS_* 环境变量会自动被读取。

4. 关于 base model 的说明

AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B") 在首次运行时也会访问 Hugging Face。若要实现完全 air-gapped 的 fine-tune,请先将模型镜像一次(同样的模式:huggingface-cli download Qwen/Qwen3-0.6B --local-dir ./qwen3-0.6b,然后复制到 PVC 或 S3),并设置 BASE_MODEL = "/mnt/shared/models/qwen3-0.6b"。随后 MLflow model registry 会像之前一样将该 fine-tune 注册为 qwen3-0.6b-sft。

步骤 3 — 提交一次性运行

在 Kubeflow Dashboard → Pipelines → Upload Pipeline 中上传 pipeline.yaml,然后点击 Create Run——或者,从 Workbench 中执行:

from kfp.client import Client

client = Client(host="<MY-KFP-ENDPOINT>")
run = client.create_run_from_pipeline_package(
    "pipeline.yaml",
    arguments=dict(
        # Omit for the synthetic-smoke default; set to a JSONL path or an
        # s3:// URI to fine-tune on a real dataset (see the section above).
        # dataset_path="/mnt/shared/datasets/alpaca.jsonl",
        num_train_epochs=1, eval_limit="50",
    ),
)
print("Run:", run.run_id)

当运行进行中时:

  • 打开 Alauda AI → Tools → MLFlow 并选择 finetune workspace。
  • qwen3-0.6b-daily-sft experiment 会显示一个名为 pipeline-<pipeline-run-id> 的 父运行,以及两个嵌套运行:train-<...>(由 HF Trainer callback 流式写入)和稍后出现的 eval-<...>(由 evaluate 步骤写入)。
  • 在 Models 下,qwen3-0.6b-sft registered model 会新增一个版本。

步骤 4 — 在 MLflow 中可视化 training metrics

  1. 在 MLflow UI 中,打开父运行 → Metrics 选项卡。
  2. 选择 loss、eval_loss 以及你添加的任何自定义 metric(perplexity、learning_rate、grad_norm)→ 点击 Chart。
  3. 若要实时查看 loss 曲线,请在 chart panel 中将 Refresh 间隔设置为 10s——当 Trainer 写入新的 step 时,MLflow 会重新获取 metric stream。

对于分类类目标,如果 accuracy 很重要,可以在 TrainingArguments 中添加一个返回 {"accuracy": ...} 的 compute_metrics 回调;report_to="mlflow" 会将 compute_metrics 返回的内容直接流式写入同一张图表。

步骤 5 — 将其安排为每日 Recurring Run

KFP 原生支持 cron 调度。将同一个 pipeline.yaml 连接到 Recurring Run,使其每天执行一次。

通过 UI

  1. Pipelines → 选择 qwen3-sft-mlflow-trustyai → Create Run。
  2. 选择 Recurring Run 作为运行类型。
  3. Trigger:Cron,表达式 0 2 * * *(每天 UTC 02:00 )。
  4. Max concurrent runs:1 ——该 pipeline 会占用一张 GPU;重叠运行会排队。
  5. Catchup:off ——跳过错过的窗口,而不是在周一早晨补跑积压任务。
  6. Parameters:num_train_epochs=1,eval_limit=200 用于 daily job(比一次性默认值更具信号)。
  7. Start。

通过 SDK

from kfp.client import Client

client = Client(host="<MY-KFP-ENDPOINT>")

# One-time: upload the pipeline as a versioned resource.
uploaded = client.upload_pipeline(
    "pipeline.yaml", pipeline_name="qwen3-sft-mlflow-trustyai")

# Attach a daily schedule.
client.create_recurring_run(
    experiment_id=client.create_experiment("qwen3-daily").experiment_id,
    job_name="qwen3-sft-daily-02utc",
    cron_expression="0 0 2 * * *",     # KFP uses 6-field cron (with seconds)
    max_concurrency=1,
    no_catchup=True,
    enabled=True,
    pipeline_id=uploaded.pipeline_id,
    version_id=uploaded.pipeline_version_id,
    params=dict(num_train_epochs=1, eval_limit="200"),
)
WARNING

Secret/mlflow-token 中的 MLflow token 会过期(Dex id token 默认有效期为 24 小时)。如果调度运行时间超过 token 生命周期,请在 fine_tune component 内从 service-account 凭据中生成 token 并在其中刷新它(参见 SDK token flow),或者运行一个小型 CronJob,以略短于 token TTL 的周期轮换 mlflow-token Secret。

步骤 6 — 在 MLflow 中比较每日评估结果

经过几天运行后,experiment 中会按天生成一行记录。MLflow 让比较只需一次点击:

  1. 在 MLflow UI 中,打开 qwen3-0.6b-daily-sft experiment。
  2. 在 runs table 中,筛选嵌套的 eval runs:tags.mlflow.runName LIKE 'eval-%'。
  3. 按 attributes.start_time DESC 排序。
  4. 勾选要比较的运行复选框(最近 7 天是个不错的起点)→ 点击 Compare。
  5. 在 Chart 选项卡中,按 tags.mlflow.runName 分组绘制 arc_easy/acc_none 和 hellaswag/acc_none。平稳的曲线意味着 fine-tune 稳定;突然明显下降通常意味着训练数据集或 base checkpoint 发生了变化。
  6. Parallel Coordinates 视图会将 training hyper-parameters 和 evaluation metrics 组合起来,方便你快速判断哪些 hyper-parameters 与最大的 evaluation 提升相关。

如果要进行程序化比较——例如输入到 Slack 告警,或者在 accuracy 日环比下降超过 2% 时让 CI gate 失败——可使用 search API:

import mlflow, pandas as pd

mlflow.set_tracking_uri("http://mlflow-tracking-server.kubeflow:5000")
mlflow.set_workspace("finetune")

df = mlflow.search_runs(
    experiment_names=["qwen3-0.6b-daily-sft"],
    filter_string="tags.mlflow.runName LIKE 'eval-%'",
    order_by=["attributes.start_time DESC"],
    max_results=14,
)[["run_id", "start_time", "metrics.arc_easy/acc_none",
   "metrics.hellaswag/acc_none"]]

df.sort_values("start_time")\
  .assign(delta=lambda d: d["metrics.arc_easy/acc_none"].diff())\
  .to_string(index=False)

若要将每日得分最高的版本提升到下游 serving 栈,请使用 Model Registry aliasing API——pipeline 已经为每个版本标记了 eval_acc:

client = mlflow.MlflowClient()
best = max(client.search_model_versions("name='qwen3-0.6b-sft'"),
           key=lambda v: float(v.tags.get("eval_acc", "0")))
client.set_registered_model_alias("qwen3-0.6b-sft", "champion", best.version)
# Downstream InferenceService references models:/qwen3-0.6b-sft@champion.

故障排查

症状检查
log_model 失败,报 NoCredentialsError / Endpoint URL errorMLflow artifact store 未配置为 S3。请参见 MLflow install 中的 High availability and storage 部分,并在 MLflow plugin 中设置 artifact bucket。然后重新运行。
deploy_for_evaluation 一直看不到 Ready=True检查 InferenceService:kubectl -n finetune describe isvc <name>。最常见原因是 pvc_path tag 中的 PVC 路径不存在,或者集群中没有可用的 KServe HF runtime image。
LMEvalJob 一直停留在 Scheduledeval job pod 处于 Pending。请检查 pod 的 node selector 和 PVC 绑定:kubectl -n finetune get pods -l app=<job-name>,并对 Pending pod 执行 kubectl describe。
LMEvalJob 无法访问 InferenceServicemodelArgs 中的 base_url 必须是 -predictor Service,而不是顶层的 InferenceService。示例使用 http://<isvc>-predictor.<ns>.svc/v1/completions——其中 -predictor 后缀和 /v1/completions 路径对于 OpenAI-compatible endpoint 都是必需的。
LMEvalJob 结束为 state=Complete, reason=Failed, message="open …/output/stdout.log: permission denied"operator 管理的 outputs PVC 属于 root,但 ta-lmes-job 容器以 UID 65532 运行,因此 driver 无法创建 stdout.log。请设置 spec.pod.securityContext.fsGroup: 65532(示例中已这样做),使挂载点对 pod 的 supplemental group 可写。
state=Complete,但尽管运行失败,pipeline 仍将其视为成功对于任何终态,state 都会变为 Complete;实际结果位于 status.reason(Succeeded / Failed / Cancelled)中。请同时检查两者,就像上面的 evaluate component 那样。
local-completions 失败,报 OSError: We couldn't connect to 'https://huggingface.co' to load this file … it looks like <name> is not the path to a directory containing a file named config.json.即使 tokenized_requests 为 "False",lm-evaluation-harness client 仍然会调用 AutoTokenizer.from_pretrained(...)——tokenizer modelArgs 条目必须是 eval pod 可访问的 repo id,或者是离线 PVC 下的本地路径。在 air-gapped 集群上,请按照 Evaluate LLM 中的 Optional: offline storage and PVC 部分操作——在 spec.offline.storage.pvcName 处挂载 PVC,填充 tokenizer 和 dataset cache,并将 tokenizer 设置为挂载路径。
父运行中缺少嵌套 MLflow runsset_experiment 在 search_runs 解析父运行之前就被调用了——但实际上并没有记录任何内容,因为父 tag 是由另一个 component 的进程写入的。请在父运行内部使用 set_tag("pipeline_run_id", run_id),并像 evaluate component 那样按该 tag 进行搜索。
Recurring Run 触发了,但运行立即失败,报 401 UNAUTHENTICATEDSecret/mlflow-token 中的 token 已在上一次成功的 pipeline 与今天的运行之间过期。请轮换它(或将 token 的生成移到 component 内部);参见步骤 5 中的警告。
KFP v2 pod 报错 container has runAsNonRoot and image will run as root(发生在 argoexec init init-container 和 kfp-launcher init-container 上)KFP v2 pod template 在 pod 级别设置了 runAsNonRoot: true,但 argoexec / kfp-launcher 镜像没有设置非 root 的 USER。在会暴露此问题的后端上(例如 DSPO 基于 Argo 的 pipeline 栈),请在运行时通过 spec.podSpecPatch 进行 patch,使 pod 级别以及每个 init-container(init + kfp-launcher)都设置 runAsUser: 1001。
Fine-tune component 失败,报 PermissionError: [Errno 13] Permission denied: '/mnt/shared/...'共享 PVC 的根目录属于 root。可以在同一个 podSpecPatch 中添加 fsGroup: 1001,或者先运行一个以 root 身份执行的一次性 pod,对 PVC 执行 chown -R 1001:1001 /mnt 完成初始化。
Fine-tune component 失败,报 modelscope_hub.errors.CacheError: [E1022] Failed to create SDK directories: [Errno 13] Permission denied: '/.modelscope'python:3.12-slim 容器将 HOME 设为 /,而 ModelScope 和 Hugging Face 默认将缓存写入 ~/。非 root pod 无法在此处写入。在 fine-tune component 中,请在首次调用 from modelscope import … / AutoModel.from_pretrained 之前,将 MODELSCOPE_CACHE、HF_HOME 和 HOME 设置为共享 PVC 上的路径(或 /tmp)。
worker node 无法访问 runtime image tagair-gapped 或受限出站的集群无法直接从 docker.io 拉取。请使用 skopeo copy 或 crane copy 将 docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0 镜像到你的内部 registry,然后将 RUNTIME_IMAGE 改为镜像后的引用,并将相应的 imagePullSecrets 添加到 pipeline ServiceAccount。
在 KFP API server 提交 pipeline 失败,报 unknown component implementation: comp-exit-handler-1后端(例如 DSPO 基于 Argo 的 pipeline 栈)不接受 KFP v2 dsl.ExitHandler 子图。请将 with dsl.ExitHandler(cleanup(...)): ev 替换为普通的 cleanup(...).after(ev),并使 cleanup 幂等化(按 label selector 删除,使其在 evaluate 已成功删除资源时也能正常工作)。
KServe pod 崩溃,报 FileNotFoundError: No such file or directory: /mnt/models/model.safetensors,即使该文件确实存在于 PVC 中KServe 容器以 UID 1000 运行(在 ClusterServingRuntime 中定义),而 fine_tune component 使用默认 umask 以 UID 1001 写入,这会产生模式 0660——组可写,但其他用户不可读。在 save_pretrained 后添加一个 chmod 处理(如示例所示):将每个目录设为 0755,每个文件设为 0644。仅在 fine-tune pod 上设置 fsGroup: 1001 并不足够——它只会改变挂载根目录的组,而不会回填子文件的模式。
evaluate component 内部出现 mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: No Experiment with id=0 existsstart_run(run_id=parent) 会附加到已有运行,但嵌套的 start_run(run_name=...) 需要一个 experiment context。请在打开任何 run 之前,先在 component 顶部调用一次 mlflow.set_experiment(experiment)。否则 MLflow 会默认使用 experiment 0,而多租户服务器不允许这样做。
MLflow create_model_version 失败,报 INVALID_PARAMETER_VALUE: Invalid model version source: '/mnt/…'. To use a local path as a model version source, the run_id request parameter has to be specified and the local path has to be contained within the artifact directory of the run specified by the run_id.,而切换到 mlflow.log_artifacts 后又报 PermissionError: [Errno 13] Permission denied: '/mlflow'Alauda MLflow plugin 默认将 artifact root 设为 /mlflow/artifacts——这是 tracking server pod 上 emptyDir 中的本地路径。客户端侧的 log_artifacts / log_model 无法写入该位置,而超出 run artifact dir 的 file:// version source 会被拒绝。可选方案有两个:(a) 将 MLflow 部署重新配置为 S3 artifact backend(参见 MLflow install 中的 High availability and storage 部分),或 (b) 跳过 model registry,改为通过父运行上的 tags 协调下游步骤——pipeline 仍然会为每次运行保留一行 MLflow 记录,并且 pvc_path、final_loss、eval_acc 都可以通过 mlflow.get_run(...).data.tags 访问。
TypeError: SFTTrainer.__init__() got an unexpected keyword argument 'tokenizer'TRL 0.12+ 已移除 tokenizer 关键字——请改为将 tokenizer 作为 processing_class=tokenizer 传入。参考 runtime image 通过 LLaMA-Factory 0.9.4 间接包含了 post-0.12 的 TRL。上面的示例使用了 processing_class=;仍然传递 tokenizer= 的旧分支需要做同样的重命名。
mlflow.transformers.log_model 失败,报 MlflowException: The task could not be inferred from the model. If you are saving a custom local model that is not available in the Hugging Face hub, please provide the 'task' argument to the log_model or save_model function.MLflow 可以从 Hub model 推断 pipeline task,但无法从本地加载(路径镜像)的 model 推断——fine-tune pod 的 trainer.model 在 AutoModelForCausalLM.from_pretrained("/mnt/shared/models/…") 之后属于后者。请在 log_model 中传入 task="text-generation"(如果是其他 model type,则传入相应的 task 字符串)。
MLflow 客户端调用失败,报 UNAUTHENTICATED: Authentication with the Kubernetes API failed. The provided token may be invalid or expired.mlflow-tracking-server 的 oauth2-proxy sidecar 没有转发 bearer token——plugin 的配置面板没有暴露这一项,必须将该 flag patch 到 Deployment 上。向 oauth2-proxy 容器的 env 中添加 OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true,等待 rollout 完成,然后重新运行。请注意,mlflow-plugin controller 会 reconcile 该 Deployment,并可能撤销此 env;在每次 operator 升级或手动重新安装后需要重新应用。

构建自定义 image

参考 image 基于 llamafactory0.9-cu126-amd64 构建,仅额外添加了 mlflow>=3.10、kserve>=0.13 和 kubernetes>=29。若要自行构建——例如切换 CUDA 版本、添加专有数据处理库,或使 base 符合 FIPS 要求——请编写一个单层 Containerfile:

# syntax=docker/dockerfile:1.7
ARG BASE_IMAGE=docker.io/alaudadockerhub/llamafactory0.9-cu126-amd64:v0.1.0
FROM ${BASE_IMAGE}

USER 0
RUN uv pip install --no-cache-dir \
        "mlflow>=3.10" \
        "kubernetes>=29" \
        "kserve>=0.13" \
        # add anything your pipeline imports:
        # "your-private-etl-lib==1.2.3" \
        ;
USER 1000
WORKDIR /workspace

使用 buildctl(或 docker buildx build)配合你的 buildkitd 进行构建,推送到内部 registry,并相应设置 RUNTIME_IMAGE。pipeline 的其他代码保持不变。

如果你的集群已经运行 Kubeflow Trainer v2,你也可以复用已发布的某个 TrainingRuntime image——llamafactory0.9-cu126-amd64:v0.1.0 已经内置 torch / HF / trl / mlflow;对于 deploy_for_evaluation + evaluate + cleanup component 只需要补充 kserve 和 kubernetes。这正是参考 image 的做法。

相关指南