Alauda Build of Gitlab Docs
  • English

    • Data migration from GitLab 14.0.12 to 17.8.z

    Problem to Solve

    Versions 3.16 and 3.18 support GitLab versions that lag behind the official versions. Upgrading GitLab to the latest official version requires more than 10 upgrades to complete. After upgrading to 4.0, the product does not provide automatic upgrades to 17.8.z. This solution addresses how to migrate data from 14.0.12 to 17.8.z.

    Considering the large version gap, we adopt a data migration approach for the upgrade.

    TOC

    TerminologyProcess OverviewData Migration PathData Migration ProcessPrerequisitesBackup the platform-deployed GitLab 14.0.12Restore backup data to all-in-one deployment of 14.0.12Rolling upgrade all-in-one deployed gitlab to 17.8.1Back up the data from 17.8.1 all-in-one instanceRestore all-in-one backup data to platform-deployed 17.8.zFAQRestore Uploads DataHow to Find the Mount Path of a PVC on a Node

    Terminology

    TermExplanation
    all-in-one GitLabAll GitLab components are packaged and deployed together in a single container or Pod. Used for rapid version upgrades.
    platform-deployed GitLabGitLab is deployed and managed by the platform using the Operator, with components separated.

    Process Overview

    Data Migration Path

    According to the official upgrade path documentation, the data migration path is as follows:

    • 14.0.12 -> 14.3.6
    • 14.3.6 -> 14.9.5
    • 14.9.5 -> 14.10.5
    • 14.10.5 -> 15.0.5
    • 15.0.5 -> 15.4.6
    • 15.4.6 -> 15.11.13
    • 15.11.13 -> 16.3.8
    • 16.3.8 -> 16.7.9
    • 16.7.9 -> 16.11.10
    • 16.11.10 -> 17.3.6
    • 17.3.6 → 17.5.5
    • 17.5.5 → 17.8.1

    Finally, back up the data from 17.8.1 and import it into the platform-deployed 17.8.z instance to complete the data migration.

    Data Migration Process

    Back up the platform-deployed GitLab and restore it to an all-in-one image deployed GitLab, then upgrade to 17.8.1 using the all-in-one image, and finally restore the data backup to the platform-deployed 17.8.z instance.

    1. Back up the platform-deployed GitLab 14.0.12 instance.
    2. Restore the backup to an all-in-one deployment of GitLab 14.0.12.
    3. Perform a rolling upgrade of the all-in-one deployment to GitLab 17.8.1.
    4. Back up the upgraded all-in-one GitLab 17.8.1 instance.
    5. Restore the backup from the all-in-one 17.8.1 to a platform-deployed GitLab 17.8.z instance.
    TIP

    For the latest GitLab instance and operator versions, please refer to the Release Note.

    Migration Duration

    The migration process involves database and repository backup/restore operations:

    • Larger databases increase migration time.
    • A higher number of repositories or larger single repositories significantly extends the duration.
    • Storage performance also impacts efficiency — using topolvm is recommended for better performance.

    Test setup:

    • One large repository (~600 MB), others are small initial repositories
    • GitLab instance: 6,700 issues, 3,600 merge requests
    Migration StepKey Factor1,000 Repos10,000 Repos
    Back up platform-deployed GitLab 14.0.12Repo size & count10 min30 min
    Restore to all-in-one GitLab 14.0.12Repo size & count20 min1 hr
    Rolling upgrade to GitLab 17.8.1Database size1 hr 30 min1 hr 40 min
    Back up all-in-one GitLab 17.8.1Repo size & count3 min10 min
    Restore to platform-deployed GitLabRepo size & count30 min1 hr

    Prerequisites

    1. Install kubectl (>=1.30.0) and yq (>=4.45.0) on the execution host.

      kubectl version --client
yq --version

# Output:
# Client Version: v1.30.0
# Kustomize Version: v5.4.2
# yq (https://github.com/mikefarah/yq/) version v4.45.4

    2. Configure environment variables:

      # Replace with the name of the old GitLab CR
export GITLAB_NAME=<old gitlab name>
export GITLAB_NAMESPACE=<gitlab deploy namespace>

    3. Prepare the PVCs required for the upgrade. These must be created in the same namespace as the old GitLab instance:

      TIP

      A simple way to estimate the required storage size for the original instance is to sum the PVC sizes used by the database and the Gitaly components.

      apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup-pvc
  namespace: ${GITLAB_NAMESPACE}
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 40Gi
  storageClassName: ceph
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: upgrade-pvc
  namespace: ${GITLAB_NAMESPACE}
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: ceph
  volumeMode: Filesystem
      1. backup-pvc: Used to store backup tarballs. The recommended capacity is twice the size of the original instance.
      2. Replace with the storage class name in your cluster.
      3. upgrade-pvc: Used to store GitLab data during the rolling upgrade process. It is recommended to provision storage equal to the total storage size of the original instance plus three times the database size.
      4. Replace with the storage class name in your cluster.

    4. Prepare the images needed for the upgrade. Download each version of the image from the attachments and push them to an image repository that the GitLab deployment cluster can pull from. Image download links:

      China Region
      Other Regions
      # amd images
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-0-12-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-3-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-9-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-10-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-0-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-4-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-11-13-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-3-8-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-7-9-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-11-10-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-3-6-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-5-5-amd.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-8-1-amd.tgz

# arm images
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-0-12-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-3-6-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-9-5-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-14-10-5-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-0-5-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-4-6-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-15-11-13-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-3-8-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-7-9-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-16-11-10-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-3-6-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-5-5-arm.tgz
https://cloud.alauda.cn/attachments/knowledge/242090509/gitlab-ce-17-8-1-arm.tgz

    5. Prepare the scripts needed for the upgrade.

      Execute the following command to generate four scripts:

    • check_gitlab.sh: Check if the GitLab pod is ready.
    • finalize_migrations.sql: Finalize the migrations.
    • monitor_gitlab.sh: Monitor the GitLab pod to check if data migration is complete.
    • rolling_upgrade.sh: Rolling upgrade the GitLab instances one by one.
    Command
    Generated Scripts
    cat << 'EOF' > check_gitlab.sh
#!/bin/bash

gitlab_pod=$1
port=${2:-30855}

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

log "Starting monitoring script for GitLab pod: ${gitlab_pod} on port: ${port}"

while true; do
  curl_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- curl -s localhost:${port} | grep "You are being")
  if [[ -z "${curl_result}" ]]; then
      log "HTTP not ready. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  log "GitLab is ready"
  break
done
EOF

cat << 'EOF' > finalize_migrations.sql
select concat(
  'gitlab-rake gitlab:background_migrations:finalize[',
  job_class_name, ',',
  table_name, ',',
  column_name, $$,'$$,
  REPLACE(cast(job_arguments as text), ',', $$\,$$),
  $$']$$
)
from batched_background_migrations WHERE status NOT IN(3, 6);
EOF

cat << 'EOF' > monitor_gitlab.sh
#!/bin/bash

gitlab_pod=$1
port=${2:-30855}

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

log "Starting monitoring script for GitLab pod: ${gitlab_pod} on port: ${port}"

while true; do
  curl_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- curl -s localhost:${port} | grep "You are being")
  if [[ -z "${curl_result}" ]]; then
      log "HTTP check not ready. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  migration_result=$(kubectl -n $GITLAB_NAMESPACE exec ${gitlab_pod} -- gitlab-psql -c "SELECT job_class_name, table_name,  column_name, job_arguments FROM batched_background_migrations WHERE status NOT IN(3, 6);" | grep "(0 rows)")

  if [[ -z "${migration_result}" ]]; then
      log "Database migration is running. Retrying in 10 seconds..."
      sleep 10
      continue
  fi

  log "GitLab is ready, all migrations are done"
  break
done
EOF

cat << 'EOF' > rolling_upgrade.sh
#!/bin/bash

image=$(kubectl -n $GITLAB_NAMESPACE get deployment gitlab -o jsonpath='{.spec.template.spec.containers[?(@.name=="gitlab")].image} ')

versions=(
  14.0.12-ce.0
  14.3.6-ce.0
  14.9.5-ce.0
  14.10.5-ce.0
  15.0.5-ce.0
  15.4.6-ce.0
  15.11.13-ce.0
  16.3.8-ce.0
  16.7.9-ce.0
  16.11.10-ce.0
  17.3.6-ce.0
  17.5.5-ce.0
  17.8.1-ce.0
)

STATE_FILE="upgrade_state.log"
touch "$STATE_FILE"

is_done() {
  grep -Fxq "$1" "$STATE_FILE"
}

mark_done() {
  echo "$1" >> "$STATE_FILE"
}

for version in "${versions[@]}"; do
  if is_done "$version"; then
    echo "Version ${version} already processed, skipping..."
    continue
  fi

  echo "Upgrading to ${version}..."
  new_image="${image%:*}:${version}"
  kubectl -n $GITLAB_NAMESPACE set image deployment/gitlab gitlab=$new_image

  echo "Waiting for the GitLab pod to start..."
  sleep 10
  kubectl -n $GITLAB_NAMESPACE wait deploy gitlab --for condition=available --timeout=3000s
  sleep 10
  kubectl -n $GITLAB_NAMESPACE wait pod -l deploy=gitlab --for=condition=Ready --timeout=3000s

  new_pod_name=$(kubectl -n $GITLAB_NAMESPACE get po -l deploy=gitlab --sort-by=.metadata.creationTimestamp  --field-selector=status.phase=Running -o jsonpath='{.items[-1].metadata.name}')
  echo "Waiting for GitLab to be ready..."
  bash check_gitlab.sh $new_pod_name

  echo "Waiting for migrations to finish..."
  kubectl -n $GITLAB_NAMESPACE cp finalize_migrations.sql $new_pod_name:/tmp/finalize_migrations.sql
  for i in {1..10}; do
    echo "Running migration tasks (attempt $i/10)..."
    if kubectl -n $GITLAB_NAMESPACE exec -ti $new_pod_name -- bash -c "gitlab-psql -t -A -f /tmp/finalize_migrations.sql > /tmp/run_migration_tasks.sh && xargs -d \"\n\" -P 3 -I {} bash -c \"{}\" < /tmp/run_migration_tasks.sh"; then
      echo "Migration tasks completed successfully"
      break
    fi
    sleep 30
  done
  bash monitor_gitlab.sh $new_pod_name
  echo "Upgraded to ${version} successfully"
  mark_done "$version"
done

echo "All upgrades completed successfully!"
EOF

    Backup the platform-deployed GitLab 14.0.12

    1. Enable the task-runner component on GitLab 14 to execute backup commands

      Commands for Platform < 3.18.0
      Commands for Platform 3.18.0+
      Output
      # generate the image repository and tag for the task-runner based on the gitaly image
image=$(kubectl get pod ${GITLAB_NAME}-gitaly-0 -n $GITLAB_NAMESPACE -o jsonpath='{.spec.containers[0].image}')
tag=${image##*:}
registry=${image%%/*}

echo "registry: ${registry}"
echo "tag: ${tag}"

kubectl patch gitlabofficials.operator.devops.alauda.io "${GITLAB_NAME}" \
  -n "$GITLAB_NAMESPACE" \
  --type=merge \
  -p "$(cat <<EOF
{
  "spec": {
    "helmValues": {
      "gitlab": {
        "task-runner": {
          "enabled": "true",
          "image": {
            "repository": "${registry}/devops/gitlab-org/build/cng/gitlab-task-runner-ce",
            "tag": "${tag}"
          },
          "resources": {
            "limits": {
              "cpu": "2",
              "memory": "4G"
            }
          }
        }
      }
    }
  }
}
EOF
)"

# Wait for the task-runner deployment to be available
echo "Waiting for the task-runner deployment to be available..."
sleep 60

# Execute the command and confirm that the ${GITLAB_NAME}-task-runner deployment exists
kubectl wait deploy ${GITLAB_NAME}-task-runner -n $GITLAB_NAMESPACE --for condition=available --timeout=3000s
kubectl get deploy -n $GITLAB_NAMESPACE ${GITLAB_NAME}-task-runner

    2. Set GitLab 14 to read-only mode

      Command
      Output
      kubectl patch deploy ${GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl patch deploy ${GITLAB_NAME}-sidekiq-all-in-1-v1 -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl scale deploy ${GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 0
kubectl scale deploy ${GITLAB_NAME}-sidekiq-all-in-1-v1 -n $GITLAB_NAMESPACE --replicas 0

    3. Patch the backup PVC for the GitLab 14 task-runner. The PVC should be named backup-pvc.

      Command
      Output
      kubectl patch deploy ${GITLAB_NAME}-task-runner -n $GITLAB_NAMESPACE --type='json' -p='
[
  {
    "op": "add",
    "path": "/metadata/annotations/skip-sync",
    "value": "true"
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/volumes/1",
    "value": {"name": "task-runner-tmp", "persistentVolumeClaim": {"claimName": "backup-pvc"}}
  }
]'

echo "Waiting for the task-runner deployment to be available..."
sleep 60

# Execute the command and confirm that the ${GITLAB_NAME}-task-runner deployment exists
kubectl wait deploy ${GITLAB_NAME}-task-runner -n $GITLAB_NAMESPACE --for condition=available --timeout=3000s
kubectl get deploy -n $GITLAB_NAMESPACE ${GITLAB_NAME}-task-runner

    4. Back up GitLab 14 data

      Command
      Output
      pod_name=$(kubectl get po -n $GITLAB_NAMESPACE -l app=task-runner,release=${GITLAB_NAME} -o jsonpath='{.items[0].metadata.name}')
kubectl exec -ti -n "$GITLAB_NAMESPACE" $pod_name -- gitlab-rake gitlab:backup:create SKIP=repositories

# Add permissions to the backup file.
kubectl exec -ti -n "$GITLAB_NAMESPACE" $pod_name -- bash -c '
chmod 777 /srv/gitlab/tmp/backups/*_gitlab_backup.tar
ls -lh /srv/gitlab/tmp/backups
'

      Back up the rails-secret by executing the following command, which will save the rails-secret to a file named gitlab14-rails-secret.yaml in the current directory.

      kubectl get secrets -n ${GITLAB_NAMESPACE} ${GITLAB_NAME}-rails-secret -o jsonpath="{.data['secrets\.yml']}" | base64 --decode | yq -o yaml > gitlab14-rails-secret.yaml

    5. Remove the task-runner component after backup is complete.

      Command
      Output
      kubectl patch gitlabofficials.operator.devops.alauda.io ${GITLAB_NAME} -n $GITLAB_NAMESPACE --type='merge' -p='
{
  "spec": {
    "helmValues": {
      "gitlab": {
        "task-runner": null
      }
    }
  }
}'

echo "Waiting for the task-runner deployment to be removed..."
sleep 30

kubectl get po -n "$GITLAB_NAMESPACE" -l app=task-runner,release=${GITLAB_NAME}

    Restore backup data to all-in-one deployment of 14.0.12

    1. Use the all-in-one image to deploy GitLab 14.0.12.

      First, set the NODE_IP environment variable to access the all-in-one GitLab instance through this IP and NodePort port.

      export NODE_IP=<node_ip>
export GITLAB_IMAGE=<registry-host>/gitlab/gitlab-ce:14.0.12-ce.0

      Create the all-in-one GitLab instance.

      Command
      Output
      kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: gitlab-http-nodeport
  namespace: ${GITLAB_NAMESPACE}
spec:
  ports:
    - appProtocol: tcp
      name: web
      port: 30855
      protocol: TCP
      targetPort: 30855
      nodePort: 30855
  selector:
    deploy: gitlab
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: ${GITLAB_NAMESPACE}
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      deploy: gitlab
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        deploy: gitlab
    spec:
      affinity: {}
      containers:
        - env:
            # External access address of the all-in-one gitlab, need to replace with the IP and nodeport port of a cluster node
            - name: GITLAB_OMNIBUS_CONFIG
              value: external_url 'http://${NODE_IP}:30855'
          image: ${GITLAB_IMAGE} # Can be replaced with an image that can be pulled
          imagePullPolicy: IfNotPresent
          name: gitlab
          ports:
            - containerPort: 30855
              name: http
              protocol: TCP
            - containerPort: 2424
              name: ssh
              protocol: TCP
          resources:
            limits:
              cpu: "8"
              memory: 8Gi
            requests:
              cpu: 4
              memory: "4Gi"
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            - mountPath: /var/opt/gitlab/backups
              name: backup
              subPath: backups
            - mountPath: /etc/gitlab
              name: gitlab-upgrade-data
              subPath: config
            - mountPath: /var/log/gitlab
              name: gitlab-upgrade-data
              subPath: logs
            - mountPath: /var/opt/gitlab
              name: gitlab-upgrade-data
              subPath: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 512Mi
        - name: backup
          persistentVolumeClaim:
            claimName: backup-pvc
        - name: gitlab-upgrade-data
          persistentVolumeClaim:
            claimName: upgrade-pvc
EOF

      Check if the all-in-one GitLab instance started successfully.

      Command
      Output
      pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
bash check_gitlab.sh $pod_name

    2. Modify the rails secret content of the all-in-one GitLab. Replace the secret_key_base, otp_key_base, db_key_base, openid_connect_signing_key, ci_jwt_signing_key fields with the corresponding fields from the backed up GitLab 14 rails secret.

      Command
      Output
      # Copy the gitlab-secrets.json to the local
pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl -n $GITLAB_NAMESPACE cp $pod_name:/etc/gitlab/gitlab-secrets.json all-in-one-gitlab-secrets.json

# Replace with the data backup from the gitlab 14 rails-secret.
KEYS=(secret_key_base otp_key_base db_key_base openid_connect_signing_key ci_jwt_signing_key)
for key in "${KEYS[@]}"; do
  echo "Replace ${key} ..."
  export KEY_VALUE=$(yq eval -r --unwrapScalar=false ".production.${key}" gitlab14-rails-secret.yaml)
  yq eval ".gitlab_rails.${key} = env(KEY_VALUE)" all-in-one-gitlab-secrets.json -i
done

# Copy the gitlab-secrets.json to the pod
kubectl -n $GITLAB_NAMESPACE cp all-in-one-gitlab-secrets.json $pod_name:/etc/gitlab/gitlab-secrets.json

      Disable unnecessary components.

      cat <<EOF >> gitlab.rb
prometheus['enable'] = false
gitlab_kas['enable'] = false
redis_exporter['enable'] = false
gitlab_exporter['enable'] = false
postgres_exporter['enable'] = false
sidekiq['enable'] = false
EOF

# Copy the gitlab.rb to the pod
kubectl -n $GITLAB_NAMESPACE cp gitlab.rb $pod_name:/etc/gitlab/gitlab.rb

    3. Sync Gitaly Data to the All-in-One GitLab

      Tip

      This step must be performed on the node where the all-in-one GitLab pod is running.

      First, locate the mount paths on the node for both the original GitLab Gitaly component's PVC (named repo-data-${GITLAB_NAME}-gitaly-0) and the upgrade PVC (named upgrade-pvc) used by the all-in-one GitLab.

      Use the rsync command to efficiently copy Gitaly data from the original PVC to the upgrade-pvc of the all-in-one GitLab instance.

      Tip

      Use rsync instead of scp — it's significantly faster and far more efficient for this type of data transfer.

      Retrieve the original instance storage type

      persistence_type=$(kubectl get GitlabOfficial.operator.devops.alauda.io -n $GITLAB_NAMESPACE $GITLAB_NAME -o jsonpath='{.spec.persistence.type}')
echo "gitlab persistence type: $persistence_type"
      Deployed with LocalPath
      Deployed with PVC or StorageClass
      origin_gitaly_local_mount_path=$(kubectl get GitlabOfficial.operator.devops.alauda.io -n $GITLAB_NAMESPACE $GITLAB_NAME -o jsonpath='{.spec.persistence.location.path}')/gitlab
origin_gitaly_node_ip=<origin_gitaly_node_ip>
all_in_one_upgrade_pvc_mount_path=<all_in_one_gitaly_mount_path>

# Run this command on the node where the all-in-one pod is located.
rsync -avh --chown=998:998 ${origin_gitaly_node_ip}:${origin_gitaly_local_mount_path}/ ${all_in_one_upgrade_pvc_mount_path}/data/git-data/repositories/

# check the ownership of the repository data.
ls -alh ${all_in_one_upgrade_pvc_mount_path}/data/git-data/repositories

      Restart the pod and wait for it to start up.

      Command
      Output
      # Restart the pod
kubectl delete po -l deploy=gitlab -n $GITLAB_NAMESPACE

echo "Waiting for the pod to start..."
sleep 10

pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
bash check_gitlab.sh $pod_name

    4. Restore the backup data.

      TIP

      During the recovery process, database-related errors may occur, for example:

      • ERROR: role "xxx" does not exist
      • ERROR: function xxx does not exist
      • ERROR: permission denied to create extension "pg_stat_statements"
      • ERROR: could not open extension control file

      These errors can be ignored, details can be found at (https://gitlab.com/gitlab-org/gitlab/-/issues/266988).

      Command
      Output
      # Set gitlab to read-only
pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- gitlab-ctl stop puma
kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- gitlab-ctl stop sidekiq

# Confirm that puma and sidekiq have been stopped
kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- gitlab-ctl status

# Execute restore
backup_id=$(kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- bash -c 'ls /var/opt/gitlab/backups/*14.0.12_gitlab_backup.tar | head -n1 | sed "s|.*/\([0-9_]*_14\.0\.12\)_gitlab_backup\.tar|\1|"')
echo "Backup ID: ${backup_id}"

echo "This step will prompt you multiple times to confirm whether to continue. \nPlease select \"yes\" each time to proceed."
kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- bash -c 'gitlab-backup restore BACKUP=${backup_id}'

      Restart the pod and wait for it to start up, then execute gitlab-rake gitlab:check SANITIZE=true to check GitLab integrity.

      Command
      Output
      # Restart the pod
kubectl delete po -l deploy=gitlab -n $GITLAB_NAMESPACE

# Waiting for the pod to start
sleep 10
pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
bash check_gitlab.sh $pod_name

kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- gitlab-rake gitlab:check SANITIZE=true

      Log in to GitLab to check if the instance restoration was successful (root user password remains the same as the original instance), and execute the following command to get the access address of the all-in-one instance.

      Command
      Output
      kubectl -n $GITLAB_NAMESPACE get pod -l deploy=gitlab -o yaml | grep external_url

    Rolling upgrade all-in-one deployed gitlab to 17.8.1

    Execute the rolling_upgrade.sh script, which will upgrade the all-in-one instance step by step to version 17.8.1 according to the upgrade path.

    bash -x rolling_upgrade.sh
    Tip

    During the rolling upgrade, the database may restart automatically. As a result, the script may show errors like the one below. This is expected — the script will automatically retry and continue.

    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"

    After the script execution is complete, enter the GitLab Web UI to check if the instance version is 17.8.1, and verify that the data is complete, including code repositories, users, issues, merge requests, etc.

    Back up the data from 17.8.1 all-in-one instance

    Stop the all-in-one instance service and back up the data. The backup files will be saved in the /var/opt/gitlab/backups/ directory.

    Command
    Output
    pod_name=$(kubectl get po -l deploy=gitlab -n $GITLAB_NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl -n $GITLAB_NAMESPACE exec -ti $pod_name -- bash -c '
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-ctl status
gitlab-rake gitlab:backup:create
'

    Stop the all-in-one instance service.

    Command
    Output
    kubectl scale deploy gitlab -n $GITLAB_NAMESPACE --replicas 0

    Restore all-in-one backup data to platform-deployed 17.8.z

    Namespace of the new gitlab instance

    The namespace of the new gitlab instance must be the same as the namespace of the old gitlab instance.

    1. Refer to the GitLab Deployment Guide to use the operator for deploying a new GitLab 17.8.z instance.

      Tip
      1. The newly deployed instance must be in the same namespace as the old instance
      2. The access method of the newly deployed instance must not conflict with the old instance, such as domain name, nodeport, etc.

      After the new instance is deployed, set the environment variable for the new instance name:

      export NEW_GITLAB_NAME=<name of the new GitLab instance>

    2. Enable the toolbox component for the new GitLab instance.

      kubectl patch gitlabofficial.operator.alaudadevops.io ${NEW_GITLAB_NAME} -n $GITLAB_NAMESPACE --type='merge' -p='
{
  "spec": {
    "helmValues": {
      "gitlab": {
        "toolbox": {
          "enabled": true,
          "resources": {
            "limits": {
              "cpu": 2,
              "memory": "4G"
            }
          }
        }
      }
    }
  }
}'

      Wait for the instance to complete redeployment.

      Command
      Output
      kubectl get po -l app=toolbox,release=${NEW_GITLAB_NAME} -n $GITLAB_NAMESPACE

    3. Restore backup data to the GitLab 17.8.z deployed by the operator

      TIP

      During the recovery process, database-related errors may occur, for example:

      • ERROR: role "xxx" does not exist
      • ERROR: function xxx does not exist
      • ERROR: must be owner of extension

      These errors can be ignored, details can be found at (https://gitlab.com/gitlab-org/gitlab/-/issues/266988).

      Close the external access of the new 17.8.z gitlab instance, and patch the toolbox component to mount the backup pvc.

      Command
      Output
      kubectl patch deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl patch deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --type='merge' -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl scale deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 0
kubectl scale deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --replicas 0

# Modify the toolbox, mount the backup pvc to the toolbox
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE -p='{"metadata":{"annotations":{"skip-sync":"true"}}}'
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/1", "value": {"name": "toolbox-tmp", "persistentVolumeClaim": {"claimName": "backup-pvc"}}}]'

      Wait for the toolbox pod to be ready, and then restore the backup data to the new GitLab instance.

      Command
      Output
      toolbox_pod_name=$(kubectl get po -n $GITLAB_NAMESPACE -l release=$NEW_GITLAB_NAME,app=toolbox -o jsonpath='{.items[0].metadata.name}')

# backup the old backup file
kubectl -n $GITLAB_NAMESPACE exec -ti $toolbox_pod_name -- bash -c '
mkdir -p /srv/gitlab/tmp/backup_tarballs
mv /srv/gitlab/tmp/backups/*gitlab_backup.tar /srv/gitlab/tmp/backup_tarballs || true
'

backup_file=$(kubectl -n $GITLAB_NAMESPACE exec -ti $toolbox_pod_name -- bash -c "ls /srv/gitlab/tmp/backup_tarballs/*17.8.1_gitlab_backup.tar | head -n1 | tr -d '\r\n'")
echo "Backup file of 17.8.1: ${backup_file}"

kubectl -n $GITLAB_NAMESPACE exec -ti $toolbox_pod_name -- backup-utility --restore --skip-restore-prompt -f "file://${backup_file}"

    4. Replace the rails-secret for the new GitLab instance.

      Command
      Output
      kubectl get secrets -n ${GITLAB_NAMESPACE} ${NEW_GITLAB_NAME}-rails-secret -o jsonpath="{.data['secrets\.yml']}" | base64 --decode | yq -o yaml > gitlab17-rails-secret.yaml
yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' gitlab17-rails-secret.yaml gitlab14-rails-secret.yaml > final-rails-secret.yml
kubectl create secret generic rails-secret -n ${GITLAB_NAMESPACE} --from-file=secrets.yml=final-rails-secret.yml

kubectl patch gitlabofficials.operator.alaudadevops.io ${NEW_GITLAB_NAME} -n $GITLAB_NAMESPACE --type='merge' -p='
{
  "spec": {
    "helmValues": {
      "global": {
        "railsSecrets": {
          "secret": "rails-secret"
        }
      }
    }
  }
}'

kubectl patch deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'
kubectl patch deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'
kubectl patch deploy ${NEW_GITLAB_NAME}-toolbox -n $GITLAB_NAMESPACE --type='json' -p='[{"op":"remove","path":"/metadata/annotations/skip-sync"}]'

kubectl delete pod -lrelease=${NEW_GITLAB_NAME} -n $GITLAB_NAMESPACE
kubectl scale deploy ${NEW_GITLAB_NAME}-webservice-default -n $GITLAB_NAMESPACE --replicas 1
kubectl scale deploy ${NEW_GITLAB_NAME}-sidekiq-all-in-1-v2 -n $GITLAB_NAMESPACE --replicas 1

      The operator will automatically redeploy the new GitLab instance. Wait for the instance to complete redeployment.

    5. Verify that the data after the upgrade is normal.

      • Enter the GitLab UI in administrator view and check whether the repositories and user data are normal.
      • Select some repositories and verify whether the code, branches, and merge requests are functioning properly.

    6. Clean up unused resources:

      • Delete the old GitLab instance and the old operator if necessary.
      • Delete the all-in-one GitLab instance if necessary.

    FAQ

    Restore Uploads Data

    The migrated data does not include avatars and attachments, so the new GitLab instance cannot display this content. To resolve this issue, you can replace the upload PVC of GitLab 17.8.z with the PVC used by 14.0.12.

    Set the environment variable for the uploads pvc name:

    export UPLOADS_PVC_NAME=<name of the uploads pvc>

    Patch the new GitLab instance to use the uploads PVC:

    kubectl patch gitlabofficials.operator.alaudadevops.io ${NEW_GITLAB_NAME} -n $GITLAB_NAMESPACE --type='json' -p='
{
  "spec": {
    "helmValues": {
      "global": {
        "uploads": {
          "persistence": {
            "enabled": true,
            "existingClaim": "${UPLOADS_PVC_NAME}"
          }
        }
      }
    }
  }
}'

    If your old GitLab instance uses HostPath storage, you must manually copy the data from the old instance's uploads directory to the /srv/gitlab/public/uploads path of the webservice component in the new instance.

    How to Find the Mount Path of a PVC on a Node

    Follow these steps to locate the mount path of a PersistentVolumeClaim (PVC) on a Kubernetes node:

    1. Get the Node IP Where the Pod is Running

      kubectl -n $GITLAB_NAMESPACE get pod <pod-name> -o jsonpath='{.status.hostIP}'

# Output:
# 192.168.xxx.xxx

    2. Get the Volume Name Bound to the PVC

      kubectl get pvc -n $GITLAB_NAMESPACE <pvc-name> -o jsonpath='{.spec.volumeName}'

# Output:
# pvc-359bb8d2-841e-406c-a5e7-e48bc701f610

    3. Find the PVC Mount Path on the Node

      Log in to the node (using the node IP from Step 1), then run:

      df -h | grep <pv-name>

# Output:
# 192.168.xx.xx:6789:/volumes/csi/csi-vol-4661bd66-a68f-42f6-b995-53bd5c3f79a8/ede3ffa1-0f77-4f44-b0e6-41ec7384484f   10G  216M  9.8G   3% /var/lib/kubelet/pods/7772096f-9b81-4698-8369-34689e36fc3a/   volumes/kubernetes.io~csi/pvc-359bb8d2-841e-406c-a5e7-e48bc701f610/mount

      The path shown in the last column, for example:

      /var/lib/kubelet/pods/7772096f-9b81-4698-8369-34689e36fc3a/volumes/kubernetes.io~csi/pvc-359bb8d2-841e-406c-a5e7-e48bc701f610/mount

      is the mount directory of your PVC on the node.