耿雨飞的个人博客

[[ slide_text ]]

[[ item.c ]]

Kubeflow 离线部署 & 镜像托管至harbor

发布时间：2023-09-22 作者：gengyufei 来源：gengyufei个人博客

Kubeflow

# 背景介绍
kubeflow 是一个基于k8s的机器学习平台，其内部集成了若干开源软件。但目前国内暂未有docker镜像加速源，提供完整的kubeflow依赖的镜像。故从国外服务器部署一次kubeflow，然后将kubeflow系统所依赖的镜像全部下载，便于后续在国内环境通过离线安装kubeflow。

本次教程
kubeflow版本：v1.5.0
离线部署代码托管于阿里云 codeup

# 前置检查
## 安装包
安装包在个人硬盘中，主要包含：
- manifests-1.5.0：github 关于kubeflow的安装清单manifests的v1.5.0版本代码
- image_pull: 自定义开发代码，主要用途：国外镜像的拉取、打tar包、镜像加载load
- pipelines-1.8.1: 独立部署pipeline组件
- kubeflow_to_harbor：将kubeflow系统组件所需image，修改image:tag并托管至harbor

## 默认存储类
由于kubeflow是基于k8s之上，并且系统组件中有一部分组件需要持久化存储，故需提前检查k8s系统是否已部署默认存储类（storageClass）
```
kubectl get sc -A
```

## kustomize 放置 /usr/bin
kustomize 二进制可执行程序，以包含于安装包的manifest目录下，执行：
```
cp ./kustomize /usr/bin
cp ./kustomize /usr/local/bin
```

# 安装kubeflow （image on harbor）
## docker image push to harbor
```
# 进入脚本目录 (视实际情况而定)
cd xxx/kubeflow_to_harbor/scripts

# 更新 脚本 (视实际情况而定)
images_tar_path = "/app/kubeflow/images_all_tar"
images_list_path = "/app/kubeflow/kubeflow-v1.5.0/kubeflow_to_harbor/images_info/images_info.txt"
manifest_path = "/app/kubeflow/kubeflow-v1.5.0/kubeflow_to_harbor/manifests-1.5.0"
harbor_url = "192.168.169.131:30002/kubeflow"

# 执行脚本
python3 images_push_harbor.py
```

## install kubeflow
```
# 执行上述脚本后，会产出目录 manifests-1.5.0_updated

cd xxx/manifests-1.5.0_updated
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done
```

## (可选) uninstall kubeflow
```
# 清理 kubeflow 对应的k8s资源
cd xxx/manifests-1.5.0_updated
while ! kustomize build example | kubectl delete -f -; do echo "Retrying to apply resources"; sleep 10; done

# 确保清理干净
## 查看所有命名空间的所有资源，是否还有kubeflow的残留资源，若有，则通过kubectl delete删除掉
kubectl get all --all-namespaces

kubectl get pvc -A
kubectl get ns
```

## 更新 configmap
```
# 背景：有一些configmap中的image仍旧是Google镜像仓库托管，需要修改为本地harbor的前缀域名

kubectl edit configmaps pipeline-install-config -n kubeflow

# 将
cacheImage: gcr.io/google-containers/busybox
# 改为
cacheImage: dockerhub.kubekey.local/kfp/google-containers/busybox

# 重启相关pod
kubectl get pods -n kubeflow | grep cache-server | awk '{print "kubectl delete pod " $1 " -n kubeflow"}' | bash
kubectl get pods -n kubeflow | grep kubeflow-pipelines-profile-controller | awk '{print "kubectl delete pod " $1 " -n kubeflow"}' | bash
kubectl get pods -n kubeflow | grep metadata-grpc-deployment | awk '{print "kubectl delete pod " $1 " -n kubeflow"}' | bash
kubectl get pods -n kubeflow | grep ml-pipeline | awk '{print "kubectl delete pod " $1 " -n kubeflow"}' | bash
```

## (可选) 更新namespace与node绑定
```
# 通过namespace指定kubeflow运行节点

# 一、给node打标签
kubectl label node <node_name> <key>=<value>
eg:
kubectl label ndoe k8s-node-1 env=kubeflow-gpu

# 验证node是否已打标签
kubectl describe node <node_name>

# 二、修改 kube-apiserver配置文件
vim /etc/kubernetes/manifests/kube-apiserver.yaml

# 将
- --enable-admission-plugins=NodeRestriction
# 改为
- --enable-admission-plugins=PodNodeSelector

# 三、修改 namespace
kubectl edit ns cert-manager
kubectl edit ns istio-system
kubectl edit ns auth
kubectl edit ns knative-eventing
kubectl edit ns knative-serving
kubectl edit ns kserve
kubectl edit ns kubeflow

# 在 metadata 下，增加
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: env=kubeflow-gpu
```

## 验证
```
# 获取 all pods 
kubectl get pods -A

# 获取namespace
kubectl get ns

## kubeflow 系统所包含的饿ns：
cert-manager
istio-system
auth
knative-eventing
knative-serving
kserve
kubeflow
kubeflow-user-example-com
```

## dashboard
```
# web url登录方法

# 获取nodeport
kubectl get svc -n istio-system

# 访问 url
<任意k8s - node-ip>:nodePort

# 账号密码
user: user@example.com
pass: 12341234
```

# 创建新的namespace for running pipeline
背景：
在部署kubeflow时，系统会自动创建一个命名空间 ```kubeflow-user-example-com```, pipeline run 运行其上。但后期业务需求，需要提供多个ns用于跑不同的pipeline run，则有了下面的内容：

```
# 在manifest目录下的相对路径
cd xxx/manifests-1.5.0_updated/common/user-namespace/base

vim params.env

# 修改 profile-name 的 value 为目标 namespace
profile-name=<target_namespace>

# kubectl  apply
kustomize build xxx/manifests-1.5.0_updated/common/user-namespace/base | kubectl apply -f -
```

# PS：kubeflow系统镜像托管于harbor

```
import os
import time

BASE_DIR = os.getcwd()

class ImageToHarbor(object):
    DEFAULT_HARBOR_URL = "dockerhub.kubekey.local/kfp"

def __init__(
        self,
        images_tar_path=BASE_DIR + '/images_tar',
        images_list_path=None,
        manifest_path=None,
        harbor_url="dockerhub.kubekey.local/kfp"
    ) -> None:
        self.images_tar_path = images_tar_path
        self.images_list_path = images_list_path
        self.harbor_url = harbor_url
        self.manifest_path = manifest_path
        self.manifest_updated_path = manifest_path + '_updated'     # use this dir to install kubeflow

def read_images_list(self):
        self.precheck()

with open(self.images_list_path, 'r') as f_r:
            for line in f_r.readlines()[1:]:
                single_image_info = line.strip().split()
                print(f"single_image_info: {single_image_info}")

# source_image_name = single_image_info[0]
                image_tag = single_image_info[1]
                image_id = single_image_info[2]
                image_name = single_image_info[3]

# load image from tar file
                self.image_clean(image_id)
                self.image_load(image_id=image_id)

# rename image
                new_image_name = image_name.replace(self.DEFAULT_HARBOR_URL, self.harbor_url.strip('/'))
                self.image_tag(image_id=image_id, new_image_name=new_image_name, image_tag=image_tag)

# update manifest
                if self.harbor_url != self.DEFAULT_HARBOR_URL:
                    self.update_manifest(image_name, new_image_name, self.manifest_updated_path)

# push harbor
                self.push_to_harbor(new_image_name, image_tag)

# clean
                self.image_clean(image_id)

def precheck(self):
        print()
        os.system(f"rm -rf {self.manifest_updated_path}")
        os.system(f"cp -r {self.manifest_path} {self.manifest_updated_path}")
        print("precheck done.")
        self._sleep()

def update_manifest(self, image_name, new_image_name, manifest_updated_path):
        print()
        cmd = f'sed -i "s#{image_name}#{new_image_name}#g" `grep "{image_name}" -rl {manifest_updated_path}`'
        print(f"update_manifest cmd: {cmd}")
        os.system(cmd)
        self._sleep()

def image_tag(self, image_id, new_image_name, image_tag):
        print()
        cmd = f"docker tag {image_id} {new_image_name}:{image_tag}"
        print(f"image tag cmd: {cmd}")
        os.system(cmd)
        self._sleep()

def image_load(self, image_id):
        print()
        cmd = f"docker load -i {self.images_tar_path}/{image_id}.tar"
        print(f"image load cmd: {cmd}")
        os.system(cmd)
        self._sleep()

def image_clean(self, image_id):
        print()
        cmd = f"docker rmi -f {image_id}"
        print(f"image clean cmd: {cmd}")
        os.system(cmd)
        self._sleep()

def push_to_harbor(self, image_name, image_tag):
        print()
        cmd = f"docker push {image_name}:{image_tag}"
        print(f"push to harbor cmd: {cmd}")
        os.system(cmd)
        self._sleep(3)
    
    def _sleep(self, seconds=1):
        time.sleep(seconds)

def run(self):
        self.read_images_list()

if __name__ == "__main__":
    tic = time.perf_counter()

images_tar_path = "/app/kubeflow/images_all_tar"
    images_list_path = "/app/kubeflow/kubeflow-v1.5.0/kubeflow_to_harbor/images_info/images_info.txt"
    manifest_path = "/app/kubeflow/kubeflow-v1.5.0/kubeflow_to_harbor/manifests-1.5.0"
    harbor_url = "192.168.169.131:30002/kubeflow"

obj = ImageToHarbor(
        images_tar_path=images_tar_path,
        images_list_path=images_list_path,
        manifest_path=manifest_path,
        harbor_url=harbor_url
    )
    obj.run()

toc = time.perf_counter()
    seconds = toc - tic
    m, s = divmod(seconds, 60)
    h, m = divmod(m, 60)
    print("该程序耗时: %02d:%02d:%02d" % (h, m, s))
```

上一篇：Docker 普通用户执行docker命令下一篇：Kubernetes ingress

你觉得文章怎么样