耿雨飞的个人博客

[[ slide_text ]]

[[ item.c ]]

Kubeflow Notebooks

发布时间：2023-04-26 作者：gengyufei 来源：gengyufei个人博客

Kubeflow

# 概要
Kubeflow Notebooks提供了一种在Kubernetes集群中，运行基于web的开发环境的方法，方法是**在pod中运行**它们。

一些主要功能包括:
- 本机支持JupyterLab, RStudio和Visual Studio Code (Code -server)。
- **用户可以直接在集群中创建笔记本容器，而不是在他们的工作站上本地创建**。
- 管理员可以为其组织**提供标准的笔记本image**，并**预装所需的软件包**。
- **访问控制由Kubeflow的RBAC管理**，使笔记本更容易在整个组织中共享

# 快速开始
## 概览
1. 安装Kuberflow
2. 浏览器中打开Kuberflow的Central Dashboard
3. 点击左侧的'**Notebooks**'
4. 点击“**New Server**” 创建一个新的notebook server
5. 指定'**notebook server**'的配置
6. 配置完notebook后，单击“CONNECT”

## 具体步骤
1. **打开kubeflow的 Central Dashboard**
2. **选择一个 namespace**
	1. 单击namespace下拉列表以查看可用的名称空间列表。
	2. 选择与Kubeflow Profile对应的名称空间
![](https://v1-6-branch.kubeflow.org/docs/images/notebooks-namespace.png)

3. **点击 Notebook Servers**
![](https://v1-6-branch.kubeflow.org/docs/images/jupyterlink.png)

4. **点击 New Server**
![](https://v1-6-branch.kubeflow.org/docs/images/add-notebook-server.png)

5. **输入一个 Name**
	1. 可以包含字母和数字，但不能有空格，例如：my-first-notebook
![](https://v1-6-branch.kubeflow.org/docs/images/new-notebook-server.png)

6. **选择一个 docker image 给notebook Server**
	1. **Custom image**: 如果你选择自定义选项，你必须指定一个Docker镜像，模板类似：```registry/image:tag```
	2. **Standard image**: 点击“image”下拉菜单查看可用镜像列表。(可以从Kubeflow管理员配置的列表中进行选择)
7. **指定notebook Server将请求的“CPU”数量**

8. **指定notebook Server将请求的“RAM”数量**

9. **指定一个“工作空间卷”作为一个PVC卷挂载到你的主文件夹**

10. **(可选)指定一个或多个“数据卷”要挂载为PVC卷**

11. **(可选)指定一个或多个额外的“配置”**
	1. 这些对应于配置文件命名空间中退出的PodDefault资源。
	2. Kubeflow将“configurations”字段中的标签与PodDefault清单中指定的财产进行匹配。
	3. 例如，在“configurations”字段中选择标签```add-gcp-secret```，以匹配包含以下配置的PodDefault清单:
	```
	apiVersion: kubeflow.org/v1alpha1
	kind: PodDefault
	metadata:
	  name: add-gcp-secret
	  namespace: MY_PROFILE_NAMESPACE
	spec:
	 selector:
	  matchLabels:
		add-gcp-secret: "true"
	 desc: "add gcp credential"
	 volumeMounts:
	 - name: secret-volume
	   mountPath: /secret/gcp
	 volumes:
	 - name: secret-volume
	   secret:
		secretName: gcp-secret
	```

12. **(可选)指定笔记本服务器将请求的任何“gpu”**
	1. Kubeflow在Pod请求中使用“限制”来为笔记本Pod提供gpu(关于调度gpu的详细信息可以在[Kubernetes文档](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ "Kubernetes文档")中找到)

13. **（可选）指定“启用共享内存”的设置**
	1. 像PyTorch这样的一些库使用共享内存进行多进程处理。
	1. 目前，Kubernetes中还没有激活共享内存的实现。
	1. 作为一种变通方法，Kubeflow在/dev/shm处装载一个空目录卷

14. **单击“LAUNCH”以使用指定的设置 创建新的notebook CRD**
	1. 应该在“notebook Server”页面上看到新笔记本服务器的条目
	2. 在“status”列中应该有一个旋转指示器。
	3. kubernetes可能需要几分钟的时间来配置notebook Server pod。
	4. 可以通过将鼠标光标悬停在“状态”列中的图标上来检查Pod的状态

15. **单击“CONNECT”以查看notebook 暴露的web界面**

# 容器-镜像
kubeflow notebook 原生支持3中类型的notebook：JupyterLab, RStudio和Visual Studio Code(代码服务器)。notebook Server作为Kubernetes Pod中的容器运行，这意味着IDE的类型(以及安装的包)取决于服务器选择的Docker 镜像。

## Images
> 官方提供了一些示例容器镜像，供参考：https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers

### Base Images
这些Base Images 为Kubeflow Notebook容器提供了一个共同的起点。

| Dockerfile     | Registry     | Notes     |
| -------- | -------- | -------- |
|[base](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/base "base")|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/base:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/base "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/base:{TAG}")	|common base image|
|[codeserver](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/codeserver "codeserver")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver:{TAG}") |base code-server (Visual Studio Code) image|
|[jupyter](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter "jupyter") | [public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter:{TAG}")	| base JupyterLab image|
|[rstudio](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/rstudio "rstudio") | [public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio:{TAG}") | base RStudio image |

### 完整 Images
以下这些image，扩展了Base Image，增加了【机器学习】常用的第三方包

| Dockerfile     | Registry     | Notes     |
| -------- | -------- | -------- |
| [codeserver-python](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/codeserver-python "codeserver-python")	| [public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:{TAG}") | code-server (Visual Studio Code) + Conda Python |
| [jupyter-pytorch (CPU)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-pytorch "jupyter-pytorch (CPU)")	 | [public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch:{TAG}")	 | JupyterLab + PyTorch (CPU) |
| [jupyter-pytorch (CUDA)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-pytorch "jupyter-pytorch (CUDA)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda:{TAG}")	|JupyterLab + PyTorch (CUDA)
|[jupyter-pytorch-full (CPU)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-pytorch-full "jupyter-pytorch-full (CPU)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full:{TAG}")	|JupyterLab + PyTorch (CPU) + common packages
|[jupyter-pytorch-full (CUDA)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-pytorch-full "jupyter-pytorch-full (CUDA)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full:{TAG}")	|JupyterLab + PyTorch (CUDA) + common packages
|[jupyter-scipy](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-scipy "jupyter-scipy")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:{TAG}")	|JupyterLab + SciPy packages
|[jupyter-tensorflow (CPU)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-tensorflow "jupyter-tensorflow (CPU)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow:{TAG}")	|JupyterLab + TensorFlow (CPU)
|[jupyter-tensorflow (CUDA)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-tensorflow "jupyter-tensorflow (CUDA)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda:{TAG}")	|JupyterLab + TensorFlow (CUDA)
|[jupyter-tensorflow-full (CPU)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-tensorflow-full "jupyter-tensorflow-full (CPU)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:{TAG}")	|JupyterLab + TensorFlow (CPU) + common packages
|[jupyter-tensorflow-full (CUDA)](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/jupyter-tensorflow-full "jupyter-tensorflow-full (CUDA)")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full:{TAG}")	|JupyterLab + TensorFlow (CUDA) + common packages
|[rstudio-tidyverse](https://github.com/kubeflow/kubeflow/tree/master/components/example-notebook-servers/rstudio-tidyverse "rstudio-tidyverse")	|[public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:{TAG}](https://gallery.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:{TAG}")	|RStudio + Tidyverse packages

### 镜像依赖关系
![](https://v1-6-branch.kubeflow.org/docs/images/notebook-container-image-chart.png)

## 自定义 Images
用户在生成Kubeflow notebook 后安装的软件包, 只能在pod生命周期内使用（除非安装到PVC支持的目录中）

为了确保在Pod重新启动期间保留包，用户需要：
- 生成包含它们的自定义图像，或者
- 确保它们安装在PVC支持的目录中

### Image Requirements
要想Kuberflow notebook使用容器镜像，需要满足以下要求：

#### 开放 http 端口 8888
- kubeflow在运行时设置一个环境变量NB_PREFIX，其中包含我们期望容器在其下侦听的URL路径
- kubeflow使用iframe，所以确保你的应用程序在HTTP响应头中设置了```Access-Control-Allow-Origin: *```

#### 以名为jovyan的用户身份运行
- jovyan的主目录应该是/home/jovyan
- jovyan的UID应该是1000

#### 使用安装在/home/jovyan的空PVC成功启动：
kubeflow在```/home/jovyan```安装了一个PVC，以在Pod重新启动时保持状态

# 提交 k8s resources
## Notebook Pod 账户
Kubeflow将 Kubernetes 账户```default-editor``` 分配给Notebook Pods。Kubernetes 账户```default-editor```绑定到```kubeflow-edit``` ClusterRole，它对许多Kubernetes资源**具有命名空间范围的权限**。

使用下述命令，获取```ClusterRole/kubeflow-edit```的RBAC列表：
```
kubectl describe clusterrole kubeflow-edit
```

## Kubectl in Notebook Pod
因为每个Notebook Pod都绑定了高度特权的Kubernetes 账户```default-editor```，所以可以在其中运行kubectl，而无需提供额外的身份验证。

例如，以下命令将创建test.yaml中定义的资源：
```
kubectl create -f "test.yaml" --namespace "MY_PROFILE_NAMESPACE"
```

# 问题排查
## notebook not starting
### 方案一： check events of Notebook
```
kubectl describe notebooks "${MY_NOTEBOOK_NAME}" --namespace "${MY_PROFILE_NAMESPACE}"
```

### 方案二： check events of Pod
```
kubectl describe pod "${MY_NOTEBOOK_NAME}-0" --namespace "${MY_PROFILE_NAMESPACE}"
```

### 方案三： check YAML of Pod
```
kubectl get pod "${MY_NOTEBOOK_NAME}-0" --namespace "${MY_PROFILE_NAMESPACE}" -o yaml
```

### 方案四： check logs of Pod
```
kubectl logs "${MY_NOTEBOOK_NAME}-0" --namespace "${MY_PROFILE_NAMESPACE}"
```

## 手动删除 notebook
### 方案一： use kubectl to delete Notebook resource
```
kubectl delete notebook "${MY_NOTEBOOK_NAME}" --namespace "${MY_PROFILE_NAMESPACE}"
```

上一篇：Kubeflow Central Dashboard 下一篇：Markdown 表格

你觉得文章怎么样