GPU主机安装CUDA驱动并加入Kubernetes集群

一、Ubuntu主机安装Nvidia CUDA 驱动

本小节参考NVIDIA Driver Installation Quickstart Guide :: NVIDIA Tesla Documentation
本节叙述如何使用包管理器在 Ubuntu 16.04 LTS 和 Ubuntu 18.04 LTS 发行版上安装 NVIDIA 驱动程序。

  1. NVIDIA 驱动程序在安装时需要依赖当前内核版本的linux kernel header和开发包。例如,linux 内核是 4.4.0,则必须安装 linux-headers-4.4.0
    $ sudo apt-get install linux-headers-$(uname -r)
    
  2. 确保 CUDA 软件源上的包优先于Canonical软件源
    $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
    $ wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
    $ sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
    
  3. 安装 CUDA 软件源的GPG公钥
    $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/7fa2af80.pub
    
  4. 安装 CUDA 软件源
    $ echo "deb http://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | sudo tee /etc/apt/sources.list.d/cuda.list
    
  5. 更新 APT 缓存并使用 CUDA 软件源安装驱动程序。可以使用 --no-install-recommends 选项安装简化版驱动程序,无需任何 X 依赖。这对于云实例上的 headless 安装特别有用。
    $ sudo apt-get update
    $ sudo apt-get -y install cuda-drivers
    
  6. 验证nVidia驱动安装结果
    $ nvidia-smi 
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce RTX 208...  Off  | 00000000:3B:00.0 Off |                  N/A |
    | 30%   28C    P8    17W / 250W |      0MiB / 11019MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   1  GeForce RTX 208...  Off  | 00000000:AF:00.0 Off |                  N/A |
    | 30%   25C    P8    12W / 250W |      0MiB / 11019MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+
    

二、安装Docker与NVIDIA Container Toolkit

本小节参考Installation Guide - NVIDIA Cloud Native Technologies documentation

  1. 安装Docker
    $ curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
    
  2. 添加nvidia-docker软件源与对应GPG 公钥
    $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
    $ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
    $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
    
  3. 安装nvidia-docker2
    $ sudo apt-get update
    $ sudo apt-get install -y nvidia-docker2
    
  4. Docker 的默认运行时改为 nvidia-container-runtime,而不是 runc
    $ vim /etc/docker/daemon.json
    {
     "default-runtime": "nvidia",
     "runtimes": {
         "nvidia": {
             "path": "nvidia-container-runtime",
             "runtimeArgs": []
         }
     },
     "registry-mirrors": ["https://hub-mirror.c.163.com"]
    }
    
  5. 重启 Docker Engine
    $ systemctl restart docker
    
  6. 验证 nvidia-docker
    $ docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce RTX 208...  Off  | 00000000:3B:00.0 Off |                  N/A |
    | 30%   27C    P8    17W / 250W |      0MiB / 11019MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    |   1  GeForce RTX 208...  Off  | 00000000:AF:00.0 Off |                  N/A |
    | 30%   25C    P8    14W / 250W |      0MiB / 11019MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+
    

三、添加主机到Kubesphere集群

  1. 修改config-sample.yaml,添加GPU主机到配置文件
    $ vim config-sample.yaml
    
  2. 使用kubekey根据配置文件自动化加入节点到Kubesphere集群
    $ ./kk add nodes -f config-sample.yaml
    
  3. 设置节点标签,打上GPU节点标签
    图形化操作,参考 Kubesphere - 节点管理
  4. 在Kubesphere集群安装k8s-device-plugin插件
    参考 调度 GPUs | Kubernetes
    $ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml
    

你可能感兴趣的:(GPU主机安装CUDA驱动并加入Kubernetes集群)