利用Dockerfile编排pbspro集群

简介

pbs Pro的全称为“PBS Professional，可移植批处理系统 (PBS) 最初由 NASA 研发设计，被称作“计算机系统的交通灯”，是一种工作负载管理器和调度软件。可以使用它来简化作业提交、跨多个平台集群分布工作负载，以及扩展到数百甚至数千 个处理器。PBS Pro的功能包括：

• 接受批处理作业
• 保持和保护作业，直到作业运行
• 运行作业
• 将输出交付给提交者

本次博客将讲述：从最基本的centos7镜像一步一步通过Dockerfile编排成一个可以运行的作业的pbs pro的分布式集群作业运算系统。

---

1. 由基础镜像构建满足条件的centos基本的环境

---

#The centos imgae's version is 7, python version to 3.2,and build the base images for install maya
### build zxj2017/base-centos7
FROM centos7

MAINTAINER  zxj

RUN yum install -y gstreamer-plugins-base.x86_64

RUN yum update -y && yum install -y \
    nano \
    csh \
    libXp \
    libXmu \
    libXpm \
    libXi \
    libtiff \
    libXinerama \
    elfutils \
    gcc \
    gamin \
    git \
    mesa-utils \
    mesa-libGL-devel \
    tcsh \
    xorg-x11-server-Xorg \
    xorg-x11-server-Xvfb \
    wget && \
    yum groupinstall -y "X Window System" && \
    yum clean all

RUN wget https://bootstrap.pypa.io/3.2/get-pip.py && \
    python get-pip.py && \
    pip install --target=/usr/local/lib/python3.2/site-packages \
        nose \
        mock \
        unittest2

# Enable playblasts with Quicktime
ENV LIBQUICKTIME_PLUGIN_DIR=/usr/autodesk/maya/lib

# Start Xvfb
# Provide an in-memory X-session for parts of Maya that require a GUI
# such as cmds.playblast()
ENV DISPLAY :99

# Run on user login, this has the limitation of being run
# each time a user logs into the Docker image. Suggestions
# are welcome to make this only run once at startup.
RUN echo "# Start Xvfb" >> ~/.bashrc && \
    echo "Xvfb :99 -screen 0 1024x768x16 2>/dev/null &" >> ~/.bashrc && \
    echo "while ! ps aux | \grep -q '[0]:00 Xvfb :99 -screen 0 1024x768x16';" >> ~/.bashrc && \
    echo "  do echo 'Waiting for Xvfb...'; sleep 1; done" >> ~/.bashrc
# Expose Python libraries to Maya
ENV PYTHONPATH=/usr/local/lib/python3.2/site-packages

使用编译命令，构建成为一个base-centos镜像

---

$#   docker build -t zxj/base-centos .

---

---

2 . 由上一步构建的镜像zxj/base-centos构建maya-centos镜像

---

#Use zxj/base-centos images,continue install the maya
###build zxj/centos-maya
FROM zxj/base-centos

MAINTAINER zxj

# Download and unpack distribution first, Docker's caching
# mechanism will ensure that this only happens once.
RUN wget http://edutrial.autodesk.com/NET17SWDLD/2017/MAYA/ESD/Autodesk_Maya_2017_EN_JP_ZH_Linux_64bit.tgz -O maya.tgz && \
    mkdir /maya && tar -xvf maya.tgz -C /maya && \
    rm maya.tgz && \
    rpm -Uvh /maya/Maya*.rpm && \
    rm -r /maya

# Make mayapy the default Python
RUN echo alias hpython="\"/usr/autodesk/maya/bin/mayapy\"" >> ~/.bashrc && \
    echo alias hpip="\"mayapy -m pip\"" >> ~/.bashrc

# Setup environment
ENV MAYA_LOCATION=/usr/autodesk/maya/
ENV PATH=$MAYA_LOCATION/bin:$PATH

# Workaround for "Segmentation fault (core dumped)"
# See https://forums.autodesk.com/t5/maya-general/render-crash-on-linux/m-p/5608552/highlight/true
ENV MAYA_DISABLE_CIP=1

# Cleanup
WORKDIR /root

使用编译命令，构建成为一个maya-centos镜像
$# docker build -t maya-centos .

---

3.由上一步构建的镜像zxj/maya-centos构建maya-pbspro镜像

---

#install pbspro on the zxj/mayabase
###build maya-centos
FROM maya-centos

MAINTAINER zxj

#ssh need dependent libraries
RUN yum install -y openssh-clients net-tools vim passwd openssl openssh-server

#config ssh
RUN yum install -y ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa  &&  cd /root/.ssh/ &&  cp id_rsa.pub  authorized_keys  &&  ssh-keygen -t rsa -P '' -f /etc/ssh/ssh_host_rsa_key  && ssh-keygen -t dsa -P '' -f /etc/ssh/ssh_host_dsa_key &&  ssh-keygen -t rsa -P '' -f /etc/ssh/ssh_host_ecdsa_key && ssh-keygen -t rsa -P '' -f /etc/ssh/ssh_host_ed25519_key && /usr/sbin/sshd


#pbspro need dependent libraries
RUN yum install -y rpm-build gcc autoconf automake libtool hwloc-devel libX11-devel libXt-devel libedit-devel libical-devel ncurses-devel postgresql-devel python-devel python-devel tcl-devel tk-devel swig expat-devel openssl-devel libXext libXft

#rpm-build the pbspro 
RUN mkdir -p /root/rpmbuild/SOURCES/
#ADD pbspro-14.1.0.tar.gz /root/rpmbuild/SOURCES/

#pbspro-14.1.0  download url https://github.com/PBSPro/pbspro/archive/v14.1.0.tar.gz

#RUN cd /root/rpmbuild/SOURCES/ && tar zcvf pbspro-14.1.0.tar.gz pbspro-14.1.0 
RUN cd /root/rpmbuild/SOURCES/ && \
    wget https://github.com/PBSPro/pbspro/archive/v14.1.0.tar.gz && \
    tar zxvf v14.1.0.tar.gz && mv v14.1.0.tar.gz pbspro-14.1.0.tar.gz
RUN cd /root/rpmbuild/SOURCES/pbspro-14.1.0 && rpmbuild -ba pbspro.spec

---

使用编译命令，构建成为一个base-centos镜像

$#   docker build -t maya-pbspro .

---

$#: docker images

1. 安装openpbs

自定义网络，便于管理

$# docker network create --subnet=172.20.0.0/16  clusterNet
$# docker network ls

创建容器，分别初始化出3个容器，一个master,两个slaver,如下：

$#  docker run -itd  --name pbs_master -h master  --net clusterNet --ip 172.20.10.1   --add-host="slaver1 :172.20.10.2" --add-host="slaver2 :172.20.10.3"  -v /pbs/:/pbs  maya-pbspro:latest bash

$#  docker run -itd  --name pbs_slaver1 -h slaver1  --net clusterNet --ip 172.20.10.2   --add-host="master :172.20.10.1" --add-host="slaver2 :172.20.10.3"  -v /pbs/:/pbs   maya-pbspro:latest bash

$# docker run -itd  --name pbs_slaver2  -h slaver2  --net clusterNet --ip 172.20.10.3   --add-host="master :172.20.10.1" --add-host="slaver1 :172.20.10.2"  -v /pbs/:/pbs  maya-pbspro:latest bash

在每个节点中的/root/rpmbuild/RPMS/x86_64
下 都有pbs pro编译好的rpm文件，分别在master和slaver节点安装这些文件
masetr节点安装pbs pro，进入master

$# docker exec -it pbs_master bash
$# yum install -y  pbspro-server-14.1.0-0.x86_64.rpm 

修改修改/etc/pbs.conf文件

PBS_SERVER=master
PBS_START_MOM=1

同理，分别进入pbs_slver1和pbs_slaver2安装 pbspro-execution-14.1.0-0.x86_64.rpm
并修改修改/var/spool/pbs/mom_priv/config文件,下以pbs_slaver1为例

$# docker exec -it pbs_slaver1 bash
$# yum install -y  pbspro-execution-14.1.0-0.x86_64.rpm 

修改修改/etc/pbs.conf文件

$clienthost master

1. 开启并提交作业
   每个节点开启pbs操作

$# /etc/init.d/pbs start

在maseter上加入slaver1,和slaver2

$# . /etc/profile.d/pbs.sh
$#  qmgr -c 'create node  slaver1'
$#  qmgr -c 'create node  slaver2'

为三个节点添加同意的UID的用户user1每个节点执行如下操作
#  useradd user1  
查看# pbsnodes -a

运行作业：

$# echo 'sleep 10' | qsub
$# qstat -a

至此，整个pbs Pro搭建完毕