Teuthology节点的部署——Ceph自动化测试平台(六)

CentOS下搭建Teuthology Ceph自动化测试平台(一)
CentOS下搭建Teuthology Ceph自动化测试平台(二)
CentOS下搭建Teuthology Ceph自动化测试平台(三)
CentOS下搭建Teuthology Ceph自动化测试平台(四)
CentOS下搭建Teuthology Ceph自动化测试平台(五)
Teuthology节点的部署——Ceph自动化测试平台(六)
Teuthology的使用与Ceph自动化测试用例的编写(一)
Teuthology的使用与Ceph自动化测试用例的编写(二)

本节是 CentOS下搭建Teuthology Ceph自动化测试平台(六),主要介绍Teuthology节点的部署,稍微改了下标题。

    • 部署准备
    • 代码修改
    • 调度者部署
    • 执行者部署
    • 日志服务器部署
    • 邮件服务

部署准备

以下内容在,我使用了新的节点,因为会产生很多的日志,最后还需要把日志做成日志服务器,方便在浏览器上查看。这个节点需要比较大的硬盘空间。该节点包含两个用户,一个是管理者,它负责生成任务JOB,并将任务JOB发布到队列中,另一个是执行者则从队列中取出JOB,然后使用slave资源节点来执行相应的任务。

首先在安装相关的依赖:

#yum install git python-pip python-virtualenv beanstalkd
#yum install mariadb-devel libev-devel libvirt-devel libffi-devel 

启动beanstackd队列:

#systemctl start beanstalkd

创建teuthology与teuthworker两个用户:

#useradd -m teuthology -g root -G root
#echo teuthology:1q2w3e | chpasswd

#useradd -m teuthworker -g root -G root
#echo teuthworker:1q2w3e | chpasswd

分别给两个账号授予passwordless sudo access权限:

#vi /etc/sudoers

# 添加下面两行

teuthology   ALL=(ALL)         NOPASSWD: ALL
teuthworker  ALL=(ALL)         NOPASSWD: ALL

添加配置文件:

#vi /etc/teuthology.yaml

# lab_domain: the domain name to append to all short hostnames
#lab_domain: example.com
lab_domain: 'abc.cn'

# The root directory to use for storage of all scheduled job logs and
# other data.
#archive_base: /home/teuthworker/archive
archive_base: /home/teuthworker/archive

# The default machine_type value to use when not specified. Currently
# only used by teuthology-suite.
default_machine_type: awesomebox

# Control how many machines need to be free in the cluster. 0 means
# Teuthology can use the entire cluster.
reserve_machines: 0

# The host and port to use for the beanstalkd queue. This is required
# for scheduled jobs.
#queue_host: localhost
#queue_port: 11300
queue_host: 127.0.0.1
queue_port: 11300


# The URL of the lock server (paddles). This is required for scheduled
# jobs.
#lock_server: http://paddles.example.com:8080/
lock_server: 'http://192.168.122.34:8080/'

# The URL of the results server (paddles).
#results_server: http://paddles.example.com:8080/
results_server: 'http://192.168.122.34:8080/'

# This URL of the results UI server (pulpito). You must of course use
# paddles for pulpito to be useful.
results_ui_server: 'http://192.168.122.34:8081/'

# Email address that will receive job results summaries.
#results_email: [email protected]

# Email address that job results summaries originate from
results_sending_email: 123456789@qq.com

# How long (in seconds) teuthology-results should wait for jobs to finish
# before considering them 'hung'
results_timeout: 43200

# Gitbuilder archive that stores e.g. ceph packages
#gitbuilder_host: gitbuilder.example.com
gitbuilder_host: '192.168.122.251'

# URL for 'gitserver' helper web application
# see http://github.com/ceph/gitserver
githelper_base_url: http://git.ceph.com:8080

# Verify the packages signatures
check_package_signatures: false

# Where all git repos are considered to reside.
ceph_git_base_url: https://github.com/Github641234230/

# Where the ceph git repo is considered to reside.
ceph_git_url: https://github.com/Github641234230/ceph.git

# Where the ceph-qa-suite git repo is considered to reside.
#ceph_qa_suite_git_url: https://github.com/ceph/ceph-qa-suite.git
ceph_qa_suite_git_url: https://github.com/Github641234230/ceph.git

# Where teuthology and ceph-qa-suite repos should be stored locally
src_base_path: /home/foo/src

# Where teuthology path is located: do not clone if present
teuthology_path: /home/foo/src/github.com_Github641234230_teuthology_master

# Whether or not teuthology-suite, when scheduling, should update
# itself from git. This is disabled by default.
automated_scheduling: false

# How often, in seconds, teuthology-worker should poll its child job
# processes
watchdog_interval: 120

# How long a scheduled job should be allowed to run, in seconds, before
# it is killed by the worker process.
max_job_time: 259200

# The template from which the URL of the repository containing packages
# is built.
#
# {host} is 'gitbuilder_host' from .teuthology.yaml
# {proj} is the value of 'project' from the job yaml file or 'ceph'
# {flavor} is the value of 'flavor' from the job yaml file or 'basic'
# {uri} is ref/tag if 'tag' is set in the job yaml file
#       or ref/branch if 'branch' is set in the job yaml file
#       or sha1/sha1 if 'sha1' is set in the job yaml file
#       or ref/master
# {pkg_type} is either 'deb' or 'rpm' depending on the host on which the
#            packages are to be installed
# {dist} If lsb_release -si is Fedora the value is:
#          Fedora 20 => fc20
#          Fedora 21 => fc21
#          etc.
#        If lsb_release -si is CentOS or RedHatEnterpriseServer it is
#          CentOS 6.5 => centos6
#          CentOS 7.0 => centos7
#          CentOS 7.1 => centos7
#          RedHatEnterpriseServer 6.4 => centos6
#          RedHatEnterpriseServer 7.0 => centos7
#          RedHatEnterpriseServer 7.1 => centos7
#          etc.
#       Everything else is whatever lsb_release -sc returns
#          Ubuntu 12.04 => precise
#          Ubuntu 14.04 => trusty
#          Debian GNU/Linux 7.0 => wheezy
#          Debian GNU/Linux 8.0 => jessie
#          etc.
# {arch} is the output of the 'arch' command on the host on which
#        the packages are to be installed
#           i386
#           x86_64
#           armv7l
#           etc.
baseurl_template: http://{host}/{proj}-{pkg_type}-{dist}-{arch}-{flavor}/{uri}
use_shaman: False

# If True, teuthology-suite verifies that a package matching the
# desired ceph branch exists in the gitbuilder. If False, no
# verification is done and teuthology-suite assumes the packages
# are either not necessary to run the task or they are created on
# demand.
suite_verify_ceph_hash: False

# If true, teuthology-suite will schedule jobs even if the required
# packages are not built.
suite_allow_missing_packages: False

# The rsync destination to upload the job results, when --upload is
# is provided to teuthology-suite.
#
#archive_upload: ubuntu@teuthology-logs.public.ceph.com:./

# The path to the SSH private key for rsync to upload to archive_upload
#
#archive_upload_key: None

# The public facing URL of the archive_upload location
#
#archive_upload_url: http://teuthology-logs.public.ceph.com/

# The OpenStack backend configuration, a dictionary interpreted as follows
#
openstack:

  # The teuthology-openstack command will clone teuthology with
  # this command for the purpose of deploying teuthology from
  # scratch and run workers listening on the openstack tube
  #
  clone: git clone http://github.com/ceph/teuthology

  # The path to the user-data file used when creating a target. It can have
  # the {os_type} and {os_version} placeholders which are replaced with
  # the value of --os-type and --os-version. No instance of a give {os_type}
  # and {os_version} combination can be created unless such a file exists.
  #
  user-data: teuthology/openstack/openstack-{os_type}-{os_version}-user-data.txt

  # The IP number of the instance running the teuthology cluster. It will
  # be used to build user facing URLs and should usually be the floating IP
  # associated with the instance running the pulpito server.
  #
  ip: 8.4.8.4

  # OpenStack has predefined machine sizes (called flavors)
  # For a given job requiring N machines, the following example select
  # the smallest flavor that satisfies these requirements. For instance
  # If there are three flavors
  #
  #   F1 (10GB disk, 2000MB RAM, 1CPU)
  #   F2 (100GB disk, 7000MB RAM, 1CPU)
  #   F3 (50GB disk, 7000MB RAM, 1CPU)
  #
  # and machine: { disk: 40, ram: 7000, cpus: 1 }, F3 will be chosen.
  # F1 does not have enough RAM (2000 instead of the 7000 minimum) and
  # although F2 satisfies all the requirements, it is larger than F3
  # (100GB instead of 50GB) and presumably more expensive.
  #
  # This configuration applies to all instances created for teuthology jobs
  # that do not redefine these values.
  #
  machine:

    # The minimum root disk size of the flavor, in GB
    #
    disk: 20 # GB

    # The minimum RAM size of the flavor, in MB
    #
    ram: 8000 # MB

    # The minimum number of vCPUS of the flavor
    #
    cpus: 1

  # The volumes attached to each instance. In the following example,
  # three volumes of 10 GB will be created for each instance and
  # will show as /dev/vdb, /dev/vdc and /dev/vdd
  #
  # This configuration applies to all instances created for teuthology jobs
  # that do not redefine these values.
  #
  volumes:

    # The number of volumes
    #
    count: 3

    # The size of each volume, in GB
    #
    size: 10 # GB

# The host running a [PCP](http://pcp.io/) manager
#pcp_host: http://pcp.front.sepia.ceph.com:44323/

# Settings for http://www.conserver.com/
use_conserver: true
conserver_master: conserver.front.sepia.ceph.com
conserver_port: 3109

# Settings for [nsupdate-web](https://github.com/zmc/nsupdate-web)
# Used by the [libcloud](https://libcloud.apache.org/) backend
#nsupdate_url: http://nsupdate.front.sepia.ceph.com/update

# Settings for https://fogproject.org/
fog:
  endpoint: http://fog.example.com/fog
  api_token: your_api_token
  user_token: your_user_token
  machine_types: ['mira', 'smithi']

archive_server: http://192.168.122.28/

smtpServer: smtp.qq.com
smtpUser: 123456789@qq.com
smtpPasswd: wheasdfaknaagnnasdjna

配置文件中,相应的内容应当按照自己的设定来更改。这里解释一些比较重要的配置。

  • lab_domain 可以配置为自己的域名,这个名字后面slave的名字会包含该字段。
  • archive_base 用于存储日志信息的,后面也会把它配置为可以浏览器访问的服务器
  • lock_server 填自己的paddles的地址
  • results_server 同样填自己的paddles的地址
  • results_ui_server 填自己的pulpito的地址
  • results_sending_email 填用于发送结果邮件的邮箱,需要与后面的smtpUser相同
  • gitbuilder_host 填写之前配置的gitbuilder的地址
  • ceph_git_base_url 前面提到过,建议将用到的代码都拉到自己的仓库,这里填自己仓库的地址
  • ceph_git_url 自己仓库中ceph的url
  • ceph_qa_suite_git_url 一样是自己仓库中ceph的url,因为ceph-qa-suite项目合并到ceph目录下了
  • src_base_path 这个是teuthology跑任务的时候,从网上拉去的代码放在的目录,这个目录的代码一般是slave节点跑测试的时候用到的,里面一般包含teuthology、ceph-cm-ansible、ceph等。这些项目是slave跑测试需要用到里面的一些内容例如脚本之类的,注意区分辖以下一小节中安装Teuthology用到的代码。也就是说,Teuthology节点跑的时候,用的是待会拉取在/home/teuthology/teuthology的代码,而slave资源节点则是在测试的时候运行的是src_base_path下的代码,而且也会用到src_base_path里面的teuthology。最好连个地方的代码保持一致。
  • teuthology_path 这个是teuthology的目录,如果指定了路径,就不会每次跑任务都从网上拉去项目到src_base_path,而是直接使用这目录下的
  • use_shaman: 是否使用项目shaman,默认使用该项目,不适用gitbuilder,这里设置为False,在teuthology的config.py中也要设置(好像是这里生效的,不记得了,所以都改改把)。
  • smtpServer 用来发邮件的邮箱的smtp服务器
  • smtpUser 发件者的用户名
  • smtpPasswd 发件者的授权码,不是密码,是授权码,需要去邮箱里设置得到

代码修改

Teuthology的一些代码修改,主要是为了适配自己的环境,关于Teuthology的代码的并不是很难懂,可以自己去看看代码,这也是解决安装或者运行过程中个出现问题的方法。

建议将修改后的代码提交到自己的仓库,然后平台运行的时候拉取的时候,就是同一份了。让然也可以拉去下来自己在本地修改。

启用gitbuilder这个项目,不使用shaman项目。对应的配置文件也需要修改:

diff --git a/teuthology/config.py b/teuthology/config.py
index 16b6987..bb0923a 100644
--- a/teuthology/config.py
+++ b/teuthology/config.py
@@ -158,7 +158,7 @@ class TeuthologyConfig(YamlConfig):
         'kojiroot_url': 'http://kojipkgs.fedoraproject.org/packages',
         'koji_task_url': 'https://kojipkgs.fedoraproject.org/work/',
         'baseurl_template': 'http://{host}/{proj}-{pkg_type}-{dist}-{arch}-{flavor}/{uri}',
-        'use_shaman': True,
+        'use_shaman': False,
         'shaman_host': 'shaman.ceph.com',
         'teuthology_path': None,
         'suite_verify_ceph_hash': True,

对于使用虚拟机的slave,应当至少新增一块磁盘。下面的修改是由于本人在新增磁盘的时候,选择的是VirtIO的接口类型,所以 sda是系统使用的磁盘,而vda是空的磁盘。这时需要修改代码,不然在创建OSD的时候,可能会移除vda这个设备,而使用sda来创建OSD。具体需不需要修改这个部分的代码,主要看自己的磁盘类型。当然你也可以,将磁盘的类型想办法用符合teuthology代码的,这样就不用修改代码了。

diff --git a/teuthology/misc.py b/teuthology/misc.py
index 59825bc..3aedd86 100644
--- a/teuthology/misc.py
+++ b/teuthology/misc.py
@@ -853,7 +853,7 @@ def get_scratch_devices(remote):

     # Remove root device (vm guests) from the disk list
     for dev in devs:
-        if 'vda' in dev:
+        if 'sda' in dev:
             devs.remove(dev)
             log.warn("Removing root device: %s from device list" % dev)

执行install的task的时候安装的ceph的包,根据需要修改,这里修改了一个包。

diff --git a/teuthology/task/install/packages.yaml b/teuthology/task/install/packages.yaml
index 1e8916a..5da1501 100644
--- a/teuthology/task/install/packages.yaml
+++ b/teuthology/task/install/packages.yaml
@@ -29,7 +29,7 @@ ceph:
   - ceph-fuse
   - cephfs-java
   - libcephfs_jni1
-  - libcephfs1
+  - libcephfs2
   - librados2
   - librbd1
   - python-ceph

这个地方还应该是一个BUG,使用gitbuilder需要修改,因为原来的第一处的修改,其实if判断中isinstance的永远返回的是Fasle,其对比的两个东西,可以自己看看对应的函数是啥,get_builder_project() 得到的并不是一个对象。这里直接使用配置来判断,get_builder_project()函数里面也是使用config里面的参数来判断的,这里直接使用config的参数判断了。

第二的地方的改动是为了修改生成的ceph.repo,将其设置为不检查gpg,不然安装包的时候,会报错,说打包的rpm包未签名。

diff --git a/teuthology/task/install/rpm.py b/teuthology/task/install/rpm.py
index b84ee60..9f8028a 100644
--- a/teuthology/task/install/rpm.py
+++ b/teuthology/task/install/rpm.py
@@ -188,8 +188,7 @@ def _yum_fix_repo_host(remote, project):
     Update the hostname to reflect the gitbuilder_host setting.
     """
     # Skip this bit if we're not using gitbuilder
-    if not isinstance(packaging.get_builder_project(),
-                      packaging.GitbuilderProject):
+    if teuth_config._defaults['use_shaman']:
         return
     old_host = teuth_config._defaults['gitbuilder_host']
     new_host = teuth_config.gitbuilder_host
@@ -201,6 +200,8 @@ def _yum_fix_repo_host(remote, project):
         args=[
             'if', 'test', '-f', repo_path, run.Raw(';'), 'then',
             'sudo', 'sed', '-i', '-e', run.Raw(host_sed_expr),
+            '-e',
+            's/gpgcheck=1/gpgcheck=0/',
             repo_path, run.Raw(';'), 'fi']
     )

邮箱修改,参考文献里面也有,使用smtp发送邮件,并在配置文件增加相应的属性。

diff --git a/teuthology/results.py b/teuthology/results.py
index 422aaab..c22cc53 100644
--- a/teuthology/results.py
+++ b/teuthology/results.py
@@ -108,7 +108,13 @@ def email_results(subject, from_, to, body):
     msg['From'] = from_
     msg['To'] = to
     log.debug('sending email %s', msg.as_string())
-    smtp = smtplib.SMTP('localhost')
+    smtpserver = config.smtpServer or 'localhost'
+    smtpuser = config.smtpUser
+    smtpasswd = config.smtpPasswd
+    smtp = smtplib.SMTP(smtpserver)
+    if smtpuser is not None and smtpasswd is not None:
+        smtp.login(smtpuser, smtpasswd)
+    #smtp = smtplib.SMTP('localhost')
     smtp.sendmail(msg['From'], [msg['To']], msg.as_string())
     smtp.quit()

调度者部署

进入teuthology用户,然后克隆项目代码。现在已经支持CentOS下的安装了。这里应该克隆自己仓库里面的teuthology代码。

#su – teuthology
#git clone https://github.com/ceph/teuthology.git
#mkdir ~/.virtualenvs
#virtualenv --system-site-packages ~/.virtualenvs/teuthology
#source ~/.virtualenvs/teuthology/bin/activate
#cd teuthology
#./bootstrap

可能会提示一些包内有安装不能运行bootstrap,去安装上就可以了,然后继续执行该脚本,它初始化了环境,安装了很多的组件。
添加环境变量,然后测试是否安装成功,是否可用。

#echo 'PATH="$HOME/teuthology/virtualenv/bin:$PATH"' >> ~/.profile
#source ~/.profile
#teuthology --version//输出版本号验证一下即可

添加资源节点slave,首先从github上拉取代码,将该脚本放到teuthology目录下。
在前面我们已将创建了一些资源节点,并克隆了很多,克隆之后,记得修改对他们的hostname和hosts文件。这里就是要将这些节点的信息收集起来,并传给数据库保存起来。这些工作都有脚本写好了,拉取create_nodes.py就可以。

#wget https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/create_nodes.py

根据设置修改一些参数,这里直接摘录参考文档的部分内容,介绍的很清楚了。

#!/usr/bin/env python
# A sample script that can be used while setting up a new teuthology lab
# This script will connect to the machines in your lab, and populate a
# paddles instance with their information.
#
# You WILL need to modify it.

import traceback
import logging
import sys
from teuthology.orchestra.remote import Remote
from teuthology.lock import update_inventory

# 这里改为你的paddles地址,这是本文一直使用的paddles地址
paddles_address = 'http://172.16.38.101:8080'

# 你想创建的机器类型,也就是为你的slave节点分个类
# 什么类型名字其实无所谓,但等会你执行任务时默认为plana
# 指定为plana类型,运行任务时可以省去指定机器类型语句
# 建议你以plana, buripa, miraas作为类型名,方便和官方统一
machine_type = 'plana'
# 前面我们配置/etc/teuthology.yaml文件时已经指定了域名,相同就行
lab_domain = 'h3c-ceph.com'
# Don't change the user. It won't work at this time.
user = 'ubuntu'
# We are populating 'typica003' -> 'typica192'
# 这里更改一下编号,从哪一号到哪一号
# 这是需要修改的最后一行,后面都不需要修改了
machine_index_range = range(17, 22)

修改hosts文件,为了让脚本能够收集到节点的信息,应该设置hosts文件。

#vi /etc/hosts
    //增加所有资源节点的hostname 与 IP 对
    Slave-hostname    IP

    hostname的组成结构是:machine_type + 3位数字 + '.' + lab_domain
    例如:plana001.xxxxx.cn

配置teuthology用户与所有slave资源节点的无密钥登陆:

#ssh-keygen
#ssh-copy-id hostname   

修改完脚本和hosts之后就可以执行脚本了。

#python create_nodes.py

然后打开网页pulpito,点击右上角的node,选择ALL,查看节点,如果看到刚才加入的资源节点就成功了。
可能会有一些报错,但是其实teuthology已经支持CentOS的脚本安装了,不要去改动bootstrap了。碰到的问题无非是网络问题和权限问题,耐心的看输出的信息,就可以解决。比如防火墙啊,selinux之类的。

执行者部署

调度者部署完毕,将资源节点添加到数据库中之后,就可以退出teuthology用户,进入teuthworker用户进行执行者部署。

#su – teuthworker
#git clone https://github.com/ceph/teuthology.git
#mkdir ~/.virtualenvs
#virtualenv --system-site-packages ~/.virtualenvs/teuthology
#source ~/.virtualenvs/teuthology/bin/activate
#cd teuthology
#./bootstrap
#python setup.py develop

添加环境变量,然后测试是否安装成功,是否可用。

#echo 'PATH="$HOME/teuthology/virtualenv/bin:$PATH"' >> ~/.profile
#source ~/.profile
#teuthology --version//输出版本号验证一下即可

下载脚本:

$wget -O https://raw.githubusercontent.com/ceph/teuthology/master/docs/_static/worker_start.sh

创建日志目录,之前的teuthology的配置文件也指定了日志路径,根据自己的设置,创建出该目录。

#mkdir -p ~/archive/worker_logs

配置无密钥登陆

#ssh-keygen
#ssh-copy-id hostname

使用worker_start来启动一个plana的工作进程。

#worker_start plana 1

如果报错,注意看输出的错误信息,可能是网络、防火墙、selinux等的问题。

日志服务器部署

这里采用nginx来构建服务器
安装nginx:

#yum install nginx

启动 nginx:

#systemctl start nginx

进入浏览器,输入本机的IP地址,可以看到nginx的欢迎界面,说明安装成功了。接下来就修改配置文件,将我们之前的目录发布出去。

#vi /etc/nginx/nginx.conf
        修改两个地方:
        default_type  text/plain
        ………
            location / {
            # 打开目录控制,使我们的页面能以目录方式呈现给用户
            autoindex on;
            root   /home/ teuthworker/archive;
        }

启动nginx:

#systemctl restart nginx

使用浏览器访问 http://192.168.122.28(本机IP),就可以看到日志的目录树结构的界面了。

这里不要忘记了回去修改网页中JOB的日志连接,应当连接到本机IP下的日志路径。
回到Paddles那一节,找到:

job_log_href_templ= 'http://qa-proxy.ceph.com/teuthology/{run_name}/{job_id}/teuthology.log'  # noqa

需要将网址改为本机IP,注意需要将上面的qa-proxy.ceph.com/teuthology字段替换掉

job_log_href_templ= 'http://192.168.122.28/{run_name}/{job_id}/teuthology.log'  # noqa

这样依赖在teuthology跑任务的时候,从web查看状态,便可以点击JOB里面的日志符号跳转到JOB相应的日志。

邮件服务

代码修改部分也提到了,将相应的代码修改,然后在 /etc/teuthology.yaml文件增加相应的属性,在前面的配置文件中也包含了。这里不再重复了

你可能感兴趣的:(CI/CD,Ceph,Teuthology,Teuthology——,Ceph的自动化测试平台)