持续交付利器-python持续集成工具Buildbot(Mozilla和谷歌等使用)介绍
简介
Buildbot是python实现的开源持续构建和持续交付工具，为Mozilla, Chromium, WebKit等知名项目使用。
与Jenkins相比，Buildbot在大陆使用者较少。原因在于Jenkins的界面相对较美观，更容易上手；Jenkins的中文文档比较丰富。但是Jenkins因为资源消耗庞大、不太方便定制而不受一些有实力的公司欢迎。这些不少把目光聚焦在Buildbot。
究竟Buildbot有哪些优点让这些公司青睐呢？Buildbot基于python网络框架Twisted，分布式做得好。Buildbot可以直接使用python包，轻松拥有上万库，具备强大的扩展能力。如果你觉得Jenkins已经轻松地满足你的需求，你不需要Buildbot。如果你在Jenkins时觉得效率低下、扩展困难、一些用python等脚本可以实现的动作在Jenkins困难重重，那么可以看看Buildbot。
本教程基于：0.8.12 linux
Buildbot是开源的自动化软件构建，测试，发布流程的框架。
Buildbot支持跨平台，分布式，并行执行jobs，与版本控制系统的灵活集成，丰富的状态报告等等。
Buildbot是一个作业调度系统：它会对作业进行排队，在所需要的资源可用时执行任务，并报告结果。
Buildbot有一个或多个主机和从机。主机监控源代码库的变化，调配从机，并给用户和开发者报告结果。从机可在多种操作系统上运行。
可以配置Python脚本到主机。这个脚本可以简单到只配置内置组件，也可以充分发挥python所长，可以动态生成的配置，定制的组件及其他任何你能想到的。
该框架基于Twisted实现，并与所有主要的操作系统兼容。
Buildbot支持持续集成，持续部署，发布管理等的。Buildbot支持持续集成测试，自动化复杂的编译系统，应用程序部署和复杂的软件发布流程管理。比CruiseControl或Jenkins更适合混合语言的环境。在 Chromium,WebKit, Firefox, Python和Twisted等有广泛的使用。
参考资料：
https://en.wikipedia.org/wiki/Buildbot
http://buildbot.readthedocs.org/en/v0.8.9
https://docs.buildbot.net/
https://www.ibm.com/developerworks/cn/linux/l-buildbot/
https://pypi.python.org/pypi/buildbot
http://buildbot.net/
快速入门
安装
pip install buildbot pip install buildbot-slave
入门实例
下面例子获取https://github.com/pyflakes/pyflakes的代码，并执行单元测试。
创建master
buildbot create-master master mv master/master.cfg.sample master/master.cfg buildbot start master
日志在master/twistd.log
创建slave
buildslave create-slave slave localhost:9989 example-slave pass buildslave start slave
日志在slave/twistd.log
用浏览器打开http://localhost:8010
点击“Waterfall Display link ”:
点击“runtests”
点击“Force Build”
结果：
点击stdio可以查看控制台输出。下面选录了部分输出。
pyflakes.test.test_imports Python26Tests test_usedAsClassDecorator ... [OK] Test test_aliasedImport ... [OK] test_assignRHSFirst ... [OK] test_differentSubmoduleImport ... [OK] test_duplicateSubmoduleImport ... [OK] test_functionNamesAreBoundNow ... [OK] test_functionsRunLater ... [OK] test_futureImport ... [OK] test_futureImportFirst ... [OK] test_ignoreNonImportRedefinitions ... [OK] test_importStar ... [OK] test_importedInClass ... [TODO]
点击“Build 0“可以查看构建的概况。
配置脚本在master/master.cfg，这实际是一个python文件，通常包含的内容：
BUILDSLAVES、CHANGESOURCES、SCHEDULERS、BUILDERS、STATUS TARGETS、PROJECT IDENTITY、DB URL等配置。
# -*- python -*-
# ex: set syntax=python:
from buildbot.plugins import *
# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.
# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}
####### BUILDSLAVES
# The 'slaves' list defines the set of recognized buildslaves. Each element is
# a BuildSlave object, specifying a unique slave name and password. The same
# slave name and password must be configured on the slave.
c['slaves'] = [buildslave.BuildSlave("example-slave", "pass")]
# 'protocols' contains information about protocols which master will use for
# communicating with slaves.
# You must define at least 'port' option that slaves could connect to your master
# with this protocol.
# 'port' must match the value configured into the buildslaves (with their
# --master option)
c['protocols'] = {'pb': {'port': 9989}}
####### CHANGESOURCES
# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes. Here we point to the buildbot clone of pyflakes.
c['change_source'] = []
c['change_source'].append(changes.GitPoller(
'git://github.com/buildbot/pyflakes.git',
workdir='gitpoller-workdir', branch='master',
pollinterval=300))
####### SCHEDULERS
# Configure the Schedulers, which decide how to react to incoming changes. In this
# case, just kick off a 'runtests' build
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
name="all",
change_filter=util.ChangeFilter(branch='master'),
treeStableTimer=None,
builderNames=["runtests"]))
c['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=["runtests"]))
####### BUILDERS
# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them. Note that any particular build will
# only take place on one slave.
factory = util.BuildFactory()
# check out the source
factory.addStep(steps.Git(repourl='git://github.com/buildbot/pyflakes.git', mode='incremental'))
# run the tests (note that this will require that 'trial' is installed)
factory.addStep(steps.ShellCommand(command=["trial", "pyflakes"]))
c['builders'] = []
c['builders'].append(
util.BuilderConfig(name="runtests",
slavenames=["example-slave"],
factory=factory))
####### STATUS TARGETS
# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.
c['status'] = []
from buildbot.status import html
from buildbot.status.web import authz, auth
authz_cfg=authz.Authz(
# change any of these to True to enable; see the manual for more
# options
auth=auth.BasicAuth([("pyflakes","pyflakes")]),
gracefulShutdown = False,
forceBuild = 'auth', # use this to test your slave once it is set up
forceAllBuilds = 'auth', # ..or this
pingBuilder = False,
stopBuild = False,
stopAllBuilds = False,
cancelPendingBuild = False,
)
c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
####### PROJECT IDENTITY
# the 'title' string will appear at the top of this buildbot
# installation's html.WebStatus home page (linked to the
# 'titleURL') and is embedded in the title of the waterfall HTML page.
c['title'] = "Pyflakes"
c['titleURL'] = "https://launchpad.net/pyflakes"
# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server (usually the html.WebStatus page) is visible. This
# typically uses the port number set in the Waterfall 'status' entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.
c['buildbotURL'] = "http://localhost:8010/"
####### DB URL
c['db'] = {
# This specifies what database buildbot uses to store its state. You can leave
# this at its default for all but the largest installations.
'db_url' : "sqlite:///state.sqlite",
}