jenkins插件学习之throttle-concurrent-builds-plugin之一

今天介绍一个插件 throttle-concurrent-builds-plugin
https://github.com/jenkinsci/throttle-concurrent-builds-plugin
This plugin allows for throttling the number of concurrent builds of a project running per node or globally.

throttle 有节流的意思，也就是限制 某个任务同时并发的个数的。下面分别的讲解 这个插件在 free style project、matrix project 等项目中的应用和异同。

throttle-concurrent-builds-plugin 使用

throttle-concurrent-builds-plugin 插件是用来限制某个job并发数的. 可以限制一个job的总的并发数量, 可以限制一个job在一个节点上的并发数量.

配置 Maximum Total Concurrent Builds 为 0

配置 Maximum Concurrent Builds Per Node 为1,

有一个job名称是 test_freestyle , 设置为可以并发执行,
有4个节点 master节点 2个空闲, node0 ~ node2 空闲也是2个.

触发test_freestyle, 多触发几个,可以发现 test_freestyle 只会在 上面的4个节点 执行, 每个节点只会执行1个.也就是2个空闲只会占用1个.
如果没有配置 throttle-concurrent-builds-plugin  插件, 那么触发的任务多了就会把2个空闲都占用了.



一切都是那么的完美, 但是一到 pipeline 的 job 时候 这个配置就不生效了.


下面我们通过插件的代码 分析一把.

ThrottleQueueTaskDispatcher 类, 继承了 QueueTaskDispatcher 类, 其中有 
public CauseOfBlockage canTake(Node node, Task task)
public CauseOfBlockage canTake(Node node, BuildableItem item)
方法, 这个方法返回null表示可以在这个节点上构建执行, 如果返回非null 表示不可执行.
其他的插件 有个  slave-prerequisites-plugin 插件 也是实现的  QueueTaskDispatcher 类, 这个插件
是执行一段用户自定义的脚本,(bat/shell脚本), 执行成功就能继续构建, 执行不成功就会一直等待.


下面通过在 ThrottleQueueTaskDispatcher 类中加上一些日志输出分析一下执行流程, 先在 private CauseOfBlockage canTakeImpl(Node node, Task task)中加上,
我们这里只看 getThrottleOption().equals("project") 的情况, 也就是job中配置的 Throttle this project alone . (另外一个配置是 Throttle this project as part of one or more categories的)
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
发现会有如上的输出, 可以发现 有8条输出, 2个的 node变量是hudson.model.Hudson@6b7450b, 表示这个是master节点.
6个是个hudson.slaves.DumbSlave的,分别是node0,node1,node2这3个节点.  个数也是和每个节点的空闲个数对应的.
max = 1 表示 配置的 throttle-concurrent-builds-plugin 插件中的 Maximum Concurrent Builds Per Node 是 1.
run = 0 表示 这个节点没有任务在执行.
这是第一次触发, 也就是当前所有节点都是空闲的.

第2次触发
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
第2次触发,可以发现前面有个节点 node1 已经有任务在执行了. 这里的输出也会同样的少一个, 只有 7个了


第3次触发
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
第3次触发,可以发现前面有2个节点 node1 / 和这次的 master 节点 已经有任务在执行了. 这里的输出也会同样的少一个 , 只有 6个了



第4次触发
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
第4次触发,可以发现前面有3个节点 node1 , node0, master 节点 已经有任务在执行了. 这里的输出也会同样的少一个 , 只有 5个了

第5次触发
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
第5次触发,可以发现前面有4个节点 node2, node1, node0, master 节点 已经有任务在执行了. 这里的输出也会同样的少一个 , 只有 4个了

第五次触发只会一只有这4条输出, 除非有个节点上的任务执行完了. 也就是 第五次触发的这个任务一直在等待, 因为他的执行条件还没满足.


当前的队列和每个节点的执行任务的情况, 每个节点只有一个在执行, 第5个任务在等待中.
Build Queue (1)
test_freestyle  cancel this build

Build Executor Status
master
    1   test_freestyle #24 terminate this build
    2   Idle        

node0
    1   Idle        
    2   test_freestyle #25 terminate this build

node1
    1   Idle        
    2   test_freestyle #23 terminate this build

node2
    1    test_freestyle #26 terminate this build
    2   Idle    


node1 -> master -> node0 -> node2    


当node1上的执行完后, 这次的输出如下.
信息: node: hudson.slaves.DumbSlave[node2], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
信息: node: hudson.model.Hudson@6b7450b, task: hudson.model.FreeStyleProject@53d9da80[test_freestyle], max: 1, run: 1
发现node1上已经有2个空闲了. 可以满足条件了, 所以队列中的那个任务就在node1上执行了.

pipeline job的执行过程, 发现也是会输出8个, 和节点总空闲数是一致的,但是发现 pipeline的job 不会 执行到 判断 tjp.getThrottleOption().equals("project")) 的地方.
因为 pipeline的job  tjp 直接就是null了.

node = {DumbSlave@21387} "hudson.slaves.DumbSlave[node0]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"


node = {Hudson@20742} 
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"


node = {DumbSlave@21750} "hudson.slaves.DumbSlave[node1]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"



node = {DumbSlave@21209} "hudson.slaves.DumbSlave[node2]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"


node = {Hudson@20742} 
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"


node = {DumbSlave@21750} "hudson.slaves.DumbSlave[node1]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"

node = {DumbSlave@21209} "hudson.slaves.DumbSlave[node2]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"

node = {DumbSlave@21387} "hudson.slaves.DumbSlave[node0]"
task = {ExecutorStepExecution$PlaceholderTask@24825} "ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#51,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/51:test_pipeline #51],cookie=null,auth=null}"

代码加上了一些判断job是否是pipeline的job.是的话就做一些特殊的处理, 感觉改动的不是特别好.
下面是经过改造后, 临时的支持一下pipeline 的job的输出, 这个是 4个节点已经跑满, 然后启动了 96 号的构建的输出.


第一次执行
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#98,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/98:test_pipeline #98],cookie=null,auth=null}, max: 1, run: 0
执行前8个都是空闲的,都是0
执行后 node1 上有一个运行了.


第二次
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#99,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/99:test_pipeline #99],cookie=null,auth=null}, max: 1, run: 1
执行前node1上有个执行了
执行后node2上有个执行了


第三次
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#100,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/100:test_pipeline #100],cookie=null,auth=null}, max: 1, run: 1
执行前node1,node2都有执行了
执行后node0上有个执行了


第四次
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#101,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/101:test_pipeline #101],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#101,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/101:test_pipeline #101],cookie=null,auth=null}, max: 1, run: 0
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#101,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/101:test_pipeline #101],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#101,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/101:test_pipeline #101],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#101,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/101:test_pipeline #101],cookie=null,auth=null}, max: 1, run: 1
执行前node0, node1, node2上都有执行了
执行后master上有个执行了


第五次
信息: node: hudson.model.Hudson@1658a444, task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#102,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/102:test_pipeline #102],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node0], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#102,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/102:test_pipeline #102],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node2], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#102,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/102:test_pipeline #102],cookie=null,auth=null}, max: 1, run: 1
信息: node: hudson.slaves.DumbSlave[node1], task: ExecutorStepExecution.PlaceholderTask{runId=test_pipeline#102,label=null,context=CpsStepContext[3:node]:Owner[test_pipeline/102:test_pipeline #102],cookie=null,auth=null}, max: 1, run: 1
第五个任务就会排队阻塞了.因为设置的条件是每个节点只能运行一个.现在是4个节点分别的都运行了一个了