BullMq: job stalled more than allowable limit
bullMq是一个nodejs版本的基于redis的任务队列，我感觉还是挺好用的，，比较方便不用自己写。
https://github.com/OptimalBits/bull
最近在生产环境会遇到job stalled more than allowable limit
问题原因是：bullmq会定期检查任务是否还在存活，但是如果有cpu耗时比较严重的，可能会阻塞这个操作，bullmq就认为任务失败了，就会重新执行这个任务。同时造成任务的重复消费。
方法一：调整一下参数：
https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.queuescheduleroptions.md
Properties
| Property | Type | Description |
|---|---|---|
| autorun? | boolean | (Optional) Condition to start scheduler at instance creation. |
| maxStalledCount? | number | (Optional) Amount of times a job can be recovered from a stalled state to the wait state. If this is exceeded, the job is moved to failed. |
| stalledInterval? | number | (Optional) Number of milliseconds between stallness checks. |
public async init() {
// Limit queue to max 1 jobs per 5 minutes.
this.myQueue = new Queue(queue1, {
redis: redisConfig,
limiter: {
max: 2, //并发数
duration: 3 * 1000
},
defaultJobOptions: {
removeOnComplete: true,
removeOnFail: true
},
settings: {
stalledInterval: 2 * 60 * 1000, // 2 分钟
maxStalledCount: 0
}
});
}方法二：使用worker操作
Stalled - BullMQ
When a job is in an active state, i.e., it is being processed by a worker, it needs to continuously update the queue to notify that the worker is still working on the job. This mechanism prevents a worker that crashes or enters an endless loop from keeping a job in an active state forever.
When a worker is not able to notify the queue that it is still working on a given job, that job is moved back to the waiting list, or to the failed set. We then say that the job has stalled and the queue will emit the 'stalled' event.
There is not a 'stalled' state, only a 'stalled' event emitted when a job is automatically moved from active to waiting state.
In order to avoid stalled jobs, make sure that your worker does not keep Node.js event loop too busy, the default max stalled check duration is 30 seconds, so as long as you do not perform CPU operations exceeding that value you should not get stalled jobs.
Another way to reduce the chance for stalled jobs is using so called "sandboxed" processors. In this case, the workers will spawn new separate Node.js processes, running separately from the main process.
main.ts
import { Worker } from 'bullmq';
const worker = new Worker('Paint', painter);
painter.ts
export default = (job) => {
// Paint something
}