2.FastRunner定时任务Celery+RabbitMQ
注意：celery版本和Python冲突问题
不能用高版本Python 用3.5以下，因为项目的celery用的django-celery == 3.2.2
python3.7 async关键字
- 冲突版本 celery3.x
- 方案一： celery3.x+python3.6
- 方案二 ： celery4.x+python3.7
解决celery执行时报错：KeyError: 'async' 报错原因：Python3.6及以上版本和celebry4.0以上版本中async关键字发生冲突。
原因分析：
1.这是因为在 python 3.7 中将 async 作为了关键字，所以当 py 文件中出现类似 from . import async, base 这类不符合python语法的语句时，Python会报错；
解决方法： 在 celery 官方的提议下，建议将 kombu下的async.py 文件的文件名改成 asynchronous；然后把引用和这个文件的所有文件的里面的async改为asynchronous；
2. 但是再修改后会报错:No module named ‘kombu.asynchronous’,搜索到的方法是回退版本,然后再次升级到最新版本,不再出现报错(有些人进回退版本就好):
pip uninstall celery && pip install celery==3.1.25;
pip install celery;不过对我都没用,因为发现回退版本后会重置async的修改,进行第一步的报错
3. 之后我从kombu下手,删除原有的kombu,并安装最新的版本后,问题解决:
pip uninstall kombu && pip install kombu
django与 python版本
Django 1.11版本兼容Python 2.7、3.4、3.5、3.6、3.7（added in 1.11.17）# 1.11.5 python<=3.6
Django 2.0版本兼容Python 3.4、3.5、3.6、3.7
Django 2.1、2.2版本兼容Python 3.5、3.6、3.7
Django 3.0版本兼容Python 3.6、3.7、3.8
djangorestframework与django python版本
djangorestframework 3.6.4 (python2.7 django1.11.5)
- Python (2.7, 3.2, 3.3, 3.4, 3.5, 3.6) # drf <= 3.8 support python2.7
- Django (1.8, 1.9, 1.10, 1.11)
djangorestframework 3.10.0
- Python (3.5, 3.6, 3.7) # drf >= 3.9 support python3.7
- Django (1.11, 2.0, 2.1, 2.2)
djangorestframework 3.11.2 (python3.7 django1.11.17 drf3.11.2)
- Python (3.5, 3.6, 3.7, 3.8)
- Django (1.11, 2.0, 2.1, 2.2, 3.0) # the last version which support django 1.11
djangorestframework 3.12(latest version 3.12.4 2021-03)
- Python (3.5, 3.6, 3.7, 3.8, 3.9)
- Django (2.2, 3.0, 3.1) # django >= 2.0
1.启动定时任务
1.安装rabbmitMQ中间件，并配置setting中的BROKER_URL（默认一般不用修改）
登陆RabbitMQ后台
- Centos7
# 注意防火墙是否开放端口
# 防火墙相关操作 https://www.jb51.net/article/147012.htm
http://ip:15672
- Windows10
http://localhost:156722. setting.py
djcelery.setup_loader()
CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'Asia/Shanghai'
# BROKER_URL = 'amqp://username:password@IP:5672//'
BROKER_URL = 'amqp://guest:[email protected]:5672//'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_RESULT_EXPIRES = 7200
CELERYD_CONCURRENCY = 1 if DEBUG else 5
CELERYD_MAX_TASKS_PER_CHILD = 403.启动定时服务
1.启动Celery Beat(相当于发送消息给RabbitMQ)
# 使用默认的celery.py启动
python3 manage.py celery beat -l info > ./logs/beat.log 2>&1
# 指定的celery.py启动,其中的 -A是指定文件
celery -A FasterRunner.celery beat -l info > ~/logs/beat.log 2>&12.启动Celery work(处理RabbitMQ中的消息)
# 使用默认的celery.py启动
celery multi start w1 -A FasterRunner -l info --logfile=./logs/worker.log
# 指定的celery.py启动,其中的 -A是指定文件
celery multi start w1 -A FasterRunner.celery -l info --logfile=./logs/worker.logcelery -A FasterRunner worker -l info 启动异步worker
python manage.py celery beat -l info 启动beat监听定时任务
4.调试定时任务
发送消息日志
# 发送消息日志
(fastrunner) [root@izwz9awyk38uq20rb3czmnz FasterRunner]# tailf logs/beat.log
[2019-06-24 18:09:00,000: INFO/MainProcess] Scheduler: Sending due task 每分钟执行一次 (fastrunner.tasks.schedule_debug_suite)
[2019-06-24 18:10:00,000: INFO/MainProcess] Scheduler: Sending due task 每分钟执行一次 (fastrunner.tasks.schedule_debug_suite)处理消息日志
# 处理消息日志
(fastrunner) [root@izwz9awyk38uq20rb3czmnz FasterRunner]# tailf ./logs/worker.log
[2019-06-24 18:09:00,172: WARNING/Worker-5] Ran 1 test in 0.138s
[2019-06-24 18:09:00,173: WARNING/Worker-5] OK
[2019-06-24 18:09:00,173: DEBUG/Worker-5]
================== Variables & Output ==================
Type | Variable : Value
------ | ---------------- : ---------------------------
--------------------------------------------------------
[2019-06-24 18:09:00,199: INFO/MainProcess] Task fastrunner.tasks.schedule_debug_suite[adda0f06-ed65-470c-a7db-7042707f3fd6] succeeded in 0.19341683201491833s: None
[2019-06-24 18:10:00,004: INFO/MainProcess] Received task: fastrunner.tasks.schedule_debug_suite[5eecf0ce-afd5-4b2a-8d60-55a184f093bf]
[2019-06-24 18:10:00,024: WARNING/Worker-1] 111
[2019-06-24 18:10:00,024: WARNING/Worker-1] 111
[2019-06-24 18:10:00,027: INFO/Worker-1] Start to run testcase: 测试用例_登陆
[2019-06-24 18:10:00,028: WARNING/Worker-1] 登陆
[2019-06-24 18:10:00,028: DEBUG/Worker-1] call hook: ${setup_hook_prepare_kwargs($request)}
[2019-06-24 18:10:00,028: INFO/Worker-1] POST /api/user/login/
[2019-06-24 18:10:00,029: DEBUG/Worker-1] request kwargs(raw): {'headers': {'Content-Type': 'application/json'}, 'verify': False, 'json': {'username': 'fastrunner', 'password': 'fastrunner'}}
[2019-06-24 18:10:00,029: DEBUG/Worker-1] processed request:
> POST http://119.23.50.82:8000/api/user/login/
> kwargs: {'headers': {'Content-Type': 'application/json'}, 'verify': False, 'json': {'username': 'fastrunner', 'password': 'fastrunner'}, 'timeout': 120}
[2019-06-24 18:10:00,160: DEBUG/Worker-1]
================== request details ==================
url : 'http://ip:8000/api/user/login/'
method : 'POST'
headers : {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '52'}
start_timestamp : 1561371000.029365
verify : False
json : {'username': 'fastrunner', 'password': 'fastrunner'}
body : b'{"username": "fastrunner", "password": "fastrunner"}'
[2019-06-24 18:10:00,161: DEBUG/Worker-1]
================== response details ==================
status_code : 200
headers : {'Server': 'nginx', 'Date': 'Mon, 24 Jun 2019 10:10:00 GMT', 'Content-Type': 'application/json', 'Content-Length': '115', 'Connection': 'keep-alive', 'Vary': 'Accept, Origin', 'Allow': 'POST, OPTIONS', 'X-Frame-Options': 'SAMEORIGIN'}
content_size : 115
response_time_ms : 131.36
elapsed_ms : 128.769
encoding : None
content : b'{"code":"0001","success":true,"msg":"login success","token":"22b3bf6afd31435e2bdce609713ed7b9","user":"fastrunner"}'
content_type : 'application/json'
ok : True
url : 'http://ip:8000/api/user/login/'
reason : 'OK'
cookies : {}
text : '{"code":"0001","success":true,"msg":"login success","token":"22b3bf6afd31435e2bdce609713ed7b9","user":"fastrunner"}'
json : {'code': '0001', 'success': True, 'msg': 'login success', 'token': '22b3bf6afd31435e2bdce609713ed7b9', 'user': 'fastrunner'}