多python环境中devstack-keystone.service无法启动的一次排查
多python环境中devstack-keystone.service无法启动的一次排查
前言
- 问题描述
- 问题排查
- 问题解决
- 番外
问题描述
devstack部署的流程直接挂了,查看终端上的问题是keystone没有起来
通过查看keystone服务的日志
journalctl -u [email protected]
Jul 07 01:25:05 cicd1 [email protected][27280]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4
Jul 07 01:25:05 cicd1 [email protected][27280]: Python version: 3.5.7 (default, Jul 7 2020, 00:31:52) [GCC 7.5.0]
Jul 07 01:25:05 cicd1 [email protected][27280]: Python main interpreter initialized at 0x11b3060
Jul 07 01:25:05 cicd1 [email protected][27280]: python threads support enabled
Jul 07 01:25:05 cicd1 [email protected][27280]: your server socket listen backlog is limited to 100 connections
Jul 07 01:25:05 cicd1 [email protected][27280]: your mercy for graceful operations on workers is 90 seconds
Jul 07 01:25:05 cicd1 [email protected][27280]: mapped 403077 bytes (393 KB) for 2 cores
Jul 07 01:25:05 cicd1 [email protected][27280]: *** Operational MODE: preforking ***
Jul 07 01:25:05 cicd1 [email protected][27280]: *** uWSGI is running in multiple interpreter mode ***
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI master process (pid: 27280)
Jul 07 01:25:05 cicd1 systemd[1]: Started Devstack [email protected].
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI worker 1 (pid: 27281, cores: 1)
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI worker 2 (pid: 27282, cores: 1)
Jul 07 01:25:05 cicd1 [email protected][27280]: running "unix_signal:15 gracefully_kill_them_all" (master-start)...
Jul 07 01:25:05 cicd1 [email protected][27280]: Traceback (most recent call last):
Jul 07 01:25:05 cicd1 [email protected][27280]: File "/usr/local/bin/keystone-wsgi-public", line 6, in
Jul 07 01:25:05 cicd1 [email protected][27280]: from keystone.server.wsgi import initialize_public_application
Jul 07 01:25:05 cicd1 [email protected][27280]: ImportError: No module named 'keystone'
查看服务的文件, 尝试手动实锤, [email protected]文件如下
root@cicd1:~# cat /etc/systemd/system/[email protected]
[Unit]
Description = Devstack [email protected]
[Service]
RestartForceExitStatus = 100
NotifyAccess = all
Restart = always
KillMode = process
Type = notify
ExecReload = /bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini
User = stack
SyslogIdentifier = [email protected]
[Install]
WantedBy = multi-user.targe
手动执行后依旧是报错
/usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini
Jul 07 01:25:05 cicd1 [email protected][27280]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4
Jul 07 01:25:05 cicd1 [email protected][27280]: Python version: 3.5.7 (default, Jul 7 2020, 00:31:52) [GCC 7.5.0]
Jul 07 01:25:05 cicd1 [email protected][27280]: Python main interpreter initialized at 0x11b3060
Jul 07 01:25:05 cicd1 [email protected][27280]: python threads support enabled
Jul 07 01:25:05 cicd1 [email protected][27280]: your server socket listen backlog is limited to 100 connections
Jul 07 01:25:05 cicd1 [email protected][27280]: your mercy for graceful operations on workers is 90 seconds
Jul 07 01:25:05 cicd1 [email protected][27280]: mapped 403077 bytes (393 KB) for 2 cores
Jul 07 01:25:05 cicd1 [email protected][27280]: *** Operational MODE: preforking ***
Jul 07 01:25:05 cicd1 [email protected][27280]: *** uWSGI is running in multiple interpreter mode ***
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI master process (pid: 27280)
Jul 07 01:25:05 cicd1 systemd[1]: Started Devstack [email protected].
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI worker 1 (pid: 27281, cores: 1)
Jul 07 01:25:05 cicd1 [email protected][27280]: spawned uWSGI worker 2 (pid: 27282, cores: 1)
Jul 07 01:25:05 cicd1 [email protected][27280]: running "unix_signal:15 gracefully_kill_them_all" (master-start)...
Jul 07 01:25:05 cicd1 [email protected][27280]: Traceback (most recent call last):
Jul 07 01:25:05 cicd1 [email protected][27280]: File "/usr/local/bin/keystone-wsgi-public", line 6, in
Jul 07 01:25:05 cicd1 [email protected][27280]: from keystone.server.wsgi import initialize_public_application
Jul 07 01:25:05 cicd1 [email protected][27280]: ImportError: No module named 'keystone'
查看/usr/local/bin/keystone-wsgi-public代码如下,发现文件头上指定了python编译器,但是还是不放心.
#!/usr/local/bin/python3.7
#PBR Generated from 'wsgi_scripts'
import threading
import sys
# 排查问题加的
print("python version")
print(sys.version)
from keystone.server.wsgi import initialize_public_application
if __name__ == "__main__":
import argparse
import socket
import sys
import wsgiref.simple_server as wss
parser = argparse.ArgumentParser(
description=initialize_public_application.__doc__,
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
usage='%(prog)s [-h] [--port PORT] [--host IP] -- [passed options]')
parser.add_argument('--port', '-p', type=int, default=8000,
help='TCP port to listen on')
parser.add_argument('--host', '-b', default='',
help='IP to bind the server to')
parser.add_argument('args',
nargs=argparse.REMAINDER,
metavar='-- [passed options]',
help="'--' is the separator of the arguments used "
"to start the WSGI server and the arguments passed "
"to the WSGI application.")
args = parser.parse_args()
if args.args:
if args.args[0] == '--':
args.args.pop(0)
else:
parser.error("unrecognized arguments: %s" % ' '.join(args.args))
sys.argv[1:] = args.args
server = wss.make_server(args.host, args.port, initialize_public_application())
print("*" * 80)
print("STARTING test server keystone.server.wsgi.initialize_public_application")
url = "http://%s:%d/" % (server.server_name, server.server_port)
print("Available at %s" % url)
print("DANGER! For testing only, do not use in production")
print("*" * 80)
sys.stdout.flush()
server.serve_forever()
else:
application = None
app_lock = threading.Lock()
with app_lock:
if application is None:
application = initialize_public_application()
加了两个print()后查看日志,实锤了果然调错了python解释器.
root@cicd1:~# /usr/local/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini
[uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini
open("./http_plugin.so"): No such file or directory [core/utils.c line 3732]
!!! UNABLE to load uWSGI plugin: ./http_plugin.so: cannot open shared object file: No such file or directory !!!
open("./python3_plugin.so"): No such file or directory [core/utils.c line 3732]
!!! UNABLE to load uWSGI plugin: ./python3_plugin.so: cannot open shared object file: No such file or directory !!!
*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jul 7 02:02:38 2020] ***
compiled with version: 5.4.0 20160609 on 05 July 2020 13:07:29
os: Linux-4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020
nodename: cicd1
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /root
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 63915
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.5.7 (default, Jul 7 2020, 00:31:52) [GCC 7.5.0]
Python main interpreter initialized at 0x225d260
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 90 seconds
mapped 403077 bytes (393 KB) for 2 cores
*** Operational MODE: preforking ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 30233)
spawned uWSGI worker 1 (pid: 30234, cores: 1)
spawned uWSGI worker 2 (pid: 30235, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
python version
3.5.7 (default, Jul 7 2020, 00:31:52)
[GCC 7.5.0]
Traceback (most recent call last):
File "/usr/local/bin/keystone-wsgi-public", line 9, in
from keystone.server.wsgi import initialize_public_application
ImportError: No module named 'keystone'
unable to load app 0 (mountpoint='') (callable not found or import error)
问题解决
rm /usr/local/bin/uwsgi
python3.7 -m pip install uwsgi
番外
之前因为些莫名情况下使得/usr/local/bin/目录下的一些openstack相关的python脚本文件指定成了python3.5
可以用下面的脚本解决下
for file in $(ls -l /usr/local/bin | awk '{print $9}' | awk 'NF > 0'); do
if grep '#!/usr/bin/python3.5' /usr/local/bin/$file; then
sed -i 's+#!/usr/bin/python3.5+#!/usr/bin/python3.7+g' /usr/local/bin/$file
fi
done