关于在openstack执行nova get-vnc-console命令，无法得到vnc url并提示服务器超时的问题描述

情景描述：

在一台openstack的控制节点上执行nova get-vnc-console UUID novnc命令，无法输出带有token的NoVNC url地址，添加debug选项再次运行该命令时，提示“504 Gateway Time-out”：

# nova --debug get-vnc-console 930egk8f-ea94-40df-a4cd-c2540221ce9c novnc
....
send: u'POST /v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930egk8f-ea94-40df-a4cd-c2540221ce9c/action HTTP/1.1\r\nHost: *.*.*.*:58774\r\nContent-Length: 39\r\nx-auth-project-id: ****\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: dd318058a3424ce18be2821c764dfc88\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n'
send: '{"os-getVNCConsole": {"type": "novnc"}}'
reply: ''
connect: (*.*.*.*, 58774) ************
send: u'POST /v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930egk8f-ea94-40df-a4cd-c2540221ce9c/action HTTP/1.1\r\nHost: *.*.*.*:58774\r\nContent-Length: 39\r\nx-auth-project-id: ****\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: dd318058a3424ce18be2821c764dfc88\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n'
send: '{"os-getVNCConsole": {"type": "novnc"}}'
reply: 'HTTP/1.0 504 Gateway Time-out\r\n'
header: Cache-Control: no-cache
header: Connection: close
header: Content-Type: text/html
DEBUG (shell:416) string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 413, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 364, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/shell.py", line 1038, in do_get_vnc_console
    utils.print_list([VNCConsole(data['console'])], ['Type', 'Url'])
TypeError: string indices must be integers, not str
ERROR: string indices must be integers, not str

查询了相关的日志（messages.log，nova的api.log，nova的compute.log），发现如下相关记录：

messages.log：

Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 ERROR nova.rpc.common [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out#0122016-02-16 09:46:56 TRACE nova.rpc.common Traceback (most recent call last):#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure#0122016-02-16 09:46:56 TRACE nova.rpc.common     return method(*args, **kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume#0122016-02-16 09:46:56 TRACE nova.rpc.common     return self.connection.drain_events(timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common     return self.transport.drain_events(self.connection, **kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common     return connection.drain_events(**kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi#0122016-02-16 09:46:56 TRACE nova.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple#0122016-02-16 09:46:56 TRACE nova.rpc.common     channel, method_sig, args, content = read_timeout(timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 ERROR nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Caught error: Timeout while waiting on RPC response.#0122016-02-16 09:46:56 TRACE nova.api.openstack Traceback (most recent call last):#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 82, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack     return req.get_response(self.application)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response#0122016-02-16 09:46:56 TRACE nova.api.openstack     application, catch_exc_info=False)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application#0122016-02-16 09:46:56 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 176, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack     return self.app(env, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack   File "
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500

nova的api.log：

2016-02-16 09:46:56 ERROR nova.rpc.common [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out
2016-02-16 09:46:56 TRACE nova.rpc.common Traceback (most recent call last):
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2016-02-16 09:46:56 TRACE nova.rpc.common     return method(*args, **kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2016-02-16 09:46:56 TRACE nova.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common     return connection.drain_events(**kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi
2016-02-16 09:46:56 TRACE nova.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple
2016-02-16 09:46:56 TRACE nova.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 93, in read_timeout
2016-02-16 09:46:56 TRACE nova.rpc.common     return self.method_reader.read_method()
2016-02-16 09:46:56 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
2016-02-16 09:46:56 TRACE nova.rpc.common     raise m
2016-02-16 09:46:56 TRACE nova.rpc.common timeout: timed out
2016-02-16 09:46:56 TRACE nova.rpc.common

2016-02-16 09:46:56 ERROR nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Caught error: Timeout while waiting on RPC response.
2016-02-16 09:46:56 TRACE nova.api.openstack Traceback (most recent call last):
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 82, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return req.get_response(self.application)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response
2016-02-16 09:46:56 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application
2016-02-16 09:46:56 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 176, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return self.app(env, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     response = self.app(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 147, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 208, in call_func
2016-02-16 09:46:56 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack     content_type, body, accept)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2016-02-16 09:46:56 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2016-02-16 09:46:56 TRACE nova.api.openstack     return method(req=request, **action_args)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/consoles.py", line 54, in get_vnc_console
2016-02-16 09:46:56 TRACE nova.api.openstack     console_type)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 100, in wrapped
2016-02-16 09:46:56 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 1549, in get_vnc_console
2016-02-16 09:46:56 TRACE nova.api.openstack     context, instance, params={"console_type": console_type})
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 151, in _call_compute_message
2016-02-16 09:46:56 TRACE nova.api.openstack     return self._cast_or_call_compute_message(rpc.call, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 143, in _cast_or_call_compute_message
2016-02-16 09:46:56 TRACE nova.api.openstack     return rpc_method(context, queue, kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
2016-02-16 09:46:56 TRACE nova.api.openstack     return _get_impl().call(context, topic, msg, timeout)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 674, in call
2016-02-16 09:46:56 TRACE nova.api.openstack     return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 343, in call
2016-02-16 09:46:56 TRACE nova.api.openstack     rv = list(rv)
2016-02-16 09:46:56 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 304, in __iter__

2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500
2016-02-16 09:47:54 INFO nova.api.openstack.wsgi [req-1dbe7cdc-2d93-4884-b72b-bc0ee7ddc51c 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] POST http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action
2016-02-16 09:47:54 ERROR nova.rpc.common [req-641f851d-a1e9-443b-8b13-58770ec42061 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out
2016-02-16 09:47:54 TRACE nova.rpc.common Traceback (most recent call last):
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2016-02-16 09:47:54 TRACE nova.rpc.common     return method(*args, **kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2016-02-16 09:47:54 TRACE nova.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common     return connection.drain_events(**kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi
2016-02-16 09:47:54 TRACE nova.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple
2016-02-16 09:47:54 TRACE nova.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 93, in read_timeout
2016-02-16 09:47:54 TRACE nova.rpc.common     return self.method_reader.read_method()
2016-02-16 09:47:54 TRACE nova.rpc.common   File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
2016-02-16 09:47:54 TRACE nova.rpc.common     raise m
2016-02-16 09:47:54 TRACE nova.rpc.common timeout: timed out
2016-02-16 09:47:54 TRACE nova.rpc.common

nova的compute.log：

Feb 16 09:45:56 upnode2 2016-02-16 09:45:56 INFO nova.api.openstack.wsgi [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] POST http://*.*.*.*58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action

Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500

 

经查58774端口是haproxy的监听端口，负载的进程为nova-api，也就是说通过HTTP方式将请求发送给nova-api服务后，无响应。

但是在对nova-api服务进行重启后，问题依然存在；nova-compute服务重启后，问题解决，原因不明。

注：可以使用如下命令取得NoVNC url（引用自https://community.rackspace.com/products/f/25/t/5933）：

# curl -H "X-Auth-Token: YOUR_AUTH_TOKEN"  \ 
       -H "Content-type: application/json" \ 
       -XPOST API_ENDPOINT_PUBLICURL/servers/CLOUD_SERVER_UUID/action \ 
       -d '{"os-getVNCConsole": {"type":"novnc"}}' | python -mjson.tool

执行成功，则会返回一条带有token的url，该地址就是要访问虚拟机的NoVNC地址，可以通过浏览器打开。