情景描述:
在一台openstack的控制节点上执行nova get-vnc-console UUID novnc命令,无法输出带有token的NoVNC url地址,添加debug选项再次运行该命令时,提示“504 Gateway Time-out”:
# nova --debug get-vnc-console 930egk8f-ea94-40df-a4cd-c2540221ce9c novnc .... send: u'POST /v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930egk8f-ea94-40df-a4cd-c2540221ce9c/action HTTP/1.1\r\nHost: *.*.*.*:58774\r\nContent-Length: 39\r\nx-auth-project-id: ****\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: dd318058a3424ce18be2821c764dfc88\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n' send: '{"os-getVNCConsole": {"type": "novnc"}}' reply: '' connect: (*.*.*.*, 58774) ************ send: u'POST /v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930egk8f-ea94-40df-a4cd-c2540221ce9c/action HTTP/1.1\r\nHost: *.*.*.*:58774\r\nContent-Length: 39\r\nx-auth-project-id: ****\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: dd318058a3424ce18be2821c764dfc88\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n' send: '{"os-getVNCConsole": {"type": "novnc"}}' reply: 'HTTP/1.0 504 Gateway Time-out\r\n' header: Cache-Control: no-cache header: Connection: close header: Content-Type: text/html DEBUG (shell:416) string indices must be integers, not str Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 413, in main OpenStackComputeShell().main(sys.argv[1:]) File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 364, in main args.func(self.cs, args) File "/usr/lib/python2.6/site-packages/novaclient/v1_1/shell.py", line 1038, in do_get_vnc_console utils.print_list([VNCConsole(data['console'])], ['Type', 'Url']) TypeError: string indices must be integers, not str ERROR: string indices must be integers, not str
查询了相关的日志(messages.log,nova的api.log,nova的compute.log),发现如下相关记录:
messages.log:
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 ERROR nova.rpc.common [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out#0122016-02-16 09:46:56 TRACE nova.rpc.common Traceback (most recent call last):#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure#0122016-02-16 09:46:56 TRACE nova.rpc.common return method(*args, **kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume#0122016-02-16 09:46:56 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common return connection.drain_events(**kwargs)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events#0122016-02-16 09:46:56 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi#0122016-02-16 09:46:56 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple#0122016-02-16 09:46:56 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)#0122016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 ERROR nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Caught error: Timeout while waiting on RPC response.#0122016-02-16 09:46:56 TRACE nova.api.openstack Traceback (most recent call last):#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 82, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack return req.get_response(self.application)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response#0122016-02-16 09:46:56 TRACE nova.api.openstack application, catch_exc_info=False)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application#0122016-02-16 09:46:56 TRACE nova.api.openstack app_iter = application(self.environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 176, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack return self.app(env, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__#0122016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)#0122016-02-16 09:46:56 TRACE nova.api.openstack File "
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500
nova的api.log:
2016-02-16 09:46:56 ERROR nova.rpc.common [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out
2016-02-16 09:46:56 TRACE nova.rpc.common Traceback (most recent call last):
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2016-02-16 09:46:56 TRACE nova.rpc.common return method(*args, **kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2016-02-16 09:46:56 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
2016-02-16 09:46:56 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi
2016-02-16 09:46:56 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple
2016-02-16 09:46:56 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 93, in read_timeout
2016-02-16 09:46:56 TRACE nova.rpc.common return self.method_reader.read_method()
2016-02-16 09:46:56 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
2016-02-16 09:46:56 TRACE nova.rpc.common raise m
2016-02-16 09:46:56 TRACE nova.rpc.common timeout: timed out
2016-02-16 09:46:56 TRACE nova.rpc.common
2016-02-16 09:46:56 ERROR nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Caught error: Timeout while waiting on RPC response.
2016-02-16 09:46:56 TRACE nova.api.openstack Traceback (most recent call last):
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 82, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return req.get_response(self.application)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response
2016-02-16 09:46:56 TRACE nova.api.openstack application, catch_exc_info=False)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application
2016-02-16 09:46:56 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 176, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return self.app(env, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack response = self.app(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 147, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 208, in call_func
2016-02-16 09:46:56 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2016-02-16 09:46:56 TRACE nova.api.openstack content_type, body, accept)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2016-02-16 09:46:56 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2016-02-16 09:46:56 TRACE nova.api.openstack return method(req=request, **action_args)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/consoles.py", line 54, in get_vnc_console
2016-02-16 09:46:56 TRACE nova.api.openstack console_type)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 100, in wrapped
2016-02-16 09:46:56 TRACE nova.api.openstack return func(self, context, target, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 1549, in get_vnc_console
2016-02-16 09:46:56 TRACE nova.api.openstack context, instance, params={"console_type": console_type})
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 151, in _call_compute_message
2016-02-16 09:46:56 TRACE nova.api.openstack return self._cast_or_call_compute_message(rpc.call, *args, **kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 143, in _cast_or_call_compute_message
2016-02-16 09:46:56 TRACE nova.api.openstack return rpc_method(context, queue, kwargs)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/rpc/__init__.py", line 68, in call
2016-02-16 09:46:56 TRACE nova.api.openstack return _get_impl().call(context, topic, msg, timeout)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 674, in call
2016-02-16 09:46:56 TRACE nova.api.openstack return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 343, in call
2016-02-16 09:46:56 TRACE nova.api.openstack rv = list(rv)
2016-02-16 09:46:56 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 304, in __iter__
2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500
2016-02-16 09:47:54 INFO nova.api.openstack.wsgi [req-1dbe7cdc-2d93-4884-b72b-bc0ee7ddc51c 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] POST http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action
2016-02-16 09:47:54 ERROR nova.rpc.common [req-641f851d-a1e9-443b-8b13-58770ec42061 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] Timed out waiting for RPC response: timed out
2016-02-16 09:47:54 TRACE nova.rpc.common Traceback (most recent call last):
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2016-02-16 09:47:54 TRACE nova.rpc.common return method(*args, **kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2016-02-16 09:47:54 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
2016-02-16 09:47:54 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 62, in wait_multi
2016-02-16 09:47:54 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 119, in _wait_multiple
2016-02-16 09:47:54 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 93, in read_timeout
2016-02-16 09:47:54 TRACE nova.rpc.common return self.method_reader.read_method()
2016-02-16 09:47:54 TRACE nova.rpc.common File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
2016-02-16 09:47:54 TRACE nova.rpc.common raise m
2016-02-16 09:47:54 TRACE nova.rpc.common timeout: timed out
2016-02-16 09:47:54 TRACE nova.rpc.common
nova的compute.log:
Feb 16 09:45:56 upnode2 2016-02-16 09:45:56 INFO nova.api.openstack.wsgi [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] POST http://*.*.*.*58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action
Feb 16 09:46:56 upnode2 2016-02-16 09:46:56 INFO nova.api.openstack [req-5865192b-5fd9-44a7-ad11-12f0adadadaf 381b738171bd4c30bb34bb982f78d943 ee6f4449201b4e48b329bea27974f2e7] http://*.*.*.*:58774/v1.1/ee6f4449201b4e48b329bea27974f2e7/servers/930ef48f-ea94-40df-a4cd-c2540221ce9c/action returned with HTTP 500
经查58774端口是haproxy的监听端口,负载的进程为nova-api,也就是说通过HTTP方式将请求发送给nova-api服务后,无响应。
但是在对nova-api服务进行重启后,问题依然存在;nova-compute服务重启后,问题解决,原因不明。
注:可以使用如下命令取得NoVNC url(引用自https://community.rackspace.com/products/f/25/t/5933):
# curl -H "X-Auth-Token: YOUR_AUTH_TOKEN" \ -H "Content-type: application/json" \ -XPOST API_ENDPOINT_PUBLICURL/servers/CLOUD_SERVER_UUID/action \ -d '{"os-getVNCConsole": {"type":"novnc"}}' | python -mjson.tool
执行成功,则会返回一条带有token的url,该地址就是要访问虚拟机的NoVNC地址,可以通过浏览器打开。