Timeout while waiting on RPC response - topic: "conductor", RPC method: "instance_get

2014-07-3120:42:26.008 5699 ERROR nova.openstack.common.periodic_task [-] Error duringComputeManager._cleanup_running_deleted_instances: Timeout while waiting on RPCresponse - topic: "conductor", RPC method:"instance_get_all_by_filters" info: ""

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task Traceback (mostrecent call last):

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py",line 180, in run_periodic_tasks

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     task(self, context)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line4945, in _cleanup_running_deleted_instances

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     for instance inself._running_deleted_instances(context):

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py",line 4983, in _running_deleted_instances

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     instances =self._get_instances_on_driver(context, filters)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 518,in _get_instances_on_driver

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     context, filters, columns_to_join=columns_to_join)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 94, ininstance_get_all_by_filters

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     columns_to_join)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/conductor/rpcapi.py", line295, in instance_get_all_by_filters

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     sort_dir=sort_dir,columns_to_join=columns_to_join)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 85, incall

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     return self._invoke(self.proxy.call, ctxt,method, **kwargs)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 63, in_invoke

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     return cast_or_call(ctxt, msg,**self.kwargs)

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py",line 130, in call

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task     exc.info, real_topic, msg.get('method'))

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task Timeout: Timeoutwhile waiting on RPC response - topic: "conductor", RPC method:"instance_get_all_by_filters" info: ""

2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task

2014-07-3120:42:31.271 5699 ERROR root [-] Unexpected exception occurred 61 time(s)...retrying.

2014-07-3120:42:31.271 5699 TRACE root Traceback (most recent call last):

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",line 78, in inner_func

2014-07-3120:42:31.271 5699 TRACE root     returninfunc(*args, **kwargs)

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 698, in _consumer_thread

2014-07-3120:42:31.271 5699 TRACE root    self.consume()

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 689, in consume

2014-07-3120:42:31.271 5699 TRACE root    it.next()

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 606, in iterconsume

2014-07-3120:42:31.271 5699 TRACE root     yieldself.ensure(_error_callback, _consume)

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 540, in ensure

2014-07-3120:42:31.271 5699 TRACE root     returnmethod(*args, **kwargs)

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 597, in _consume

2014-07-3120:42:31.271 5699 TRACE root    nxt_receiver = self.session.next_receiver(timeout=timeout)

2014-07-3120:42:31.271 5699 TRACE root   File"", line 6, in next_receiver

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line665, in next_receiver

2014-07-3120:42:31.271 5699 TRACE root     ifself._ecwait(lambda: self.incoming, timeout):

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line50, in _ecwait

2014-07-3120:42:31.271 5699 TRACE root     result =self._ewait(lambda: self.closed or predicate(), timeout)

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line571, in _ewait

2014-07-3120:42:31.271 5699 TRACE root     result =self.connection._ewait(lambda: self.error or predicate(), timeout)

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line214, in _ewait

2014-07-3120:42:31.271 5699 TRACE root    self.check_error()

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line207, in check_error

2014-07-31 20:42:31.2715699 TRACE root     raise self.error

2014-07-3120:42:31.271 5699 TRACE root InternalError: Traceback (most recent call last):

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line650, in write

2014-07-3120:42:31.271 5699 TRACE root    self._op_dec.write(*self._seg_dec.read())

2014-07-3120:42:31.271 5699 TRACE root   File"/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, inwrite

2014-07-3120:42:31.271 5699 TRACE root     ifself.op.headers is None:

2014-07-3120:42:31.271 5699 TRACE root AttributeError: 'NoneType' object has no attribute'headers'

2014-07-3120:42:31.271 5699 TRACE root

2014-07-3120:42:31.271 5699 TRACE root

2014-07-3120:42:34.763 5699 WARNING nova.openstack.common.loopingcall [-] task runoutlasted interval by 50.015705 sec

上面报错出现的原因：由于时间同步出现问题，需要把openstack的时间和时间服务器同步，重启这计算节点的openstack-nova-compute进程