2014-07-3120:42:26.008 5699 ERROR nova.openstack.common.periodic_task [-] Error duringComputeManager._cleanup_running_deleted_instances: Timeout while waiting on RPCresponse - topic: "conductor", RPC method:"instance_get_all_by_filters" info: "<unknown>"
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task Traceback (mostrecent call last):
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py",line 180, in run_periodic_tasks
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task task(self, context)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line4945, in _cleanup_running_deleted_instances
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task for instance inself._running_deleted_instances(context):
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/compute/manager.py",line 4983, in _running_deleted_instances
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task instances =self._get_instances_on_driver(context, filters)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 518,in _get_instances_on_driver
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task context, filters, columns_to_join=columns_to_join)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 94, ininstance_get_all_by_filters
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task columns_to_join)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/conductor/rpcapi.py", line295, in instance_get_all_by_filters
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task sort_dir=sort_dir,columns_to_join=columns_to_join)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 85, incall
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task return self._invoke(self.proxy.call, ctxt,method, **kwargs)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 63, in_invoke
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task return cast_or_call(ctxt, msg,**self.kwargs)
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py",line 130, in call
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task exc.info, real_topic, msg.get('method'))
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task Timeout: Timeoutwhile waiting on RPC response - topic: "conductor", RPC method:"instance_get_all_by_filters" info: "<unknown>"
2014-07-3120:42:26.008 5699 TRACE nova.openstack.common.periodic_task
2014-07-3120:42:31.271 5699 ERROR root [-] Unexpected exception occurred 61 time(s)...retrying.
2014-07-3120:42:31.271 5699 TRACE root Traceback (most recent call last):
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",line 78, in inner_func
2014-07-3120:42:31.271 5699 TRACE root returninfunc(*args, **kwargs)
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 698, in _consumer_thread
2014-07-3120:42:31.271 5699 TRACE root self.consume()
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 689, in consume
2014-07-3120:42:31.271 5699 TRACE root it.next()
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 606, in iterconsume
2014-07-3120:42:31.271 5699 TRACE root yieldself.ensure(_error_callback, _consume)
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 540, in ensure
2014-07-3120:42:31.271 5699 TRACE root returnmethod(*args, **kwargs)
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py",line 597, in _consume
2014-07-3120:42:31.271 5699 TRACE root nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-07-3120:42:31.271 5699 TRACE root File"<string>", line 6, in next_receiver
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line665, in next_receiver
2014-07-3120:42:31.271 5699 TRACE root ifself._ecwait(lambda: self.incoming, timeout):
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line50, in _ecwait
2014-07-3120:42:31.271 5699 TRACE root result =self._ewait(lambda: self.closed or predicate(), timeout)
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line571, in _ewait
2014-07-3120:42:31.271 5699 TRACE root result =self.connection._ewait(lambda: self.error or predicate(), timeout)
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line214, in _ewait
2014-07-3120:42:31.271 5699 TRACE root self.check_error()
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line207, in check_error
2014-07-31 20:42:31.2715699 TRACE root raise self.error
2014-07-3120:42:31.271 5699 TRACE root InternalError: Traceback (most recent call last):
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line650, in write
2014-07-3120:42:31.271 5699 TRACE root self._op_dec.write(*self._seg_dec.read())
2014-07-3120:42:31.271 5699 TRACE root File"/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, inwrite
2014-07-3120:42:31.271 5699 TRACE root ifself.op.headers is None:
2014-07-3120:42:31.271 5699 TRACE root AttributeError: 'NoneType' object has no attribute'headers'
2014-07-3120:42:31.271 5699 TRACE root
2014-07-3120:42:31.271 5699 TRACE root
2014-07-3120:42:34.763 5699 WARNING nova.openstack.common.loopingcall [-] task runoutlasted interval by 50.015705 sec
上面报错出现的原因:由于时间同步出现问题,需要把openstack的时间和时间服务器同步,重启这计算节点的openstack-nova-compute进程