为什么openstack中的oslo模块总喜欢发生代码冲突？ (by quqi99)

作者：张华  发表于：2014-02-02
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

(http://blog.csdn.net/quqi99 )

       在我的印象中，每当我急切地想用devstack快速地搭起一个openstack环境去修改一些急迫的issue时，一般就不会很顺利，多半都是python模块发生冲突。
这不，你会发现这两天的devstack（2015年2月2日）100%会运行失败的，不是少这个模块就是缺那个模块。已经在这上面花费两天时间了，定位到应该是这个BP造成的（http://specs.openstack.org/openstack/oslo-specs/specs/kilo/drop-namespace-packages.html）， 它说oslo这个名空间容易发生冲突，它的思想是将“from oslo.foo import bar”换成“from oslo_foo import bar”。但究竟是为什么，它没说清楚。
1, 创建两个实验用的python模块
mkdir -p pkg1/ns/pkg1 pkg2/ns/pkg2
echo "from setuptools import setup, find_packages; setup(name='pkg1', packages=find_packages(), namespace_packages=['ns'])" > pkg1/setup.py
echo "__import__('pkg_resources').declare_namespace(__name__)" > pkg1/ns/__init__.py
touch pkg1/ns/pkg1/__init__.py
echo "from setuptools import setup, find_packages; setup(name='pkg2', packages=find_packages(), namespace_packages=['ns'])" > pkg2/setup.py
echo "__import__('pkg_resources').declare_namespace(__name__)" > pkg2/ns/__init__.py
touch pkg2/ns/pkg2/__init__.py


2, 使用python setup.py install来安装ns/pkg1模块。
cd pkg1
python setup.py install --single-version-externally-managed --record record.txt
cd ..
安装后，在record.txt文件中我们看到这个步骤生成了下列的文件：
/usr/local/lib/python2.7/dist-packages/ns/pkg1/__init__.py
/usr/local/lib/python2.7/dist-packages/ns/pkg1/__init__.pyc
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info/dependency_links.txt
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info/namespace_packages.txt
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info/PKG-INFO
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info/SOURCES.txt
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7.egg-info/top_level.txt
/usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7-nspkg.pth
它生成了pkg1-0.0.0-py2.7-nspkg.pth这个文件，其内容如下， 它创建了ns模块，并将其加入到了__path__内建变量中：
hua@hua-ThinkPad-T440p:/tmp/oslo/pkg2$ cat /usr/local/lib/python2.7/dist-packages/pkg1-0.0.0-py2.7-nspkg.pth
import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('ns',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not ie and sys.modules.setdefault('ns', types.ModuleType('ns'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)


3, 使用python setup.py develop来安装ns/pkg2模块。
cd pkg2
python setup.py develop
安装后，在/usr/local/lib/python2.7/dist-packages/easy-install.pth文件中多了如下一行：
/tmp/oslo/pkg2


4, 由于pkg1-0.0.0-py2.7-nspkg.pth加载ns/pkg1模块创建了ns模块，并将其加入到了__path__内建变量中。这样pkg2/ns/__init__.py文件（__init__.py也是用来创建模块的）中的“__import__('pkg_resources').declare_namespace(__name__)"永远没有机会运行。这样就造成了导入ns/pkg1模块时会成功（python -c 'import ns.pkg1'）， 但是导入ns/pkg2模块时会失败。
hua@hua-ThinkPad-T440p:/tmp/oslo/pkg2$ python -c 'import ns.pkg1'
hua@hua-ThinkPad-T440p:/tmp/oslo/pkg2$ python -c 'import ns.pkg2'
Traceback (most recent call last):
  File "", line 1, in
ImportError: No module named pkg2


5, 将上述的ns/pkg1与ns/pkg2两个模块看成oslo.config及oslo.messaging不就知道为什么oslo里有些子模块不work了。


6, 解决方法一，在pkg1-0.0.0-py2.7-nspkg.pth中添加pkg2/ns/__init__.py中这行“__import__('pkg_resources').declare_namespace(__name__)"添加到末尾，且保证easy-install.pth与pkg1-0.0.0-py2.7-nspkg.pth的文件名是字典有序的。这样“python -c 'import ns.pkg2'”也可以成功执行了。


7, 解决方法二，修改python setup.py develop命令，但它也生成{pkg}-nspkg.pth这样的pth文件，其内容如下, 其中/path/to/sourcecode与easy-install.pth中的重复：
/path/to/sourcecode
import pkg_resources; pkg_resources.declare_namespace('ns')


8, 解决方法三，如这个BP提到的去掉oslo这个名空间，直接将类似于oslo.i18n这样的模块变成oslo_i18n模块。这样，例如，原来的oslo/i18n/log.py被重构为oslo_i18n/log.py，但为了向前兼容，将原来的oslo/i18n/log.py的内容改为"from oslo_i18n.log import *"即可。

我们应该尽量避免使用pip install -e (也就是develop模式）模式。
1, 彻底清除残留oslo代码， sudo rm -rf /usr/local/lib/python2.7/dist-packages/oslo*
2, 删除其他openstack工程的develop模式的安装包，如：sudo rm -rf /usr/local/lib/python2.7/dist-packages/neutron*
3, 如有，请清除sudo rm -rf /usr/local/lib/python2.7/dist-packages/easy-install.pth中的相关项。
4, 删除openstack源代码目录的oslo相关工程，杜绝以develop模式安装。

9, 终极绝招

今天使用devstack仍然有下列错误，费了九牛二虎之力，解决了(原因是什么时候在keystone目录下执行过: python setup.py install命令），终级办法是：

1, 清理干净所有和openstack相关的python模块

sudo rm -rf /usr/local/lib/python2.7/dist-packages/oslo*
sudo rm -rf /usr/local/lib/python2.7/dist-packages/*keystone*
sudo rm -rf /usr/local/lib/python2.7/dist-packages/*glance*
sudo rm -rf/usr/local/lib/python2.7/dist-packages/*swift*
sudo rm -rf /usr/local/lib/python2.7/dist-packages/*cinder*
sudo rm -rf /usr/local/lib/python2.7/dist-packages/*nova*
sudo rm -rf /usr/local/lib/python2.7/dist-packages/*neutron*

2, 也要删除/usr/local/lib/python2.7/dist-package/easy-install.pth中的所有相关的条目。

3, 所有主项目如keystone, glance, swift, cinder, nova, horizon, neutron以develop形式安装（如sudo pip install -e /bak/openstack/keystone）, 其他client工程最好不要以develop形式安装。

hua@hua-ThinkPad-T440p:/bak/openstack/devstack$ /bak/openstack/keystone/bin/keystone-manage db_sync
Traceback (most recent call last):
  File "/bak/openstack/keystone/bin/keystone-manage", line 30, in
    from keystone import cli
  File "/bak/openstack/keystone/keystone/cli.py", line 22, in
    from keystone import assignment
  File "/bak/openstack/keystone/keystone/assignment/__init__.py", line 15, in
    from keystone.assignment import controllers  # noqa
  File "/bak/openstack/keystone/keystone/assignment/controllers.py", line 24, in
    from keystone.assignment import schema
  File "/bak/openstack/keystone/keystone/assignment/schema.py", line 13, in
    from keystone.common import validation
  File "/bak/openstack/keystone/keystone/common/validation/__init__.py", line 16, in
    from keystone.common.validation import validators
  File "/bak/openstack/keystone/keystone/common/validation/validators.py", line 16, in
    from keystone import exception
  File "/bak/openstack/keystone/keystone/exception.py", line 18, in
    from keystone.common import config
  File "/bak/openstack/keystone/keystone/common/config.py", line 16, in
    from oslo import messaging
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/__init__.py", line 18, in
    from .exceptions import *
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/exceptions.py", line 13, in
    from oslo_messaging.exceptions import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/__init__.py", line 18, in
    from .notify import *
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/__init__.py", line 23, in
    from .notifier import *
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/notifier.py", line 32, in
    help='Driver or drivers to handle sending notifications.'),
  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1039, in __init__
    item_type= types.MultiString(),
AttributeError: 'module' object has no attribute 'MultiString'

20150203继续遇到坑

昨天按上述方法已经将AttributeError: 'module' object has no attribute 'MultiString'这个错误解决了（昨天是由于不知什么在keystone目录下执行sudo python setup.py install命令，这样使用了keystone的旧文件造成的）。但今天 直接在Bash里运行devstack都没问题，但只要一在eclipse用pydev运行时就报上面一模一样的错，奇了怪了，将Bash运行与eclipse运行的sys.path打出来都是一样的。最后我将/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py的334行由from oslo.config import types改成from oslo_config import types后问题消失。如果有问题，应该在Bash里也有问题啊，到现在百思不得其解，有知道的同学告诉我，谢谢。

接着，发现无论是bash启动的-agent还是eclipse启动的-agent都无法创建qrouter- 名空间，用eclipse调试发现卡在一个queue那里了，想起我改过monkey_patch，还原，正常了。

再接着，就是HeartbeatTimeout: heartbeat timeout这种错，或者下列这种错，ok，既然看到了timeout字眼，一下子就想到了去修改/etc/neutron/neutron.conf中的 qpid_heartbeat=1800，非常顺利，就是这个问题。

下面这个错，是得把monkey patch 放在oslo.messaging前面，所以可以把monkey patch移到__init__中解决。

https://bugs.launchpad.net/oslo.messaging/+bug/1288878

2015-02-03 22:59:02.932 INFO neutron.agent.l3.agent [req-54853ae1-1be4-46eb-ad1c-e32ad44dcf8f None None] L3 agent started
2015-02-03 23:00:06.015 INFO oslo_messaging._drivers.impl_qpid [req-54853ae1-1be4-46eb-ad1c-e32ad44dcf8f None None] Connected to AMQP server on 172.16.1.1:5672
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/bak/openstack/neutron/neutron/agent/l3/agent.py", line 1128, in _process_router_update
    self._process_router_if_compatible(router)
  File "/bak/openstack/neutron/neutron/agent/l3/agent.py", line 1082, in _process_router_if_compatible
    self._process_added_router(router)
  File "/bak/openstack/neutron/neutron/agent/l3/agent.py", line 1093, in _process_added_router
    adv_svc.AdvancedService.after_router_added, ri)
  File "/bak/openstack/neutron/neutron/agent/l3/event_observers.py", line 35, in notify
    getattr(observer, method_name)(*args, **kwargs)
  File "/bak/openstack/neutron-aas/neutron_aas/services//_service.py", line 63, in after_router_added
    device.sync(self.context, [ri.router])
  File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
    return f(*args, **kwargs)
  File "/bak/openstack/neutron-aas/neutron_aas/services//device_drivers/ipsec.py", line 686, in sync
    context, self.host)
  File "/bak/openstack/neutron-aas/neutron_aas/services//device_drivers/ipsec.py", line 475, in get__services_on_host
    return cctxt.call(context, 'get__services_on_host', host=host)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
    retry=self.retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
    timeout=timeout, retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 417, in send
    retry=retry)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 408, in _send
    raise result
AttributeError: _oslo_messaging_localcontext_8b5daa55496f41b7b9ec24a2109969fe
Traceback (most recent call last):


  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
    executor_callback))


  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 188, in _dispatch
    localcontext.clear_local_context()


  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/localcontext.py", line 55, in clear_local_context
    delattr(_STORE, _KEY)


AttributeError: _oslo_messaging_localcontext_8b5daa55496f41b7b9ec24a2109969fe

2014-02-26， 经过近一个月的痛苦摸索，上面的问题（ 在bash里pycharm里运行代码没问题，但是eclipse里运行有问题）终于搞定。原来是多了个目录 /usr/lib/python2.7/dist-packages/oslo/ (它应该是在/usr/local/lib/python2.7目录），而eclipse的模块加载目录将/usr/local/lib/python2.7的顺序放到了/usr/lib/python2.7之前。