OpenStack CEPH Liberty 统一存储 bug解决
1， 问题：
glance 镜像 raw 格式，启动云主机报错
2015-12-13 11:53:27.597 11654 INFO nova.virt.libvirt.driver [req-db7b8ac1-8218-4ab0-b769-9c4d4379843b e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Creating image 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [req-db7b8ac1-8218-4ab0-b769-9c4d4379843b e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Instance failed to spawn 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Traceback (most recent call last): 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in _build_resources 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] yield resources 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] block_device_info=block_device_info) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2437, in spawn 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] admin_pass=admin_password) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2861, in _create_image 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] instance, size, fallback_from_host) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6282, in _try_fetch_image_cache 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] size=size) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] *args, **kwargs) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 777, in create_image 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] prepare_template(target=base, max_size=size, *args, **kwargs) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] return f(*args, **kwargs) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 230, in fetch_func_sync 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] fetch_func(target=target, *args, **kwargs) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2853, in clone_fallback_to_fetch 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] backend.clone(context, disk_images['image_id']) 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 808, in clone 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] if self.driver.is_cloneable(location, image_meta): 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 193, in is_cloneable 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] if image_meta.disk_format != 'raw': 2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] AttributeError: 'dict' object has no attribute 'disk_format'
2，问题分析
1，查找问题定位：/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py
import ipdb;ipdb.set_trace()
if image_meta.disk_format != 'raw':
reason = ("rbd image clone requires image format to be "
"'raw' but image {0} is '{1}'").format(
url, image_meta.disk_format)
LOG.debug(reason)
return False
2, ipdb 启动服务追踪
[root@AthComputeLiberty10325 ~]# ipdb /usr/bin/nova-compute --config-file=/etc/nova/nova.conf
3，发现蛛丝马迹
AttributeError: "'dict' object has no attribute 'disk_format'"
> /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py(194)is_cloneable()
193
--> 194 if image_meta.disk_format != 'raw':
195 reason = ("rbd image clone requires image format to be "
ipdb> args
self = <nova.virt.libvirt.storage.rbd_utils.RBDDriver object at 0x5f41850>
image_location = {'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}
image_meta = {'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta.disk_format
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> image_meta.disk_format()
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> image_meta.disk_format.values()
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> dir(image_meta.disk_format)
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> args
self = <nova.virt.libvirt.storage.rbd_utils.RBDDriver object at 0x5f41850>
image_location = {'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}
image_meta = {'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta
{'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> type(image_meta)
<type 'dict'>
ipdb> image_meta
{'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta['disk_format']
u'raw'
3，解决问题bug:
修改前：
if image_meta.disk_format != 'raw':
修改后：
if image_meta['disk_format'] != 'raw':
179
180 def is_cloneable(self, image_location, image_meta):
181 import ipdb;ipdb.set_trace()
182 url = image_location['url']
183 try:
184 fsid, pool, image, snapshot = self.parse_url(url)
185 except exception.ImageUnacceptable as e:
186 LOG.debug('not cloneable: %s', e)
187 return False
188
189 if self._get_fsid() != fsid:
190 reason = '%s is in a different ceph cluster' % url
191 LOG.debug(reason)
192 return False
193
194 # Add william , image_meta.disk_format != 'raw'
195 # so ...
196 if image_meta['disk_format'] != 'raw':
197 reason = ("rbd image clone requires image format to be "
198 "'raw' but image {0} is '{1}'").format(
199 url, image_meta.disk_format)
200 LOG.debug(reason)
201 return False
4, 完成操作
2015-12-13 13:18:36.659 19827 INFO nova.virt.libvirt.firewall [req-7d3b097b-701d-4352-a8cc-87bc6fb3f9f1 e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Called setup_basic_filtering in nwfilter 2015-12-13 13:18:36.660 19827 INFO nova.virt.libvirt.firewall [req-7d3b097b-701d-4352-a8cc-87bc6fb3f9f1 e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Ensuring static filters 2015-12-13 13:18:37.418 19827 INFO nova.compute.manager [-] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] VM Started (Lifecycle Event) 2015-12-13 13:18:37.441 19827 INFO nova.virt.libvirt.driver [-] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Instance spawned successfully. 2015-12-13 13:18:37.525 19827 INFO nova.compute.manager [req-a92e6077-014c-421c-9fcc-9d21e6f7d420 - - - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] During sync_power_state the instance has a pending task (spawning). Skip.