openstack victoria 对接华为存储OceanStor 5110 v5（Kunpeng）

1. 环境

本文主要侧重于victoria 版本对接华为存储OceanStor 5110 v5（Kunpeng）的步骤记录和问题记录，云平台的部署等相关问题本文不做介绍

HOST OS：Ubuntu 20.04.3 LTS
OpenStack：victoria （kolla），详情如下：

kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "victoria"
openstack_tag: "victoria"

存储：华为存储OceanStor 5110 v5（Kunpeng），万兆交换机
网络：具备管理、业务、存储等隔离网络条件

---

2. 华为存储配置

华为存储上需要做三项配置：

• 导入LICNESE文件，以支持cinder的对接
• 创建资源资源池，用于OpenStack cinder对接，其中主机组、LUN组、主机、映射等均不用做
• 配置管理地址和业务网络地址，分别用于API和业务使用

---

3. cinder配置

OpenStack的部署我们这里略过，直接介绍cinder配置相关信息，由于我们这块采用了kolla部署的openstack云平台，所以所有服务均运行于容器内，配置文件也通过映射模式映射到容器内的，配置文件（/etc/kolla/目录下）修改会有些差异，可根据各自的环境做适当调整。

github有华为openstack_cinder_driver配置文档：https://github.com/Huawei/OpenStack_Driver/tree/50ff55d2cad7d70bb4d26f5975326c2061556c41/ConfigDoc/zh
对接不同OpenStack版本的驱动代码：https://github.com/Huawei/OpenStack_Driver/tree/master/Cinder
cinder配置部分均在cinder-volume节点操作。

3.1 驱动下载导入

通过git clone https://github.com/Huawei/OpenStack_Driver.git 或者直接网页下载后解压，我们采用了当时最新的OpenStack_Driver-2.3.RC3

docker cp OpenStack_Driver-2.3.RC3 cinder_volume:/tmp
docker exec -it -u 0 cinder_volume bash
cd /tmp/OpenStack_Driver-2.3.RC3/Cinder/Victoria
mv * /var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/

3.2 cinder_huawei_conf.xml

创建如下文件，并编辑如下

vim /etc/kolla/cinder-volume/cinder_huawei_conf.xml

  
    V5
    iSCSI
    https://x.x.x.x:8088/deviceManager/rest/
    admin
    ******
  
  
    Thin
    4
    xxx
  
  
          
  

根据我们所采用的产品、协议和账号密码等填写Product、Protocol、RestURL，Username、Userpassword、StoragePool等信息，这些信息均在第二章所配置的。Userpassword项填写admin密码，RestURL项中的x.x.x.x填写管理地址，StoragePool项填写需要对接的资源池，TargetIP项填写业务地址。

3.3 config.json

添加cinder_huawei_conf.xml文件映射
vim /etc/kolla/cinder-volume/config.json

		...
        {
            "source": "/var/lib/kolla/config_files/cinder_huawei_conf.xml",
            "dest": "/etc/cinder/cinder_huawei_conf.xml",
            "owner": "cinder",
            "perm": "0700"
        },
        ...

3.4 cinder.conf

采用iSCSI协议，enabled_backends以逗号分割，可以接多个后端，我们仅用一个，如下：
vim /etc/kolla/cinder-volume/cinder.conf

[DEFAULT]
enabled_backends = huawei_backend
[huawei_backend]
volume_driver = cinder.volume.drivers.huawei.huawei_driver.HuaweiISCSIDriver
cinder_huawei_conf_file = /etc/cinder/cinder_huawei_conf.xml
volume_backend_name = huawei_backend
use_multipath_for_image_xfer = true
enforce_multipath_for_image_xfer = true

做完以上配置，重启服务

docker restart cinder_volume

---

4. multipath配置

存储（cinder-volume）和计算（nova-compute）节点都需要配置
针对存储配置多路径
vim /etc/kolla/multipathd/multipath.conf

defaults {
   user_friendly_names yes
   find_multipaths yes
}

blacklist {
  devnode "^sda$"
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^sd[b-d]"
  devnode "^nbd*"
}

devices {
  device {
    vendor                            "HUAWEI"
    product                           "XSG1"
    path_grouping_policy              "group_by_prio"
    path_checker                      tur
    prio                              alua
    detect_checker                    no
    failback                          immediate
    path_selector                     "round-robin 0"
    dev_loss_tmo                      30
    fast_io_fail_tmo                  5
  }
}

---

5. 计算节点

配置nova-compute支持多路径及cpu_mode为透传
vim /etc/kolla/nova-compute/nova.conf

[libvirt]
volume_use_multipath = True
cpu_mode=host-passthrough

docker restart nova_compute

6. OpenStack使用后端存储

创建卷类型并关联后端如下：

cinder type-create huawei_backend
cinder type-key huawei_backend set volume_backend_name="huawei_backend"

通过cinder type-list获取存储后端信息，通过glance image-list获取镜像信息等：

cinder type-list

+--------------------------------------+----------------+---------------------+-----------+
| ID                                   | Name           | Description         | Is_Public |
+--------------------------------------+----------------+---------------------+-----------+
| 81a35e1a-70d5-4d19-9873-7c9a40cd7cda | huawei_backend | -                   | True      |
+--------------------------------------+----------------+---------------------+-----------+

创建卷（启动卷含image id，亦可创建空卷）选择对应的后端：

cinder create --volume-type 81a35e1a-70d5-4d19-9873-7c9a40cd7cda--image-id 678f7dab-2c3b-44d3-9f9e-b758ddc1cbe8 --name volume-with-huawei 200

---

7. 问题记录

• 对接华为存储LICENSE invalid问题
  参看cinder-volume.log日志详情如下：

2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server     raise exception.VolumeBackendAPIException(data=msg)
2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create lun {'NAME': '7636c1b0-3c7e9475b344d8138b0476', 'PARENTID': '3', 'DESCRIPTION': 'volume-7636c1b0-dcfa-4703-aaab-a806a3bd2ae7', 'ALLOCTYPE': '1', 'CAPACITY': 20971520, 'ENABLESMARTDEDUP': False, 'ENABLECOMPRESSION': False, 'MIRRORPOLICY': '1'} error.
2022-04-10 10:53:06.985 103 ERROR oslo_messaging.rpc.server result: {'data': {}, 'error': {'code': 1077951771, 'description': 'The license format is incorrect or the feature included in the license file is invalid.', 'suggestion': 'Download a valid license file. If the problem persists, contact technical support engineers.'}}.

解决办法：找华为存储厂商，要LICENSE，然后通过存储页面导入LICENSE文件

• 存储挂载Volume device not found at及重复initiator问题
  nova-compute日志详情如下：

2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/utils.py", line 68, in _wrapper
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     return r.call(f, *args, **kwargs)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     do = self.iter(retry_state=retry_state)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 368, in iter
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     raise retry_exc.reraise()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 186, in reraise
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     raise self.last_attempt.result()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     return self.__get_result()
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     raise self._exception
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     result = fn(*args, **kwargs)
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 784, in _connect_multipath_volume
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]     raise exception.VolumeDeviceNotFound(device='')
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f] os_brick.exception.VolumeDeviceNotFound: Volume device not found at .
2022-04-12 11:42:40.852 7 ERROR nova.compute.manager [instance: 0f48b1a2-6365-4e90-abc9-9549bcb1956f]
2022-04-12 11:42:40.892 7 ERROR nova.volume.cinder [req-8036223c-164a-4bb4-9655-7889218a6567 5375c23f2bf445aa87b0518d3db820df 3e3013c5092b475980fd4e0fb9015bd7 - default default] Delete attachment failed for attachment dde284a2-1e4e-43c0-9610-6d9abea24fcb. Error: Volume attachment could not be found with filter: attachment_id = dde284a2-1e4e-43c0-9610-6d9abea24fcb. (HTTP 404) (Request-ID: req-0cbfdf63-5749-4302-a512-2aaf85c0002e) Code: 404: cinderclient.exceptions.NotFound: Volume attachment could not be found with filter: attachment_id = dde284a2-1e4e-43c0-9610-6d9abea24fcb. (HTTP 404) (Request-ID: req-0cbfdf63-5749-4302-a512-2aaf85c0002e)
2022-04-12 11:42:40.964 7 ERROR oslo_messaging.rpc.server [req-8036223c-164a-4bb4-9655-7889218a6567 5375c23f2bf445aa87b0518d3db820df 3e3013c5092b475980fd4e0fb9015bd7 - default default] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .

cinder-volume.log日志详情如下：

2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     conn = self.initialize_connection(volume, properties)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "", line 2, in initialize_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/coordination.py", line 151, in _synchronized
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     return f(*a, **k)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_driver.py", line 70, in initialize_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     self.configuration)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_flow.py", line 2475, in initialize_iscsi_connection
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     engine.run()
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     for _state in self.run_iter(timeout=timeout):
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     failure.Failure.reraise_if_any(er_failures)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     failures[0].reraise()
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     six.reraise(*self._exc_info)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     raise value
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     result = task.execute(**arguments)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/huawei_flow.py", line 1444, in execute
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     initiator, host_id, alua_info)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/rest_client.py", line 613, in associate_iscsi_initiator_to_host
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     initiator, host_id)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils   File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/huawei/rest_client.py", line 93, in _assert_result
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils     raise exception.VolumeBackendAPIException(data=msg)
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Add initiator iqn.1994-05.com.redhat:92a5f47e3aa5 to host 25 error.
2022-04-12 16:40:21.634 104 ERROR cinder.volume.volume_utils result: {'data': {}, 'error': {'code': 1077950338, 'description': 'The initiator has been added to another host.', 'suggestion': 'Specify a valid initiator and add it to the host.'}}.

解决办法：通过查看存储发现，挂载时候有两个启动器，且一个离线一个在线，绑定到离线的主机上了。而且由于是容器化部署，经过摸排发现是iscsid和nova_compute容器里面InitiatorName不一致导致的，每个计算节点又是同一个InitiatorName，故每个节点InitiatorName需要重新生成，且保证iscsid和nova_compute里面InitiatorName一致。
具体操作如下：

docker exec -it -u 0 iscsid bash
echo "InitiatorName=`/sbin/iscsi-iname`" > /etc/iscsi/initiatorname.iscsi
exit

docker cp iscsid:/etc/iscsi/initiatorname.iscsi /opt
docker cp /opt/initiatorname.iscsi nova_compute:/etc/iscsi/initiatorname.iscsi
docker restart iscsid nova_compute

• 基于镜像创建卷signature_verified问题
  执行从镜像创建卷，镜像拷贝一段时间后报错：Failed to copy metadata
  to volume: Glance metadata cannot be updated, key signature_verified exists for
  volume id b75f957c-0ba4-4099-8706-866c68012779，最终导致创建卷失败。
  解决办法：

vim /etc/kolla/cinder-volume/cinder.conf
[DEFAULT]
verify_glance_signatures = disabled

docker restart cinder_volume

• 调整实例大小报危险，发生错误稍后重试
  查看horizon日志/var/lib/docker/volumes/kolla_logs/_data/horizon/horizon.log如下：
  解决办法：
  https://review.opendev.org/c/openstack/horizon/+/829686/

• 虚拟机热迁移same host问题
  nova-compute日志报错：

ERROR nova.virt.libvirt.driver [req-acade39d-5762-4ba6-809b-883d55323aff - - - - -] [instance: ae3d9d67-1520-4ebd-b1e3-891a7a53a291] Live Migration failure: internal error: Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009: libvirt.libvirtError: internal error: Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009

解决办法：
各计算节点通过uuidgen生成UUID
vim /etc/kolla/nova-libvirt/libvirtd.conf

host_uuid  = "c536f595-8eb8-4a72-a0da-46ceca198576"

docker restart nova_compute nova_libvirt

---

8. 参考文献

https://github.com/Huawei/OpenStack_Driver/tree/master/Cinder
https://github.com/Huawei/OpenStack_Driver/tree/50ff55d2cad7d70bb4d26f5975326c2061556c41/ConfigDoc/zh
https://docs.openstack.org/cinder/queens/configuration/block-storage/drivers/huawei-storage-driver.html
https://blog.csdn.net/weixin_34111819/article/details/91730809