Cinder Multiple-backends
Cinder Multiple-backends
2017/11/15
在我们的实际生产环境中，openstack后端存储需要与多种类型的存储对接，如商业的SAN存储和分布式ceph存储。即使是同种类型的存储，也可以做性能区分，分为冷盘和热盘，存储不同类型的数据。本文主要介绍如何实现在ceph存储根据不同磁盘介质类型创建不同的卷类型，并与openstack进行对接。为用户提供不同级别的存储性能服务。
效果展示
先看下配置完成后的效果
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder service-list
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | node07 | nova | enabled | up | 2017-11-15T03:20:09.000000 | - |
| cinder-scheduler | node07 | nova | enabled | up | 2017-11-15T03:20:06.000000 | - |
| cinder-volume | node07@rbd-1 | nova | enabled | up | 2017-11-15T03:20:08.000000 | - |
| cinder-volume | node07@rbd-2 | nova | enabled | down | 2017-11-03T02:31:31.000000 | - |
| cinder-volume | node07@rbd-sata | nova | enabled | up | 2017-11-15T03:20:06.000000 | - |
| cinder-volume | node07@rbd-ssd | nova | enabled | up | 2017-11-15T03:20:07.000000 | - |
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
查看已创建的卷类型
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder type-list
+--------------------------------------+------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| cc790d0a-17d9-480f-8236-863549882fa1 | ssd | | True |
| ccf55837-ed2d-4e9f-84b8-5dadb30ef725 | sata | | True |
+--------------------------------------+------+-------------+-----------+查看volume-type和backend的关系
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder extra-specs-list
+--------------------------------------+------+-------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+------+-------------------------------------+
| cc790d0a-17d9-480f-8236-863549882fa1 | ssd | {'volume_backend_name': 'rbd-ssd'} |
| ccf55837-ed2d-4e9f-84b8-5dadb30ef725 | sata | {'volume_backend_name': 'rbd-sata'} |
+--------------------------------------+------+-------------------------------------+
这里的extra_specs字段就指明了两者的关系。而这里的rbd-ssd就是cinder.conf中volume_backend_name=rbd-ssd.
开始试验
本次试验目的是创建2个不同的卷类型，分别为ssd和sata，也就是用ssd磁盘和sata磁盘创建了2个不同的性能pool。具体如何指定osd创建pool，我在上一篇博客中已经说明，这里不再赘述。下面看具体操作命令:
创建2个volume type：ssd和sata
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder type-create ssd
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder type-create sata指定该volume type的backend为new_lvm_backend
(cinder-volume)[cinder@node07 /var/lib/cinder]$cinder type-key ssd set volume_backend_name=rbd-ssd
(cinder-volume)[cinder@node07 /var/lib/cinder]$cinder type-key sata set volume_backend_name=rbd-sata
创建完之后查看对应关系：
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder extra-specs-list
+--------------------------------------+------+-------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+------+-------------------------------------+
| cc790d0a-17d9-480f-8236-863549882fa1 | ssd | {'volume_backend_name': 'rbd-ssd'} |
| ccf55837-ed2d-4e9f-84b8-5dadb30ef725 | sata | {'volume_backend_name': 'rbd-sata'} |
+--------------------------------------+------+-------------------------------------+在cinder.conf中指定rbd-ssd和rbd-sata的详细信息
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cat /etc/cinder/cinder.conf
[DEFAULT]
debug = False
log_dir = /var/log/kolla/cinder
use_forwarded_for = true
use_stderr = False
enable_v1_api = false
osapi_volume_workers = 5
volume_name_template = volume-%s
glance_api_servers = http://172.21.195.17:9292
glance_num_retries = 1
glance_api_version = 2
os_region_name = RegionOne
enabled_backends = rbd-1,rbd-sata,rbd-ssd ##新增2个backend,默认只有一个rbd-1
osapi_volume_listen = 172.21.195.17
osapi_volume_listen_port = 8776
api_paste_config = /etc/cinder/api-paste.ini
nova_catalog_info = compute:nova:internalURL
auth_strategy = keystone
transport_url = rabbit://openstack:[email protected]:5672
#default_volume_type = rbd-1
[oslo_messaging_notifications]
driver = noop
[database]
connection = mysql+pymysql://cinder:[email protected]:3306/cinder
max_retries = -1
[keystone_authtoken]
auth_uri = http://172.21.195.100:5000
auth_url = http://172.21.195.100:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = lbGlcrSDGzrKYQ0mtWXHlTwkiLMf2sujDuhJK3BJ
memcache_security_strategy = ENCRYPT
memcache_secret_key = pZya97OGQCWM3zlHr9VsdQb7ro98TVCCPu2CTQe1
memcached_servers = 172.21.195.17:11211
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[rbd-1]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = rbd-1
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = 5
rbd_user = cinder
rbd_secret_uuid = 41b0cb1c-1c55-4163-aaed-eb080299d124
report_discard_supported = True
#以下为新增内容
----------
[rbd-ssd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = rbd-ssd
rbd_pool = ssd_pool ##指定该backend使用的pool，这里的ssd_pool是之前创建好的，用的是ssd的osd
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = 5
rbd_user = cinder
rbd_secret_uuid = 41b0cb1c-1c55-4163-aaed-eb080299d124 ##UUID保持一致
report_discard_supported = True
[rbd-sata]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = rbd-sata
rbd_pool = sata_pool ##指定该backend使用的pool，这里的sata_pool是之前创建好的，用的是sata的osd
rbd_secret_uuid = 41b0cb1c-1c55-4163-aaed-eb080299d124 ##UUID保持一致
rbd_user = cinder
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = 5
report_discard_supported = True
----------
[privsep_entrypoint]
helper_command = sudo cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf
修改完之后保存重启cinder-api/cinder-volume/cinder-scheduler服务之后，看到我们一开始看到的内容：
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder service-list
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | node07 | nova | enabled | up | 2017-11-15T03:20:09.000000 | - |
| cinder-scheduler | node07 | nova | enabled | up | 2017-11-15T03:20:06.000000 | - |
| cinder-volume | node07@rbd-1 | nova | enabled | up | 2017-11-15T03:20:08.000000 | - |
| cinder-volume | node07@rbd-2 | nova | enabled | down | 2017-11-03T02:31:31.000000 | - |
| cinder-volume | node07@rbd-sata | nova | enabled | up | 2017-11-15T03:20:06.000000 | - |
| cinder-volume | node07@rbd-ssd | nova | enabled | up | 2017-11-15T03:20:07.000000 | - |
+------------------+-----------------+------+---------+-------+----------------------------+-----------------+
使用新的volume_type创建volume
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder create --name volume001 --volume-type ssd 10
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-15T03:44:34.000000 |
| description | None |
| encrypted | False |
| id | 69465c86-78f4-4b50-87a5-28b44c37b955 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volume001 |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | e2e6c07e82424b5baa5132855141aaaa |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | db19e7d461f0433186547a14357ddbf8 |
| volume_type | ssd |
+--------------------------------+--------------------------------------+
(cinder-volume)[cinder@node07 /var/lib/cinder]$ cinder create --name volume002 --volume-type sata 10
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-11-15T03:45:43.000000 |
| description | None |
| encrypted | False |
| id | 07289a14-c8f1-4eae-8e6f-e6ee9a077b1f |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volume002 |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | e2e6c07e82424b5baa5132855141aaaa |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | db19e7d461f0433186547a14357ddbf8 |
| volume_type | sata |
+--------------------------------+--------------------------------------+登录到Horizon页面中查看