Cinder LVM backend cinder-volume service down

目录

问题

[stack@manager ~]$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | hostgroup               | nova | enabled | up    | 2018-12-26T13:32:26.000000 | -               |
| cinder-volume    | hostgroup@tripleo_iscsi | nova | enabled | down  | 2018-12-23T04:23:33.000000 | -               |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+

volume.log

ERROR cinder.service [-] Manager for service cinder-volume hostgroup@tripleo_iscsi is reporting problems, not sending heartbeat. Service will appear "down".

cinder.conf

[tripleo_iscsi]
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
volumes_dir=/var/lib/cinder/volumes
volume_group = cinder-volumes
iscsi_protocol=iscsi
iscsi_ip_address=172.18.22.74
volume_backend_name=tripleo_iscsi
iscsi_helper=lioadm

调查

2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager [req-d703742c-2989-47af-9b15-3e927c0e09d6 - - - - -] Failed to initialize driver.
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager Traceback (most recent call last):
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 423, in init_host
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     self.driver.check_for_setup_error()
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 301, in check_for_setup_error
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     self.configuration.lvm_suppress_fd_warnings))
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 105, in __init__
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     if self._vg_exists() is False:
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 138, in _vg_exists
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     run_as_root=True)
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     result = self.__execute(*args, **kwargs)
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     return processutils.execute(*cmd, **kwargs)
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 394, in execute
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager     cmd=sanitized_cmd)
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --noheadings -o name cinder-volumes
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager Exit code: 5
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager Stdout: u''
2018-12-26 13:33:03.104 163092 ERROR cinder.volume.manager Stderr: u'File descriptor 10 (/dev/urandom) leaked on vgs invocation. Parent PID 163094: /usr/bin/python2\n  Volume group "cinder-volumes" not found\n  Cannot process volume group cinder-volumes\n'

定位到 Cannot process volume group cinder-volumes。

解决

[root@overcloud-controller-0 ~]# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda2      xfs       447G   47G  400G  11% /
devtmpfs       devtmpfs   63G     0   63G   0% /dev
tmpfs          tmpfs      63G  103M   63G   1% /dev/shm
tmpfs          tmpfs      63G  2.1M   63G   1% /run
tmpfs          tmpfs      63G     0   63G   0% /sys/fs/cgroup
tmpfs          tmpfs      13G     0   13G   0% /run/user/979
tmpfs          tmpfs      13G     0   13G   0% /run/user/977
tmpfs          tmpfs      13G     0   13G   0% /run/user/0
tmpfs          tmpfs      13G     0   13G   0% /run/user/1000

[root@overcloud-controller-0 ~]# dd if=/dev/zero of=/cinder/volumegroup/disk.img count=102400 bs=1MB
102400+0 records in
102400+0 records out
102400000000 bytes (102 GB) copied, 362.07 s, 283 MB/s

[root@overcloud-controller-0 ~]# ll -h /cinder/volumegroup/disk.img
-rw-r--r--. 1 root root 96G Dec 26 13:42 /cinder/volumegroup/disk.img

[root@overcloud-controller-0 ~]# losetup /dev/loop0 /cinder/volumegroup/disk.img
[root@overcloud-controller-0 ~]# pvcreate /dev/loop0
  Physical volume "/dev/loop0" successfully created.
[root@overcloud-controller-0 ~]# vgcreate cinder-volumes /dev/loop0
  Volume group "cinder-volumes" successfully created
[root@overcloud-controller-0 ~]# vgs
  VG             #PV #LV #SN Attr   VSize  VFree
  cinder-volumes   1   0   0 wz--n- 95.36g 95.36g

[root@overcloud-controller-0 ~]# systemctl restart openstack-cinder-volume

需要注意的是，当服务器重启后依旧会出现同样的问题，这是因为使用文件模拟 VG，服务器重启后需要重新 Mount，可以通过下述方式来解决：

$ cat /etc/rc.local 
losetup /dev/loop0 /cinder/volumegroup/disk.img