Cinder中的iscsitarget和tgt的使用
OpenStack中的Cinder-volume组件的iSCSI target volume可以通过iscsitarget或者tgt提供支撑的，通常情况下使用iscsitarget，但是有的时候会出现一些问题，比较常见的就是cinder-volume无法启动、或者无法创建volume，而解决问题的时候甚至都搞不清楚自己用的到底是tgt还是iscsitarget，因为两个软件都安装了，本文就给大家大概的介绍下这两个软件的安装和出现问题时的解决办法。
环境说明：Ubuntu 12.04+OpenStack Grizzly
组件说明：tgt和iscsitarget使用不同的管理工具，tgt使用的是tgtadm，iscsitarget使用的是ietadm，体现在步骤5中的iscsi_helper属性。
ietadm： iSCSI Enterprise Target Administration Utility ---------- iscsitarget
tgtadm： Linux SCSI Target Administration Utility ---------- tgt
安装步骤
1、安装软件
apt-get install -y cinder-api cinder-scheduler cinder-volume
说明：使用默认的tgt，只需要安装以上软件即可，并且跳过步骤3，直接进入步骤4。
apt-get install -y cinder-api cinder-scheduler cinder-volume iscsitarget open-iscsi iscsitarget-dkms说明：使用iscsitarget，软件列表中虽然没有tgt组件，但是会通过依赖自己安装上。
2、创建数据库
CREATE DATABASE cinder; GRANT ALL ON cinder.* TO 'cinderUser'@'%' IDENTIFIED BY 'cinderPass';
3、tgt和iscsitarget都是使用3260端口，两个服务不能共存，启动的时候tgt在iscsitarget之前启动，抢先占用3260端口，所以如果使用iscsitarget前需要停止tgt服务
service tgt stop为了防止操作系统重启后tgt占用iscsitarget端口，可以将tgt的启动文件删除或者修改名称
mv /etc/init/tgt.conf /etc/init/tgt.conf.disable配置iscsitarget可用
sed -i 's/false/true/g' /etc/default/iscsitarget
重启相关iSCSI服务
service iscsitarget restart service open-iscsi restart
注意：如果使用默认的tgt，则跳过步骤3。
4、修改/etc/cinder/api-paste.ini
[filter:authtoken] paste.filter_factory = keystone.middleware.auth_token:filter_factory service_protocol = http service_host = 10.68.19.61 service_port = 5000 auth_host = 10.68.20.61 auth_port = 35357 auth_protocol = http admin_tenant_name = service admin_user = cinder admin_password = service_pass signing_dir = /var/lib/cinder5、修改/etc/cinder/cinder.conf
[DEFAULT] rootwrap_config=/etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:[email protected]/cinder api_paste_config = /etc/cinder/api-paste.ini iscsi_helper=ietadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone iscsi_ip_address=10.68.20.61若是使用tgt，内容为：
[DEFAULT] rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:[email protected]/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes
6、数据库同步
cinder-manage db sync7、创建VG
pvcreate -ff /dev/sdb vgcreate cinder-volumes /dev/sdb8、重启所有的cinder服务
cd /etc/init.d/; for i in $( ls cinder-* ); do sudo service $i restart; done
常见问题
问题一、操作系统重启或者iscsitarget、tgt服务重启之后，cinder-volume服务无法启动，什么原因造成的，如何解决？原因分析：出现该问题的是由于iscsitarget和tgt发生冲突，通常是启动的服务与cinder-volume中配置使用的管理工具不一致造成，两个组件使用同一个端口，系统重启后tgt服务会先于iscsitarget启动，占用了3260端口导致iscsitarget启动失败，而此时若/etc/cinder/cinder.conf文件中的iscsi_helper属性为ietadm时，cinder-volume服务就无法启动，同样，若/etc/cinder/cinder.conf文件中的iscsi_helper属性为tgtadm，而启动的服务是iscsitarget，cinder-volume服务也无法启动。
解决办法：
1、使用lsof命令查看侦听3260端口的服务到底是iscsitarget还是tgt：lsof -i:3260
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ietd 13025 root 7u IPv4 18939 0t0 TCP *:3260 (LISTEN) ietd 13025 root 8u IPv6 18940 0t0 TCP *:3260 (LISTEN)ietd进程表示当前服务是iscsitarget，对应的iscsi_helper属性应该为ietadm。
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tgtd 14564 root 4u IPv4 21457 0t0 TCP *:3260 (LISTEN) tgtd 14564 root 5u IPv6 21458 0t0 TCP *:3260 (LISTEN) tgtd 14565 root 4u IPv4 21457 0t0 TCP *:3260 (LISTEN) tgtd 14565 root 5u IPv6 21458 0t0 TCP *:3260 (LISTEN)tgtd进程表示当前服务是tgt，对应的iscsi_helper属性应该为tgtadm。
2、修改/etc/cinder/cinder.conf文件配置
ietd--iscsitarget--ietadm方案配置内容：
[DEFAULT] rootwrap_config=/etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:[email protected]/cinder api_paste_config = /etc/cinder/api-paste.ini iscsi_helper=ietadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone iscsi_ip_address=10.68.20.61tgtd--tgt--tgtadm方案配置内容：
rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:[email protected]/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes3、重启cinder-volume服务
service cinder-volume restart
问题二、系统重启或者iscsitarget服务重启的之后，创建第X个volume的时候失败，继续创建却没有问题，错误日志如下，是什么原因造成，如何解决？
2014-01-02 16:33:19 ERROR [cinder.volume.manager] volume volume-08cf713a-0d62-44c9-8bc0-8367783102f2: create failed
2014-01-02 16:33:19 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
rval = self.proxy.dispatch(ctxt, version, method, **args)
File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 288, in create_volume
LOG.error(_("volume %s: create failed"), volume_ref['name'])
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 281, in create_volume
model_update = self.driver.create_export(context, volume_ref)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 496, in create_export
chap_auth)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 229, in create_iscsi_target
self._new_target(name, tid, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 288, in _new_target
**kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 73, in _run
self._execute(self._cmd, *args, run_as_root=True, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 190, in execute
cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf ietadm --op new --tid=5 --params Name=iqn.2010-10.org.openstack:volume-08cf713a-0d62-44c9-8bc0-8367783102f2
Exit code: 239
Stdout: ''
Stderr: 'File exists.\n'原因分析：iscsitarget服务重启的时候，会将target挂载重新进行排序并且会恢复部分已经删除的挂载信息，服务重启之前的挂载信息为：
root@controller:/var/log/cinder# cat /proc/net/iet/volume
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
lun:0 state:0 iotype:fileio iomode:wt blocks:8388608 blocksize:512 path:/dev/cinder-volumes/volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
root@controller:/var/log/cinder# cat /proc/net/iet/session
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
sid:1125899977949696 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
sid:844424950907392 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
sid:281475031040512 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:nonetid=3的挂载盘已经删除，但是此时重新启动iscsitarget服务之后，会发现，ietadm会将原来删除掉的tid=3的挂载信息恢复，只不过不是活动状态，信息如下：
root@controller:/var/log/cinder# cat /proc/net/iet/volume
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
lun:0 state:0 iotype:fileio iomode:wt blocks:8388608 blocksize:512 path:/dev/cinder-volumes/volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
tid:3 name:iqn.2010-10.org.openstack:volume-42da15af-f951-4dec-bf54-3fe6a6240aac
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-42da15af-f951-4dec-bf54-3fe6a6240aac
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
root@controller:/var/log/cinder# cat /proc/net/iet/session
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
sid:844425001239040 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:3 name:iqn.2010-10.org.openstack:volume-847be27e-7ed7-4fd2-bc5e-716c20051965
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
sid:1125899927618048 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
sid:281475031040512 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none此时，cinder数据库的iscsi_targets表中的id3的target的volume_id已经更新为null，再执行创建volume的时候，会执行命令：
sudo cinder-rootwrap /etc/cinder/rootwrap.conf ietadm --op new --tid=3 --params Name=iqn.2010-10.org.openstack:volume-08cf713a-0d62-44c9-8bc0-8367783102f2
从而发生错误提示“File exists.”，这个tid=3的值是取自iscsi_targets表volume_id为null的主键最小值。
解决办法：
方法一、删除没有活动的挂载
ietadm --op delete --tid=3
说明：有的时候还会出现tid挂载错位，即ietadm中显示挂载了volume-xxx的tid为5，但是在cinder数据库的iscsi_targets表中却是6，这个时候需要手动进行调整
方法二、使用tgt替换iscsitarget
1、停止iscsitarget服务
service iscsitarget stop2、启动tgt服务
service tgt start注意：若tgt无法正常启动，错误提示：
Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service tgt stop initctl: Unknown job: tgt Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the stop(8) utility, e.g. stop tgt检查tgt的启动文件是否正常/etc/init/tgt.conf，本文的安装过程中就曾将该文件改名防止其启动。
3、修改/etc/cinder/cinder.conf内容
rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:[email protected]/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes