raw是默认的格式,格式简单,容易转换为其他的格式。需要文件系统的支持才能支持sparse file
创建image
# qemu-img create -f raw flat.img 10G
Formatting 'flat.img', fmt=raw size=10737418240
如果我们ls则看到
ls -lh flat.img
-rw-r--r-- 1 root root 10G Jun 30 22:27 flat.img
但是并不真正占用10G
# du -h flat.img
0 flat.img
# qemu-img info flat.img
image: flat.img
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: 0
我们可以用dd来制作sparse file
下面的命令制作的是非sparse的文件
# dd if=/dev/zero of=flat1.img bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.66135 s, 1.6 GB/s
block size是1024k,共1000个block
# qemu-img info flat1.img
image: flat1.img
file format: raw
virtual size: 1.0G (1048576000 bytes)
disk size: 1.0G
下面的命令制作的是sparse的文件
# dd if=/dev/zero of=flat2.img bs=1024k count=0 seek=2048
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000141061 s, 0.0 kB/s
# qemu-img info flat2.img
image: flat2.img
file format: raw
virtual size: 2.0G (2147483648 bytes)
disk size: 0
seek的意思是将文件的结尾设在那个地方。
如何copy一个sparse文件呢
# dd if=/dev/zero of=flat3.img bs=1024k count=1 seek=2048
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0010721 s, 978 MB/s
# qemu-img info flat3.img
image: flat3.img
file format: raw
virtual size: 2.0G (2148532224 bytes)
disk size: 1.0M
# cp --sparse=never flat3.img flat3-never.img
这样拷贝会占用整个2G空间
# qemu-img info flat3-never.img
image: flat3-never.img
file format: raw
virtual size: 2.0G (2148532224 bytes)
disk size: 2.0G
# cp --sparse=always flat3.img flat3-always.img
这样拷贝原来写入的1M的0也会被去掉
# qemu-img info flat3-always.img
image: flat3-always.img
file format: raw
virtual size: 2.0G (2148532224 bytes)
disk size: 0
# cp --sparse=always flat3-never.img flat3-never-always.img
这样拷贝原来的2G都会被清理掉
# qemu-img info flat3-never-always.img
image: flat3-never-always.img
file format: raw
virtual size: 2.0G (2148532224 bytes)
disk size: 0
qcow2是动态的,相对于raw来说,有下列的好处:
qcow2的格式如下:
qcow2的数据是存储在data clusters里面的,每个cluster是512 byte sector
为了能够管理这些cluster,qcow2保存了两层的Table,L1 table指向L2 Table,L2 Table管理data cluster.
所以在image里面的offset会被解析成三部分,L1 Table Pointer先找L1,L1 Table Pointer+ offset[0]是L1中的一个entry,读出来便是L2 Table Pointer, L2 Table Pointer + offset[1]是L2中的一个entry,读出来便是data cluster pointer, data cluster pointer +offset[3]便是数据所在的位置。
每一个data cluster都有RefCount,因为可能有多个snapshot都引用每一个数据块
这是backing file的用处,一个qcow2的image可以保存另一个disk image的改变,而不影响另一个image
root:/var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7# qemu-img info disk
image: disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 450M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/45a5886083c924b1fbe412b2ad92f6ce47381610
Format specific information:
compat: 1.1
lazy refcounts: false
root:/var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7# qemu-img info /var/lib/nova/instances/_base/45a5886083c924b1fbe412b2ad92f6ce47381610
image: /var/lib/nova/instances/_base/45a5886083c924b1fbe412b2ad92f6ce47381610
file format: raw
virtual size: 5.0G (5368709120 bytes)
disk size: 1.4G
上面的输出表示有一个raw的image是/var/lib/nova/instances/_base/45a5886083c924b1fbe412b2ad92f6ce47381610,我们基于他生成一个虚拟机,但是不想改变这个image的内容,所以我们基于它生成了qcow2的image :/var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7/disk。
一开始新的image是空的,读取的内容都从老的image里面读取。
当一个data cluster被写入,发生改变的时候,在新的image里面创建一个新的data cluster,这就是copy on write的意义。
snapshot其实也是copy on write的一种应用,但是两者有微妙的不同。
广义来讲,有两种snapshot,一中时internal snapshot,一种是external snapshot
internal snapshot是qcow2中的snapshot table所维护的snapshot,所有的snapshot都是在同一个文件中维护。
在monitor中,savevm, loadvm都是针对internal snapshot来操作的
我们创建一个虚拟机
# qemu-system-x86_64 -enable-kvm -name ubuntutest -m 2048 -hda ubuntutest.img -hdb ubuntutest1.img -boot menu=on -vnc :19 -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no
当我们在monitor中执行savevm后,ubuntutest.img和ubuntutest1.img都会在image内部打一个snapshot
$ qemu-img info ubuntutest.img
image: ubuntutest.img
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 2.1G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 snapshot1 390M 2014-06-28 04:04:42 01:00:59.268
2 vm-20140630230902 239M 2014-06-30 23:09:02 00:04:07.783
Format specific information:
compat: 1.1
lazy refcounts: false
$ qemu-img info ubuntutest1.img
image: ubuntutest1.img
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 134M
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
2 vm-20140630230902 0 2014-06-30 23:09:02 00:04:07.783
Format specific information:
compat: 1.1
lazy refcounts: false
然而在monitor中info block的时候,还是指向的这两个文件。
其中的原理是,当打一个snapshot后,会在snapshot table中建立一项,但是起初是空的,包含L1 table的一个复制,当L2 table或者data cluster改变的时候,则会将原来的数据复制一份,由snapshot的L1 table来维护,而原来的data cluster已经改变,在原地。
external snapshot则往往采用上面的copy on write的方法,当打snapshot的时候,将当前的image不再改变,创建一个新的image,以原来的image作为backing file,然后虚拟机使用新的image。
在monitor中,snapshot_blkdev ide0-hd1 ubuntutest-snapshot.img qcow2,就是采用的这种方式,当这个命令执行完毕后,在monitor中info block
而且我们发现,在HOST机器上多了一个文件
$ qemu-img info ubuntutest-snapshot.img
image: ubuntutest-snapshot.img
file format: qcow2
virtual size: 4.0G (4294967296 bytes)
disk size: 964K
cluster_size: 65536
backing file: ubuntutest1.img
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
backing file可以是raw,也可以是qcow2,但是一旦打了snapshot,新的格式就是qcow2了。
两者很相似,稍微的不同是:
virsh有命令支持snapshot
对已internal snapshot
root:/etc/neutron# virsh snapshot-list instance-0000000a
Name Creation Time State
------------------------------------------------------------
root:/etc/neutron# virsh snapshot-create instance-0000000a
Domain snapshot 1404158715 created
root:/etc/neutron# virsh snapshot-list instance-0000000a
Name Creation Time State
------------------------------------------------------------
1404158715 2014-07-01 04:05:15 +0800 running
root# ps aux | grep instance-0000000a
libvirt+ 9057 0.0 1.2 6846900 821876 ? Sl Jun26 5:34 qemu-system-x86_64 -enable-kvm -name instance-0000000a -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 5d6cb926-5237-4b73-a7b4-e9af851023e7 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2014.1,serial=80590690-87d2-e311-b1b0-a0481cabdfb4,uuid=5d6cb926-5237-4b73-a7b4-e9af851023e7 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000000a.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:fe:2a:41,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:8 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
root 17709 0.0 0.0 10468 924 pts/1 S+ 04:06 0:00 grep --color=auto instance-0000000a
root# qemu-img info /var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7/disk
image: /var/lib/nova/instances/5d6cb926-5237-4b73-a7b4-e9af851023e7/disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 1.1G
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/45a5886083c924b1fbe412b2ad92f6ce47381610
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 1404158715 725M 2014-07-01 04:05:15 117:35:52.160
Format specific information:
compat: 1.1
lazy refcounts: false
对于external snapshot
# virsh snapshot-create-as Instance01 snap1-Instance01 "Instance01 snapshot description" --diskspec vda,file=/home/cliu8/images/instance01-snapshot.img --disk-only --atomic
error: internal error: unable to execute QEMU command 'transaction': Could not open '/home/cliu8/images/instance01.img': Could not open '/home/cliu8/images/instance01.img': Permission denied: Permission denied
从网上查,是因为下面的问题:
-------------------------------------------------------------------------------------
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1004606
When you attempt to create a virtual machine snapshot onto a separate disk, this fails with an error message. The error message is different depending on whether you use snapshot-create or snapshot-create-as but both approaches fail.
The intention is to be able to take a backup of a live VM by freezing the running image file and applying snapshot deltas to a separate file so the backup can save the filesystem image without it being modified during the backup. (i.e. standard practice in VM world)
Format being used is QCOW2 for the disk file which supports snapshots. Manually using qemu-img to create a snapshot on the file does work without any problems.
If you use snapshot-create-as :
snapshot-create-as winxppro3 snapname --disk-only --diskspec hda,snapshot=external,file=/srv/virtual-machines/tester/snap01.qcow2
error: internal error unable to execute QEMU command 'blockdev-snapshot-sync': An undefined error has ocurred
Or if you use snapshot-create and an XML file specification :
virsh # snapshot-create winxppro3 snapspec.xml
error: argument unsupported: unable to handle disk requests in snapshot
snapspec.xml is :
PC specification is :
etc. etc.
The path, excluding extension, is the same as the original image.
Unfortunately the extension can be chosen by the user:
"Name
The name for this snapshot. If the name is specified when initially creating the snapshot, then the snapshot will have that particular name. If the name is omitted when initially creating the snapshot, then libvirt will make up a name for the snapshot, based on the time when it was created."
(from http://libvirt.org/formatsnapshot.html) with the name parameter)
Assuming my_dom disk image is stored under /nfs/diskimages/my_dom.img, a command like the following:
# snapshot-create-as my_dom my_snap --disk-only
will create /nfs/diskimages/my_dom.my_snap changing domain definition XML to use this file instead of /nfs/diskimages/my_dom.img (/nfs/diskimages/my_dom.img will be a backing file for /nfs/diskimages/my_dom.my_snap)
However I fear it's not that simple because even if I try to use a snapshot name like mysnap.img, snaphot still fails because the original image name is removed from the apparmour profile dinamically created/maintained by libvirt under /etc/apparmor.d/libvirt. The original filename is replaced with the new image name.
So, to sum up, I think the following might be needed in order to make disk-only snapshot work
1) virt-aa-helper (/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper) should be able to read virtual machine image files even if the extension isn't imq/qcow2/...
2) dynamically created profiles for libvirt (/etc/apparmor.d/libvirt/libvirt-XXXX.files) should retain the old image filename
Please, be aware that after the "snapshot-create" command fails, the corresponding profile under /etc/apparmor.d/libvirt/ isn't coherent anymore with the real filename for virtual images.
I found a that looks better than deactivating apparmor.
I found here ( http://libvirt.org/drvqemu.html#securitysvirtaa ) the information that Apparmor is „just“ used for protecting the vm host and that there is a TEMPLATE under /etc/apparmor.d/libvirt/ that can be modified. In that TEMPLATE I included one of my own rules and under /etc/apparmor.d/local/usr.sbin.libvirtd i added a similar rule.
Yes Theodor, please share your apparmor-rules :)
For the moment i am following this guide to disable just the libvirt-apparmor-profile:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-security-policies "Configure Apparmor (Ubuntu)"
$ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
$ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
Stop running virtual machines.
$ sudo service apparmor restart
$ sudo apparmor_status
Start machines again.
In the /etc/apparmor.d/local/usr.sbin.libvirtd file I just created one rule to give libvirtd read'n'write access to the images in my storage pool with the following line:
"/var/lib/libvirt/images/*.img" rw,
As preliminary: I have created my own naming convention for my overlays, these are used for incremental backups to another server. This convention can be found in my abstractation and has to be adjusted to your own needs.
First of all I've created my own abstraction as /etc/apparmor.d/local/abstraction-libvirt-storage. This file gives the clients access to the important images like that:
"/var/lib/libvirt/images/*.base.img" rw,
"/var/lib/libvirt/images/*.base.img" rw,
"/var/lib/libvirt/images/*.stable_overlay.img" rw,
"/var/lib/libvirt/images/*.running.img" rw,
The /etc/apparmor.d/libvirt/TEMPLATE file is a source for all rule files in /etc/apparmor.d/libvirt/. There you need to source the abstraction-libvirt-storage so the TEMPLATE looks similar to this one (adjust to your own needs):
profile LIBVIRT_TEMPLATE {
#include
#include
}
It is also possible to put the information of the abstraction-libvirt-storage file directly into the TEMPLATE but a change on some of the rules would require to edit multiple files ( /etc/apparmor.d/libvirt/*)
I hope this will help. This adjustments should be fine for safety requirement, because the host should still be secured against guests and thats the only thing you can do with libvirt+apparmor.
http://libvirt.org/drvqemu.html#securitysvirtaa
Configure Apparmor (Ubuntu)
Check to see whether AppArmor is installed on your machine. If not, you can skip this section.
In Ubuntu AppArmor is installed and enabled by default. You can verify this with:
$ dpkg --list 'apparmor'
Disable the AppArmor profiles for libvirt
$ ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
$ ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
$ apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
$ apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1096125
Hello,
i have a plain installation of Ubuntu 12.10 server,
i installed package ubuntu-virt-server and added the user to the kvm and libvirtd group. (qemu-kvm 1.2 libvirt 0.9.13)
After that i can run kvm virtual machines.
Now i am trying to get a snapshot of the disk (first step to achieve live backup of virtual machines)
virsh snapshot-create-as [vmname] snapshot1 "snapshot" --disk-only --atomic
but i am getting the following error:
error: internal error unable to execute QEMU command 'transaction': Could not open '/var/lib/libvirt/images/ubu1204.snapshot1'
(i even tryed with permission 777 )
Hi Serge,
i made a snapshot after disabling apparmor and it works!
Here are the commands used:
/etc/init.d/apparmor stop
update-rc.d -f apparmor remove
reboot now
virsh snapshot-create-as ubu-server-64-10.04.3 snap1 --disk-only
So i changed the target package of the bug to apparmor
Thank you again
affects:
kvm (Ubuntu) → apparmor (Ubuntu)
----------------------------------------------------------------------------
我们可以在/etc/libvirt/qemu.conf将apparmor去掉
security_driver = "none"
重启service libvirt-bin restart
重启虚拟机
virsh destroy Instance02
virsh start Instance02
# virsh snapshot-create-as Instance02 snap1-Instance02 "Instance02 snapshot description" --diskspec vda,file=/home/cliu8/images/instance02-snapshot.img --disk-only --atomic
Domain snapshot snap1-Instance02 created
# qemu-img info instance02-snapshot.img
image: instance02-snapshot.img
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 2.2M
cluster_size: 65536
backing file: /home/cliu8/images/instance02.img
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
在创建一个
# virsh snapshot-create-as Instance02 snap2-Instance02 "Instance02 snapshot description" --diskspec vda,file=/home/cliu8/images/instance02-snapshot2.img --disk-only --atomic
Domain snapshot snap2-Instance02 created
# virsh snapshot-list Instance02 --tree
snap1-Instance02
|
+- snap2-Instance02
root:/home/cliu8/images# qemu-img info instance02-snapshot.img
image: instance02-snapshot.img
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 2.2M
cluster_size: 65536
backing file: /home/cliu8/images/instance02.img
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
root:/home/cliu8/images# qemu-img info instance02-snapshot2.img
image: instance02-snapshot2.img
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536
backing file: /home/cliu8/images/instance02-snapshot.img
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
看了我们需要单独的一节来阐述snapshot和AppArmor security deriver了
image格式之间可以转换
raw可以转换为qcow2
root:/home/cliu8/images# qemu-img info flat1.img
image: flat1.img
file format: raw
virtual size: 1.0G (1048576000 bytes)
disk size: 1.0G
root:/home/cliu8/images# qemu-img convert -f raw -O qcow2 flat1.img flat1.qcow2
root:/home/cliu8/images# qemu-img info flat1.qcow2
image: flat1.qcow2
file format: qcow2
virtual size: 1.0G (1048576000 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
有时候qcow2也可以转换为qcow2,转换的过程中,没用的data cluster就被去掉了
root:/home/cliu8/images# qemu-img info ubuntutest.img
image: ubuntutest.img
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 2.1G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 snapshot1 390M 2014-06-28 04:04:42 01:00:59.268
2 vm-20140630230902 239M 2014-06-30 23:09:02 00:04:07.783
Format specific information:
compat: 1.1
lazy refcounts: false
root:/home/cliu8/images# qemu-img convert -f qcow2 -O qcow2 ubuntutest.img ubuntutest.qcow2
root:/home/cliu8/images# qemu-img info ubuntutest.qcow2
image: ubuntutest.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 1.4G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
还可以压缩
root:/home/cliu8/images# qemu-img convert -c -f qcow2 -O qcow2 ubuntutest.img ubuntutest-compress.qcow2
root:/home/cliu8/images# qemu-img info ubuntutest-compress.qcow2
image: ubuntutest-compress.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 465M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
还可以加密
root:/home/cliu8/images# qemu-img convert -o encryption -f qcow2 -O qcow2 ubuntutest.img ubuntutest-encrypt.qcow2
Disk image 'ubuntutest-encrypt.qcow2' is encrypted.
password:
root:/home/cliu8/images# qemu-img info ubuntutest-encrypt.qcow2
image: ubuntutest-encrypt.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 1.4G
encrypted: yes
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
我们运行一个虚拟机
# qemu-system-x86_64 -enable-kvm -name ubuntutest -m 2048 -hda ubuntutest-encrypt.qcow2 -boot menu=on -vnc :19 -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=n
发现一开始虚拟机并不启动
进入monitor输入用户名密码,方才启动
也可以在monitor中修改密码
block_passwd ide0-hd0 newpassword
image可以扩大,首先必须instance在shutdown的情况下,另外是必须raw format才能enlarge,最后扩大的空间既不会被partition,也不会被format
# qemu-img resize ubuntutest.img +10G
qemu-img: Can't resize an image which has snapshots
qemu-img: This image does not support resize
root:/home/cliu8/images# qemu-img resize ubuntu-14.04.qcow2 +10G
Image resized.
root:/home/cliu8/images# qemu-img info ubuntu-14.04.qcow2
image: ubuntu-14.04.qcow2
file format: qcow2
virtual size: 15G (16106127360 bytes)
disk size: 461M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
看了qcow2如果没有snapshot也能enlarge