Xen disk images vs LVM

In setting up a Xen virtualized web server with about 15 GB of static files I questioned the wisdom of using a disk image as apposed to LVM. I read a bunch of forums about which is better and it seems the disk images are easier to manage because you can just copy to make backups etc.. LVM you just can't move it and pop it up whereever you want but you have more flexibility in resizeing and backing them up within the host OS. However what I didn't find on the forums is performance tests. People theororized about the speed difference but it seems nobodies sat down and done some tests. It's my goal that after summer vacation to put together a performance whitepaper comparing all the virtual machine solutions. For now I'll just put up xen disk images against LVM.

So I created a 10 GB empty disk image using dd and I created a 10 GB LVM Logical volume and assigned both of them to the domU.

name = "centtest"
memory = "512"
disk = [ 'tap:aio:/srv/xen/centos5.img,xvda,w', 'tap:aio:/srv/xen/centtest.img,xvdb,w', 'phy:/dev/vgsys/lvtest,xvdc,w' ]
vif = [ 'bridge=xenbr0', ]
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot = 'restart'
on_crash = 'restart'

Test 1 - disk image

The disk image /dev/xvdb1 was mounted as /media/test1. The images itself was stored on a 500 GB SATA drive.

/dev/xvdb1            10317828    154236   9639476   2% /media/test1

[root@localhost ~]# bonnie++ -s 2048 -d /media/test1 -u root

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 2G 6065 15 70595 37 20649 2 26878 53 57270 0 135.8 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17008 99 +++++ +++ 26079 90 16002 94 +++++ +++ 29094 99
localhost.localdomain,2G,6065,15,70595,37,20649,2,26878,53,57270,0,135.8,0,16,17008,99,+++++,+++,26079,90,16002,94,+++++,+++,29094,99

Test 2 - LVM

The LVM Logical Volume was created on the same 500 GB SATA drive and mounted in the domU as /media/test2

/dev/xvdc             10321208    154236   9642684   2% /media/test2

[root@localhost ~]# bonnie++ -s 2048 -d /media/test2 -u root

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost.locald 2G 34825 92 50898 26 24833 4 35920 69 77594 1 137.9 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 15556 92 +++++ +++ 26198 96 16281 98 +++++ +++ 28508 100
localhost.localdomain,2G,34825,92,50898,26,24833,4,35920,69,77594,1,137.9,0,16,15556,92,+++++,+++,26198,96,16281,98,+++++,+++,28508,100
INIT: version 2.86 reloading

LVM soundly trounced the disk image in Sequential Output - Per Chr by about 600% and bettered it by about 20-30% in the other tests. The on interesting one though was Sequential Output - Block where the disk image pulled ahead of LVM by about 20%. This I can't account for and will need to retest because it doesn't make a lot of sense. All of the Sequential Create/Delete stats were within 10% of each other.

Test 3 - Dom0 - LVM

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
vs.soundlinu 10048M 38679 94 74047 42 30272 6 40389 84 75538 2 138.7 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 13678 88 +++++ +++ 28937 97 12704 84 +++++ +++ 26902 94
vs.soundlinuxtraining.com,10048M,38679,94,74047,42,30272,6,40389,84,75538,2,138.7,0,16,13678,88,+++++,+++,28937,97,12704,84,+++++,+++,26902,94

The last test was done in Dom0 and isn't an exact duplicate of the other tests because I have 5 GB of ram available and needed to bump the file size to get the test to run. In the future I'll have a more apples to apples comparison. What is interesting is that the DomU LVM is within 15-20% of the Dom0 LVM performance most of the time and only the very strange results for domU Sequential Output - Block falling further behind.

I think that I need to do more testing but I was surprised that a disk image did as well as it did.