Hard Drive Archive

You do have backups, do you?

Selecting best backend for LXD

How it started

I detected crazy iowait with all read/write operations. Things was reasonable with a few containers, but after a while containers count grow, and even read/write limits in most loaded containers didn’t make any difference. And after I several simple dd read/write tests with sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync in container I was shocked. It was 15.4 MB/s in ZFS separated partition. While on raw xfs filesystem on same SSD I had near 95 MB/s!
Next thing I tested, was LXD on another production machine with plain(directory) backend. And it was only 1-2% difference in speed vs raw filesystem. After that I disabled prefetch for ZFS with echo 1 > /sys/module/zfs/parameters/zfs_prefetch_disable, but it didn’t helped things, and I got 22.2 MB/s on zfs backend. This was the sign to move small server out of ZFS.

Storage backends supported functions from storage-backends.md:

Feature Directory Btrfs LVM ZFS
Optimized image storagenoyesyesyes
Optimized container creationnoyesyesyes
Optimized snapshot creationnoyesyesyes
Optimized image transfernoyesnoyes
Optimized container transfernoyesnoyes
Copy on writenoyesyesyes
Block basednonoyesno
Instant cloningnoyesyesyes
Nesting supportyesyesnono
Restore from older snapshots (not latest)yesyesyesno
Storage quotasnoyesnoyes

Test begins

I tested directory (plain) backend in separated partition on same SSD, mounted under /var/lib/lxd and as whole / partition with XFS and EXT4 filesystems. BTRFS in separated partition and as part of /. LVM with XFS and EXT4 filesystems. I also made tests using different mount options. However I was not able to test LVM backend with storage.lvm_mount_options since is not in repository lxd version, yet.


LXD version: 2.0.4
Kernel: 4.4.0-40-generic
CPU: Intel(R) Atom(TM) CPU  230   @ 1.60GHz

How I tested: sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync 4 times and
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75 2 times, middle values in the table.

Options Raw (host)  
 dd (MB/s)fio (runt, msec)
/, xfs, noatime,nobarrier92156416
/, btrfs, noatime,ssd,nobarrier94116566
/, btrfs, noatime,ssd,nobarrier,nodatacow88151841
/var/lib/lxd, xfs, noatime,nobarrier93154046
/var/lib/lxd, ext4, noatime,nobarrier92168451
/var/lib/lxd, btrfs, noatime,ssd,nobarrier89136560
Options Directory (in container)  
 dd (MB/s)fio (runt, msec)
/, xfs, noatime,nobarrier86164339
/var/lib/lxd, xfs, noatime,nobarrier76170480
/var/lib/lxd, ext4, noatime,nobarrier83172358
Filesystem LVM (in container)  
 dd (MB/s)fio (runt, msec)
Options Btrfs (in container)  
 dd (MB/s)fio (runt, msec)
/, btrfs, noatime,ssd,nobarrier94118584
/var/lib/lxd, btrfs, noatime,ssd,nobarrier87138856

Results and Conclusions

BTRFS shows best values and best performance on SSD. Disabling COW will make things worse for btrfs.
There is some slight difference in read/write performance vs host. Without tweaking LVM backend mount options, ext4 shows better results, then xfs. ZFS performance is awful. There no reason to mess with ZFS, unless you build fat zraid. Directory is good choice for production with less than 5 containers count, or if you don’t need to move and create new containers frequently. If you don’t like BTRFS, LVM is second best choice.

BTRFS Tweaks

btrfstune mostly useful if you need to enable some option, when it was not activated on formating time. Partition needs to be unmounted, to use.
btrfs filesystem balance / for btrfs balance/rebalance, could be useful after enabling skinny-metadata.
btrfs quota enable / to enable quotas, need for quotas in containers.

Written on October 15, 2016

No comments

You today

Comments are closed