Sunday, April 22, 2007

Linux LVM2 and md arrays

For some time now, I've been running a debian server with several disks in a RAID5 md array with LVM2 layered over the top. At the time this seemed like the usual path that people were taking; ie. Use the md tools to create your redundant drive array, and use LVM2 over the top to get a flexible volume management system. Coming from a Veritas Volume Manager background, this always seemed a bit dodgy (Veritas manages the redundancy and the logical volumes). However, this setup has proved reliable and straight forward to work on. Admitedly I haven't had a drive failure, but I've never had a situation where the LVM2 or md information needs some form of repair.

I still don't mirror my root filesystem ... but thats a personal choice.

At the time I setup this disk array, there were some notes indicating that the Linux device-mapper stuff (which effectively implements the core of LVM2) would implement mirroring. It looks like that is the case now, though it's not exactly mentioned in many places on the net. The main resource seems to be redhat who I think maintain the LVM2 stuff.

Anyway, you can now do mirroring without an underlying md array. It's really quite simple. If you have another drive you need to pvcreate at least a partition on it and then vgextend and finally lvconvert to do the actual mirroring. Here's an example from an LVM'd debian setup

pvcreate /dev/hdd1
vgextend debian /dev/hdd1
lvconvert --corelog -m1 /dev/debian/usr

Not too sure what the corelog does. I kept on getting an insufficient extents error until I added it. You can do a 'lvs' to see the progress of the mirror. It works pretty well. Like most LVM operations you don't need to umount anything while they're running.

But you still need to use md if you want RAID5. I don't think the device mapper implements RAID5 yet. One thing that is good about current (post kernel 2.6.16) md setups is that you can add a disk to a RAID5 array and dynamically expand it (google for mdadm raid5 reshaping). If you understand how RAID5 works, you'll realise that adding a disk is a complicated process involving restriping data across all the disks in the array. When I set up my array this reshaping feature was not available so I deliberately split each drive up into about 5 or 6 partitions to create multiple RAID5 instances .... which suffice to say allows me to add another drive and reshape it myself (it's a long process, but doable)