Comments on: When the links go bouncing…

By: brerrabbit

brerrabbit — Tue, 13 Jul 2010 21:12:48 +0000

Just so you know, the effective limit on the number of hosts in an ESX cluster for 4.0 (haven’t checked 4.1 since it just came out this morning) is 256 – # number of LUNs you intend to present to the hosts.

In other words, in 4.0 there’s a hard limit of 256 LUNs per cluster, and each host brings a LUN into the count….ESX has a local VMFS LUN where the serv console runs, and I suspect that there’s a hidden VMFS partition on ESXi, while probably tiny, still counts toward the limit

By: Thomas Suiter

Thomas Suiter — Mon, 12 Jul 2010 19:42:06 +0000

Not to be argumentative… but I think we are talking the same thing. Clvm is this: lvm2 with an extra daemon (clvmd) to handle locking and storage. It is basically part of lvm, just open /etc/lvm/lvm.conf on any RedHat box (Advanced Platform to Fedora) and you’ll find references to it. RedHat does separate the two and only includes the clvm daemon in certain channels, so if you want their binary you have pay for it but there’s nothing really special or unique about it.

By: dim

dim — Sun, 11 Jul 2010 06:26:05 +0000

Well, you’re in for a surprise – no clvm in there, plane lvm2 instead, with an extra daemon to handle locking and storage management.

If you are interested in actually seeing the product you might want to contact your sales rep, otherwise it’s just guesses that don’t lead anywhere

By: InsaneGeek

InsaneGeek — Sat, 10 Jul 2010 20:54:42 +0000

My understanding of RHEV with block storage, it generally uses lvm volumes to carve & present the storage. I could be wrong (as there is very limited docs on the tech behind the RHEV management layer… assuming it’s similar tech if you weren’t using their management) but I believe RHEV uses RHCS for mgmt of clustering Xen and now KVM with RHEL6, at least that’s the pitch from our RHEL sales reps over the years. Since RHCS has pretty much always used clvm from their Sistina purchase years ago for clustered logical volumes without contradictory info, I’m assuming they are continuing to use it here with RHEV clusters (there are a number of guides on making XEN Linux clusters with clvm and/or GFS).

By: dim

dim — Sat, 10 Jul 2010 06:29:35 +0000

What does clvm have to do with RHEV?

By: InsaneGeek

InsaneGeek — Sat, 10 Jul 2010 04:08:20 +0000

Unfortunately technology can hardly ever fully cover a sysadmin mistake. I don’t know of any volume manager that will prevent an admin from adding different types of luns into a volume group (vxvm, ZFS, lvm, etc).

If you want to play dangerous, you can also have a vmfs filesystem span across luns as well. Generally against best practices (but it’s there if you really want to do it) and it doesn’t *have* to be the same protection type. You do have to intentionally want to do it and know how to do it… so much, much less likely to happen (but possible).

As to the claim of avoiding the filesystem from a performance perspective… not completely sure on that claim. We too have had a very large single cluster of ESX servers (52x nodes) running in production… and yes it was very much out of spec (not my idea but ran for year+ that way). There are certain times when locks happen that have an affect on performance within VMFS: create vmdk, metadata change (i.e. grow vmdk or snapshot), poweron/off guest or host. There are certain similar times when lvm2 cluster type locks also occur. LVM2 snapshots growing don’t have the same lock affects but other operations do.

I haven’t run a RHEV system, but having tried to run a 90TB gfs2 filesystem under RHEL 5… I have to say had nothing but problems with clvm (patch on top of patch on top of patch and still nothing but hanging problems) some rather painful as well where it required a full cluster outage. VMFS I’ve found just works… additionally if you are that twitchy about the filesystem being an issue then you can do RDM and avoid filesystems completely (but that is damn icky for management) and use raw luns for guests.

By: dim

dim — Wed, 07 Jul 2010 22:11:11 +0000

>But in RHEV when you create a storage domain you’re not
>able to “VMotion” a system between domains right?
You probably mean “storage vmotion” – that’s currently an offline only feature (so you can move VMs that aren’t running), but qemu has it in the upstream, so RHEV will as well soon enough

>Also – since the VG’s are essentially LVM2, how do
>they handle locking when multiple hypervisors are
>hitting VM’s?
that’s what the SPM is for, and why it’s so important.

Not having an FS makes RHEV storage handling much faster in the long run, and also removes the host number limitation (seen a cluster of ~70 hosts, and that’s only because I ran out of machines to add)

>So the mistake here was when the admin threw all
>storage into the same domain and didn’t bother to
>check on mirroring.
yup, not a “best practice” in anyone’s book

By: Jesse

Jesse — Wed, 07 Jul 2010 22:03:19 +0000

But in RHEV when you create a storage domain you’re not able to “VMotion” a system between domains right? Doesn’t that pose serious limitations? I mean with VMWare it’s easy to move a VM from one storge type to another without downtime, which I’ve found to be invaluable in many instances… (I’ve run test boxes on NFS storage and then migrated it to FC storage when the test system goes into production.)

Also – since the VG’s are essentially LVM2, how do they handle locking when multiple hypervisors are hitting VM’s? VMFS handles this very well to a point, (I think 16-20 machines is the recomended limit per lun) but I’ve never known LVM2 to be particularly tolerant of block-level sharing…

So the mistake here was when the admin threw all storage into the same domain and didn’t bother to check on mirroring. (I have not been allowed to have any kind of real view into the system, so I don’t know, I just handed them the storage they requested.)

By: dim

dim — Wed, 07 Jul 2010 19:58:45 +0000

Well, there are different ways of working with the storage domains. You have to understand what you are doing of course
Since a storage domain (SD) is, on the storage level, a VG, you can expand it with relevant LUNs (which are in effect PVs) thus making it larger. Here is where the mistake took place actually.
On the other hand, you can always add new SDs, if the LUNs (PVs) are based on different storage types, and create your VMs with virtual disks in the different SDs, according to the VM’s requirements, e.g. have a NIS server VM on a SD that is based on a slower LUN, and put the database VMs on the faster LUNs.

As long as all the hosts in the same cluster can access all the relevant LUNs (and they should), this will work.

I don’t think you can actually “sense”, on the host side, if the LUNs you are presented with are different (might be a nice feature for dm-mp to have I guess — does powerpath do that?) so you have to know your storage when creating a setup of course

note to self – this is good KB material

By: Jesse

Jesse — Wed, 07 Jul 2010 16:49:39 +0000

Touche – as well as the guy who decided to simply hit the power button rather than actually troubleshoot the problem.

The way I look at it, and from what i read, RHEV doesn’t really give you the option of what to put where, does it? can someone answer that question? I mean with VMWare you simply don’t stripe VMFS across luns, you create the luns to be the size you need them, and that’s that. So if/when there is a crash, there is no risk of any issues with consistency.