My week with Clariion
by Jesse on Jan.05, 2008, under Clariion
I spent the week following New Years in Chelmsford, Mass doing some Clariion work.
Basically customer is seeing major performance loss on an older CX700. The CX700 is fully loaded (as far as slots full, not necessarily space – lots of 73G drives.)
A few things to note on the Clariion
- It’s not a true “Enterprise Class” storage system. Pretending it is only sets you up for trouble. There is a reason that the Clariion is almost exclusively sold into the SME market.
- If you are going to try to use a CX700 for “real” work, be mindful of the layout of the disks. Unlike the Symmetrix, where you put the disks on the back-end directly affects performance.
- Get intimately familiar with the “Migrate Lun” option. This allows you to become the equivalent of Symm Optimizer. If you identify a hot bus, enclosure, or disk, you can move luns off that disk to other physicals.
A few of the gotchas:
- LUN’s in ATA disk raid-groups must all be owned by the same SP. The reason being is that by design the ATA disks are single-ported. If the lun on a particular disk is owned by a different SP, that IO has to cross the backplane between SP’s to get routed out the correct SP.
- Spread the IO as far as possible. If you are on a CX3-80 with 4 busses and 4 enclosures, each enclosure should be on a separate bus, create a raid-group within each enclosure, a lun in each raid-group, and then a metalun to cross all of them. This means that (assuming 4+1 Raid-5) each IO is distributed across 20 physical spindles. The management is a pain but it mitigates any issues with one application bringing the array to it’s knees.
- You’re not married to the physical placement. You can easily migrate luns between raid-groups, to and from metaluns, even to larger devices if needed. And this can be done online with minimal host impact.
- Last, but most important. The first 5 drives in the array (Bus0, Enclosure0, Disk0-4) are off-limits. If due to storage constraints you must use them, the only thing you can put on them is static storage. Heavy IO to these disks *WILL* cause the array as a whole to slow down dramatically.
An example of this would be putting something like a 500G Lun for an exchange recovery storage group. It’s only used in rare circumstances, and never that heavily.
January 6th, 2008 on 6:40 pm
I am supposed to start a new contract with a company just starting out into the SAN world. The environment will have brand spanking new CLARiions. What issues did you find out about performance? I am always interested in what people did that requires PS to come out and fix it. And hopefully, not repeat that mistake especially since I am not a CLARiion guy.
January 6th, 2008 on 9:53 pm
Fun stuff – when you start showing them the amazing you can do with a SAN it’s fun to watch their faces.
The catch is to make sure you balance. All the things that you assume the Symm is doing for you has to be done by you on a clariion.
Remember also that load balancing isn’t done across SP’s, it’s done across ports on an SP. So each HBA has to be zoned to two separate ports, one on each SP, for redundancy.
The way it’s usually set up:
HBA0 –> SPA0/SPB0
HBA1 –> SPA1/SPB1
This way if SP-A owns the lun, the load balancing happens through SPA0/SPA1. If SP-B owns it, the load balancing happens through SPB0/SPB1.
As a result, the usual drive to connect/zone like a symm (SAN-A connected to FA8xx, SAN-B connected to FA-9xx) has to be avoided. SAN-A should be connected to SPA0/SPB0, SAN-B should be connected to SPA1/SPB1, etc. (On Clariions with 4 front-end ports, use evens/odds)
Email me if you run into specifics while you’re there.
January 7th, 2008 on 9:03 am
I don’t take care of a lot of Clariions anymore, but where I used to work we did our cabling a bit differently. On fabric A, we’d put SPA0,SPB1,SPA2,SPB3 and of fabric B we’d put SPB0,SPA1,SPB2,SPA3. I originally wanted to do odd/even, but we were advised by EMC to go this way instead. The upside is in the case of MirrorView for replication. MirrorView can only use ports SPA3 and SPB3. If you go with odd/even cabling, a FC fabric outage has a 50/50 chance of causing all your mirrors to fail. If you’ve got SPA3 and SPB3 on separate fabrics MirrorView will continue to operate. I don’t recall if MirrorView losing connection between Clariions will force a LUN trespass, so if it doesn’t I guess you’re still out half your mirrors if you drop a whole fabric. It’s been a while…
January 7th, 2008 on 3:42 pm
Tape monkey:
Things we do on the Clariions:
1) 6+1 raid5 groups: This allows us to fit 2x raid groups nicely into a single shelf and have a hot spare on the shelf. Or even better use raid6, especially with ATA drives if your system supports it. (as our ATA’s have aged the failure rate has increased dramatically, and I’ve had a couple of: 1x drive failed, and received a read error during the rebuild and the data on that part of the disk was lost).
2) Everything is a 4 way meta, I came to that number because 4 is normally easily divisble into most storage requests (100gb = 4x 25gb) and fit nicley into the number of shelves we had on each of our clariion units (most had 8x). On 5400rpm ATA drives it changed our performance from ~30MB/sec to >150MB/sec. (as you can see this has a big performance impact, but you might have to check the type of workloads… we are very random so this works well for us. If you are highly sequential it might not be beneficial to stripe on top of the raid5 stripe)
3) Use your own lun numbers (ALU) and keep track of the ones you used and don’t reuse them rather than the letting the clariion define them. The Clariion will happily re-use ALU numbers and can make things confusing for others: i.e. if using the default numbers provided by the clariion
Assume you have raidgroups 1-8 and want to create 2x meta devices with 4x members
Create luns with ALU number of 1,2,3,4 in raidgroup 1,2,3,4 respectively
Create meta of luns, meta ALU # = 1
Create luns with ALU number of 2,3,4,5 in raidgroup 5,6,7,8 respectively
Create meta of luns, meta ALU # = 2
There is nothing technically wrong with this configuration but it can confuse the hell out of people who aren’t well versed on it, as now you have multiple luns numbered 2 (2x are meta members and 1x is the meta head).
4) Everything has 4x paths to the storage, normally I pick ports 0&1 or 2&3 so I don’t have to do a tresspass in case of a path failure and there is a standard way of doing things. We have the SP’s ports connect the same way jm does it for mirrorview.
5) Increase your SP write cache, you might think you want a “50/50 balance” between read & write, but almost allways you want to increase the write and only leave a small amount of read cache on the SP. Your particular workload may change this, but in general you want this; you don’t want to get into a situation where you have writes delayed to the host because it or some other host decided to send a “burst” of data.
6) If you have puchased the software licenses for mirrorview, snapview, etc. Deffinetly spend some time on planning out the reserve-lun pool and keep an eye on it when in production to make sure you don’t run out of space and or impact performance. We spread out our reserve luns across the whole array with lots of 20GB luns to avoid hot-spots, rather than in just a couple of raid groups (again we have a highly-random workload which lends itself well to spreading out wide). The one detriment is that you can’t control which luns are used in the pool, so in a bad situation a snapview session’s reserve lun could be in the same raidgroup as the source, so you’ll be copying from and to the same spindles causing a lot of head movement (this hasn’t caused us any pain over the years but should be noted as workloads differ).
January 7th, 2008 on 9:10 pm
Thanks everyone. Its great to see different viewpoints on the same product. I know some of the things written, but others were definitely new.
Sorry Sangod, I have to agree with jm on his SP cabling. I’ve seen it like that but never had a good reason explained.
InsaneGeek, thanks for the config tips, those will definitely come in handy. But as far as #3, I think I have to sit in front of Navisphere to figure that one out. I am sure I did it before, but reading it confuses the hell out of me.
January 7th, 2008 on 11:11 pm
I agree – I had forgotten about the Mirrorview aspect of the Clariion and the limitations it imposes. That’s probably why it’s still best practice to use the “D” processor for SRDF as well, just a hold-over. (though pre-clariion days it was the only way it was supported, so that may still not be a valid explanation)
You also have to remember I cut my Clariion teeth when there were only ports 0 and 1.
So are we in agreement -
SAN-A A0, B1, A2, B3
SAN-B B0,A1,B2,A3
Right?
February 19th, 2008 on 3:33 am
We pretty much do it the same way. In our old data centers with FC4700s we do HBA0->Brocade_1->SPA0/SPB0 and HBA1->Brocade_2->SPA1/SPB1. We went from the Clariions to DMX-2s in the next engineering release but the one or two CX-700s we have are setup like Sangod says above.
February 20th, 2008 on 12:48 am
It makes sense – I’ve always believed that the best design is consistent. If you’re going to do it one way, keep doing it that way.
The A0,B1,A2,B3 / B0,A1,B2,A3 configuration makes the most sense in Mirrorview implmentations, so that A3 and B3 are in different Fabrics, and as such maintain their redundancy.
If you’re going to do it that way for Mirrorview, do it for any implementation, so that the next guy has an idea of what to expect when the customer decides that they need Mirrorview implemented.