50Micron.com

Tonight I have started the process of converting my own CX300 to a CX3-20c.  (yay!  upgrading from SERIOUSLY out-of-date hardware to MODERATELY out-of-date hardware, right?)

Why?  Because it’s there.  Since EMC doesn’t offer free training to sub-sub-sub-contractors such as myself, it falls to me to learn what I can where I can. 

Besides.  It’s fun.

So far it’s been pretty simple.  Printed out the 63 page guide from the Clariion Proceedure Generator, and was briefly intimidated by it before I realized that 80% of it is completely useless.

But, because I want the experience, I’m going through each step of it.

First thing I did was backed up the vault pack.  This particular CX300 has no data on it, so it was a simple process to swap the five vault drives on at a time.  Though in the interest of doing it gracefully, I did bind a 1G lun across the five drives so that I could use the proactive sparing to gracefully remove each drive.  (The option isn’t available unless you have a bound lun on the raid group)

Obviously I want to come out of this with a working CX300 as well as the CX3-20.   (And of course my fear is ending up with not one but TWO doorstops at the end of this process)

Ran CRC2.  Interesting application, might come in handy in the future, because  when it comes down to it, it gives you a LOT of information about your clariion that you don’t get from the GUI.

Luckily this is a situation where there isn’t anything i need to keep on the array.  One thing stands out though, I had to delete the little 1G lun I bound on the vault pack to get the CRC2 check to pass, because apparently a number of the system partitions need to expand.  (Life would have sucked if I didn’t have the ability to wipe the drives)

After deleting the 1G lun, CRC2 passed and all was good with the world.

Skipped the next 6 pages, which describe how to make room in the rack for the new equipment.

Why?

No rack in my basement anymore, I guess a desk will do.

Let’s just say rack-space really isn’t the issue here.

So the only thing I wasn’t able to find was a copy of EMCRemote.  Though I have an older one from back in my EMC days, hopefully the protocols haven’t changed much.  <crosses fingers>

So Unisphere Service Manager (Formerly Navisphere Service TaskBar) is installed, and the first step is to install the target platform conversion-prep software.  A pretty straight-forward install.  Once this package is installed you’re comitted. (or should be)

I had a bit of a worry loading the conversion-image, SPB didn’t come back before USM timed out…which seems like something that might just happen in these older, slower devices, right? 

The ConversionPrep, ConversionImage, and most importantly, the new Utility partition all seem to have installed correctly, as with the HSConversionB package. 

Now sadly, there are no descriptions as to what each NDU does, though logically:

ConversionPrep handles the settings – one of the things that you verify after the ConversionPrep is loaded is that write-cache is disabled (which it always was because I don’t have a SPS cable for this model) and such.

ConversionImage and Utility Partition are pretty self-explanitory, the CX3 looks for things in different places, requires different drivers, etc. 

HSConversionB was the stumper.  HardwareSwap?  Any ideas?

Last step was to shut-down the array.  This will be my stopping point. 

Stay tuned, same bat-time, same bat-rss-feed.

Ok, given the changes to the storage arena I’ve been working on a revised “Tiering system” to incorporate all of the levels of data…importance?

My version of Storage Tiering is (or should be) as follows:

• Tier-1    – Symmetrix/Replicated – High Performance/Criticial Data
• Tier-2    – Symmetrix/NonReplicated – High Performance/Non-Criticial Data
• Tier-3   – Symmetrix/SATA/Replicated – High-Medium Performance/Critical Data
• Tier-4   – Symmetrix/SATA/NonReplicated – High-Medium Performance/Non-Critical Data
• Tier-5    – Clariion/FC/Replicated – Medium Performance/Critical Data
• Tier-6    – Clariion/FC/NonReplicated – Medium Performance/Non-Critical Data
• Tier-7    – Clariion/SATA/Replicated – Low Performance/Critical Data
• Tier-8    – Clariion/SATA/NonReplicated – Low Performance/Non-Critical Data
• Tier-9    – CelerraNAS/Replicated – Network Attached/Critical Data
• Tier-10  – CelerraNAS/NonReplicated – Network Attached/Non-Criticial Data
• Tier-11  – Atmos – Network Attached / Low Performance
• Tier-12  – Centerra (Content Addressable Storage) – Low Performance Archive / Highly Available
• Tier-13  – Primary Tape-In-Library (Automatic loading on demand via HSM)
• Tier-14  – Primary Tape-Out-Of-Library (Manual Intervention Required)

“Critical Data” vs. “Non-Critical Data” is simply a matter of how long you can be without the data should a failure or accidental deletion occur.  As all data is available in Tier8/9 storage (in theory).

I’ve also considered using Tier1/Tier1B to describe DMX storage vs. Clariion storage, given that there is a LOT of overlap in performance characteristics these days…

Oh, and iSCSI would be somewhere between 10 and 13….

Any thoughts?

There comes a time in every geek’s life….

A look of wonder crossing his face….

A new toy, a new project, a new endeavour….

A beloved wife saying something along the lines of “You bring that damned thing in here and I swear to god…”

This is one of those times.

Over the course of the next few weeks I’ll be moving 50micron.com and it’s assorted supporting systems off of the “OLD” storage (the 8 year old Dell PowerVaults I’ve been running on) and moving to a brand new (for me) NS500..

Now a few things I plan on making happen.  This NS500 will become an NS500G and CX500, because there is no way in hell I’m going to continue to live with a captured back-end when there is so much fibrechannel storage to be had, just out of reach..  (No iSCSI in this household, period, end of discussion)

Almost 9TB of storage, between the rack of 73G drives (pictured, sitting on top of the NAS) and the rack of 500G SATA disks (not pictured, too heavy for the bakers rack it’s all sitting on right now.)

Ohboyohboyohboyohboy.  This is the kind of thing real geeks live for.

You know, it floored me recently when i heard that someone had said that a CX3-20 couldn’t be used as a dumparea for Tivoli.

What floored me even more is when they said this would not perform as well as a Symmetrix 8430.

8430.

Symm4.

Almost 10 year old hardware.

Huh?

Back when I was working for the student loan company in Sterling, we ran all of our backups to a single CX500 with 15 Terabytes behind it. Worked without any issue and was screaming fast, to the point that 18 hours (to tape) of backups was compressed into 8 hours (to disk.)

{sarcasm}Now correct me if I’m wrong, the CX3-20 is a tad faster than the older CX500, right?{/sarcasm}

As with any array, the bulk of the issue arises from whether or not the disks are laid out appropriately. If you try to run single-disk LUNs you’re probably going to die of old-age before a backup finishes. But if you stripe it appropriately and make sure the LUN ownership is correct, you can do wonders with EMC’s “Tier-2″ array.

Got into a scary situation this week.  Got called into help with a customer with a Mirrorview implementation.

Situation was:  Customer had Mirrorview/S set up within the existing switch environment, replication worked perfectly.

Then they reconfigured the switches to run FCIP so they could start replication to a remote site.  This is where things went badly.

First off – Cisco sets the Gig/E ports on the 9216i for jumbo-frames.  (MTU defaults to 2300)

This is a great idea for Fibrechannel replication, because a fibrechannel frame is 2114 bytes and this allows an entire FC frame to be sent within an ethernet packet.

Problem is that the default MTU on most network environments is 1500.  Now the *REAL* problem is that when  you first connect the GIg/E ports on the 9216i to a 6509 or other switch – it will at first appear to work perfectly…..until you try to pass data.

When you try to pass data across this link, the DF (Don’t Fragment) bit is set and the larger frames get dropped.  This causes an ISL connection between switches to flap, which causes no end of issues.  The fabrics will segment and re-join repeatedly until the first time you do anything that causes a reconfiguration, like updating the zoneset.  If you do that during a cycle where the ISL is going up and down, the vsan’s will fragment and stay fragmented because it will not be able to re-merge the fabrics.

So I come into this situation and the switches are so badly configured that it takes me a day just to get the ISL’s up and stable.  I set the MTU to 1500 on the switches, took the gig-e links down, and went to each switch and (carefully) deleted each vsan that didn’t belong on that switch.  (In addition to this being set up incorrectly, all three vsans merged when the swtiches were first connected due to the ISL’s being configured incorrectly)

Now the Clariion issue is still open.  A normal mirrorview configuration is as follows:

Source_SPAx –> Target_SPAx  (Where ‘x’ is the highest SP port #)

Source_SPBx –> Target_SPBx   (same here)

Now when the customer’s Clariion’s are zoned this way (in this case SPB3 to SPB1) nothing shows up in the Connectivity Status window.   But when I reverse the zoning, running SPA3 to SPB1, it shows up fine.  (Unfortunately Mirrorview doesn’t work in that configuration.

That’s where we stand.  A “simple 15 minute FCIP fix” is coming to the end of it’s third day.

I spent the week following New Years in Chelmsford, Mass doing some Clariion work.

Basically customer is seeing major performance loss on an older CX700.  The CX700 is fully loaded (as far as slots full, not necessarily space – lots of 73G drives.)

A few things to note on the Clariion

• It’s not a true “Enterprise Class” storage system.  Pretending it is only sets you up for trouble.  There is a reason that the Clariion is almost exclusively sold into the SME market.

• If you are going to try to use a CX700 for “real” work, be mindful of the layout of the disks.  Unlike the Symmetrix, where you put the disks on the back-end directly affects performance.

• Get intimately familiar with the “Migrate Lun” option.  This allows you to become the equivalent of Symm Optimizer.  If you identify a hot bus, enclosure, or disk, you can move luns off that disk to other physicals.

A few of the gotchas:

• LUN’s in ATA disk raid-groups must all be owned by the same SP.  The reason being is that by design the ATA disks are single-ported.  If the lun on a particular disk is owned by a different SP, that IO has to cross the backplane between SP’s to get routed out the correct SP.

• Spread the IO as far as possible.  If you are on a CX3-80 with 4 busses and 4 enclosures, each enclosure should be on a separate bus, create a raid-group within each enclosure, a lun in each raid-group, and then a metalun to cross all of them.  This means that (assuming 4+1 Raid-5) each IO is distributed across 20 physical spindles.  The management is a pain but it mitigates any issues with one application bringing the array to it’s knees.

• You’re not married to the physical placement.  You can easily migrate luns between raid-groups, to and from metaluns, even to larger devices if needed.  And this can be done online with minimal host impact.

• Last, but most important.  The first 5 drives in the array (Bus0, Enclosure0, Disk0-4) are off-limits.  If due to storage constraints you must use them, the only thing you can put on them is static storage.  Heavy IO to these disks *WILL* cause the array as a whole to slow down dramatically.

An example of this would be putting something like a 500G Lun for an exchange recovery storage group.  It’s only used in rare circumstances, and never that heavily.