50Micron.com

Tonight I have started the process of converting my own CX300 to a CX3-20c.  (yay!  upgrading from SERIOUSLY out-of-date hardware to MODERATELY out-of-date hardware, right?)

Why?  Because it’s there.  Since EMC doesn’t offer free training to sub-sub-sub-contractors such as myself, it falls to me to learn what I can where I can. 

Besides.  It’s fun.

So far it’s been pretty simple.  Printed out the 63 page guide from the Clariion Proceedure Generator, and was briefly intimidated by it before I realized that 80% of it is completely useless.

But, because I want the experience, I’m going through each step of it.

First thing I did was backed up the vault pack.  This particular CX300 has no data on it, so it was a simple process to swap the five vault drives on at a time.  Though in the interest of doing it gracefully, I did bind a 1G lun across the five drives so that I could use the proactive sparing to gracefully remove each drive.  (The option isn’t available unless you have a bound lun on the raid group)

Obviously I want to come out of this with a working CX300 as well as the CX3-20.   (And of course my fear is ending up with not one but TWO doorstops at the end of this process)

Ran CRC2.  Interesting application, might come in handy in the future, because  when it comes down to it, it gives you a LOT of information about your clariion that you don’t get from the GUI.

Luckily this is a situation where there isn’t anything i need to keep on the array.  One thing stands out though, I had to delete the little 1G lun I bound on the vault pack to get the CRC2 check to pass, because apparently a number of the system partitions need to expand.  (Life would have sucked if I didn’t have the ability to wipe the drives)

After deleting the 1G lun, CRC2 passed and all was good with the world.

Skipped the next 6 pages, which describe how to make room in the rack for the new equipment.

Why?

No rack in my basement anymore, I guess a desk will do.

Let’s just say rack-space really isn’t the issue here.

So the only thing I wasn’t able to find was a copy of EMCRemote.  Though I have an older one from back in my EMC days, hopefully the protocols haven’t changed much.  <crosses fingers>

So Unisphere Service Manager (Formerly Navisphere Service TaskBar) is installed, and the first step is to install the target platform conversion-prep software.  A pretty straight-forward install.  Once this package is installed you’re comitted. (or should be)

I had a bit of a worry loading the conversion-image, SPB didn’t come back before USM timed out…which seems like something that might just happen in these older, slower devices, right? 

The ConversionPrep, ConversionImage, and most importantly, the new Utility partition all seem to have installed correctly, as with the HSConversionB package. 

Now sadly, there are no descriptions as to what each NDU does, though logically:

ConversionPrep handles the settings – one of the things that you verify after the ConversionPrep is loaded is that write-cache is disabled (which it always was because I don’t have a SPS cable for this model) and such.

ConversionImage and Utility Partition are pretty self-explanitory, the CX3 looks for things in different places, requires different drivers, etc. 

HSConversionB was the stumper.  HardwareSwap?  Any ideas?

Last step was to shut-down the array.  This will be my stopping point. 

Stay tuned, same bat-time, same bat-rss-feed.

Oh 50micron.com….how I’ve neglected you.  But it’s been for good reason.

You see, the little nothing company that I quite literally started in my basement is growing up…  A few dozen customers and suddenly I’m finding that the electricity and heat problems in my basement were *NOT* going to tolerate a second tray of disks, let alone higher end vmware servers.

Customers?  Yeah.  Customers.   I host a few dozen websites for people for a nominal fee, provide off-site storage for a few local small-business customers who aren’t happy with the idea of keeping their backups in the same room with their servers, and the like.  It’s come a long way from running a webserver that I’d hand-out space on to friends from time to time.  Long and short, it’s starting to actually pay the bills (or at least a portion of them.)

So I’m moving.  Moved really, I moved everything to a “temporary” CX500 in my basement, and hauled the original CX500 and NS502G have moved to a colo site in Springfield, VA. 

Hello HopOne Internet Corp.   We’ve been together for barely a month and yet bits of you are already annoying the hell out of me.  Like trying to charge me $400 “setup fee” to plug a power-strip into a recepticle, or the fact that the 20a circuit you sold me was limited by the 15a circuit-breaker in the strip, you know, the little things.

But it’s working.  The hardware is up and running, and I started writing this post while the Celerra there was upgrading to 5.6.49 DART.

Also new to the mix is a pair of Dell PowerEdge 1950.  Not new by any stretch but dual, quad-core processors and 16GB of RAM/ea will make for a good pair of VMware servers.  The old servers?  On the block (See below)

So that’s it.  We keep evolving, slowly but surely.

As far as hardware goes, the 3 Gen-2 PowerEdge 1850 servers I used to run on are for sale if anyone is interested.  $450/each, including shipping via FedEx ground to anywhere FedEx ground will ship from here.  (Contental US)

These include

Dual 3.2Ghz Xeon Processors
*8* GB of ram.
PERC Raid (I think 4i or 5i)
Rapid Rails / Bezels
Dual Power Supplies
Dual 18G 15K rpm Drives.

Let me know if you’re interested.  They won’t run ESX4.1, and won’t virtualize a 64bit OS, but they’re great for lab / playtoys.

Downtime last night – This time (almost) planned.

In my constant effort to stay no more than a couple of steps behind current technology, I have started aquiring Dell 1850′s to replace my 2650′s.

I decided to go with the 1850′s because they are smaller form-factor, which means that they’ll fit in my cramped little rack and allow me room to add the third rack of disks to the Clariion.  I had originally purchased the 2650′s because not having a SAN in the basement I needed something in the way of real storage.

One happy side-effect is that apparently the PC3200 DDR2 memory that the 1850′s requires is in much higher supply than the PC2100 that the old 2650′s took.  This makes it cheaper to run more member and thereby fewer servers.

The whole thing was sold to the finance committee as a power saving/cost saving effort.  The best part (and real reason) is that this upgrade has enabled me to go from ESX 3.5 to vSphere4.  (The 2650 doesn’t support vSphere4 due to it’s lack of 64 bit support)

So far my impression is vSphere4 is that it’s a pretty solid upgrade to ESX 3.5, however I’m sure that once I get a chance to dive into it I’ll find a lot more coolness burined under the covers.

Oh – and the 2650′s are probably hitting ebay soon, should anyone want them – let me know ahead of time and I’ll cut you a sweet deal for being a reader.

I did not stay up way too late upgrading (playing with) the new CX.

I did not accidentally plug my new CX into a consumer-grade UPS but accidentally plug it into the SURGE outlet and not the BATT outlet.

I did not plug it into the same wall circuit that the aux AC is on.

I did not plug it into the same wall circuit that the Brinks Security system is on.

I did *NOT* trip the circuit breaker thereby causing the house alarm to go off at almost midnight, waking the entire family and generating a new level of annoyed from my wife that I can’t remember seeing in the recent past.

And most importantly….I did not do all of this halfway through the DART load on the NS half of the box.

That’s gonna sting.

There comes a time in every geek’s life….

A look of wonder crossing his face….

A new toy, a new project, a new endeavour….

A beloved wife saying something along the lines of “You bring that damned thing in here and I swear to god…”

This is one of those times.

Over the course of the next few weeks I’ll be moving 50micron.com and it’s assorted supporting systems off of the “OLD” storage (the 8 year old Dell PowerVaults I’ve been running on) and moving to a brand new (for me) NS500..

Now a few things I plan on making happen.  This NS500 will become an NS500G and CX500, because there is no way in hell I’m going to continue to live with a captured back-end when there is so much fibrechannel storage to be had, just out of reach..  (No iSCSI in this household, period, end of discussion)

Almost 9TB of storage, between the rack of 73G drives (pictured, sitting on top of the NAS) and the rack of 500G SATA disks (not pictured, too heavy for the bakers rack it’s all sitting on right now.)

Ohboyohboyohboyohboy.  This is the kind of thing real geeks live for.

This may be a more well-known reference than I earlier thought.

I grew up watching Happy-Days.  The show was great until the episode where Fonzi jumped the shark-tank.  After that it pretty much went down-hill quickly.

Hence the term “Jumped the shark” or “Jumping the shark” has come to mean any single event that marks the point where something degenerates into crap.

My VMWare NFS server jumped the shark this weekend.  It was hilarious.  I had a beautifully quiet afternoon on Friday, from about 14:30 on my blackberry was quiet.  Turns out that the NFS server that I use for storage experienced an unexplained (and apparently barely logged) kernel panic and rebooted.

In the process, the 6 adapters, in what I can only guess was a techno-square-dance, all switched places and lost their bonding configuration.

All went south, right in the middle of one of my busiest travel weeks as far as work goes.  So my wife, god bless her, earned her stripes this weekend as I walked her through ‘ifconfig eth0 10.1.1.10′ and ‘ping 10.1.1.254′ etc.  trying to figure out what happened.

Still don’t know.  But with everything down (including this site) my first priority was to get it all back online, troubleshoot later.  (When my desktop goes down I know why, I have an inquisitive 3 year old with a fetish for power-buttons), but the server power buttons are protected by a key – for that very purpose.

So I ordered a bunch of 146G drives for the hosts, and I’m going to move criticial apps back to internal storage until I figure out what in the hell happened and how to fix it.  It might give me an opportunity to eval. some new FC Target toys I’ve been thinking about.

Who knows.  No more shark-jumping though. 

Ok – So I’m going to put the question out here.

Has anyone heard of a decent fibrechannel target mode package?  I’ve been doing some research and have found a number of open NAS appliances, but only a few (and none of them open) that do target-mode.

Open-E – seems to be about the most solidly built, but expensive, retailing at about $1,600 US.  Their “Data Storage Server (DSS)” software seems to be the most robust.  It does offer NIC Teaming and the like, but I’ve not actually seen it work as of yet.  (They have a demo-bootable CD) but I’ve not really been able to get it configured, mostly due to lack of time, also due to lack of interest – I don’t like the idea of putting storage out there that is going to evaporate in 60 days.  If the folks at Open-E want to provide me a usable copy of the software I’d be happy to give it a serious exercising.

FreeNAS - as a rule – don’t trust any software for “production” data that is still in the 0.x phase of it’s lifecycle.  Let me know when you get a 1.x version out and I’ll have a look.

OpenFiler - probably one of the more mature open source products on the market – no support for Fibrechannel target mode is an instant turn-off.  In my eval of it I was also not able to discern an easy way to bond network interfaces together.  I’ve considered trying to SSH into the box and do it via linux but never got around to trying.  If it will support true LACP it might be worth investigating.  I’m still not sold on iSCSI as being “production-worthy” because ethernet being what it is, I question reliability.  (I don’t even like running NFS when it comes down to it.)

So that being said – does anyone have any suggestions for FC target-mode appliances?

A couple of years ago, I picked up an old Clariiion FC5300 wholesale (free) from a junk-pile at one of my customers.  I played with it, it was nice, but I couldn’t figure out why I should use it when I had 73+G drives available to me.

I started the FC@Home project then.  Because I thought it would be cool to have fibrechannel running in my home system.

Well I got rid of the FC5300 because the 30 x 18G Full-Height drives were just too much to power and cool.

A few weeks ago I decided that I needed to do it again.  (I posted something of it earlier)  Got an old EMC/Brocade DS16B2 switch, a PowerVault 224F JBOD, and started playing.

Well the first thing I found is that I could never use JBOD for the purposes I wanted to.  I wanted to put together some redundant shared storage for my VMWare servers so I could play with VMotion and Clustering.  While I could share individual disks, RAID wasn’t an option and I refuse to use unprotected storage.

So I scoured Ebay and found a PowerVault 660F to add to the 224F.  Now the 224F came with 14x 18G drives, the 660F came with 14x36G drives.  I paid under $200 (not including shipping) for each of the two racks.  They are 3U units, don’t pull a tremendous amount of power, and are as difficult to cool as any drive array (it’s the drives that cause the heat, not the array)

Another $100 or so in cables (The HSSDC->HSSDC jumpers that were required for between the units) and I was good to go.  I already had some DB-9 FC –> SC-Duplex converts, as well as some SC–>LC cables, so that part was easy.  I found someone who off-loaded a bunch of old Qlogic QLA2200 HBA’s (9 for $50) and the whole things was done.

I initially had issues getting it recognized, but on a whim I called Dell support.  The tech informed me that this was so far out of support he really couldn’t help me, then proceeded to spend about an hour helping me out.  Turns out that the Array Manager software that you use to manage the thing doesn’t work with the latest / greatest QLogic drivers.  I had to back-rev them to v8.x and suddenly it worked perfectly.  (He also told me it was never ever going to work on Win2k3 – a fact I’ve happily disproven.)

I just got it carved, and all but one of my VM’s are moved over to it.  I have about 500G of Raid-5 Storage available with 2 Hot-Spares (Since I don’t know the history of the drives, I figured better safe than sorry).

So far so good.  Performance is great, though I’m only going through one switch I have redundant RAID controllers, so that’s at least something.  As soon as I find someone dumping a second DS16B2 I’ll probably incorporate that into the mix as well.

So I set up the 2-node VMWare cluster, and set it for DRS just to see if it works the way they say it will.  (I’m also curious because I have less memory in the second node than the first, if it will be aware of that.)  I have a third 2650 I got here because some newbie on Ebay didn’t realize that the particular error message he got on boot meant simply that there was no operating system on the disks.  As soon as I get the rail-kit I’m going to mount this puppy up and make it a 3-node cluster.

I’m such a geek.