50Micron.com

Downtime last night – This time (almost) planned.

In my constant effort to stay no more than a couple of steps behind current technology, I have started aquiring Dell 1850′s to replace my 2650′s.

I decided to go with the 1850′s because they are smaller form-factor, which means that they’ll fit in my cramped little rack and allow me room to add the third rack of disks to the Clariion.  I had originally purchased the 2650′s because not having a SAN in the basement I needed something in the way of real storage.

One happy side-effect is that apparently the PC3200 DDR2 memory that the 1850′s requires is in much higher supply than the PC2100 that the old 2650′s took.  This makes it cheaper to run more member and thereby fewer servers.

The whole thing was sold to the finance committee as a power saving/cost saving effort.  The best part (and real reason) is that this upgrade has enabled me to go from ESX 3.5 to vSphere4.  (The 2650 doesn’t support vSphere4 due to it’s lack of 64 bit support)

So far my impression is vSphere4 is that it’s a pretty solid upgrade to ESX 3.5, however I’m sure that once I get a chance to dive into it I’ll find a lot more coolness burined under the covers.

Oh – and the 2650′s are probably hitting ebay soon, should anyone want them – let me know ahead of time and I’ll cut you a sweet deal for being a reader.

I did not stay up way too late upgrading (playing with) the new CX.

I did not accidentally plug my new CX into a consumer-grade UPS but accidentally plug it into the SURGE outlet and not the BATT outlet.

I did not plug it into the same wall circuit that the aux AC is on.

I did not plug it into the same wall circuit that the Brinks Security system is on.

I did *NOT* trip the circuit breaker thereby causing the house alarm to go off at almost midnight, waking the entire family and generating a new level of annoyed from my wife that I can’t remember seeing in the recent past.

And most importantly….I did not do all of this halfway through the DART load on the NS half of the box.

That’s gonna sting.

There comes a time in every geek’s life….

A look of wonder crossing his face….

A new toy, a new project, a new endeavour….

A beloved wife saying something along the lines of “You bring that damned thing in here and I swear to god…”

This is one of those times.

Over the course of the next few weeks I’ll be moving 50micron.com and it’s assorted supporting systems off of the “OLD” storage (the 8 year old Dell PowerVaults I’ve been running on) and moving to a brand new (for me) NS500..

Now a few things I plan on making happen.  This NS500 will become an NS500G and CX500, because there is no way in hell I’m going to continue to live with a captured back-end when there is so much fibrechannel storage to be had, just out of reach..  (No iSCSI in this household, period, end of discussion)

Almost 9TB of storage, between the rack of 73G drives (pictured, sitting on top of the NAS) and the rack of 500G SATA disks (not pictured, too heavy for the bakers rack it’s all sitting on right now.)

Ohboyohboyohboyohboy.  This is the kind of thing real geeks live for.

This may be a more well-known reference than I earlier thought.

I grew up watching Happy-Days.  The show was great until the episode where Fonzi jumped the shark-tank.  After that it pretty much went down-hill quickly.

Hence the term “Jumped the shark” or “Jumping the shark” has come to mean any single event that marks the point where something degenerates into crap.

My VMWare NFS server jumped the shark this weekend.  It was hilarious.  I had a beautifully quiet afternoon on Friday, from about 14:30 on my blackberry was quiet.  Turns out that the NFS server that I use for storage experienced an unexplained (and apparently barely logged) kernel panic and rebooted.

In the process, the 6 adapters, in what I can only guess was a techno-square-dance, all switched places and lost their bonding configuration.

All went south, right in the middle of one of my busiest travel weeks as far as work goes.  So my wife, god bless her, earned her stripes this weekend as I walked her through ‘ifconfig eth0 10.1.1.10′ and ‘ping 10.1.1.254′ etc.  trying to figure out what happened.

Still don’t know.  But with everything down (including this site) my first priority was to get it all back online, troubleshoot later.  (When my desktop goes down I know why, I have an inquisitive 3 year old with a fetish for power-buttons), but the server power buttons are protected by a key – for that very purpose.

So I ordered a bunch of 146G drives for the hosts, and I’m going to move criticial apps back to internal storage until I figure out what in the hell happened and how to fix it.  It might give me an opportunity to eval. some new FC Target toys I’ve been thinking about.

Who knows.  No more shark-jumping though. 

Ok – So I’m going to put the question out here.

Has anyone heard of a decent fibrechannel target mode package?  I’ve been doing some research and have found a number of open NAS appliances, but only a few (and none of them open) that do target-mode.

Open-E – seems to be about the most solidly built, but expensive, retailing at about $1,600 US.  Their “Data Storage Server (DSS)” software seems to be the most robust.  It does offer NIC Teaming and the like, but I’ve not actually seen it work as of yet.  (They have a demo-bootable CD) but I’ve not really been able to get it configured, mostly due to lack of time, also due to lack of interest – I don’t like the idea of putting storage out there that is going to evaporate in 60 days.  If the folks at Open-E want to provide me a usable copy of the software I’d be happy to give it a serious exercising.

FreeNAS - as a rule – don’t trust any software for “production” data that is still in the 0.x phase of it’s lifecycle.  Let me know when you get a 1.x version out and I’ll have a look.

OpenFiler - probably one of the more mature open source products on the market – no support for Fibrechannel target mode is an instant turn-off.  In my eval of it I was also not able to discern an easy way to bond network interfaces together.  I’ve considered trying to SSH into the box and do it via linux but never got around to trying.  If it will support true LACP it might be worth investigating.  I’m still not sold on iSCSI as being “production-worthy” because ethernet being what it is, I question reliability.  (I don’t even like running NFS when it comes down to it.)

So that being said – does anyone have any suggestions for FC target-mode appliances?

A couple of years ago, I picked up an old Clariiion FC5300 wholesale (free) from a junk-pile at one of my customers.  I played with it, it was nice, but I couldn’t figure out why I should use it when I had 73+G drives available to me.

I started the FC@Home project then.  Because I thought it would be cool to have fibrechannel running in my home system.

Well I got rid of the FC5300 because the 30 x 18G Full-Height drives were just too much to power and cool.

A few weeks ago I decided that I needed to do it again.  (I posted something of it earlier)  Got an old EMC/Brocade DS16B2 switch, a PowerVault 224F JBOD, and started playing.

Well the first thing I found is that I could never use JBOD for the purposes I wanted to.  I wanted to put together some redundant shared storage for my VMWare servers so I could play with VMotion and Clustering.  While I could share individual disks, RAID wasn’t an option and I refuse to use unprotected storage.

So I scoured Ebay and found a PowerVault 660F to add to the 224F.  Now the 224F came with 14x 18G drives, the 660F came with 14x36G drives.  I paid under $200 (not including shipping) for each of the two racks.  They are 3U units, don’t pull a tremendous amount of power, and are as difficult to cool as any drive array (it’s the drives that cause the heat, not the array)

Another $100 or so in cables (The HSSDC->HSSDC jumpers that were required for between the units) and I was good to go.  I already had some DB-9 FC –> SC-Duplex converts, as well as some SC–>LC cables, so that part was easy.  I found someone who off-loaded a bunch of old Qlogic QLA2200 HBA’s (9 for $50) and the whole things was done.

I initially had issues getting it recognized, but on a whim I called Dell support.  The tech informed me that this was so far out of support he really couldn’t help me, then proceeded to spend about an hour helping me out.  Turns out that the Array Manager software that you use to manage the thing doesn’t work with the latest / greatest QLogic drivers.  I had to back-rev them to v8.x and suddenly it worked perfectly.  (He also told me it was never ever going to work on Win2k3 – a fact I’ve happily disproven.)

I just got it carved, and all but one of my VM’s are moved over to it.  I have about 500G of Raid-5 Storage available with 2 Hot-Spares (Since I don’t know the history of the drives, I figured better safe than sorry).

So far so good.  Performance is great, though I’m only going through one switch I have redundant RAID controllers, so that’s at least something.  As soon as I find someone dumping a second DS16B2 I’ll probably incorporate that into the mix as well.

So I set up the 2-node VMWare cluster, and set it for DRS just to see if it works the way they say it will.  (I’m also curious because I have less memory in the second node than the first, if it will be aware of that.)  I have a third 2650 I got here because some newbie on Ebay didn’t realize that the particular error message he got on boot meant simply that there was no operating system on the disks.  As soon as I get the rail-kit I’m going to mount this puppy up and make it a 3-node cluster.

I’m such a geek.