50Micron.com

This may be a more well-known reference than I earlier thought.

I grew up watching Happy-Days.  The show was great until the episode where Fonzi jumped the shark-tank.  After that it pretty much went down-hill quickly.

Hence the term “Jumped the shark” or “Jumping the shark” has come to mean any single event that marks the point where something degenerates into crap.

My VMWare NFS server jumped the shark this weekend.  It was hilarious.  I had a beautifully quiet afternoon on Friday, from about 14:30 on my blackberry was quiet.  Turns out that the NFS server that I use for storage experienced an unexplained (and apparently barely logged) kernel panic and rebooted.

In the process, the 6 adapters, in what I can only guess was a techno-square-dance, all switched places and lost their bonding configuration.

All went south, right in the middle of one of my busiest travel weeks as far as work goes.  So my wife, god bless her, earned her stripes this weekend as I walked her through ‘ifconfig eth0 10.1.1.10′ and ‘ping 10.1.1.254′ etc.  trying to figure out what happened.

Still don’t know.  But with everything down (including this site) my first priority was to get it all back online, troubleshoot later.  (When my desktop goes down I know why, I have an inquisitive 3 year old with a fetish for power-buttons), but the server power buttons are protected by a key – for that very purpose.

So I ordered a bunch of 146G drives for the hosts, and I’m going to move criticial apps back to internal storage until I figure out what in the hell happened and how to fix it.  It might give me an opportunity to eval. some new FC Target toys I’ve been thinking about.

Who knows.  No more shark-jumping though. 

Well – for the last three weeks I’ve designed more NAS/iSCSI systems than I really want to admit to.

I understand what is so alluring about network storage, it’s cheap, plentiful, and it utilizes a technology that every data-center already has in place.

I have to admit that the customer I’m currently working on has done what none other has dared do until now. iSCSI on Symmetrix.

My first question of course is…..Why? Does it make sense to spend millions on a storage array only to refuse to spend $25k on a couple of switches? The only time I could see this as being even remotely useful would be a situation where you have something along the lines of a giant linux distributed computing cluster, though for simplicity’s sake I would probably opt for NFS over iSCSI.

Customer of course asked me what I thought – and I bit my tongue so hard I was afraid he’d caught that I wasn’t sold on the idea. I gave them the politic answer, of course being that “It’s a great, inexpensive way to connect a large number of hosts to a single pool of storage.” which of course neither answered his question nor committed me to an opinion.

Truthfully – I’ve seen it work a few times now, and with a few exceptions about powerpath and different subnets, it seems fairly robust. (Powerpath for windows works out of the box, PowerPath for linux requires that each interface be in a different subnet, which complicates things)

The only thing I can suggest if you’re planning an iSCSI SAN, is that at the very minimum you want to create a separate VLAN for the iSCSI traffic or risk having issues with broadcast traffic interfering with production traffic. Ideally you’ll create a separate physical storage network and n’ere the twain shall meet.

1. iSCSI is a viable alternative to FC for Small infrastructures.

2. I learned that no matter how well prepared for an install you are, the techie-gods will always throw curve-balls at you.

3. I’ve learned that Linux and PowerPath requires that multiple iSCSI HBA’s in a single host are not supported.
(Author’s note – this is not entirely true – see comment #3 below)

4. I’ve learned that seeing mice (yes, plural) running around a datacenter while you’re crawling around on the floor running cables is creepy.

Nuff said. I just got off 20 almost-straight hours in a row (I napped between 3am and 6am this morning) doing what should have been a very very simple install.

Needless to say it wasn’t. I’m going to bed.

-J