50Micron.com

I just got the following job posting and it made me, literally, laugh out loud, spitting latte all over my laptop.

If your sales rep allows you to do something like this, it’s a fair bet that s/he hates you (or is planning to buy your company out of bankruptcy later).

“WANTED: VMWare 1-month resident to assist with new deployment/planning around 200VM’s and new Celerra NS480′s being purchased by client. Will probably end up primarily being VM’s using NFS on NS Celerra Replication will be enabled between (2) NS480′s.”

The key points are:

200VM’s

Celerra

**NFS**

Replicator

Ewww…..

Did I mention NFS?

Someone actually sold this?  Even if the customer comes to you direct and says “this is what I want…” the answer should be “In the interests of protecting you from yourself, I can’t allow you to do this.”

I don’t care how much the deal is worth.

This may be a more well-known reference than I earlier thought.

I grew up watching Happy-Days.  The show was great until the episode where Fonzi jumped the shark-tank.  After that it pretty much went down-hill quickly.

Hence the term “Jumped the shark” or “Jumping the shark” has come to mean any single event that marks the point where something degenerates into crap.

My VMWare NFS server jumped the shark this weekend.  It was hilarious.  I had a beautifully quiet afternoon on Friday, from about 14:30 on my blackberry was quiet.  Turns out that the NFS server that I use for storage experienced an unexplained (and apparently barely logged) kernel panic and rebooted.

In the process, the 6 adapters, in what I can only guess was a techno-square-dance, all switched places and lost their bonding configuration.

All went south, right in the middle of one of my busiest travel weeks as far as work goes.  So my wife, god bless her, earned her stripes this weekend as I walked her through ‘ifconfig eth0 10.1.1.10′ and ‘ping 10.1.1.254′ etc.  trying to figure out what happened.

Still don’t know.  But with everything down (including this site) my first priority was to get it all back online, troubleshoot later.  (When my desktop goes down I know why, I have an inquisitive 3 year old with a fetish for power-buttons), but the server power buttons are protected by a key – for that very purpose.

So I ordered a bunch of 146G drives for the hosts, and I’m going to move criticial apps back to internal storage until I figure out what in the hell happened and how to fix it.  It might give me an opportunity to eval. some new FC Target toys I’ve been thinking about.

Who knows.  No more shark-jumping though. 

A few have noticed the site was down for an extended period this week.  I learned a few things this week.

I set up my FC system and was so excited to get it moving that I neglected to adequately test my equipment.  I bought used equipment, with used drives, and put real data on them after a whopping 2 days of light testing.  I never stress-tested the drives, didn’t do any kind of exercizing of them to validate that they were worthy of production data.

I also neglected to functionally test the array.  While it did offer the ability to configure a hot-spare, I didn’t check to see if the hot-spare was functional before I moved data over to it.  (Seeing that it was configured was enough for me)

So what happened was this.  I was running on the system and all was well until a drive failed.  The hot-spare didn’t invoke on it’s own, and while one drive was in a failed state, the second drive failed.  Needless to say I lost half my luns and three of them were corrupted beyond repair. 

Luckily I’m one of the old hold-outs.  I have a tape backup system consisting of a Veritas 6.0 environment with an ATL tape library.  I was able to restore to within 48 hours of the failure using tape.

My *NEW* storage back-end of consists of a Dell 2650 with 5x 146G drives.  I installed CentOS5 with a 512GB NFS-mount partition and mounted them to my VMWare servers.  The most interesting part is I realized that by bonding the network interfaces I’m getting the same bandwidth I got out of the 2x 1Gig fibrechannel ports.

Not being a network guy though, does anyone have any suggestions for optimising NFS for storage applications?