50Micron.com

I started this blog in September, 2006. I was working as a storage administrator for “Loan To Learn” a small student loan company in Sterling, Virginia. It was an amazing challenge. We built an enterprise environment from the ground up in amazing time. Overcoming odds, battling beasts, etc. It was great fun.

It’s a pity that the pilot of that particular airliner didn’t see the mountain looming and smacked straight into the side of it without blinking..

The blog was a great place to vent, to talk about the discoveries and problems in running a day-to-day environment. (Something I had only done once before, in 1997 when I was just starting in the industry)

I received a tremendous amount of help from all of you, and I’m hoping the information contained within these “pages” was/is helpful.

Long story short, I think I’ve come to the point where I think this blog has outlived it’s usefulness, at least as a day-to-day diary, both to me and everyone around me. It’s been difficult for me of late because I don’t feel it’s fair to blog about my current clients. They haven’t consented to such public favor (or ridicule, as the case sometimes is.)

My last engagement, government agency, hurt my career.  Spending 2+ years working with a group who have an abhorrent fear of new technologies, or in fact doing ANYTHING new, kept me well behind the curve, technically.  It’s only through sheer luck (and a few people on the west coast that still know my name) that I got this engagement where I actually got my hands on my first VMAX…only one, a year later than just about everyone else I know.  Talked them into their first NAS device while I was there (Celerra)

Sadly, these days I’m still not as in the ‘cutting edge’ mix of things, but I’m diving in wherever I can, and I still have my home projects to keep my skills up.  I’m trying to dive into the design work wherever I can…  But all in all, I don’t have a lot of time to play with the newest and bestest things, it’s a lot of day-to-day crap.

That and I’m realizing that while I’m a good engineer…I kinda suck as a writer.

I think the biggest thing is…I just don’t have that much to rant about anymore..  That’s good right?

Keep in touch, I can be reached at jg (at) 50micron dot com or via twitter @50micron

Ok, I can say it in a sentence.

A contractor is someone you hire to do a job, a consultant is someone you hire to fix a problem.

I’ve done both, but in the last 8 years I’ve been primarily a “Consultant.”  My job is to fix whatever perceived problem.

Some companies might have a backup problem.  You streamline their process and reduce redundancy, and poof, backup problem solved.

Some companies might have a replication problem.  You analyze their environment then recommend and implement changes.

Some companies have a data management problem.  You simplify storage, identify Tiers, move storage to where it best suits the orginization.  (IE Static Image data doesn’t belong on Tier-1 Symmetrix)

Some companies have a culture problem.

Here I got nothing.

But when your culture problem interferes with the consulting that you are asking me to do, I bristle.

When your culture problem causes me to wait 8 months of a 1 year contract before I’m given the tools to do my job, I boil.

When your culture problem is making me feel like I should take up golf.  I start looking at dice.com for something better.  (I hate golf)

Maybe it’s just that I *LOVE* what I do.

I do.  I love what I do.  I get paid to do what I love.  Which is why I can’t stand seeing people who are either A> There to collect a paycheck and maybe if they’re lucky a pension. or B> try and create their little empire so they can brag to colleagues about how much money they have to spend this year on nothing.

It boggles the mind.

Anil Gupta had an interesting post on his blog that I found interesting:

Anil’s Blog Post

Included in his post was a peice of marketing fluff from Craig Nunes, VP Marketing at 3Par, about how great their arrays are.

Of course they are…especially if you are VP of their marketing department.  It’s there job to promote, therefore from an engineering standpoint anything they say should be taken with a grain of salt.

Truthfully I can’t say one way or another, because looking across a CoLo cage at one a few weeks ago was actually the first time I’ve ever SEEN a 3Par array.

It was pretty.

I tend to follow one rule when it comes to storage arrays:

The effectiveness of an array is inversely proportional to the count and speed of the blinking lights on the exterior of the array.

Any time I see lots of blinking lights I think to myself – what are those blinking lights covering for?

I offer as proof:

1.  Symmetrix – black box.  No blinking lights, very few lights at all on the exterior actually, and the ones on the interior are mostly covered up even when the doors are open.

2. Clariion – it’s been interesting to see the maturation of the Clariion product over the years.  The old FC (and IP) series had the typical blinking green lights on the outside, plus the amber lights for “Alert” statuses.  As the line has matured they’ve gone further and further away from the lights.  Note that the AX, being a lower-tier than the CX, has more lights on it.

3. Centerra – other than the activity lights on the network switch in the back of the cabinet, I’m hard pressed to know if these are powered on.

I’d offer more examples, but it’s 6pm and time for me to head home.

/Jg

One of the things that happens when you run a storage/technology blog, is that you will regularly get emails from PR firms with press-releases hoping that you’ll write a post touting their product or at least link back to their site to improve their ratings in the serach engines.

My problem is, that while I’m an avid storage/technology blogger, I have an science/engineering background.  What this translates to is this.

If you haven’t seen it happen, it’s a hypothesis.  If you haven’t put the functionality to the test, don’t assume it’s real.  And if it sounds too good to be true, it probably is.

My wife, having spent many years in marketing, was quick to explain to me how marketing really works.  “Play up the good, discount the bad, and forget the mention the fatal flaws.”

So when a new storage hot-shot approached me to write an article about their new hardware, and gave me the specs as they saw them, i refused, flat out.

If you tell me your product does 100,000+ IOPS that’s fine.  I’ve not seen it so I have only your numbers to go on, and for all I know your numbers were invented over Bacardi’s at the local pub.  Send me a demo and I’ll be happy to try it out and report the numbers.  Just beware it better do what you say it’s going to because I will report the real numbers, good or bad.

I have a few years in R&D under my belt, I know how testing is done.  I also know how any test can and usually is skewed to show off the strengths of a product.

When I worked for another non-emc storage vendor and had to tell them that their new array wasn’t capable of pushing more than 13MB/sec, they asked me not to put that particular fact in my report.  When I refused, i was suddenly “laid off.”  (of course a short time later the rest of the company was, so that may not have been quite the cause/effect I like to intimate.)

That’s marketing…sadly it’s not science.  And the one truth is marketing people have difficulty tolerating engineering people.

I’m surpised my wife puts up with me actually.

This article in e-week.com says it all. 

Storage has always been good to me.  In 2001 I got laid off while working for MTI (right before they imploded and were “reborn” as an EMC reseller – with a huge influx of capital from EMC) but was back in business almost immediately.  (Read the article in ComputerWorld)

Storage has always given me the flexibility to a> pick my jobs, and b> stay employed despite the market.

While EMC can be a bit of a “fair-weather friend” when it comes to employment, there has always been, and will always be a strong demand for people who understand storage and SAN concepts.  Someone who knows the internals of EMC, HDS, or any of the big players in the Tier-1 storage world will always have a place to go, and will always have a big advantage in the negotiations.

Once your name is out there, it’s usually not hard to keep it out there.  Not quite a year later, I’m still getting calls from ex-employers and old co-workers asking if I’d like to go back to it.

Most of the engineers I’ve known who have been laid off by EMC get hired by the service partners who have built up a whole industry around supplying EMC with the labor and expertise they need.  Most of them are, like me, ex-EMC’ers, some are just naturally bright and quick to pick up.  The partners pay well because of their lower overhead, and raise the bar for EMC when the almost systematic reaction to the lay-offs occurs.  The “re-hire” usually proves too expensive for EMC and they end up with either a more expensive or lower-quality brand of employee, sometimes forcing even more business to the partner, etc.

Anyway, like I said, it’s a good time to be us.  Just make sure you keep up the knowledge.  Read all you can, try new things, etc.  The hardest part about going back to EMC after being away from their hardware for a year was playing “catch-up” on the technology improvements.

I had to re-write this one, I didn’t like my first run at it.

As one part of my duties, I manage the companies one and only fileserver.  We run on a fairly simple rule, if it’s a file share it goes on the Celerra.  There are *NO* windows fileservers in our environment.

Setting this up was fun.  We run CAVA (Celerra Anti Virus Agent) to protect against those nasty windows virii that are constantly running around (my Linux desktop at work is immune) and the Celerra fileserver (It’s an NS500G with a Clariion back-end for anyone who cares) has been wildly successful.  Only one real problem in the six months I’ve had it up and running, and that was when one of our applications, “Argo” (a credit decisioning engine of the sort used by banks – www.argodata.com) dumped 150 gig worth of trace files into the netowrk file system and the filesystem in particular tried to start a second auto-expansion before the first one had completed.

Now the fun begins.  With the auto-expansion set I’ve been able to take a more hands-off approach to managing it.  I probably should have kept a closer eye.

Did you know that in EMC Control Center there are more NAS management tools for Network Appliance (www.netapp.com) then there are for their own NAS front-end?  The one thing I needed was a report of when my storage pool got to within 10% so I could plan on either allocating more LUNS to the NAS or to purchase more.  (The cause of my last post)

Such a report apparently doesn’t exist.  So when I logged in today for my weekly check to find that these greedy people I work with have in three months managed to chew through 2 terabytes of storage.

So now I have to scramble to come up with more “temporary” storage.  (It’s been my experience that nothing is more permanent than a temporary storage assignment.)  I found some – but really hope that the Exchange environment doesn’t grow before we get the new Symmetrix spun up, because it no longer has anywhere to grow to.

 

http://www.mcdata.com/about/news/releases/2006/0808c.html

Given the choice between brocade and McData I’ll usually choose McData.  Now that I don’t have a choice, I think I’ll stick with Cisco.

Truthfully Brocade needed McData.  McData has a solid director class switch, one that will withstand the tests of time and the rigors of a large-scale production datacenter.  Brocade never really has.  What they’ve had is a collection of blade-mounted edge switches linked together with a pseudo-fabric back-end.  Either way, it was a joke. 

I remember specifically a series of incidents down at a DoD client in Norfolk, VA.  Huge fabric.  4 64 port Brocade DS-12000B “directors”  (EMC wouldn’t even give it an “ED” designation usually reserved for “Enterprise Director” and instead gave it a “DS” designation for “Departmental Switch”.

Out of the four enclosures that were sent, one of them was dead on it’s face, and the rest of them underwent many many failures, blades dropping off-line, losing power, the whole nine yards.

The funny part is, I remember seeing this switch when it was in it’s infancy at the company’s headquarters in northern california.  It just didn’t look right.  The only thing I could see that they did correctly was to at least mount the swtich boards vertically, which allows for better heat disipation.

The one thing I really have to say for Brocade is that their command line interface is much easier to deal with that Mcdata’s ever was.  With common-sense command structures and a consistent format it was easy to get into a routine while doing zoning or creating aliases or whatnot.  I was never comfortable with Mcdata’s command line, mostly because EMC discouraged it’s use at just about every turn.  (Cisco’s command line is a bit weird, but if you’ve used IOS you’ll fall right in with it) 

That’s the great part about the Cisco, is that as an IT Manager, you can actually put the SAN equipment into the realm of the Network gurus without too much additional training.  (Save for zoning and theory and the like)  (That plus the fact that once the MDS switches were released, I literally don’t think I went on another engagement that was a new McData install)

I would like to invite comments on this – I’m curious as to what other people see as coming out of BrocData.  More speed?  Better reliability?  10G?   Or just a little more moderate competition for Cisco?

Jesse