50Micron.com

I would have much less problem with you putting Americans out of work by shipping our valuable support jobs to India if you could at least do us one favor:

At least find people capable of actually providing support.  Your rep, Anshu Kitchloo, seems to have tremendous difficulty understanding basic sentence structure (let alone technology) and of responding to a question without having to run every line of web-chat through an interpreter would be a tremendous plus.

Only then will I stop cringing at the suggestion that I open up a ticket.

Love, Me.

I have seen the future of data storage and I weep for it.

A few random rants at 2am after a data migration didn’t go because I’m not willing to kill a backup process that has a hold of my mount-point.

—-

Engineering seems to be a thing of the past.  We’ve graduated to the world of “that’s good enough for launch, we’ll fix the bugs in future releases” in spades.

I’m so tired of hearing people explain to me that a solution is “good enough” or compromising technically due to cost, when we all know that a badly engineered product will suck the life and profitability out of any company.

Most vendors, and EMC is included in this generalization, have taken to solving hardware engineering problems with software.  Enter the “appliance”.  Consumer grade hardware thrown in with crappy software designed in some third-world country without any thought to the long term failure rates on such combination.

Poor engineers make the mistake of assuming that an overly complex solution must be better than a simple one.  IBM GPFS, Sun Shared QFS, EMC Celerra MPFS are all examples of a psychotically overcomplicated solution to a very simple problem that can be solved with NFSv4 and a decent network back-end.

The more moving parts you throw at a problem, the more chances something stupid, or someone stupid, is going to foul up the mix.

And the more external vendors you buy your parts from the more your chances of having to deal with a problem that is of SOMEONE ELSE’s making, since you’ve given up control of your product and Quality is truly a thing of the past.  (You’ve also made your product *WAY* more expensive than it needs to be, because you’re not supporting the margin of everyone else between you and the supplier)

And if you buy the cheapest product to do a job, remember that it’s cheap for a reason.  IBM has been known to give up to 90% discounts on the XiV platform in order to compete with EMC.

90% – wow.  Reminds me of the usual assumption about guys and big trucks.

If you’ve got to give that much of a discount to convince people to buy your product, then it’s not much of a product now is it.

—-

I’m going to do a giveaway.  I never do giveaways, but I have some collectable “State of Hawaii” starbucks cards in my desk drawer so it sounds like fun.  I also have nothing of substance to write about because I’m bored out of my skull.

Cards have $5 on them and will be awarded randomly to an entry / comment picked by random number generator on Friday, 9/10/10.  If I get a lot of entries I’ll throw in a few extra cards just for grins.

I’m looking for “You might be a redneck if…” ideas as they relate to IT and Technology.

A few of mine:

“Your IT department might be redneck if your Disaster Recovery solution involves building a new datacenter.”

“Your IT department might be redneck if you bought your servers on Ebay.”  (That’s me by the way)

“Your IT department might be redneck if your solution to too much cable under the floor is to raise the floor.”

“Your IT department might be redneck if you use tape for live data.”

Ok, 8 days.  Go.

…

SEWP BOWL?  Really?  And who said the government doesn’t have a sense of humor.

(from http://www.sewp.nasa.gov)

It’s funny, but I always get this funny look when I talk about technology being alive, evolving, and maybe even slightly sentient.

This is a non-storage post, but I never claimed to write only about storage, just mostly.

We dumped cable TV today.  And it got me to thinking on how certain technologies just become obsolete after a while.

We did it because the only ones who watched TV in our house were our kids, and they only watched two channels.  Nickelodeon, and Nick Jr, and becaue Nick Jr was a part of the “premium” cable package we ended up spending $100 a month or more so that my 5 year old could learn Spanish from a diminutive latina with a talking backpack and an oddly-dressed monkey for a friend.

Now they get that same education from the same shows, but they get it from Netflix streaming through the Sony Playstation.  Net cost $16.99/month.

But the cost savings are only part of it.  My kids are no longer assaulted by endless advertising, commercials for shows that are NOT on their age level, and are no longer scheduling their lives around what shows come on at what times.

It really is an amazing freedom when you think about it.  And it definitely signifies a technology that is finally moving away from “the way it’s always been done” to ways that may, finally, make more sense and put the consumer in control.

In a world where bandwidth is cheaper and cheaper, it really surprises me that more people aren’t making this change.

In a true DR environment where Synchronous replication is used, it’s best to have two routes from source to target, or at the very least a switched route that can dynamically re-route in semi-real-time.

Everyone knows the story.  The link is up, everything is good, source ack’s a write to the host when the target acks it.  The link is down, replication is halted, source ack’s to the host when write is committed to cache on the source.

(Or, in this case, you have two optical routes but somehow managed to put it all through the same DWDM tray, which then failed, taking out both routes)

But i’ve seen it happen more often than not.  The “Bouncing” link.  Up, down, up down up down etc etc etc..

Very few storage systems handle that well.  Mostly because when the link is half-way there the system gets torn between the requirement (in synchronous replication) to acknowledge the link.

The good news is most host operating systems handle it wonderfully.  Sun records such events as “Retryable disk errors”, Windows and AIX I don’t think even report it.

Enter RedHat Linux, or in this case, RHEV.  RHEV uses a standard lvm2 volume group with virtual disks as logical volumes within the volume group.  Simple enough right?

Well what if you have disks from different disk subsystems?  What if  you have some mirrored and some not (the usual reason for that would be test/dev and production in the same environment. (Though putting dev/test and production in the same cluster is kinda nutty)

The situation I just saw was this.  4x 500G volumes, only ONE of them mirrored.   RHEV apparently put them all in the same volume group.

You *NEVER* put mirrored and non-mirrored volumes in the same volume group.  If for no other reason than the disk on the target array is USELESS without it’s partner disks.

In this case we had one disk out of 4 that was dropping on and off-line, some admin gets the idea to reboot the host – which of course attempts to close the volume group.  When it can’t flush those writes to disk the behavior gets a little unpredictable.  Most likely the shutdown will hang, causing some overzealous admin to go hit the power-switch…

Data loss ensues because there are cached-writes that haven’t been committed.

And they call me for help with it.  meanwhile, the freeware VMWare ESXi environment, that is also replicated, and that *I* have been pushing hard for enterprise-wide adoption of, blows right through the 36 hours of random problems with not even a sigh.

The problem with calling me for help with it, is I can just SMELL someone trying to blame the data-loss on EMC, and I want NOTHING to do with it.  So I tell them to open up a support ticket with RedHat.

Oops, they didn’t buy support.  Apparently when you throw in support the cost-benefit analysis vs. VMWare that makes it too expensive..

FML

I worked for 18 straight hours on Friday.

I’m a bit of a stick in the mud when it comes to change.

I dislike it.

A *LOT*

So what did I do…gave up my years-long love-affair with Blackberry and bought a new Android-based phone.

I’ve sort of had it spelled out to me recently that I resist change even when it makes sense.  Fearing the unknown and all that.

So this is my gesture. 

I managed to hit Sprint.Com during one of the 18 whole hours they had the HTC EVO in stock.  My son wanted to spend his end-of-year-report-card-money on one and in playing with it in the store, I had to admit I was pretty impressed.

First off the screen.  The EVO sports a 4.3″ diag. screen which puts the rest of the droid-based phones to shame.  (Even the new HTC Droid “Incredible” isn’t so incredible at a paltry 3.7″)

Setup was easy, but not as easy as it could have been.  After several unsuccessful attempts at activating online I was forced to call in.  The unsuccessful activations were due to the fact that the phones were tied to the number they were purchased under, and I was trying to cross-activate them.

So far so good.  The only notible exception is the inability to sync my windows “Notes” using exchange active-sync.  I depend on those pretty heavily so will have to find a work-around for it. 

I’ll update when i actually get a chance to use this in a “work” situation. 

So if consulting firm A talks to consultant B about an engagement, and then decides to go with consultant C because he’s cheaper it’s totally understandable.

If consultant C then calls consultant B for help with that engagement, consulting firm A should expect to get a bill for the time.

I mean first off, its rude.

Secondly, none of us work for free.

Now I’m glad to help when i can, but when it goes from “i’ve got a few questions” to “give me the step-by-step on how to do this” I have to draw the line.

</rant>

Every time I turn around it seems I seem to be running into the same question.

Is it better to be multi-vendor or single source?

Well the easy answer to that is, it depends.  Different vendors do things differently, work better/worse with some hardware, etc.

The arguments in favor of a single-vendor solution is easy.  Cost, Simplicity, Management, Interoperability.

Even if you’re buying a more expensive solution, there can STILL be major cost savings.

First, in staffing.  When you maintain multiple vendors, you have to maintain support-staff knowledgable for each vendor.

If you’ve got a storage team that consists of 5 people, and two of them work almost exclusively on Veritas Netbackup.  You *MIGHT* be lucky if you get one subject matter expert capable of doing Tier1 (IE Symmetrix) one for Tier2 (Clariion) and one for NAS (Celerra) .

But throw in HDS, IBM DSxxxx, XiV, IBM GPFS, IBM HPSS, NetApp, SONAS, Sun StorEdge, etc. etc. etc.  And what do you have?

You either have an overworked staff (and as i’ve discussed, union protected salaried federal employees aren’t known for 70 hour weeks) or stuff just plain doesn’t get done.

If you don’t spend the money on staffing, you *WILL* spend the money in support and professional services.  Now support is one thing.  If my XiV or Symm or whatever loses a harddrive, I expect the vendor to own that problem and fix it.

They will *NOT* however send people out to help with day-to-day provisioning without a pretty hefty P.O. associated with it.

And the last reason for single-vendor options is simple.  I want stuff that is going to work together.  Now yes, functionality costs, but one of the things I like about EMC is that when it comes down to it, it *ALL* works together.  I can move data from Symm to Clariion or vice-versa using SanCopy, I can migrate fileservers to celerra and within storage tiers as needed.

There is nothing worse than needing to expand one storage system by 20TB and having the storage somewhere else, but unusable.  It means you’re wasting money buying storage you already have.  (Especially when your purchase cycle is 4-6 months on average.)

Not a happy thing to explain to the boss.

“Yes we have 80TB of Clariion avaialble, but the IBM DS4800 is running short so I need to spend an extra $100k on disks.”

“Yes, I know this isn’t budgeted, but the data grew faster than we’d expected.”

(Of course, you can span filesystems across arrays, as long as it’s not replicated data, because you can’t get a consistent split when half of your extents are on one array and half on another)

Just a quick question as it’s *WAY* past my bedtime. 

Has anyone else ever heard of a customer that a sales-person or tech would quit rather than set foot in?

I’m remembering my days back at Disney in California.  (over 7 years ago, so I’m assuming the statute of limitations has expired on that one)

I got called up for a 6-week scripting engagement, and 18 months later I had to actually quit the consulting firm I was working for and move across the country to get away from them.  (Happiest place on earth?–Not if you’re in their IT dept.)

These words actually came from one of their “cast members”:

“Once you work for the mouse, you ALWAYS work for the mouse.”

When I told them i was moving to Virginia/DC to do goverment work, the response was even more ominous…

“The sun never sets on the ears…you can’t move far enough away.”

In the end, the experience I got at Disney (Scripting TimeFinder to back up AIX/DB2/SAP) was invaluable and one of the guys there basically taught me more about kornshell scripting than I think I’ve learned from any single source since then.

But the environment was toxic.  I’m curious as to whether anyone else has stories like these or if I’m the only one who runs into them?