50Micron.com

This is what happens when a computer-geek spends too much time in one hotel.

Do I really have to say anything to go along with this picture, or does that pretty much cover it?

I learned what “Host Isolation Response” was today.  Well I already knew what it was, but I learned that in a VMWare cluster, if you leave it at the default, then if the network goes away between the clustered hosts, the HOST then RESPONDS to this ISOLATION by shutting your entire environment down.

Oops.

Not that anyone would notice, but from 1:30 to 2:00pm EST the system was offline because I (ironically) unplugged the switch briefly to put a battery behind it.  Needless to say it’s better now, but it wasn’t quite the “momentary interruption” I had hoped for.

Not going to be long with this one, needless to say I’m prepping to start traveling again.  Not totally excited about it, but I hear Seattle is nice during the summer in much the same way Virginia isn’t, so at least there’s an upside.

It’s cold in this data center.

I went to stand behind the IBM XIV to stay warm.  I actually had to stand 5′ from the box to keep from being uncomfortably HOT.

Just an observation…

I think that every network security department operates under the following motto:

“If you can do you’re job, we’re not doing ours.”

The customer I’m with has instituted a massively long password scheme. Now I’m not going into details, because that would be telling and would give even the greenest of hackers enough to walk right in.

My point is this. If you make a password scheme overly complicated or tedious, what you’re going to find, and I say this as an absolute, that 20-40% of your users will WRITE THEIR PASSWORD DOWN AND PUT IT WITHIN REACH OF THEIR WORKSTATION.

Hello social engineering.

Here’s a note.

If youre hardware has a driver that requires you compile it before use, and that can’t be easily upgraded.

Your hardware isn’t ready for prime-time.

It’s why I like Emulex. You plug it in and it works. You upgrade the driver with rpm and it works.

You used to have great stuff. Not so much any longer.

Transitioning “dial-home” away from modem has got to be the crappiest idea there is.

When the network is down, remote support is down.

I’m sitting in a datacenter that was powered off this weekend (don’t ask) On recovering, one Symm failed to IML.

The network is still down.

I can’t even START troubleshooting the Symm until it’s back up.

Start of a craptastic weekend in my book.

Lesson Learned – keep the modems JUST IN CASE.

I would have much less problem with you putting Americans out of work by shipping our valuable support jobs to India if you could at least do us one favor:

At least find people capable of actually providing support.  Your rep, Anshu Kitchloo, seems to have tremendous difficulty understanding basic sentence structure (let alone technology) and of responding to a question without having to run every line of web-chat through an interpreter would be a tremendous plus.

Only then will I stop cringing at the suggestion that I open up a ticket.

Love, Me.

I’m going to do a giveaway.  I never do giveaways, but I have some collectable “State of Hawaii” starbucks cards in my desk drawer so it sounds like fun.  I also have nothing of substance to write about because I’m bored out of my skull.

Cards have $5 on them and will be awarded randomly to an entry / comment picked by random number generator on Friday, 9/10/10.  If I get a lot of entries I’ll throw in a few extra cards just for grins.

I’m looking for “You might be a redneck if…” ideas as they relate to IT and Technology.

A few of mine:

“Your IT department might be redneck if your Disaster Recovery solution involves building a new datacenter.”

“Your IT department might be redneck if you bought your servers on Ebay.”  (That’s me by the way)

“Your IT department might be redneck if your solution to too much cable under the floor is to raise the floor.”

“Your IT department might be redneck if you use tape for live data.”

Ok, 8 days.  Go.

In a true DR environment where Synchronous replication is used, it’s best to have two routes from source to target, or at the very least a switched route that can dynamically re-route in semi-real-time.

Everyone knows the story.  The link is up, everything is good, source ack’s a write to the host when the target acks it.  The link is down, replication is halted, source ack’s to the host when write is committed to cache on the source.

(Or, in this case, you have two optical routes but somehow managed to put it all through the same DWDM tray, which then failed, taking out both routes)

But i’ve seen it happen more often than not.  The “Bouncing” link.  Up, down, up down up down etc etc etc..

Very few storage systems handle that well.  Mostly because when the link is half-way there the system gets torn between the requirement (in synchronous replication) to acknowledge the link.

The good news is most host operating systems handle it wonderfully.  Sun records such events as “Retryable disk errors”, Windows and AIX I don’t think even report it.

Enter RedHat Linux, or in this case, RHEV.  RHEV uses a standard lvm2 volume group with virtual disks as logical volumes within the volume group.  Simple enough right?

Well what if you have disks from different disk subsystems?  What if  you have some mirrored and some not (the usual reason for that would be test/dev and production in the same environment. (Though putting dev/test and production in the same cluster is kinda nutty)

The situation I just saw was this.  4x 500G volumes, only ONE of them mirrored.   RHEV apparently put them all in the same volume group.

You *NEVER* put mirrored and non-mirrored volumes in the same volume group.  If for no other reason than the disk on the target array is USELESS without it’s partner disks.

In this case we had one disk out of 4 that was dropping on and off-line, some admin gets the idea to reboot the host – which of course attempts to close the volume group.  When it can’t flush those writes to disk the behavior gets a little unpredictable.  Most likely the shutdown will hang, causing some overzealous admin to go hit the power-switch…

Data loss ensues because there are cached-writes that haven’t been committed.

And they call me for help with it.  meanwhile, the freeware VMWare ESXi environment, that is also replicated, and that *I* have been pushing hard for enterprise-wide adoption of, blows right through the 36 hours of random problems with not even a sigh.

The problem with calling me for help with it, is I can just SMELL someone trying to blame the data-loss on EMC, and I want NOTHING to do with it.  So I tell them to open up a support ticket with RedHat.

Oops, they didn’t buy support.  Apparently when you throw in support the cost-benefit analysis vs. VMWare that makes it too expensive..

FML

I worked for 18 straight hours on Friday.

So if consulting firm A talks to consultant B about an engagement, and then decides to go with consultant C because he’s cheaper it’s totally understandable.

If consultant C then calls consultant B for help with that engagement, consulting firm A should expect to get a bill for the time.

I mean first off, its rude.

Secondly, none of us work for free.

Now I’m glad to help when i can, but when it goes from “i’ve got a few questions” to “give me the step-by-step on how to do this” I have to draw the line.

</rant>