50Micron.com

Security is a good thing….until it isn’t.

Security isn’t a good then when it interferes needlessly with productivity.  By needlessly I mean to say when you don’t get the security you’re looking for but instead make it harder for your people to do their job than needs to be.

A few examples:

1.   Company “A” hires consultants to perform day-to-day tasks.  Company “A” then refuses to give them access to the troubleshooting tools and software downloads they are supposed to be supporting.

2.   Company “B” decides that it’s employees can’t be trusted.  (If you can’t trust an employee, why are they an employee?)  Company “B” then decides to lock down PC workstations so that *NO* software can be installed or removed by said employee.  Company “B” instructs their helpdesk to ignore all requests for installation of needed software.

3.   Company “C” requires an contractor to be on-call for 24×7 support.  Company “C” refuses to grant said contractor remote access to support the equipment he’s on-call to support, forcing a 45 minute drive in the event of an emergency.  Company “C” then reams the contractor for not being timely in his/her support.

4.   (My Favourite)  Company “D” gets *VERY* creative with Windows Group Policies on a workstation, rendering said workstation a paperweight.  Company “D” neglects to block access to the system BIOS and allows booting from USB only to allow any user to introduce any unlocked/unguarded operating system in the world into their environment by virtue of a thumbdrive.

In my career, I’ve been said employee/contractor in every one of these instances.

(Just an aside -  my favorite gotcha came from watching a help-desk guy come in and disable the USB ports in the bios of a system only to be rudely reminded that the keyboard and mouse are USB (and that they don’t make PS2 connections for them any longer))

My point is this:  If you’re going to implement security make sure it’s effective security that also allows your employees to do their jobs.

If it’s not effective security – IE going to show a security benefit (that benefit being a quantifiable improvement in the security of your data or the stability of your environment) don’t bother with it – you do nothing but alienate the people you hire to work for you and make them want to go elsewhere.

Contrary to popular belief, there are still elsewheres to go.

So I had a failure – Tuesday night last week, which caused me (forced me, really) to write this post:

I Miss the Good Old Days

Now in the grand scheme of things, it was probably a bit snarky, but at 4am I think I legally am not responsible for my actions.

But the bottom line is on Tuesday at about 5pm I called to open a hardware case.  Hardware.  The high-school kid who answers the phone routes me into the Software group.  Correct me if I’m wrong, but that’s kindof the opposite of hardware.  (And no, if this kid was a college graduate, please god tell me what college he graduated from so i can forbid my son from going there)

2 hour wait for the call-back, 2 hours of trading email back and forth before I convince him this is a hardware case and to please route it to the hardware people.

He never does the transfer.

In the meantime, it’s about 2am and I go fix the bloody problem myself, restart my change scripts and all is happy.  I got home about 4 that morning after an 18 hour day.

Leap forward to Thursday night.  Same process, different array, same failure.  Now it’s 10pm and I call in and STRESS to the triage guy that this is a HARDWARE case.  He routes me to hardware.  75 minutes to call-back, 2 hours to fix the problem and step the script through to the end, plus I get handy knowledge like root-cause analysis and a set of steps to ensure it doesn’t happen again.

My point is this.  So many problem can be avoided if you simply LISTEN to the person who is calling in.  Assess their skill level and if someone asks to be transferred to a specific group (ESPECIALLY if he knows the actual name of the group he wants to be transferred to – means he’s done this before)

Long and the short of it was on Tuesday, a failed drive on the target array was failed when the script started.  This caused it to error out because it saw invalid tracks existing between the mirrors.

Same thing happened on Thursday, different drive, different symm, and out of 4,000 volumes it happened to hit a volume I was working with both times.

I should play the lottery…   You’ll know if I win too because my blackberry doesn’t work on the beach in Cozumel.

When the triage guys @ EMC actually listened to the person calling in the problem and directed the call appropriately. (Just had one I specifically asked for PSE and they routed me to software for some unknown reason, now, three hours later, it’s been re-routed to PSE)

When the support specialist working a call would stay through the end of the problem, and didn’t give you the “Sorry my shift is over, please explain your problem to a new guy for 45 minutes before you do anything else productive”

Follow-up. It’s now 0100 the next morning. I’ve been working on this problem for 8 hours straight. The software guy who was originally assigned went home without turning the call over to someone else. I’ve gotten not a single call since before 9pm.

This is not support people. This is the opposite of support. I’ve since fixed the problem myself, without the help of the SAC/PSE folks. The sad part is if I had done this 8 hours ago I could have been home eating my corned-beef / cabbage and drinking my Guiness.

Totally missed St. Patrick’s day, the only religious holiday I actually observe.

Not happy.

Ok – I see the surface benefits of a third party replication appliance, such as Recoverpoint. I even sat through a long discussion on it. I’m still totally on the fence but there are a few questions I need satisfactorally answered before I can recommend this to my customer.

I want to ask my readers, (all five of you) What’s your take on Recoverpoint or any appliance-based replication standard vs. array-based SRDF, Mirrorview/S, and the like?

There were a few points in the presentation that I had to cry foul.

“There is no appreciable latency in the appliance.”

Sorry, that doesn’t wash. Any time you put an intermediary device into a fibre path, you’re going to introduce latency. The real question (and answer) is “Is the latency introduced by said appliance mitigated by the compression gain.”

“Fully Synchronous Replication”

Synchronous replication exists when the IO is not acknowledged back to the host until the write is completed to disk on BOTH SIDES. Because of obvious latency issues this is not possible without A> Sufficient Bandwidth B> low-latency. Since the Recoverpoint product seems to respond to the source array as soon as the write is journaled, that means there is a potential for data-loss if the source building becomes a smoking hole in the ground. (Or even if something as simple as a regional power-failure happens, provided such a failure affects the circuit as well as the datacenter.)

Anyway – I really need to know – are my concerns off base or am I just being protectionist of the technologies I already know?

This is how NOT to run fibre-cables

Just FYI – it’s ok to run fibre cables under the floor.

It is, however, advised that you have a raised floor to run them under.

This is *NOT* ok.  (Yes, that is an APC Battery Backup that the cables run directly under….)

Nuff said.