50Micron.com

Hi there.  Remember me?  I’m your friendly (not) neighborhood (probably not) storage blogger (ok, well sometimes)

So the situation keeps coming up, and it’s worthy of a post, so here I am.  You’ve all (all six of you) probably gathered that I’m not here to promote anything.  I don’t play favorites, and I certainly don’t get any money from the stupid ads I placed on the sidebar… (why are those still there?)  Hell I’m well aware that I am sorely lacking as a writer even.

Tonight’s topic:

Storage Security

Why are companies locking the storage admins out of the hosts?

Why for the love of pete?  I have a customer where the storage-admin’s job stops at the connection to the server, for ‘security reasons’

It’s a useless endeavor, it doesn’t gain you ANYTHING as far as security goes, and in fact *WILL* end up costing you more than you would ever have dreamed of saving.

it also makes your storage admins feel untrusted and unappreciated… (but employers don’t care about that so much these days.)

So, in a nutshell, I will list the three common reasons for locking the storage people out of the server environments, and why doing so is a complete waste:

Scenarios

• My computers have sensitive information on them that the storage people shouldn’t have access to.

If you trust someone to manage your storage environment you trust them with your data.  I can name two different ways off the top of my head that a storage admin could gain access to data without ever going NEAR the server, either physically or over the network.  And one of those would be COMPLETELY UNTRACEABLE.

Long story short, the storage admin has access to the data.  Just get used to that fact and stop making up ways to make their lives more difficult.

If you have doubts about the people you’re hiring, look at your hiring practices.

• The storage admin could inadvertently crash the host.

Well gee.  Anyone with access to the power cord could do that.  Again, can think of at LEAST two different ways a storage admin could do that without even trying, and that happen on a daily basis.  (Remove device masking, remove zones)  Again – you’re fixing the chicken-coop with the fox inside.

Try trusting the people you hire to do what you pay them to do.

• The storage admin doesn’t require access.

Well, this is kind of a generalization.  Many companies practice a “if your job doesn’t directly relate to the server you aren’t granted access to it.  If troubleshooting only extended to the point where the server connected to the SAN the above would be a true statement.  But as with most systems, there are inter-relationships that are crucial.  Multi-path software, HBA management software, Drivers/Firmware, *ALL* are a part of the storage environment.

And the bottom line is this:  Storage touches EVERYTHING.

If, like most sane companies, backup is included in the storage job, that’s 100% everything, otherwise there are SOME occasions where non-SAN attached hosts don’t require storage-admin access.

Troubleshooting in an environment where the storage admin’s access ends at the HBA connection can take HOURS longer than it would normally take, and requires at least twice the manpower.

Storage doesn’t stop at the physical layer.  Storage management software counts!

My scenario – Here’s why giving your storage administrator access to the servers *WILL* save you money.

It’s 4:15 on a friday afternoon. The dual-port PCI-e HBA you put into the server (to save money and slots which are tight in 1U servers) has failed. Not the port (which, granted, is infinitely more likely) but the chip itself. The SAN storage for the host is down.

As the storage admin, I got a page when the switch ports went dark. Assuming the storage environment is managed properly, I instantly know what host is experiencing the problem. (it’s also safe to assume that the host owner knows because his disks are MISSING)

Now as the storage admin, I’ve tested the connections, the switch ports and I’ve narrowed it down to an HBA issue. The host needs to be shut down (assuming it’s not Windows and blue-screened at the first sign of trouble)

Now if I have to coordinate the reboot, the installation of the new HBA’s, flash up-to-date firmware, pull WWPN’s, rezone, remask and reboot the host again, we’re talking about time. Maybe not much, maybe the host admin is on the ball, and maybe if you’re clever you can zone/mask before the initial boot, but you still need to flash firmware to stay within supportability and not risk further problems.

I’ve done this. If I’m doing it myself the system is back up by now, and the only thing i need the application owner to do is validate the app is functioning correctly.

If you don’t have access but are sitting in the same room with the person it’s still fairly simple but takes a little longer, though not much.

So let’s hope the failure happens during business hours—If it’s after hours, you’ve got two people driving in instead of one. Hours of downtime, total, that is, if you’re lucky enough to be able to get ahold of the host admin.

Now this came about because I had an outage happen. A VMWare lun disappeared and the owners of the “secure” vmware environment were nowhere to be found. (on what planet is it ok for an IT person to not respond to a page?)

Myself and the owners of the “unsecure” vmware environment sat around for a while twiddling our thumbs before the decision was made that the host owner wasn’t going to get back to us and the management decision was made to leave it for the night.

That’s a whole night this host will be down because the people who were there didn’t have the information needed to finish fixing the problem.

I’ve said it before, I’ll say it again.  If you don’t trust the people you hire, maybe who has access to what isn’t your primary problem.

It really does seem to be the question…  the sad part is how many people I talk to in my travels don’t really understand what cloud even is, let alone what the pros and cons are of moving your applications into it.

Background – a company is considering moving probably 3,000-5,000+ users to gmail as a ‘corporate’ email system…  They are running exchange currently…

Apparently, they don’t read the news and have missed out on the multiple spectacular failures of services like Google, Amazon and the like.

Cloud services are GREAT if you are running a small business, don’t want to / can’t afford an IT budget, or just plain don’t want to deal with it.

If you’re a billion dollar corporation with a multi-million dollar IT infrastructure already in place.  Outsourcing email seems a bit…odd.

Granted, if you are this company, you are obviously going to get the top-of-the-line service, dedicated support personel, etc.  You’re also buying plausible deniability should data-loss put you in jeopardy under subpoena. (While “I disposed of the data” is bad, “The company I was outsourcing to lost it” is not as bad.)

“Honest your honor, we had the emails but Google deleted them by accident.”

*DISCLAIMER – I’m not implying that google would ever do something like this on purpose, using them as a generic, like Xerox.

** It’s Google’s fault…they’re big enough to have become the verb.

***Does anyone actually own a Xerox branded machine anymore?

So if you’re SuperMegaCorp, LLC…you pay for the real service.  You get dedicated support staff, a private line to call, etc.  But to be honest, you might as well keep it in house because hey, you already have the staff, the datacenter, the VMWare farm, etc.  At that point you’re talking a few dollars in licensing and you’ve got email address for your thousands of employees for pennies each.  (Ok, yes, add in replication, backup, etc and it gets a bit higher, but the point is you’ve already comoditized it. (is too a word))

But think about it this way.  The company you’re contracting too has to pay for the same things *YOU* have to pay for.  *PLUS* they have to make enough of a profit to keep their shareholders off their back.  They do get a bit of a discount for bulk licensing, hardware, etc…

But what you GET for hosting it in house is immeasurable.  You get control.

At my last gig I heard the following phrase over and over again.  “I want one neck to choke.” (Oddly enough it was the argument given for moving AWAY from their previously preferred vendor, but you get the idea.)

When the email admin works for you, you have one neck to choke.  You get immediate results. Or you get the pleasure of firing someone.  (Can be fun in the right circumstances, ask The Donald.)

Now say you hosted with Amazon, just for grins.

Not only are your hosts down, potentially THOUSANDS of other hosts are down as well.  Now while we would like to believe they have a thousand techs on staff to give each customer equal time…let’s face it.  it’s not going to happen.  They  have, EXTREMELY generously, 10 technicians per thousand customers.  The techs will bring hosts up as soon as they can…

In an egalitarian society, odds are quite simply about 1000:1 against your site being the first one brought up…  990:1 against it being the second, etc.  See where I’m getting?  Eventually they’ll get around to it, but unless they figured out time travel and can loop back and do them all at the same point in time…you’re out of luck.  Yes, you’ve probably got a 99.999% uptime guarantee…but read the small print of your contract…  Their liability to you cannot exceed the cost of the hosting, if that, or some similiar legalease that limits their liability for downtime and, god forbid, data loss.

But this is not an egalitarian society…  Pure capitalism and “he who has the most gold gets their email back first.” If you’re with Amazon, well they host some PRETTY big sites…including their own.  Netflix comes to mind.  So in a downtime event if it comes down to bringing Joe the Plumber’s CRM app or Netflix’s east-coast streaming…which one do you think is going to get priority?

Right.

I have one neck to choke…  50Micron is hosted by Catbytes… the company that I do my consulting through.  Reason being that I maintain the lab anyway for “play” (officially: self-education and training) purposes, it’s easy for me to spin up an extra VM and put Exchange on it, a couple of CentOS Mailscanners, a few webservers, etc, even off-site replication of backups over a 10MBit link to a “DR” site (that happens to be in my basement)  (If someone wants to donate another CX3-20i or a couple of FCIP bridges I’ll have block-level replication. )

When Amazon EC2 had their issues, suspiciously I had a pretty major crash as well… (As did the customer I was working for at the time, don’t get me started on my paranoid theories.)

But when my stuff breaks… It’s my fault, it’s my responsibility, and *I* am the only one in line.  If I had hosted with Google or Amazon I might have been down for weeks…

I was back up in about 2 hours.  The time it took me to cycle the environment remotely.

Yes…building an IT infrastructure if you already have one can be pricey..  Paying someone else for hosting when you already HAVE an IT infrastructure just plain doesn’t make sense.

P.S. The funniest part is I’m now hosting about a half-dozen servers for friends/family (not free, I’m ugly, not stupid; and co-lo cages are NOT cheap) and about 40-50 websites that I’ve gotten via friends and word-of-mouth…

Of course my guarantee is as follows:

“Best effort, and you have to realize I have a day job that by it’s very nature comes first.” 

Sitting here running RDF create scripts for a data push this weekend and going over the days events in my head.

One of the things you get as a consultant is the ability to get a glimpse of the political machinations of many different companies and to get a first-hand view of what does and doesn’t work.

One thing I’ve seen is about a million different attempts at integrating storage into various systems departments.  It never works.  It always ends up with departmental pissing contests over who owns what, and usually results in a company or orginization buying more storage than they need to in order to pacify the different warring factions.

Storage belongs by itself. Pure and simple, the only way I’ve ever seen it work storage is a department in and of itself, with it’s own staff, it’s own budget, and a little autonomy and freedom to make decisions, and to act with the peace of mind that you’re not having to work around someone else’s changes.

The main reason for this is that server people don’t have the time to understand the dynamics of a truly heterogenious storage environment.  Network people understand firewalls and routing (something that *STILL* puzzles me to a certain extent), etc.

A good storage person knows the basics of as many operating systems as they can.

For instance – the current environment I’m working in has the following systems:

• AIX
• Mainframe
• VMS
• AS/400
• Windows
• Linux
• VMWare

A good storage person knows the gotchas of each server, but may not know even how to log into the system.

For instance – for each of the systems listed above:

• AIX – mount the pseudo device powerpath creates (hdiskpowerX).  AIX is sensitive to D_ID changes (Switch port changes) but if you’re using the LVM there are no real worries, just have to be careful.
• Mainframe – Three words – Long Wave SFP’s
• VMS – Is actually sensitive to the SYMDEV number.  if you’re doing a data migration you have to move the data to the same SYMDEV number.
• AS400 – Boot from SAN using a Load Source Emulator – use the serial cable included with it to configure the boot device.  The boot device has to be on a separate port than the data devices.  Make sure Emulation is set correctly.
• Windows – Dynamic disks cause hell with replication and TimeFinder – don’t use them.
• Linux – make sure you use disk/partition labels so you can avoid issues if the LUN order changes.
• VMWare – SPC2 bit needs to be set on FA’s for DRS/HA Clustered hosts.  Best bet is to do this using Symmask to avoid conflicts with other hosts sharing these ports.

A good storage department would include:

Tier-1 (Symmetrix) expert

Tier-2 (Clariion) expert

Backup person

NAS person

just my thoughts.

Brocade bought SBS.

I don’t know how many of you happen to have looked at the resume I had posted – but I spent a couple of years at Strategic Business Systems (www.sbsplanet.com).

I’m not sure what Brocade is hoping to get out of this.  SBS doesn’t do sales, and doesn’t even really have any influence in the buying process.

SBS has been a pretty successful company – grown by leaps and bounds.  I would never go back to them because they wield their non-compete agreement like a battle-axe and use every opportunity as a chance to hook someone in.

The real problem is that Brocade as a switch manufacturer is on it’s way out.  From a 90% install base they really have nowhere to go but down, and Cisco is gaining very quickly.

I’m not a big fan of Brocade.  I have a brocade switch in my home SAN not because of any preference, but because they are cheap on Ebay.   Their ASIC’s are slow and their licensing is oppressive.

Does this make sense to anyone?

I guess this is safe now.

http://www.washingtonpost.com/wp-dyn/content/article/2007/08/01/AR2007080102602.html?nav=rss_politics/congress

I guess now that I don’t work for them anymore, I can at least tell people who I worked for.  I haven’t seen the agreement they will want me to sign in order to receive my severance, but I’m going to assume it’s the usual non-disclosure.  And maybe even something saying that I won’t say anything negative about the company flounder….

Truthfully, I don’t need to disclose company information, most of it is going to be public record in very short order.

I always said this company was not built to last.  I hated to be right though.  It’s hard to accept the fact that you spent the last 18 months building something that is going to get tossed out the window like a rancid hamburger.

Ok, maybe that wasn’t the right visual.

As to the question of ethical issues, i.e. whether or not the non-profit status of the company was  abused - I’m not in a position to speculate (publicly) on that right now.  Some things I know and some things I don’t…