50Micron.com

It really does seem to be the question…  the sad part is how many people I talk to in my travels don’t really understand what cloud even is, let alone what the pros and cons are of moving your applications into it.

Background – a company is considering moving probably 3,000-5,000+ users to gmail as a ‘corporate’ email system…  They are running exchange currently…

Apparently, they don’t read the news and have missed out on the multiple spectacular failures of services like Google, Amazon and the like.

Cloud services are GREAT if you are running a small business, don’t want to / can’t afford an IT budget, or just plain don’t want to deal with it.

If you’re a billion dollar corporation with a multi-million dollar IT infrastructure already in place.  Outsourcing email seems a bit…odd.

Granted, if you are this company, you are obviously going to get the top-of-the-line service, dedicated support personel, etc.  You’re also buying plausible deniability should data-loss put you in jeopardy under subpoena. (While “I disposed of the data” is bad, “The company I was outsourcing to lost it” is not as bad.)

“Honest your honor, we had the emails but Google deleted them by accident.”

*DISCLAIMER – I’m not implying that google would ever do something like this on purpose, using them as a generic, like Xerox.

** It’s Google’s fault…they’re big enough to have become the verb.

***Does anyone actually own a Xerox branded machine anymore?

So if you’re SuperMegaCorp, LLC…you pay for the real service.  You get dedicated support staff, a private line to call, etc.  But to be honest, you might as well keep it in house because hey, you already have the staff, the datacenter, the VMWare farm, etc.  At that point you’re talking a few dollars in licensing and you’ve got email address for your thousands of employees for pennies each.  (Ok, yes, add in replication, backup, etc and it gets a bit higher, but the point is you’ve already comoditized it. (is too a word))

But think about it this way.  The company you’re contracting too has to pay for the same things *YOU* have to pay for.  *PLUS* they have to make enough of a profit to keep their shareholders off their back.  They do get a bit of a discount for bulk licensing, hardware, etc…

But what you GET for hosting it in house is immeasurable.  You get control.

At my last gig I heard the following phrase over and over again.  “I want one neck to choke.” (Oddly enough it was the argument given for moving AWAY from their previously preferred vendor, but you get the idea.)

When the email admin works for you, you have one neck to choke.  You get immediate results. Or you get the pleasure of firing someone.  (Can be fun in the right circumstances, ask The Donald.)

Now say you hosted with Amazon, just for grins.

Not only are your hosts down, potentially THOUSANDS of other hosts are down as well.  Now while we would like to believe they have a thousand techs on staff to give each customer equal time…let’s face it.  it’s not going to happen.  They  have, EXTREMELY generously, 10 technicians per thousand customers.  The techs will bring hosts up as soon as they can…

In an egalitarian society, odds are quite simply about 1000:1 against your site being the first one brought up…  990:1 against it being the second, etc.  See where I’m getting?  Eventually they’ll get around to it, but unless they figured out time travel and can loop back and do them all at the same point in time…you’re out of luck.  Yes, you’ve probably got a 99.999% uptime guarantee…but read the small print of your contract…  Their liability to you cannot exceed the cost of the hosting, if that, or some similiar legalease that limits their liability for downtime and, god forbid, data loss.

But this is not an egalitarian society…  Pure capitalism and “he who has the most gold gets their email back first.” If you’re with Amazon, well they host some PRETTY big sites…including their own.  Netflix comes to mind.  So in a downtime event if it comes down to bringing Joe the Plumber’s CRM app or Netflix’s east-coast streaming…which one do you think is going to get priority?

Right.

I have one neck to choke…  50Micron is hosted by Catbytes… the company that I do my consulting through.  Reason being that I maintain the lab anyway for “play” (officially: self-education and training) purposes, it’s easy for me to spin up an extra VM and put Exchange on it, a couple of CentOS Mailscanners, a few webservers, etc, even off-site replication of backups over a 10MBit link to a “DR” site (that happens to be in my basement)  (If someone wants to donate another CX3-20i or a couple of FCIP bridges I’ll have block-level replication. )

When Amazon EC2 had their issues, suspiciously I had a pretty major crash as well… (As did the customer I was working for at the time, don’t get me started on my paranoid theories.)

But when my stuff breaks… It’s my fault, it’s my responsibility, and *I* am the only one in line.  If I had hosted with Google or Amazon I might have been down for weeks…

I was back up in about 2 hours.  The time it took me to cycle the environment remotely.

Yes…building an IT infrastructure if you already have one can be pricey..  Paying someone else for hosting when you already HAVE an IT infrastructure just plain doesn’t make sense.

P.S. The funniest part is I’m now hosting about a half-dozen servers for friends/family (not free, I’m ugly, not stupid; and co-lo cages are NOT cheap) and about 40-50 websites that I’ve gotten via friends and word-of-mouth…

Of course my guarantee is as follows:

“Best effort, and you have to realize I have a day job that by it’s very nature comes first.” 

Point of reference – A few months ago I wrote a post that I never ended up publishing that started with the line:

“My gods I need to work with technology that wasn’t conceived of in the 1990s.”

With that in mind, in case you’re wondering where I’ve been this past month or so…

I’ve been playing with this beast…

8 Engine VMAX

225 400G SSD Drives (90 TB Raw)

Direct Attached to *ONE* host.

Biggest.  Thumbdrive.  Ever.

Well I was saying I needed to get some serious hands-on VMAX experience.  When you put a request like that out there, sometimes the universe answers LOUDLY.

**ALERT** I’ve had to…modify this post so it won’t offend someone who doesn’t realize that the storage community is very small and that word will get out regardless…

I’m unemployed.

Unexpectedly too.  Unexpected because right up until the day they told me to go home because I wasn’t getting paid, everyone assured me that the contract renewal was in the bag.

I’m such a sucker.  Believing people like that.  Never again.  I’ll also never believe anyone who tells me “don’t worry about it, I’ve got you covered if there’s a gap.”

It’s ok, next gig is on the horizon already…  And it looks like it will be something that while geographically unpleasant, will be a great job I can learn a LOT from, and truly excel at, which for me is key, because I’ve spent the past two years trying to shoe-horn new ideas into the heads of people who think a new idea is like anthrax, to be avoided at all costs.

(And with that I’d like to say hello to the nice folks at the NSA.  Please forgive me, it was an analogy, if a badly placed one.)

Consulting sucks sometimes.  The worst part of course is not knowing where you’ll be working from year to year, or the fact that you have to keep your eyes open, in permanent recruiter mode.

Of course the money is great, and if you tend to go stagnant on doing the same thing over, and over, and over again…It’s nice to be able to change.

It’s a pity that with being yanked out of an environment with no notice comes no turnover on the projects, and that there are a few implementations that I was in the middle of that might blow up if not tended to properly and in the right time-frame, which sadly isn’t far off.

(Ok, first anthrax and then the phrase ‘blow-up’ – the boys in black are DEFINITELY knocking on my door tonight.)

So the real question is…who is going to get saddled with picking up where I left off, *AND* are they going to ask me to help…

Can’t wait until that happens to give me the opportunity to lecture someone on the value of giving notice.

The government’s “Continuing Resolution” will be expiring a week from today.  As a government contractor, this directly affects me.

They have two choices.  They can pass ANOTHER C.R. or they can actually pass a budget.

I don’t post political statements here too often.  However I don’t know about you, but from where I stand this travesty that the House has floated is a disaster.  1.2 million jobs lost by the estimates I’m hearing, and to top it all of, it doesn’t do SQUAT to balance the budget because the places that need to be cut / reformed, IE Defense, etc. are off the table.  So this will be for nothing.

If the posturing peacocks on capitol hill don’t get their collective crap together and one side (or the other) forces the government to shut down. I may have some time on their hands.

Part of me is hoping that cooler heads prevail.

Part of me is looking forward to a little time off.   I’m told it is actually a CRIMINAL offense for me to work if the government shuts down.

Bring it.

Sitting here running RDF create scripts for a data push this weekend and going over the days events in my head.

One of the things you get as a consultant is the ability to get a glimpse of the political machinations of many different companies and to get a first-hand view of what does and doesn’t work.

One thing I’ve seen is about a million different attempts at integrating storage into various systems departments.  It never works.  It always ends up with departmental pissing contests over who owns what, and usually results in a company or orginization buying more storage than they need to in order to pacify the different warring factions.

Storage belongs by itself. Pure and simple, the only way I’ve ever seen it work storage is a department in and of itself, with it’s own staff, it’s own budget, and a little autonomy and freedom to make decisions, and to act with the peace of mind that you’re not having to work around someone else’s changes.

The main reason for this is that server people don’t have the time to understand the dynamics of a truly heterogenious storage environment.  Network people understand firewalls and routing (something that *STILL* puzzles me to a certain extent), etc.

A good storage person knows the basics of as many operating systems as they can.

For instance – the current environment I’m working in has the following systems:

• AIX
• Mainframe
• VMS
• AS/400
• Windows
• Linux
• VMWare

A good storage person knows the gotchas of each server, but may not know even how to log into the system.

For instance – for each of the systems listed above:

• AIX – mount the pseudo device powerpath creates (hdiskpowerX).  AIX is sensitive to D_ID changes (Switch port changes) but if you’re using the LVM there are no real worries, just have to be careful.
• Mainframe – Three words – Long Wave SFP’s
• VMS – Is actually sensitive to the SYMDEV number.  if you’re doing a data migration you have to move the data to the same SYMDEV number.
• AS400 – Boot from SAN using a Load Source Emulator – use the serial cable included with it to configure the boot device.  The boot device has to be on a separate port than the data devices.  Make sure Emulation is set correctly.
• Windows – Dynamic disks cause hell with replication and TimeFinder – don’t use them.
• Linux – make sure you use disk/partition labels so you can avoid issues if the LUN order changes.
• VMWare – SPC2 bit needs to be set on FA’s for DRS/HA Clustered hosts.  Best bet is to do this using Symmask to avoid conflicts with other hosts sharing these ports.

A good storage department would include:

Tier-1 (Symmetrix) expert

Tier-2 (Clariion) expert

Backup person

NAS person

just my thoughts.

The difference between consulting and contracting.

Contractors are there to perform a task.

Consultants are there to solve a problem.

Now there are Consultants who perform contracting work – it’s the way we stay in business when there isn’t consulting work to be done.  (What I’m doing now)

I prefer consulting.

Ok, related to my last post only slightly, so I thought it merited it’s own post.

I’m coming to the point where I’ve spent a larger portion of my career consulting than I did actually working as an employee of a company.

I worked from 1995-2001, and other than 18 months (wasted) with a financial services company, I’ve been consulting ever since.

So I’m going through my resume trying to figure out the best way to highlight consulting experience.  I’m starting to think that the conventional chronological format may no longer suit my needs, at least not in the sense of “who you work for and what did you do while you were there”

My resume currently has entries like this:

Company Name / Date

• Project
  • Duties

• Project
  • Duties

Since which company we’re consulting through is largely irrelevant (very few of the consulting firms offer their employees more than dedicated head-hunting service (along with removal of 30% of your earnings), is it considered ok to remove them from the list?

And then again – the bigger question is, how do you document dry spells?  A colleague of mine recently conveyed his concern at being on a project that has been largely idle for months now.  (A dream job in a lot of respects)  He said (paraphrasing) “What do I tell them (in an interview) I did during this this period?  Got my golf score down into the 80′s?”

As consultants we can’t really take advantage of some of the training that’s out there, as we have to come out-of-pocket for the training, the travel, as well as the time-off.  (I’m not a big fan of Video or CBT training – I learn by doing it)

So what do you do, just count the number of basic Clariion designs you’ve done?

I’ve lost track.

Periodically I’ll put myself through the interview process.  Every once in a while I find a job that really gets my attention and warrants the time.  It’s safe to say that when I interview I really want the job (I don’t let it get to that point if I’m not interested, ask the 20-30 recruiters who contact me every week, my usual response is “sorry – no bid.”  But one of the added bonuses of interviewing is that you get to brush up on your interviewing skills.

I went on a job interview a couple of weeks ago for a major university in the Washington area.  (Limited number of options for who this is, I just won’t use the name out of a sense of professionalism)

I came out of the interview feeling like I Aced the interview, my resume had them all talking, just about everything they’re planning on doing in the near-to-distant future I’ve done at least once in my past.  I answered almost every technical question they could through at me.

So why didn’t I get the job?  A large part of it is that I’m mostly trial-by-fire taught.  (a friend calls it Death-by-PDF) The long and the short of it, no degree.  I should have guessed that not having a degree would handicap me greatly going into the academic world.

Now don’t get me wrong – I’m not heart-broken that I didn’t get the job.  I do very well right now and this job was probably going to mean a 20-30 percent reduction in salary.  (Though other benefits kind of made up for the loss)

My first hint of trouble was going into the interview with someone who held the title of Deputy CIO.  Seems the CIO is retiring in a few years and this guy is slated to replace him.  He only recently got his degree, having first gone through the air-force and through a number of jobs in-between, then pursuing his degree.

So I ask this open question.  What degree would one recomend for a storage professional?  a Management/Information Systems degree seems a bit vague, and might qualify, anyone going into Business Administration wouldn’t be well suited for hands-on design, planning, implementation,  – (though most of the MBA’s I know can’t even manage a business to save their lives)

I spent my coming-of-age building IBM PC-XT clones.  Running online bulletin-boards written (by me) in BASIC (when my connection speed was a screaming 2400baud) and generally trying to find out as much as I could about as much as I could, where my contemporaries were studying philosophy and religion I was digging through code manuals and trying to find new and better ways to write code.

For the last ten years I’ve been, as I call it, “Storage-Centric.”  From my first disk-to-disk data-migration in 1998 I was completely hooked.  And it’s all gotten better since.

So the certain University has chosen, I presume, to hire someone degreed over someone qualified.  Who knows.  Maybe I’ll end up working for them as a consultant in the future.

Not likely though.   I’m too mouthy.

IBM ‘Blindsided’ by Federal Contract Suspension

If there ever was an *OUCH* moment this is it.  IBM – who running jokes have referred to as “Inferior but Marketable” may not be marketable anymore.  It seems that they’ve been caught up in an $80 million bid-rigging scandal that has them blacklisted from bidding on new projects with the federal government.

I would say this is a good thing, but the main alternative now is EDS…..

Ok, changed venues again.  Going to do some work for some smaller customers.  Down side is I won’t be involved in the huge datacenter migrations I’ve done in the past, up side is that I’ll get more hands-on stick time and be able to keep the skills sharp.  So far it’s a *LOT* of NAS work, which is fine by me.

One thing I noticed is that the big data centers, you know, the big Telco sites that seem to run amok in the mid-Atlantic and north-east have an inexplicable “hands-off” approach to systems management.

I was in one site recently when I found that not even the local admins had access to half the systems in the data center. They had outsourced almost ALL of their SA work to India. Even the storage manager, who had WAY above average skills in the Systems Administration arena, was not allowed to log into a box even to check the status of PowerPath.

Now there is a right-way and a wrong-way to do things. Forcing your storage guy to sit on his hands while someone half-a-world away troubleshoots a storage problem is insane. Trust your employees to do the job you pay them to do, and that includes consultants.

Nothing is worse than being in a ‘hands-off’ data center. You can quadruple the amount of time it’s going to take to get something done. The best consulting jobs are the ones where they turn the keyboard over to you and walk away, expecting it to be completed when they get back from their coffee break. Because nine times out of ten, without someone triple-checking everything you do, you can get it done in no-time flat and have time to catch a movie afterwards.

What’s worse is forcing someone to wait. It’s bad enough when an employee has to wait days for something to get done, but when a consultant you’re paying upwards of $200/hr to have on-site, don’t make him wait two days for outputs he needs to do his job – Let’s see, using the above as an example, $200/hr x 8 hrs x 2 days = $3,200….. As a stockholder in some of these companies, I object. I don’t like wasting other people’s money any more than I like wasting my own.

(By the way – $200 / hr falls into the category of guesses – I go out of my way not to know exactly what I’m costing the customer in the end.)

/SG