50Micron.com

Sitting here running RDF create scripts for a data push this weekend and going over the days events in my head.

One of the things you get as a consultant is the ability to get a glimpse of the political machinations of many different companies and to get a first-hand view of what does and doesn’t work.

One thing I’ve seen is about a million different attempts at integrating storage into various systems departments.  It never works.  It always ends up with departmental pissing contests over who owns what, and usually results in a company or orginization buying more storage than they need to in order to pacify the different warring factions.

Storage belongs by itself. Pure and simple, the only way I’ve ever seen it work storage is a department in and of itself, with it’s own staff, it’s own budget, and a little autonomy and freedom to make decisions, and to act with the peace of mind that you’re not having to work around someone else’s changes.

The main reason for this is that server people don’t have the time to understand the dynamics of a truly heterogenious storage environment.  Network people understand firewalls and routing (something that *STILL* puzzles me to a certain extent), etc.

A good storage person knows the basics of as many operating systems as they can.

For instance – the current environment I’m working in has the following systems:

• AIX
• Mainframe
• VMS
• AS/400
• Windows
• Linux
• VMWare

A good storage person knows the gotchas of each server, but may not know even how to log into the system.

For instance – for each of the systems listed above:

• AIX – mount the pseudo device powerpath creates (hdiskpowerX).  AIX is sensitive to D_ID changes (Switch port changes) but if you’re using the LVM there are no real worries, just have to be careful.
• Mainframe – Three words – Long Wave SFP’s
• VMS – Is actually sensitive to the SYMDEV number.  if you’re doing a data migration you have to move the data to the same SYMDEV number.
• AS400 – Boot from SAN using a Load Source Emulator – use the serial cable included with it to configure the boot device.  The boot device has to be on a separate port than the data devices.  Make sure Emulation is set correctly.
• Windows – Dynamic disks cause hell with replication and TimeFinder – don’t use them.
• Linux – make sure you use disk/partition labels so you can avoid issues if the LUN order changes.
• VMWare – SPC2 bit needs to be set on FA’s for DRS/HA Clustered hosts.  Best bet is to do this using Symmask to avoid conflicts with other hosts sharing these ports.

A good storage department would include:

Tier-1 (Symmetrix) expert

Tier-2 (Clariion) expert

Backup person

NAS person

just my thoughts.

The difference between consulting and contracting.

Contractors are there to perform a task.

Consultants are there to solve a problem.

Now there are Consultants who perform contracting work – it’s the way we stay in business when there isn’t consulting work to be done.  (What I’m doing now)

I prefer consulting.

Ok, related to my last post only slightly, so I thought it merited it’s own post.

I’m coming to the point where I’ve spent a larger portion of my career consulting than I did actually working as an employee of a company.

I worked from 1995-2001, and other than 18 months (wasted) with a financial services company, I’ve been consulting ever since.

So I’m going through my resume trying to figure out the best way to highlight consulting experience.  I’m starting to think that the conventional chronological format may no longer suit my needs, at least not in the sense of “who you work for and what did you do while you were there”

My resume currently has entries like this:

Company Name / Date

• Project
  • Duties

• Project
  • Duties

Since which company we’re consulting through is largely irrelevant (very few of the consulting firms offer their employees more than dedicated head-hunting service (along with removal of 30% of your earnings), is it considered ok to remove them from the list?

And then again – the bigger question is, how do you document dry spells?  A colleague of mine recently conveyed his concern at being on a project that has been largely idle for months now.  (A dream job in a lot of respects)  He said (paraphrasing) “What do I tell them (in an interview) I did during this this period?  Got my golf score down into the 80′s?”

As consultants we can’t really take advantage of some of the training that’s out there, as we have to come out-of-pocket for the training, the travel, as well as the time-off.  (I’m not a big fan of Video or CBT training – I learn by doing it)

So what do you do, just count the number of basic Clariion designs you’ve done?

I’ve lost track.

Periodically I’ll put myself through the interview process.  Every once in a while I find a job that really gets my attention and warrants the time.  It’s safe to say that when I interview I really want the job (I don’t let it get to that point if I’m not interested, ask the 20-30 recruiters who contact me every week, my usual response is “sorry – no bid.”  But one of the added bonuses of interviewing is that you get to brush up on your interviewing skills.

I went on a job interview a couple of weeks ago for a major university in the Washington area.  (Limited number of options for who this is, I just won’t use the name out of a sense of professionalism)

I came out of the interview feeling like I Aced the interview, my resume had them all talking, just about everything they’re planning on doing in the near-to-distant future I’ve done at least once in my past.  I answered almost every technical question they could through at me.

So why didn’t I get the job?  A large part of it is that I’m mostly trial-by-fire taught.  (a friend calls it Death-by-PDF) The long and the short of it, no degree.  I should have guessed that not having a degree would handicap me greatly going into the academic world.

Now don’t get me wrong – I’m not heart-broken that I didn’t get the job.  I do very well right now and this job was probably going to mean a 20-30 percent reduction in salary.  (Though other benefits kind of made up for the loss)

My first hint of trouble was going into the interview with someone who held the title of Deputy CIO.  Seems the CIO is retiring in a few years and this guy is slated to replace him.  He only recently got his degree, having first gone through the air-force and through a number of jobs in-between, then pursuing his degree.

So I ask this open question.  What degree would one recomend for a storage professional?  a Management/Information Systems degree seems a bit vague, and might qualify, anyone going into Business Administration wouldn’t be well suited for hands-on design, planning, implementation,  – (though most of the MBA’s I know can’t even manage a business to save their lives)

I spent my coming-of-age building IBM PC-XT clones.  Running online bulletin-boards written (by me) in BASIC (when my connection speed was a screaming 2400baud) and generally trying to find out as much as I could about as much as I could, where my contemporaries were studying philosophy and religion I was digging through code manuals and trying to find new and better ways to write code.

For the last ten years I’ve been, as I call it, “Storage-Centric.”  From my first disk-to-disk data-migration in 1998 I was completely hooked.  And it’s all gotten better since.

So the certain University has chosen, I presume, to hire someone degreed over someone qualified.  Who knows.  Maybe I’ll end up working for them as a consultant in the future.

Not likely though.   I’m too mouthy.

IBM ‘Blindsided’ by Federal Contract Suspension

If there ever was an *OUCH* moment this is it.  IBM – who running jokes have referred to as “Inferior but Marketable” may not be marketable anymore.  It seems that they’ve been caught up in an $80 million bid-rigging scandal that has them blacklisted from bidding on new projects with the federal government.

I would say this is a good thing, but the main alternative now is EDS…..

Ok, changed venues again.  Going to do some work for some smaller customers.  Down side is I won’t be involved in the huge datacenter migrations I’ve done in the past, up side is that I’ll get more hands-on stick time and be able to keep the skills sharp.  So far it’s a *LOT* of NAS work, which is fine by me.

One thing I noticed is that the big data centers, you know, the big Telco sites that seem to run amok in the mid-Atlantic and north-east have an inexplicable “hands-off” approach to systems management.

I was in one site recently when I found that not even the local admins had access to half the systems in the data center. They had outsourced almost ALL of their SA work to India. Even the storage manager, who had WAY above average skills in the Systems Administration arena, was not allowed to log into a box even to check the status of PowerPath.

Now there is a right-way and a wrong-way to do things. Forcing your storage guy to sit on his hands while someone half-a-world away troubleshoots a storage problem is insane. Trust your employees to do the job you pay them to do, and that includes consultants.

Nothing is worse than being in a ‘hands-off’ data center. You can quadruple the amount of time it’s going to take to get something done. The best consulting jobs are the ones where they turn the keyboard over to you and walk away, expecting it to be completed when they get back from their coffee break. Because nine times out of ten, without someone triple-checking everything you do, you can get it done in no-time flat and have time to catch a movie afterwards.

What’s worse is forcing someone to wait. It’s bad enough when an employee has to wait days for something to get done, but when a consultant you’re paying upwards of $200/hr to have on-site, don’t make him wait two days for outputs he needs to do his job – Let’s see, using the above as an example, $200/hr x 8 hrs x 2 days = $3,200….. As a stockholder in some of these companies, I object. I don’t like wasting other people’s money any more than I like wasting my own.

(By the way – $200 / hr falls into the category of guesses – I go out of my way not to know exactly what I’m costing the customer in the end.)

/SG

1. iSCSI is a viable alternative to FC for Small infrastructures.

2. I learned that no matter how well prepared for an install you are, the techie-gods will always throw curve-balls at you.

3. I’ve learned that Linux and PowerPath requires that multiple iSCSI HBA’s in a single host are not supported.
(Author’s note – this is not entirely true – see comment #3 below)

4. I’ve learned that seeing mice (yes, plural) running around a datacenter while you’re crawling around on the floor running cables is creepy.

Nuff said. I just got off 20 almost-straight hours in a row (I napped between 3am and 6am this morning) doing what should have been a very very simple install.

Needless to say it wasn’t. I’m going to bed.

-J

Well – after moving along for a few weeks at a nice leisurely pace, I find myself working on six different projects.  Loads of fun, especially when three of them are just similar enough to get the details confused.

Got an iSCSI install next weekend though, this should be interesting.  I think I have a handle on how the Celerra does iSCSI, so the only real trick will be setting up the hosts correctly.  It’s a mixture of RHEL3, RHEL4, and CentOS5, which makes it even more interesting.

Another thing I got to play with last week was the McData Eclipse series FCIP router.   When tied to a pair of Brocade switches (one on each side of the FCIP tunnel) I found them to be almost impossible to use.  I’ve got quite a bit of FCIP experience in different replication scenarios, and it still took me almost 6 hours to get these connected.  Talk about having to pay attention to detail, this was painful.  I’d be infinitely happier with a Cisco 9216i with a 14/2 blade in it and be done with it.  Eliminates the need for 90% of the make-work that had to be done to get this thing running.

In McData’s defense though, I came in unprepared, it was my understanding going into the engagement that the McData was already set up and I came in and found not only wasn’t it set up for FCIP, (it was set up for iSCSI) the tunnel hadn’t been built.  So all we really had was a CE who came in, set the IP’s and ran like heck for the door.

I also had a Celerra NSX install that same weekend.  The NSX is an interesting piece, very much like the old CNS boxes, albeit much smaller/faster.  Modular setup makes it very expandable.

What I don’t get, is why, in this day and age, you are still required to use a floppy to boot the control-station installation CD.  Bootable CD’s have been around for quite some time, and in fact the NS502g I ran when I was at the evil empire even booted perfectly off the CD.

The NSX required however a serial console connection into the server, and had the bios locked so you couldn’t change the boot order.  Add that to the fact that EMC put the wrong crossover serial cable in the box with it (Mail-Female) meant a 2 hour install ended up taking 8 hours when you factored in driving around looking for a USB floppy drive (to create the boot disk) and a null-modem adapter (First time I’ve set foot in a radio shack in 10 years).

This week I’m off to sunny (?) Florida to do an ECC install and then off to Texas to do the NS20/iSCSI install.  *THAT* should be worth writing about.

Thanks.

You know – there is something to be said for working from home.  I get to pick my hours, I get to relax in shorts and a T-Shirt when most of my colleagues are out there in at the very least some form of “Business Casual” attire.

But sometimes I prefer being in the mix of it.  For one thing, I find it’s easier to get things done when I don’t have three kids stomping around the house starting fights with each-other. (Usually just outside my office door so I can’t help but hear them)  I also have too many of my own “pet” projects that I’m much more interested in completing than the stupid Visio diagram I have to do for a very basic install because the customer requested it.

For the most part things work out.  For the rest of the time, well I end up blogging instead of Visio’ing.

So I bought a Dell PowerVault 224F array on Ebay last week.  14 drives, Fibrechannel, the whole nine yards.  I also found a bunch of 18G seagate drives that are in reasonable shape (they are all passing the initial smoke test at least, I’ve yet to put data on them.)

I’ve always suspected that making the move to Fibrechannel in the house was going to be a pain, but if I want to play with things such as VMotion or virtual clusters, I’m going to need some form of shared back-end.  SCSI works, but is a pain in the tail with all the different flavours, is it Ultra 160 or Ultra320?  LVD, HVD, BVD?  Is the termination correct?  Did a bend a $%&@% pin when I plugged the damned cable in behind the rack in the dark?

So I picked up the array and drives, I have an old Brocade switch floating around that has it’s full compliment of licenses (you need TL mode to go from Loop to FCSW), a lot of old (but still sealed, for what it’s worth) Qlogic PCI Controllers for the servers (so I can take advantage of the load-balancing that comes with VMWare 3.5) and am going to give it a try.

Eventually my goal is to have all my VM’s running on their own individual drives, and all VM’s shared between the two servers for both load and redundancy.

</jg>

We did it.   Migrated the hosts/data.   Production is now running in Kansas, DR in Georgia, and the old datacenters in NY/NJ are one step closer to being shut down.

Interesting couple of things I learned today.

SRDF/A is a great technology for replicating over long distances while maintaining what they call a “dependent-write-consistent” state.  It means that even though the replication is being taken care of asynchronously, with minimal performance impact to the host, that in the event of a failure you’re going to lose a minimal amount of data.  (In our case, when it was running the R2 disks were about 45-60 seconds behind the R1.)

We also performed a “failure” (disconnected both Gig-E ports to simulate the Kansas site dropping out) and brought the DR hardware up as primary, then reconnecting, unmounting, and restarting the SRDF/A session.

The only downside I’ve found with SRDF/A is that it’s a royal pain to stop and restart the replication.  In cases like this one, where once a week they take the R2′s offline to run a 20-hour backup off them, they are putting themselves at unneeded risk.  It’s a situation where TimeFinder/SNAP would be a great benefit.  You snap the R2′s at midnight and back them up, thereby leaving your R2′s in sync with your R1′s for the duration.  You can also then mount the SNAP volumes to a separate media server thereby avoiding having to re-configure the DR server as a temporary media server.

It’s just a thought.

It’s always a great feeling when you hit the deadline dead-on, especially when you’re dealing with a situation where the requirements keptchanging throughout the project, even to the point of having to add new devices at the last minute.

Oh well, on to the next.  At least the next is going to keep me closer to home.   Small-scale data migration from DMX2 to DMX2 within the same room, this should be a cake-walk.