50Micron.com

1. iSCSI is a viable alternative to FC for Small infrastructures.

2. I learned that no matter how well prepared for an install you are, the techie-gods will always throw curve-balls at you.

3. I’ve learned that Linux and PowerPath requires that multiple iSCSI HBA’s in a single host are not supported.
(Author’s note – this is not entirely true – see comment #3 below)

4. I’ve learned that seeing mice (yes, plural) running around a datacenter while you’re crawling around on the floor running cables is creepy.

Nuff said. I just got off 20 almost-straight hours in a row (I napped between 3am and 6am this morning) doing what should have been a very very simple install.

Needless to say it wasn’t. I’m going to bed.

-J

Well – after moving along for a few weeks at a nice leisurely pace, I find myself working on six different projects.  Loads of fun, especially when three of them are just similar enough to get the details confused.

Got an iSCSI install next weekend though, this should be interesting.  I think I have a handle on how the Celerra does iSCSI, so the only real trick will be setting up the hosts correctly.  It’s a mixture of RHEL3, RHEL4, and CentOS5, which makes it even more interesting.

Another thing I got to play with last week was the McData Eclipse series FCIP router.   When tied to a pair of Brocade switches (one on each side of the FCIP tunnel) I found them to be almost impossible to use.  I’ve got quite a bit of FCIP experience in different replication scenarios, and it still took me almost 6 hours to get these connected.  Talk about having to pay attention to detail, this was painful.  I’d be infinitely happier with a Cisco 9216i with a 14/2 blade in it and be done with it.  Eliminates the need for 90% of the make-work that had to be done to get this thing running.

In McData’s defense though, I came in unprepared, it was my understanding going into the engagement that the McData was already set up and I came in and found not only wasn’t it set up for FCIP, (it was set up for iSCSI) the tunnel hadn’t been built.  So all we really had was a CE who came in, set the IP’s and ran like heck for the door.

I also had a Celerra NSX install that same weekend.  The NSX is an interesting piece, very much like the old CNS boxes, albeit much smaller/faster.  Modular setup makes it very expandable.

What I don’t get, is why, in this day and age, you are still required to use a floppy to boot the control-station installation CD.  Bootable CD’s have been around for quite some time, and in fact the NS502g I ran when I was at the evil empire even booted perfectly off the CD.

The NSX required however a serial console connection into the server, and had the bios locked so you couldn’t change the boot order.  Add that to the fact that EMC put the wrong crossover serial cable in the box with it (Mail-Female) meant a 2 hour install ended up taking 8 hours when you factored in driving around looking for a USB floppy drive (to create the boot disk) and a null-modem adapter (First time I’ve set foot in a radio shack in 10 years).

This week I’m off to sunny (?) Florida to do an ECC install and then off to Texas to do the NS20/iSCSI install.  *THAT* should be worth writing about.

Thanks.

You know – there is something to be said for working from home.  I get to pick my hours, I get to relax in shorts and a T-Shirt when most of my colleagues are out there in at the very least some form of “Business Casual” attire.

But sometimes I prefer being in the mix of it.  For one thing, I find it’s easier to get things done when I don’t have three kids stomping around the house starting fights with each-other. (Usually just outside my office door so I can’t help but hear them)  I also have too many of my own “pet” projects that I’m much more interested in completing than the stupid Visio diagram I have to do for a very basic install because the customer requested it.

For the most part things work out.  For the rest of the time, well I end up blogging instead of Visio’ing.

So I bought a Dell PowerVault 224F array on Ebay last week.  14 drives, Fibrechannel, the whole nine yards.  I also found a bunch of 18G seagate drives that are in reasonable shape (they are all passing the initial smoke test at least, I’ve yet to put data on them.)

I’ve always suspected that making the move to Fibrechannel in the house was going to be a pain, but if I want to play with things such as VMotion or virtual clusters, I’m going to need some form of shared back-end.  SCSI works, but is a pain in the tail with all the different flavours, is it Ultra 160 or Ultra320?  LVD, HVD, BVD?  Is the termination correct?  Did a bend a $%&@% pin when I plugged the damned cable in behind the rack in the dark?

So I picked up the array and drives, I have an old Brocade switch floating around that has it’s full compliment of licenses (you need TL mode to go from Loop to FCSW), a lot of old (but still sealed, for what it’s worth) Qlogic PCI Controllers for the servers (so I can take advantage of the load-balancing that comes with VMWare 3.5) and am going to give it a try.

Eventually my goal is to have all my VM’s running on their own individual drives, and all VM’s shared between the two servers for both load and redundancy.

</jg>

We did it.   Migrated the hosts/data.   Production is now running in Kansas, DR in Georgia, and the old datacenters in NY/NJ are one step closer to being shut down.

Interesting couple of things I learned today.

SRDF/A is a great technology for replicating over long distances while maintaining what they call a “dependent-write-consistent” state.  It means that even though the replication is being taken care of asynchronously, with minimal performance impact to the host, that in the event of a failure you’re going to lose a minimal amount of data.  (In our case, when it was running the R2 disks were about 45-60 seconds behind the R1.)

We also performed a “failure” (disconnected both Gig-E ports to simulate the Kansas site dropping out) and brought the DR hardware up as primary, then reconnecting, unmounting, and restarting the SRDF/A session.

The only downside I’ve found with SRDF/A is that it’s a royal pain to stop and restart the replication.  In cases like this one, where once a week they take the R2′s offline to run a 20-hour backup off them, they are putting themselves at unneeded risk.  It’s a situation where TimeFinder/SNAP would be a great benefit.  You snap the R2′s at midnight and back them up, thereby leaving your R2′s in sync with your R1′s for the duration.  You can also then mount the SNAP volumes to a separate media server thereby avoiding having to re-configure the DR server as a temporary media server.

It’s just a thought.

It’s always a great feeling when you hit the deadline dead-on, especially when you’re dealing with a situation where the requirements keptchanging throughout the project, even to the point of having to add new devices at the last minute.

Oh well, on to the next.  At least the next is going to keep me closer to home.   Small-scale data migration from DMX2 to DMX2 within the same room, this should be a cake-walk.

I saw an article in E-Week/Careers and had to comment.

I’ve been the victim of overzealous non-compete enforcement, almost twice actually. 

The problem is that it’s not that they can actually enforce it, it’s that no matter what you do they try.

It cost me over $8,000 leaving a certain Alpharetta, GA-based storage consulting firm, even though my departure did nothing to violate the agreement I signed.  I left the metropolitan area (specifically per the agreement) and went through major pains not to work at any of the client sites I was sub-contracted out to.

Over $8,000 of legal fees later they randomly dropped it stating that they didn’t believe they had a claim.

A non-compete is a license-to-harass.  It’s a way for a company to get past an evidencary hearing and make a technologist’s life hell should they decide that maybe their career isn’t moving ahead fast enough or that the 80-hour work-weeks are getting to be too much.

I left my first consulting firm after they refused to take me off an assignment that was killing me and causing major family problems.  I was driving from Laguna Beach, CA to Burbank, CA, a 60 mile round trip that would take me on average 2-3 hours to drive in each direction.  I took the assignment because it was supposed to be a 6-week scripting project that turned into almost 18 months.  The project itself was fun but when you spend that much time in a car things start to suffer – health, family, and sanity being among them.

The non-compete agreement I had with them stated, quite plainly that I wasn’t allowed to consult in any metropolitan region where I had performed services for them.  So that left Phoenix Metro, LA Metro and SanDiego Metro out of the picture.

So I moved to Virginia.  Not quite the LA metro area, right?   Well they sued me anyway, forcing me to retain a lawyer to explain to them exactly why they were idiots.  He did it, but then sent me the bill, which is when I really realized that I was the idiot.

My last consulting firm was for a Virginia based company.  When I started making noises about leaving, I was pulled aside and told in no-uncertain terms that I would be vigorously pursued if I did something stupid like signing on with EMC directly.

Which is why I ended up at the last private company I worked for, wasting 18 months of my life on yet another no-future position waiting for my non-compete to expire.

They almost got me back in March, but luckily I had the forsight to come back on the condition that they let me amend the non-compete.  Which I did, removing any reference to the 18 month wait and making absolutly sure that working for EMC wouldn’t be considered competition.

Now I work for myself, slowly starting my own firm (a firm of one so far).  But I am postiive if it ever grows beyond a firm of one, that I’ll never even think about having someone sign a non-compete…

The biggest problem with consulting is that unless you live *IN* a major metro area, you have to travel *TO* a major metro area.

To complicate matters, when you’re in a specialty like storage, the jobs, even in the major metro areas, can be few and far between, so that leaves travel.

So you book the trip, make the drive to the airport, go through security, wait for the flight, if you’re lucky your flight is on time and you cram yourself into the center seat (because no-one allows you to pre-book seats these days) get to the rental car counter, find out what kind of peice of junk Rent-A-Wreck will provide you, and drive to a hotel where they will put you in the room right next to the elevator, (or, in my case on my last trip, right next to the 24 hour exercize room)

All because most companies don’t trust consultants to remote in to do their work. 

So if you’re going to pay hundreds of thousands of dollars for, say, a data migration, don’t you think you might try something so novel as trusting the people you’ve hired to do the job?

Probably not.  But then again, it keeps us employed, and in demand.

/jg

The joys of data migrations. 

One of the most common problems is the standard practice of most companies to avoid upgrading whenever possible.  The “if it ain’t broke, don’t fix it” mentality.

I could spend days and days on that particular brand of suicide.  For now I’ll just replace that addage with a new one.

If you don’t upgrade it now when you can do it in a controlled fashion, you will end up doing it when your life depends on it with very little planning.

So on the 17th, a customer is going to have to take an application down on an *OLD* Symmetrix 4.8 system to upgrade from 5265 code to 5267 code.  (two major code revs up, from 5×65 to 5×66, then from 5×66 to 5×67, and neither can be loaded on-line)

All of this has to happen *JUST* so we can move the data off this symm and onto a “not-so” old Symm 5.0 that will then me packed up to be moved out of state.

First off, the idea that you can simply turn off a Symm and ship it across the country is nuts.  Anytime you get a system with that many moving parts (harddrives) that have been spinning for that length of time and simply “turn it off” you run the risk of multiple hard-disk failures.  And as we all know, any time you have multiple hard-disk failures in an array, you run the risk of losing both halfs of a mirror.  Hell I cringe at turning off my desktop PC because I know that there is always the chance it’s not going to come back up, and I’ve got Raid-1 (160G) on my boot devices and Raid-5 (500G) on my data volumes, so I’m reasonably protected.

Secondly, why are we moving a Symm that is going to hit EOSL before too long?  Doesn’t it make sense to go ahead and upgrade to the latest and greatest hardware, get a free support renewal (included with the purchase of new hardware) and get the latest and greatest features/functionality?  Of course we’re moving a bunch of Sun E3000/E4000/E6000 class hardware.  These are the systems I cut my admin teeth on back in 1996 when I first started out in datacenter operations.  They were old 6 years ago.

Next time someone asks you the correct way to move a datacenter, the correct answer is “twin the hardware and replicate” followed by “trade-in.”

Bekins Moving should never be an option.

This is a large part of what drives me nuts about customers.   (If they didn’t pay the bills, I’d be for letting them all drown in the sea of bad decisions they make)  The unwillingness to spend a little extra to do it right.

Let me give you an example.  I’m working now on a data-center consolidation.  Two datacenters in the New-York area that have been around since the 80′s are being consolidated into other datacenters further south. 

One is being closed, and the other is staying online, presumably long enough for them to realize that it’s also out of date.  About a dozen hosts are SAN attached. and are being moved.

Now here is the scary part – We’re talking about a collection of Sun E3500′s, I think the most powerful unit they have is an E5000.  All running Sybase and are still running Solaris 2.7.  The real kicker is that the hosts are using JNI SBus cards – these have not been supported in over 5 years (the company is long since a memory)  It was a real challenge even to FIND a batch of old Emulex LP9002-S cards to replace their JNI cards with.

The Symms we’re migrating off of are a collection of 4.0′s, 4.8′s, and a few 5.0′s – and they’re keeping most of the 4.0 and 4.8 symms and retiring the 5.0.s….    Huh?

So what would be the best idea?

If you said “Buy a pair of UE10K’s to go with the new DMX3′s and move all of the hosts into it” you’d be right.    The wonderful thing about VM based hosts, like the Sun Ultra Enterprise series and the AIX p-series, is the ability to consolidate multiple smaller, older hosts onto it.  You save a mint just in floor-space and power/cooling bills.  Not to mention the HBA’s you don’t have to buy.  Figure this company bought in the neighborhood of 30 Emulex LP9002′s, even conservatively priced out at $500/ea, that’s a chunk of change.  When one UE10K with 6 or 8 LP10000 HBA’s could have done the same work, and at 4gig no less.

They spent the money on a pair of DMX3′s, source and target, presumably because they had to (I dont think you can buy a Symm5 outside of Ebay anymore)  But they are actually going to spend the next three months MOVING antique hosts 1500 miles and hoping they survive the trip.  The good news is that I get overtime, and I’m averaging 60-70 hours a week right now trying to play into their little psychosis.

And where does it all end?   Quite simply they are going to find themselves upgrading the hardware anyway, probably after a catastrophic failure, so they just wasted a million dollars moving stuff they are going to throw in the trash in a year or two.

Ok, not really storage related, but it’s my site, so I get to bend the rules a bit, right?

So going to private practice consulting has proven to be more complex than I had originally anticipated.

First – I’m not an accountant, and while I’m learning the process quickly, I think I’m going to have to hire one to keep myself in the IRS’s good graces.

Second (and to the point of this post. I’ve had to pursue the process of taking on individual insurance coverage for myself and my family.

I never realized what a hairy deal it was. I went to “Anthem – Blue Cross/Blue Shield” and filled out the application. According to the quote I received after I filled out my personal data, I was going to save about $500 over the COBRA benefits that my last company offered.

So I signed up. And 4 days before my new coverage was going to take effect, I got the letter stating that neither me nor my 5 year old son could be covered, because I’m overweight and my son has been diagnosed with mild autism.

They pointed me to another policy, for “high-risk” individuals, that was going to cost me $1,000 a month *JUST FOR THE TWO OF US*, (plus the $450 a month for my wife and two “healthy” kids.)

Now I figured that the autism was going to be an issue, and was prepared to rider the coverage and pay for his speech therapy out of pocket. However they didn’t even offer me the option. Not only did they not offer me the option, they said that he wasn’t eligible for coverage for 10 YEARS. So because he has a little difficulty with correctly responding to people when they ask him questions, he’s being denied *ALL* access to health care.
I think I’m jumping on the universal healthcare bandwagon at this point. Insurance companies should not be allowed to let profit dictate health decisions.
Luckily for me, I’m still eligible for coverage under COBRA, and have signed up, it’s $500-$600 more a month than i would be spending, but at least we’re covered.

And the funny part is, it’s through Anthem.

First off I’m sorry for the long delay between posts – all of a sudden I find I have more work than i know what to do with.   This is a good thing, right?

So the second part of my post is:

1099-Direct

This is probably the easiest type of consulting to do.  You work for an agency, Norwin, K-Force, and the like – they have a staff in place than handles the paperwork, invoicing, and such, and all you do is collect your check.  To start you go to the IRS website and fill out a simple form for an EIN – that is an employer ID number.  You fill it out as a “sole proprietor” and at tax time, all of the money that is made by the “Company” is transferred to you directly.

Pros 

• Simplicity.  Most of us don’t want to spend a lot of time hassling with LLC paperwork, paying corporate income taxes as well as the hassle of paying yourself out of the company.  Just be sure you’re putting your 25% away for the quarterly taxes, because if, at the end of the year, you haven’t kept that up, well the fines are huge.  (If when you file your 1040 you owe more than 25,000, they peanize you for underwithholding, *AND* then require you to pay estimated tax throughout the rest of the year.
• No Hussling.  If you’re working for an agency, you don’t have to get out and hussle up your next assignment, someone else is going to do it for you.

Cons

• You still work for someone else.  Someone else dictates your schedule, you still have to request time off whereas when you’re doing project-based work, all you have to do is schedule yourself some vacation in between projects. 
• Your “company” will never grow because it’s just you.  You don’t get to take advantage of the fact that being a “Corporate Consultant” puts you in a great position to recruit other consultants.
• Benefits – you’re paying cash for benefits.  The quote I got for my family of five from Anthem was about $750/mo.  (just to provide a baseline)

That’s all I can think off off the top of my head, anyone got anything else?