50Micron.com

Ok, the way government budgeting/purchasing scares the hell out of me as an engineer, and as a tax-payer it gets me absolutely barking.

It starts off in the budget.  Department heads go around and ask all of their people how much money they’re going to need to do their job.  When supplied with this information, Department head then goes and multiplies that number by 2 and adds about 10 million for good measure just to be sure.

Those budgets are then fed up the food chain.  Every person who handles the budget adds 10-20% for good measure, their little pet project, or anything else they can come up with.

This number goes in front of congress.  Congress immediately approves the budget because they don’t know a SAN box from a NAS box from a Kleenex box.  (Seriously, like 16 brain cells between them)

Budget approved, spending starts.

About six months from the end of the fiscal year, someone realizes that “hey – we have all this money left over for some reason.”

Here’s the painful part.  They start making stuff up to spend it on.  Absolutely ridiculous stuff when it comes down to it – like storage rooms full of Sun thin-workstations or uselessly huge laptops that are barely deserving of the name “laptop” (and don’t have a serial port I might add, annoying for people who have to do ground-up switch configurations)

Why?  Because:

If they don’t spend it this year they won’t be able to justify requesting a budget increase next year.

If they don’t spend it they will get their hands slapped for requesting too much money this year.

(The greatest reason EVER given to me by a government employee as to why they spend money like this:  “If we don’t spend it now we won’t have it next year.”)

My brain hurts.  And as a taxpayer, this is the kind of stuff that gets me positively barking.

Spend money – it keeps the economy moving and allows people like me to keep working.

HOWEVER – spend it smartly.  Don’t throw $1.2 million dollars away on a solution when a $600K solution will do the same job with fewer moving parts.

In fact, it’s funny – when you start spending money smartly, the side effect is you can usually get more.  So instead of buying $1,200,000 worth of storage for a Data-center, you can get $1,200,000 worth of storage and fully equip both a data-center *AND* a Disaster Recovery site.

You know – actually protecting the data you’re entrusted to protect might be a neat idea…

By the way, all examples here are hypothetical…right?

Just a bit of Celerra fun.

The content on this site is on a Celerra NS500 back-end.  It’s protected in a show of great trust and support by nothing but Celerra SnapSure.  (30 daily checkpoints)

In the six or so months I’ve been running this there are now more than two dozen sites here, including this one.

Of all of them, the first person to actually need the SnapSure backup was…

…

Drum roll please.

…

Me.

(As if you didn’t see that coming)

I “somehow” managed to clobber the header.php file which resulted in this blog turning a bright shade of nothing but white for a few minutes.  luckily, it was *THAT* simple to type

cp .ckpt/<datestamp>/header.php ./header.php

and the problem was instantly solved.

Thanks EMC. <grin>

Ok, I can say it in a sentence.

A contractor is someone you hire to do a job, a consultant is someone you hire to fix a problem.

I’ve done both, but in the last 8 years I’ve been primarily a “Consultant.”  My job is to fix whatever perceived problem.

Some companies might have a backup problem.  You streamline their process and reduce redundancy, and poof, backup problem solved.

Some companies might have a replication problem.  You analyze their environment then recommend and implement changes.

Some companies have a data management problem.  You simplify storage, identify Tiers, move storage to where it best suits the orginization.  (IE Static Image data doesn’t belong on Tier-1 Symmetrix)

Some companies have a culture problem.

Here I got nothing.

But when your culture problem interferes with the consulting that you are asking me to do, I bristle.

When your culture problem causes me to wait 8 months of a 1 year contract before I’m given the tools to do my job, I boil.

When your culture problem is making me feel like I should take up golf.  I start looking at dice.com for something better.  (I hate golf)

Maybe it’s just that I *LOVE* what I do.

I do.  I love what I do.  I get paid to do what I love.  Which is why I can’t stand seeing people who are either A> There to collect a paycheck and maybe if they’re lucky a pension. or B> try and create their little empire so they can brag to colleagues about how much money they have to spend this year on nothing.

It boggles the mind.

Ok, I have no problem with tape.  It’s a *GREAT* backup medium when your requirement is portability for massive amounts of data and you’re not replicating said data.

If I had to ship 400TB of backups to Iron-Mountain, to protect against the earthquake-to-end-all-earthquakes tape would be my FIRST choice (though maybe, as a GIANT CAVE – Iron Mountain might not be.)

But… (and this is where it gets fun)

I have a customer who *LOVES* tape.

Wants to have it’s children loves it.

Uses it as primary storage loves it.

Now if you:

A> Have a few hundred terabytes of data to Archive.

B> Have millions of dollars to spend on giant room-sized storagetek libraries, and the space, power, and cooling that that entails.

C> Really love tape.

and most importantly

D> Live in the early 1980s

Then Archival to tape is *SO* the way to go.

The argument given is as follows.  “Tape is cheaper than Disk”

Well yes, on a terabyte for terabyte scale tape might be cheaper…maybe if you exclude the hardware.

But if you throw something along the lines of EMC’s Atmos product, or even Centerra, or I’d even go so far as to say the NetApp box appealed to me at one point.  (Now that the Celerra supports File Level Retention, I’ve been cured of that.)

Because when  you throw in modern options like replication and, dare I say it, DEDUPLICATION, Disk rapidly becomes the better, faster, more cost effective way to store your long-term data.

Now I wouldn’t recommend anyone go out and buy a DMX-4 for Archival purposes..  (Though if you want to let me know ahead of time so I can buy some EMC stock. – I’m not currently holding any.)

I checked, and the only Tape vs. Disk comparisons I could find on-line were done by storage vendors, each of which has their own agenda (and big surprise, the analysis came out favouring whatever they were selling), so none of them are valid in the grand scheme of things.  (I have a few things to say about marketing and statistics, but that’s a different post)

The things I look for when judging where to store data…

A> How many copies of the data do I need?

This is often overlooked and a question not asked.  How many copies of a piece of data do you really need?  And how many do you currently have?  I’ve been in one data center recently where they LITERALLY have boxes of old tapes stacked up along the walls.  (Note: Storing your backups WITH the system you’re backing up doesn’t do much in the event of a fire or natural disaster)

B> How long to I need to keep the data?

Retention policies are a big catch for a lot of people.  For “Backup” purposes (see my last post) I say two full backups are all that is really required.  If there is any kind of a likelihood that some critical corruption could be missed for weeks (or months) than adjust your backup strategy accordingly.  (or find a better way of auditing your production data for errors)

C> Does my data have to be portable?

Ok, this is aimed specifically at Tape.  The answer is this.  If you have a remote DR facility and a high-speed connection between them, there is absolutely NO REASON to go to tape for portability.  By virtue of Replication (whether it be the production data or VTL) you’ve already moved your data off-site.  Now if you’ve only got one data centre and it’s sitting right on the San Andreas fault line (I’ve actually worked here – not joking) then send tapes off-site.

Lots of them.

5 or 6 times a day if you can.

D> Am I storing a copy of production or my only copy?

If you’re storing a copy of production (running) then chances are you’re not going to need the backup.  If you’re protecting yourself against someone hitting the delete key accidentally, then maybe Celerra (SnapSure – periodic checkpoints that even the users can access themselves) or Centerra (Don’tEvenThinkAboutDeletingThis) are better options.

If you’re storing a copy of something so you can make room for something else, than backup tape is probably not your best option.  Consider an archiving solution like Atmos or Centerra, or even a Celerra with File Level Retrieval enabled – and version 5.6.44 and later supports de-duplication (both single-instance storage and compression) natively.

E> Do I have the money to spend now, or am I willing to spend more over time to keep the initial investment down.  (This is a valid question – and I’d like to know if anyone has any ideas on which would be the cheaper initial investment.

Just remember that you have to count the floor-space as well.  Something many people forget when scoping out storage buys.

if I want 150TB of storage and I want to do it with tape, what’s the supporting hardware going to cost me?  (A single CX4-240 with one rack of disks can provide up to about 220TB of storage with current drive-sizes.

A final note.  Remember with any “portable” backup solution that you have to keep your backups safe.  Tapes, like disks, don’t respond well to things like…well…dropping.  Anytime you transport a medium from one location to another physically you put that data at risk.

Just my .02 cents.

Downtime last night – This time (almost) planned.

In my constant effort to stay no more than a couple of steps behind current technology, I have started aquiring Dell 1850′s to replace my 2650′s.

I decided to go with the 1850′s because they are smaller form-factor, which means that they’ll fit in my cramped little rack and allow me room to add the third rack of disks to the Clariion.  I had originally purchased the 2650′s because not having a SAN in the basement I needed something in the way of real storage.

One happy side-effect is that apparently the PC3200 DDR2 memory that the 1850′s requires is in much higher supply than the PC2100 that the old 2650′s took.  This makes it cheaper to run more member and thereby fewer servers.

The whole thing was sold to the finance committee as a power saving/cost saving effort.  The best part (and real reason) is that this upgrade has enabled me to go from ESX 3.5 to vSphere4.  (The 2650 doesn’t support vSphere4 due to it’s lack of 64 bit support)

So far my impression is vSphere4 is that it’s a pretty solid upgrade to ESX 3.5, however I’m sure that once I get a chance to dive into it I’ll find a lot more coolness burined under the covers.

Oh – and the 2650′s are probably hitting ebay soon, should anyone want them – let me know ahead of time and I’ll cut you a sweet deal for being a reader.

Microsoft, T-Mobile Apologize For Data Loss, Offer Month Credit

File this under whups.  Microsoft loses data.  That’s not a big surprise.

But I’m in a situation where Microsoft is recommending to a customer that they use almost exactly the same technology to protect their new exchange environment – there is a HUGE part of me that wants to stand up and scream that this is *NOT* a good idea.

Nevermind that in the past I’ve tried a number of times explaining to them some of the shortcomings of their design.

1. That using DAS in an enterprise environment when there is a multi-million-dollar replicated SAN already at their disposal is foolish.

2. They are going to replicate over a 50% saturated gigabit IP network when there is an 8b DWDM Fibrechannel connection available might leave their production and DR environments out of sync.

3. They are going to set all of this up on Hyper-V when VMware offers load-balancing, HA and an amazingly scalable environment is a bit short-sighted.

It’s obvious to me that the genius who designed this cluster____ pulled the design directly from a Microsoft white-paper..

But look at the Microsoft/T-Mobile debacle and ask yourself…  Is the Microsoft way always the right way?

My answer would be quite solidly…no.

Ok, that was too cutsey for such a classy establishment.

When you’re building a SAN, everything should play together, in the same SAN box if you will (with my apologies to QLogic.)

When you start putting in multiple stand-alone SAN islands you increase your maintenance overhead exponentially.  You also prevent the very thing that make a SAN a huge advantage over DAS.

Everything can see everything.

If you have a host and need to throw a certain type of storage at it, you can do that easily.  (and if it’s already cabled you can do it from your living-room)

However, if SAN A/B are connected to one group of hosts/storage, and SAN C/D are connected to a second group of storage, and SAN E (no redundancy) is connected to even a third, you run into a problem.

*NOW* if I want Host_A (Connected to SAN A/B) to see Storage_C (Connected to SAN C/D) I have to do much more than a simple zoning change.

In the end, this is where a few well placed ISL connections can come in handy.  VSAN them off so they don’t cause the fabrics to merge, create an IVR zone to route across them, and then presto.  Host_A can see Storage_C with a minimum of fuss.

Or maybe a *REAL* core-edge topology even.  Where you put a core switch with 24-port (fully subscribed) blades ISL’d to an edge switch, which maybe has the 4/4/40 configuration (4 8-Gig ports, 4 dedicated 4Gig ports, and 40 shared 4Gig ports)

And put one person in charge of it.  Preferably someone with a touch of OCD.

The fundamental difference between BACKUP and ARCHIVE.

A backup is there to help you deal with a crisis such as “My datacenter is a smoking hole in the ground now what do I do?” or something not quite as dramatic like “A virus ate my data.”  You recover from the backup to the last known good and all is happy, right?  Well except for the two or three days that might have gone since your last good backup…  (Was in one lawfirm that lost a drive only to find out their backups hadn’t been running for two months.. came back two weeks later to find a COMPLETE change in personnel had gone on while I was gone – lawyers are not very forgiving when they lose two months worth of email.)

An archive is data that, while not “Active” still might be required on a day-to-day basis.  Film / Video / Image archives are a good candidate for and example of that.

So on a disk-based archive you have some platform, ostensibly EMC/Legato DiskExtender or Rainfinity or something along those lines – that will move the data from “Active” storage to “Archive” storage.  In some applications you can even set up a true HSM, moving data that hasn’t been accessed to Tier-2(Enterprise SATA) and even Tier-3(yes, tape) as it ages, only to be recalled to Tier-1 when it’s accessed.

More often than not I’m brought face to face with people who don’t understand that very subtle difference.  One of my recent customers is actually doing it appropriately, using DX and a smallish Centerra to archive data that, while retention is required, is almost never actually accessed.

Then there are the people who use backup technology for archival purposes.

I’m pretty “old school” when it comes down to it.

Tape is for backup.  Tape is *NOT* supposed to be used as nearline storage when there are equally inexpensive (and more reliable) disk methods out there.

My main complaint about tape as archive: You don’t know if it’s bad until you try to read it.  And time you read it the simple act of moving the tape into a tape drive that was manufactured under less than ideal conditions means you are putting your data at risk.

Spending millions of dollars on a new Room-Sized tape library doesn’t make sense when Centerra storage is fairly inexpensive *AND* provides redundancy of the data automatically.

Spending more millions of dollars on three of them is lunacy when one EMC Atmos set up could provide redundancy and a single namespace for recall.  (and if you go whole hog, geographically relevant retrieval is an option to, so you automatically get it from the closest copy.)

It pains me to see it done wrong.  Especially when it involves trying to shoe-horn two more STK monsters into an already cramped datacenter when the work of it could be done in a couple of floor-tiles of spinning disks.

But apparently the brain-trust that runs the facilities for my customer doesn’t.

If you’ve ever seen a datacenter go completely black, it’s a scary thought.

If you’ve ever seen someone actually manage to take out the power to *JUST* the servers (leaving the lights and AC functional)

Well that’s just a work of art…

That we’re *STILL* recovering from.