50Micron.com

Just a quick question as it’s *WAY* past my bedtime. 

Has anyone else ever heard of a customer that a sales-person or tech would quit rather than set foot in?

I’m remembering my days back at Disney in California.  (over 7 years ago, so I’m assuming the statute of limitations has expired on that one)

I got called up for a 6-week scripting engagement, and 18 months later I had to actually quit the consulting firm I was working for and move across the country to get away from them.  (Happiest place on earth?–Not if you’re in their IT dept.)

These words actually came from one of their “cast members”:

“Once you work for the mouse, you ALWAYS work for the mouse.”

When I told them i was moving to Virginia/DC to do goverment work, the response was even more ominous…

“The sun never sets on the ears…you can’t move far enough away.”

In the end, the experience I got at Disney (Scripting TimeFinder to back up AIX/DB2/SAP) was invaluable and one of the guys there basically taught me more about kornshell scripting than I think I’ve learned from any single source since then.

But the environment was toxic.  I’m curious as to whether anyone else has stories like these or if I’m the only one who runs into them?

Ok, I have no problem with tape.  It’s a *GREAT* backup medium when your requirement is portability for massive amounts of data and you’re not replicating said data.

If I had to ship 400TB of backups to Iron-Mountain, to protect against the earthquake-to-end-all-earthquakes tape would be my FIRST choice (though maybe, as a GIANT CAVE – Iron Mountain might not be.)

But… (and this is where it gets fun)

I have a customer who *LOVES* tape.

Wants to have it’s children loves it.

Uses it as primary storage loves it.

Now if you:

A> Have a few hundred terabytes of data to Archive.

B> Have millions of dollars to spend on giant room-sized storagetek libraries, and the space, power, and cooling that that entails.

C> Really love tape.

and most importantly

D> Live in the early 1980s

Then Archival to tape is *SO* the way to go.

The argument given is as follows.  “Tape is cheaper than Disk”

Well yes, on a terabyte for terabyte scale tape might be cheaper…maybe if you exclude the hardware.

But if you throw something along the lines of EMC’s Atmos product, or even Centerra, or I’d even go so far as to say the NetApp box appealed to me at one point.  (Now that the Celerra supports File Level Retention, I’ve been cured of that.)

Because when  you throw in modern options like replication and, dare I say it, DEDUPLICATION, Disk rapidly becomes the better, faster, more cost effective way to store your long-term data.

Now I wouldn’t recommend anyone go out and buy a DMX-4 for Archival purposes..  (Though if you want to let me know ahead of time so I can buy some EMC stock. – I’m not currently holding any.)

I checked, and the only Tape vs. Disk comparisons I could find on-line were done by storage vendors, each of which has their own agenda (and big surprise, the analysis came out favouring whatever they were selling), so none of them are valid in the grand scheme of things.  (I have a few things to say about marketing and statistics, but that’s a different post)

The things I look for when judging where to store data…

A> How many copies of the data do I need?

This is often overlooked and a question not asked.  How many copies of a piece of data do you really need?  And how many do you currently have?  I’ve been in one data center recently where they LITERALLY have boxes of old tapes stacked up along the walls.  (Note: Storing your backups WITH the system you’re backing up doesn’t do much in the event of a fire or natural disaster)

B> How long to I need to keep the data?

Retention policies are a big catch for a lot of people.  For “Backup” purposes (see my last post) I say two full backups are all that is really required.  If there is any kind of a likelihood that some critical corruption could be missed for weeks (or months) than adjust your backup strategy accordingly.  (or find a better way of auditing your production data for errors)

C> Does my data have to be portable?

Ok, this is aimed specifically at Tape.  The answer is this.  If you have a remote DR facility and a high-speed connection between them, there is absolutely NO REASON to go to tape for portability.  By virtue of Replication (whether it be the production data or VTL) you’ve already moved your data off-site.  Now if you’ve only got one data centre and it’s sitting right on the San Andreas fault line (I’ve actually worked here – not joking) then send tapes off-site.

Lots of them.

5 or 6 times a day if you can.

D> Am I storing a copy of production or my only copy?

If you’re storing a copy of production (running) then chances are you’re not going to need the backup.  If you’re protecting yourself against someone hitting the delete key accidentally, then maybe Celerra (SnapSure – periodic checkpoints that even the users can access themselves) or Centerra (Don’tEvenThinkAboutDeletingThis) are better options.

If you’re storing a copy of something so you can make room for something else, than backup tape is probably not your best option.  Consider an archiving solution like Atmos or Centerra, or even a Celerra with File Level Retrieval enabled – and version 5.6.44 and later supports de-duplication (both single-instance storage and compression) natively.

E> Do I have the money to spend now, or am I willing to spend more over time to keep the initial investment down.  (This is a valid question – and I’d like to know if anyone has any ideas on which would be the cheaper initial investment.

Just remember that you have to count the floor-space as well.  Something many people forget when scoping out storage buys.

if I want 150TB of storage and I want to do it with tape, what’s the supporting hardware going to cost me?  (A single CX4-240 with one rack of disks can provide up to about 220TB of storage with current drive-sizes.

A final note.  Remember with any “portable” backup solution that you have to keep your backups safe.  Tapes, like disks, don’t respond well to things like…well…dropping.  Anytime you transport a medium from one location to another physically you put that data at risk.

Just my .02 cents.

The fundamental difference between BACKUP and ARCHIVE.

A backup is there to help you deal with a crisis such as “My datacenter is a smoking hole in the ground now what do I do?” or something not quite as dramatic like “A virus ate my data.”  You recover from the backup to the last known good and all is happy, right?  Well except for the two or three days that might have gone since your last good backup…  (Was in one lawfirm that lost a drive only to find out their backups hadn’t been running for two months.. came back two weeks later to find a COMPLETE change in personnel had gone on while I was gone – lawyers are not very forgiving when they lose two months worth of email.)

An archive is data that, while not “Active” still might be required on a day-to-day basis.  Film / Video / Image archives are a good candidate for and example of that.

So on a disk-based archive you have some platform, ostensibly EMC/Legato DiskExtender or Rainfinity or something along those lines – that will move the data from “Active” storage to “Archive” storage.  In some applications you can even set up a true HSM, moving data that hasn’t been accessed to Tier-2(Enterprise SATA) and even Tier-3(yes, tape) as it ages, only to be recalled to Tier-1 when it’s accessed.

More often than not I’m brought face to face with people who don’t understand that very subtle difference.  One of my recent customers is actually doing it appropriately, using DX and a smallish Centerra to archive data that, while retention is required, is almost never actually accessed.

Then there are the people who use backup technology for archival purposes.

I’m pretty “old school” when it comes down to it.

Tape is for backup.  Tape is *NOT* supposed to be used as nearline storage when there are equally inexpensive (and more reliable) disk methods out there.

My main complaint about tape as archive: You don’t know if it’s bad until you try to read it.  And time you read it the simple act of moving the tape into a tape drive that was manufactured under less than ideal conditions means you are putting your data at risk.

Spending millions of dollars on a new Room-Sized tape library doesn’t make sense when Centerra storage is fairly inexpensive *AND* provides redundancy of the data automatically.

Spending more millions of dollars on three of them is lunacy when one EMC Atmos set up could provide redundancy and a single namespace for recall.  (and if you go whole hog, geographically relevant retrieval is an option to, so you automatically get it from the closest copy.)

It pains me to see it done wrong.  Especially when it involves trying to shoe-horn two more STK monsters into an already cramped datacenter when the work of it could be done in a couple of floor-tiles of spinning disks.

Ok, given the changes to the storage arena I’ve been working on a revised “Tiering system” to incorporate all of the levels of data…importance?

My version of Storage Tiering is (or should be) as follows:

• Tier-1    – Symmetrix/Replicated – High Performance/Criticial Data
• Tier-2    – Symmetrix/NonReplicated – High Performance/Non-Criticial Data
• Tier-3   – Symmetrix/SATA/Replicated – High-Medium Performance/Critical Data
• Tier-4   – Symmetrix/SATA/NonReplicated – High-Medium Performance/Non-Critical Data
• Tier-5    – Clariion/FC/Replicated – Medium Performance/Critical Data
• Tier-6    – Clariion/FC/NonReplicated – Medium Performance/Non-Critical Data
• Tier-7    – Clariion/SATA/Replicated – Low Performance/Critical Data
• Tier-8    – Clariion/SATA/NonReplicated – Low Performance/Non-Critical Data
• Tier-9    – CelerraNAS/Replicated – Network Attached/Critical Data
• Tier-10  – CelerraNAS/NonReplicated – Network Attached/Non-Criticial Data
• Tier-11  – Atmos – Network Attached / Low Performance
• Tier-12  – Centerra (Content Addressable Storage) – Low Performance Archive / Highly Available
• Tier-13  – Primary Tape-In-Library (Automatic loading on demand via HSM)
• Tier-14  – Primary Tape-Out-Of-Library (Manual Intervention Required)

“Critical Data” vs. “Non-Critical Data” is simply a matter of how long you can be without the data should a failure or accidental deletion occur.  As all data is available in Tier8/9 storage (in theory).

I’ve also considered using Tier1/Tier1B to describe DMX storage vs. Clariion storage, given that there is a LOT of overlap in performance characteristics these days…

Oh, and iSCSI would be somewhere between 10 and 13….

Any thoughts?

You know, it floored me recently when i heard that someone had said that a CX3-20 couldn’t be used as a dumparea for Tivoli.

What floored me even more is when they said this would not perform as well as a Symmetrix 8430.

8430.

Symm4.

Almost 10 year old hardware.

Huh?

Back when I was working for the student loan company in Sterling, we ran all of our backups to a single CX500 with 15 Terabytes behind it. Worked without any issue and was screaming fast, to the point that 18 hours (to tape) of backups was compressed into 8 hours (to disk.)

{sarcasm}Now correct me if I’m wrong, the CX3-20 is a tad faster than the older CX500, right?{/sarcasm}

As with any array, the bulk of the issue arises from whether or not the disks are laid out appropriately. If you try to run single-disk LUNs you’re probably going to die of old-age before a backup finishes. But if you stripe it appropriately and make sure the LUN ownership is correct, you can do wonders with EMC’s “Tier-2″ array.

Ok, I’ve been playing more and more with VMware lately.  All of it personal, because the work opportunities just haven’t really presented themselves.

In relocating one of my “production” servers to the Fibre array I purchased recently, I ran into a problem.  I realized that I was doing it wrong and tried to cancel out of a disk move.

Every vmware vmfs disk is made up of two parts.  The actual virtual disk is contained in a file ending in “-flat.vmdk” then there is a header file that is named the same way, minus the “-flat”.

In my particular mistake somehow the -flat file got moved but the header file didn’t.  So when I went to re-mount the disk under the VM, it was just gone.

To give you an idea of the level of panic that was going on, the name of the disk that was lost was “finance.vmdk”.  Yes, this is the root disk of the server that runs my accounting package for work.  Not a happy time for me”

I played with it, I scoured the vmfs volumes to ensure that it didn’t get redirected to the wrong lun, I searched VMWare’s knowledge base (a useless endeavor) and was getting ready to rebuild the server when I had an idea.

I renamed the remaining flat file to “finance-temp-flat.vmdk” and went into the console and created a new disk of exactly the same size.  I then deleted the -flat file that was created, and renamed ‘finance-temp-flat.vmdk” to “finance-flat.vmdk” .

I restarted the virtual machine, and lo and behold, it booted without effort.

I then immediately shut it down and backed it up.

I then exhaled.

-SG

It’s ironic that the process of automating backups can sometimes take longer than running the backups manually.

Of course the process is SUPPOSED to pay off in the long run, right?

I’m starting the long and tedious process of converting my Veritas network-based backups to TimeFinder split backups.  it’s tedious, boring work, but it’s got to be done methodically (and well documented) so that should I win the lottery (or get hit by a bus) the next guy should be able to take over.

What I really need is a good scheduler.  I’ve done this before in an enterprise environment using Tivoli’s “Maestro” software but I don’t think the environment I’m in now is nearly big enough to rate that kind of expenditure.  (Not to mention I think if I went up the corporate food chain asking for $100,000 for scheduling software, I’d get laughed out of the county)

Some people just don’t see the value in automation….that is until the guy who knows everything quits….again.

 

http://www.ontrackpowercontrols.com/

Some of you who are forced to deal with MS Exchange might want to take a look at this.  For about $1,000 retail, the basic version of Ontrack’s “Power Controls” software is the best investment you could ever make.

I ran into the point this week where my Exchange “Brick Level” backs were taking more than a day to complete.  So the long and the short of it the next backup was ready to start before the prior one finished.

Going to TimeFinder/SNAP based backups was always the plan, but this little fiasco today accelerated that project.  But what I needed was to be able to do a mailbox or item-level restore.

OnTrack PowerControls does just that.  You can open the .EDB file directly and extract whatever you need out of it.  Search by name or subject or just select an entire mailbox and either drag it directly to your production exchange server or into a .PST file that the software will create for you just for this purpose.

Amazing stuff.  The other peice you get using the full EDB file instead of the Veritas Exchange client backup is you get everything in the deleted items folder as well, and if you’ve set retention for 30 days or so, it means that you get it even if the user has emptied the deleted items in religiously.  (Somehow I always end up cleaning up after the disgruntled employee who feels it necessary to dump everything he’s got to make life miserable on the people he’s abandoning)

And now, instead of having to rely on the MAPI connection into the exchange server for my backups, I just SNAP the volumes off, back-up the flat files, and I’m good to go.  I’ve got enough to re-create the server *AND* do a single-item restore in one backup.

And our SNAP based backups run, literally 1,000 times faster than the old fashioned way of doing it.

The backups work wonderfully, and the restore of a plain backup works as well.

What I’m having trouble with is something that’s so simple in Oracle and DB2 that it’s stumping me that I can’t find any documentation on it for SQL.

In any real RDBMS, you would restore the datafiles to disk and mount the database in a recovery mode, and then roll whatever transaction logs you want to into the database.  I’ve even seen it done with Oracle and DB2 where specific transactions can be edited out of the transaction log, IE to remove a transaction that corrupted a database while preserving subsequent writes.

But in SQL if you’re restoring a database you have to use the stupid GUI.

I’ll keep plugging away, but if anyone has any ideas I’d be greatful.  (Not being a Microsoft guy, I don’t have anything like this in my little bag-of-tricks.)

 Jesse

 

So I’m going through my MP3 library and the backups, and got to thinking.

How many times do you need to back up static content?f

I’ve got about 100G, but an MP3 library, like a lot of static storage, it doesn’t change much.

This means that every time i run a full backup of the drive, I’m backing up the same data every week.  Since I’ve only got DLT-7000 drives, that means a full backup of my environment is taking 4-5 tapes.

The alternative is to do a full backup of the data once, and keep doing differentials going forward.

It makes single file or directory restores very easy, but makes it a nightmare if I ever have to restore the whole drive.

It also makes for an expensive backup plan, because you never recycle tapes.