50Micron.com

Just a quick question as it’s *WAY* past my bedtime. 

Has anyone else ever heard of a customer that a sales-person or tech would quit rather than set foot in?

I’m remembering my days back at Disney in California.  (over 7 years ago, so I’m assuming the statute of limitations has expired on that one)

I got called up for a 6-week scripting engagement, and 18 months later I had to actually quit the consulting firm I was working for and move across the country to get away from them.  (Happiest place on earth?–Not if you’re in their IT dept.)

These words actually came from one of their “cast members”:

“Once you work for the mouse, you ALWAYS work for the mouse.”

When I told them i was moving to Virginia/DC to do goverment work, the response was even more ominous…

“The sun never sets on the ears…you can’t move far enough away.”

In the end, the experience I got at Disney (Scripting TimeFinder to back up AIX/DB2/SAP) was invaluable and one of the guys there basically taught me more about kornshell scripting than I think I’ve learned from any single source since then.

But the environment was toxic.  I’m curious as to whether anyone else has stories like these or if I’m the only one who runs into them?

Ok, I have no problem with tape.  It’s a *GREAT* backup medium when your requirement is portability for massive amounts of data and you’re not replicating said data.

If I had to ship 400TB of backups to Iron-Mountain, to protect against the earthquake-to-end-all-earthquakes tape would be my FIRST choice (though maybe, as a GIANT CAVE – Iron Mountain might not be.)

But… (and this is where it gets fun)

I have a customer who *LOVES* tape.

Wants to have it’s children loves it.

Uses it as primary storage loves it.

Now if you:

A> Have a few hundred terabytes of data to Archive.

B> Have millions of dollars to spend on giant room-sized storagetek libraries, and the space, power, and cooling that that entails.

C> Really love tape.

and most importantly

D> Live in the early 1980s

Then Archival to tape is *SO* the way to go.

The argument given is as follows.  “Tape is cheaper than Disk”

Well yes, on a terabyte for terabyte scale tape might be cheaper…maybe if you exclude the hardware.

But if you throw something along the lines of EMC’s Atmos product, or even Centerra, or I’d even go so far as to say the NetApp box appealed to me at one point.  (Now that the Celerra supports File Level Retention, I’ve been cured of that.)

Because when  you throw in modern options like replication and, dare I say it, DEDUPLICATION, Disk rapidly becomes the better, faster, more cost effective way to store your long-term data.

Now I wouldn’t recommend anyone go out and buy a DMX-4 for Archival purposes..  (Though if you want to let me know ahead of time so I can buy some EMC stock. – I’m not currently holding any.)

I checked, and the only Tape vs. Disk comparisons I could find on-line were done by storage vendors, each of which has their own agenda (and big surprise, the analysis came out favouring whatever they were selling), so none of them are valid in the grand scheme of things.  (I have a few things to say about marketing and statistics, but that’s a different post)

The things I look for when judging where to store data…

A> How many copies of the data do I need?

This is often overlooked and a question not asked.  How many copies of a piece of data do you really need?  And how many do you currently have?  I’ve been in one data center recently where they LITERALLY have boxes of old tapes stacked up along the walls.  (Note: Storing your backups WITH the system you’re backing up doesn’t do much in the event of a fire or natural disaster)

B> How long to I need to keep the data?

Retention policies are a big catch for a lot of people.  For “Backup” purposes (see my last post) I say two full backups are all that is really required.  If there is any kind of a likelihood that some critical corruption could be missed for weeks (or months) than adjust your backup strategy accordingly.  (or find a better way of auditing your production data for errors)

C> Does my data have to be portable?

Ok, this is aimed specifically at Tape.  The answer is this.  If you have a remote DR facility and a high-speed connection between them, there is absolutely NO REASON to go to tape for portability.  By virtue of Replication (whether it be the production data or VTL) you’ve already moved your data off-site.  Now if you’ve only got one data centre and it’s sitting right on the San Andreas fault line (I’ve actually worked here – not joking) then send tapes off-site.

Lots of them.

5 or 6 times a day if you can.

D> Am I storing a copy of production or my only copy?

If you’re storing a copy of production (running) then chances are you’re not going to need the backup.  If you’re protecting yourself against someone hitting the delete key accidentally, then maybe Celerra (SnapSure – periodic checkpoints that even the users can access themselves) or Centerra (Don’tEvenThinkAboutDeletingThis) are better options.

If you’re storing a copy of something so you can make room for something else, than backup tape is probably not your best option.  Consider an archiving solution like Atmos or Centerra, or even a Celerra with File Level Retrieval enabled – and version 5.6.44 and later supports de-duplication (both single-instance storage and compression) natively.

E> Do I have the money to spend now, or am I willing to spend more over time to keep the initial investment down.  (This is a valid question – and I’d like to know if anyone has any ideas on which would be the cheaper initial investment.

Just remember that you have to count the floor-space as well.  Something many people forget when scoping out storage buys.

if I want 150TB of storage and I want to do it with tape, what’s the supporting hardware going to cost me?  (A single CX4-240 with one rack of disks can provide up to about 220TB of storage with current drive-sizes.

A final note.  Remember with any “portable” backup solution that you have to keep your backups safe.  Tapes, like disks, don’t respond well to things like…well…dropping.  Anytime you transport a medium from one location to another physically you put that data at risk.

Just my .02 cents.

Ok, I’m curious as to whether anyone has an answer for this.

Why don’t more people boot VMWare ESX from the SAN?

It occurred to me the other night that I have 2 36G drives in each of my servers that I use possibly 10G of, when I already have a High-Availability storage solution at my fingertips.  I’ve got plenty of storage space, not even including the vault drives.

So I tried it.  I took one of my off-line VMware boxes. (I use DPM so at any given time 2 of my 3 VMWare hosts are probably in StandBy mode) and popped the drives out of it.

I turned it on, went into the BIOS and disabled the onboard RAID controller and enabled the boot BIOS on one of the Emulex HBA.s

I created an 18G lun on the clariion and assigned it to the host as LUN0 and poof, I have a boot disk.

Worked like a charm.  The one surprise (pleasant) is that VMWare seems aware of the multi-pathed boot device even without any form of powerpath on the system.  (That was my biggest concern)

So now I have my VMWare infrastructure running on a host with ZERO fixed-disk drives spinning in it.

So has anyone else tried this and know of any gotchas involved that I may not have run across yet?  I’ve done windows and Linux native boot-from-san many many times, but this is my first attempt at VMWare.

I’ve not however tried pulling a path to see just HOW resilient it is…I should probably should try that before I convert the other two systems to diskless operation, right?

Ok – certain people do certain things well.

I’m a storage administrator/architect.  If you present me a problem I will *ALWAYS* look at it from a storage standpoint.  If you present me with a non-storage problem, I’ll try and make it fit.

I’ve identified four types of systems engineer-type-people:

Storage people

Server people

Network people

Desktop people

I think that just about anyone in IT either fits into one of those four roles or supports one of those roles.

Now when you are looking to solve a problem, the solution you get depends on who you go to.  If you ask a desktop person to solve a network problem for instance, they will probably come up with something under the desk.  (IE throwing a linksys router under a desk.)

If you try and throw a server person a storage role, you’re going to get a server solution to that role.

Enter IBM GPFS.

GPFS is a server solution to a storage problem.  It’s obvious that the person who came up with the idea of solving a storage problem by loading software on a server is not a storage person.

POSIT:  Mutliple hosts in a web-farm need access to data.  Filesystems need to be R/W to an ingest server and R/O to the web-content servers.

Storage Solution:  NAS/NFS – Trunked connection to a real backbone and multiple Apache webserver front-ends running at 1G to play out data.  (Fastest data transfer is going to be the 45MB/Sec backbone coming into the building, so a single Gigabit connection can handle it.   F5 Round-robin load-balancer to distribute the front-end load.  (might also be proposed by Savvy network people, who tend to understand NAS)

Server Solution:  IBM GPFS solution.  Over a million dollars in net-new server hardware + software licensing (not including storage).   Each host accessing storage requires HBA’s, Drivers, fast RELIABLE network. and a level of complexity unheard of even in government.

From what I can tell, and maybe someone can give me a little more insight, works very much like Sun’s Shared QFS.  A metadata server acts as a gatekeeper telling which member servers can access which blocks on which disks.  There is still no simultaneous disk access because a SCSI lock is a SCSI lock.

Now from a storage standpoint, this is rife with problems.

First off, it would seem that if network access was compromised during a write data integrity could easily be compromised.

Secondly, Other than block-level mirroring of the underlying disks, I can’t see a good way to replicate this.  And block-level mirroring of the underlying disks would require an identical infrastructure at the remote/DR site wouldn’t it?  That is of course assuming that the metadata can be mirrored.

Now in database uses or other types of distributed computing I can see it being VERY valuable.  But for flat file storage and web retrieval I can’t think of a single good reason to use something so obnoxiously complicated.  Especially when EMC Celerra, NetApp, or just about any of the other higher-end NAS appliances would cost *SO MUCH* less and be *SO MUCH* more reliable.

/EndOfRant

I just got the following job posting and it made me, literally, laugh out loud, spitting latte all over my laptop.

If your sales rep allows you to do something like this, it’s a fair bet that s/he hates you (or is planning to buy your company out of bankruptcy later).

“WANTED: VMWare 1-month resident to assist with new deployment/planning around 200VM’s and new Celerra NS480′s being purchased by client. Will probably end up primarily being VM’s using NFS on NS Celerra Replication will be enabled between (2) NS480′s.”

The key points are:

200VM’s

Celerra

**NFS**

Replicator

Ewww…..

Did I mention NFS?

Someone actually sold this?  Even if the customer comes to you direct and says “this is what I want…” the answer should be “In the interests of protecting you from yourself, I can’t allow you to do this.”

I don’t care how much the deal is worth.