50Micron.com

Yeah – one of those posts.  But I’m not talking about the latest blockbuster, and I’m not even talking about anything even remotely “Matrix-Like” (though it does make the list)

I’m talking about the movies us techie-types hate to admit that we loved.

Here they are:

Honorable Mention: Short Circuit – campy but loveable.  Good movie for the kids.

10. Weird Science – John Hughes at the height of his mediocrity.

9. The Net – Love me some Sandra Bullock, but who believes that hackers are going to take over the world with Macs? (or that Ms. Bullock is going to be the one to stop them?)

8. Independence Day – See “The Net”: Jeff Goldbloom isn’t believable as a hacker/nerd/geek in the first place.  Hacking an alien species with a Macbook is beyond bad.

7. Hackers – Angelina Jolie before she really was anyone, and yes, we get to see her websites.  Plot-line was fun, dialog was horrible, tech was sadly misrepresented – the scary part though is I’ve known people like Cereal Killer….

6. Tron – So nerdy even the nerds won’t own it.  Stay tuned – rumors of a Tron2 running around, I saw a pirated preview on youtube (Link – Here)

5. Real Genius – Val Kilmer plays a slightly left-of-center genius.  Great laughs, good teen/college movie.

4. Stargate – ok, not so much a nerd/geek show but the scientist saves the day every time. (and gets the cute slave-chick in the process)

3. The Matrix (Any one of them) – There is something to be said for the ‘willful suspension of disbelief’ – I thoroughly enjoyed the movies and one day want to learn kung-fu by ramming a needle into the back of my head.

3. Blade Runner – Can’t hate anything with Harrison Ford in it, but it took a lot of hemming and hawing over whether this went above or below The Matrix.  It is a serious marvel.  Just got it on Blu-Ray and it’s SPECTACULAR.

2. Wargames – My inspiration.  That and when I was growing up I was about as nerdy as Matthew Broderick…well…..is.  (And he still managed to get Sarah Jessica Parker)

1. Star-Trek – Didn’t think I’d forget this did you?  Most of the movies were great, once you got past the ones that sucked (Voyage Home, anyone?).  The movies are here as an afterthought because they were pure marketing.  An attempt to capitalize on the popularity instead of furthering the idea.   The series rocked though, and I will always have a warm-spot in my heart for Gene Roddenberry.

We are a mainframe shop, and some time ago, when we converted from DMX2 to DMX3 hardware EMC informed us that we would need to convert to Timefinder/Clone since our target devices would be RAID5.

We have recently learned that when performing a Clone1 to R2 that a full push of all tracks will always be invoked … which is not acceptable .. we need to be able to fallback with incremental restores (Clone1 to R2, R2 to R1, R1 to Standard should all be incremental if source site is still intact as in a power outage). EMC is now telling us that we will need to convert to Timefinder/Mirror clone emulation in order to have this capability … and that a full push of all data will be necessary to perform this conversion.

1. Does this sound correct?

2. With Timefinder/Mirror clone emulation can we be IPLed from a Clone1 image and concurrently invoke a restore from Clone1 to R2? (The manual seems to indicate that the volumes will be ‘not ready’ in this situation). If so, will the resulting R2 image be ‘consistent’?

3. Currently with Timefinder/Clone we do not invoke R2 to Clone1 remotely from the source site LPAR but instead invoke this process from an LPAR at the DR site (where the R2 and Clone1 images reside) since execution time is three times longer if invoked ‘over the wire’ from the production DMX3 site (EMC claims it has something to do with passing all of the commands back and forth). Should we expect to experience the same elongated elapsed times for Timefinder/Mirror clone emulation R2 to Clone1 if we try and invoke it from the production DMX3 site?

4. Any other considerations we should be aware of if we convert from Timefinder/Clone to Timefinder/Mirror clone emulation?

Thanks!

It’s ironic that the process of automating backups can sometimes take longer than running the backups manually.

Of course the process is SUPPOSED to pay off in the long run, right?

I’m starting the long and tedious process of converting my Veritas network-based backups to TimeFinder split backups.  it’s tedious, boring work, but it’s got to be done methodically (and well documented) so that should I win the lottery (or get hit by a bus) the next guy should be able to take over.

What I really need is a good scheduler.  I’ve done this before in an enterprise environment using Tivoli’s “Maestro” software but I don’t think the environment I’m in now is nearly big enough to rate that kind of expenditure.  (Not to mention I think if I went up the corporate food chain asking for $100,000 for scheduling software, I’d get laughed out of the county)

Some people just don’t see the value in automation….that is until the guy who knows everything quits….again.

 

You know, everytime I start thinking of “Cloning” I am afraid the far-right is going to burn me in effigy, just on the principle of it.

But in this case, I’m talking about cloning disks within an array for a data migration.

The decision was made to move our Microsoft Exchange (Corporate Email) from the Tier-1 (Raid-1+SRDF) to Tier-2 (3+1 Raid-5) storage.  I guess the logic is that losing a day of email won’t hurt us horribly.  (Ok, maybe it will, but I’ll get into the solution to that problem in a different post)

In this case, I’ve got the following drives:

7x 84G Metavolumes
4x 33G Metavolumes
2x single hypervolumes.

The new Exchange administrator, who I am actually most impressed with so far, would like to add 3x 200+G Metavolumes to the mix.  The main reason for the move is that we’re rapidly running out of Tier-1 storage, and need to save it for expansion, production growth, etc.  (Or buy new disks, but that’s yet another story)

So I am going to use this opportunity to demonstrate the power of TimeFinder/Clone.

I’ve created the new volumes on the Raid-5, mapped them to the front end ports, and prepared the masking scripts to move the device masking from the old devices to the new. 

Now in the old days, the way you’d do this is; Shut the hosts down, do a bit-by-bit copy of the disks (if you’re lucky you can do them in parallel, otherwise it’s single-threaded) change the pointers on the host, and bring them back up, hoping everything is exactly how you left it.  Net downtime could be in the neighborhood of several hours, if you’re lucky.

This is a new beast.  Enter TimeFinder/Clone.  Now I have the blank devices.  I do things in a slightly different order.

1. Create the clone session – this establishes the pairing of devices.
2. Shut down the Exchange services.
3. (each node) Unmount the existing disks using admhost or symntctl
4. (each node) Change the device masking so Exchange sees it’s new disks
5. (each node) re-scan the bus to remove the entries for the old disks and create the entries for the new luns. (at this time, it will see all new blank disks)
6. Shut the cluster hosts down.
7. “Activate” the clone session 
8. Bring the first “Active” node of exchange up
9. Bring the passive node up.

Net downtime is about an hour.  The reason being, once the clone session is activated, a background copy is started.  However from the target side, any reads to “invalid” tracks on the target disks actually get serviced from the source disks.  As far as the host is concerned it’s all the original data.  

As the copy progresses, more and more of the reads are serviced from the new disks.

When the array receives a write to a track that hasn’t been copied to the target yet, that track is first copied, THEN the write is processed on the target disks only.  This preserves your production disks in their original state.

With the advent of TF/Clone, I’m surprised anyone still uses BCV’s.  They’re so “old-tech”  The main hang-up of course was the fact that while you (in theory) could protect a BCV using Raid-1, the performance hit you took during establish and split operations was so bad that it wasn’t worth it.  With TF/Clone you can go from Raid-1, to 3+1 Raid5, to 7+1 Raid5, etc. etc.  Without minimal performance impact.

The only downside comes of course when you’re cloning production volumes while they’re in use.  Since reads are being serviced by the production disks and not the clone disks (technically, the tracks you’re reading are simply being copied to the target while the host read is serviced from the cached track) you’re impacting the production spindles during the copy process.

It’s a cool bit of magic – and it’s really fun to play with the minds of people who don’t understand the technology.

A user posted this as a response to “The Many flavors of EMC TimeFinder“  I felt it rated it’s own post.

———————————————

Q: “My experience is mostly using IBM Sharks. I’m now working in a very large EMC environment, foucsed on backup. I’m wondering if the TimeFinder/Clone’s cloned volume can be permanently mounted on another host. Specifically, I want to avoid importing a nd exporting DGs from Veritas every time I need to ’split’ the mirror, as I would have to do with TineFinder’/Mirror. “

A: TimeFinder/Clone can in fact be a permanent copy of the data.  As long as the either of the copy options are set, the -precopy option, which copies all data before the session is activated, or the -copy option, which performs a full background copy of the data while the target (clone) device is available to the backup/development host, is used.

The default, (no switches used) will not result in a full clone and as such is only available while the clone session is in the “Active” state.

Q: What I’m looking for a is a point in time copy which is mounted on a backup media server while the production data disk are mounted on the production server. The application on the production server would be paused, then the clone would happen, then the application could be restarted. The cloned data would “magically” appear on the clone volume set mounted on the backup media server.

A: If you’re doing this simply for backup, then copy-on-access is easier and more flexible.  COA allows you to create and activate a session, copy the data, and terminate it immediately.  Truthfully TimeFinder/SNAP is equally up to this task (as it’s largely the same thing) but in my opinion you get more flexibility by purchasing clone instead of snap, though you do spend more money on disks in the process.

Unfortunately, as I’m assuming you’re talking about a Windows environment, there isn’t much “Magical” about it. 

Q: The largest issue I’ve run into is the insistence by a few EMC folks that the clone volume must be mounted on the prod host, then unmounted and mounted on the backup host.

A: Again, assuming you’re talking about Windows, this is incorrect.  You can’t remount the snap or clone of a production volume back to the same host because windows is largely dependant on the Signature of the drive.  Doing so can actually confuse windows into thinking that there are two copies of the disk and cause data corruption.  (I can’t say anything, the same is true of AIX, which, if you’re using LVM and not raw disks, has the same signature (they call it PVID) dependency.)

I’ve worked in several environments where the target volumes are simply unmounted, synched, and remounted.  So long as the target ID and the signature don’t change, this is not usually an issue.

The benefit to using clone is that you can leave the session active, or use one of the ‘copy’ options to produce a full copy.  Then, restoring from the disk in the case of a failure becomes a real option.  You simply reverse the sync, remount the production volumes, and start the application or database from there.  (Ok, it’s not simple, but I’m not sure even I have the drive space to post the full procedure here.  )

 

I’m not really bashing their workstations, i’m actually quite fond of Vista on my laptop.

However, when it comes to servers, I view being in an environment where Microsoft is the PRIMARY operating system by a factor of 20:1 as a form of torture akin to having my finger-naiils pulled out or being tied to a chair and forced to listen to “Barney” all day.

What I hate most about Microsoft – (and if I keep this up, I’m going to have to rename this site to Microsoft-Hates-Me.Com) – is that it can’t handle the simplest tasks.

For a split-mirror backup, whether it be TimeFinder/Mirror, TF/Clone, or TF/SNAP, the process is the same:

1.  Freeze the database / filesystem
2.  Snap the volumes.
3.  Thaw the database / filesystem
4.  Mount the volumes on your media server host.
5.  Back the filesystems up.
6.  Unmount the volumes from the media server
7.  Terminate the Snap session

Seems pretty basic.  Microsoft seems to have trouble with #4 and #6.  Seems this “Super OS” they’ve got can’t handle the idea that SCSI devices might go on and off the bus at different times. 

EMC gives a tool, TFIM (TimeFinder Integration Modules) at at least allows you to perform the commands that Microsoft doesn’t even make available, mount, unmount, flush, etc.   But god forbid you reboot a host while the SNAPS are inactive or the BCV’s are established (and thereby not ready to the host).  You’re screwed.

Can *SOMEONE* please write a decent SCSI driver for Windows?  Please?

Let’s face it, if it’s Oracle, DB2, or anything along those lines, I can snap a copy and back it up with my eyes closed.

MSSQL, being a pretend database, has me stumped.  I’m so used to archive logs that I’m not even sure how to use TF/Snap to back up the database.

This is my understanding.  MSSQL doesn’t do “Archive Logging” in the traditional sense.

In a “REAL” database system the process is as follows:

1.  You put the database into “Hot Backup” mode.  In Oracle this quiesces the data files and writes all changes to the transaction log.  (When you take the database out of backup mode, the transactions in the log are then played into the database)

2.  When the above is complete, you can issue a command in one form or another to switch out the last transaction log, which closes one file and opens the next one, and then back up the database files along with the closed transaction log files via whatever file level backup process you have in place, whether it be TimeFinder or just having NetBackup pull the files from that server.

At Disney we did just that, with DB2, and moved in the neighborhood of 250+ Terabytes to tape every night.

At a number of other sites I’ve done the exact same process with Oracle.

Enter MSSQL, a Playschool excuse for an RDBMS, and I’m stumped.  See – the problem is there is never more than one “database.LDF” file for logging.  How am I supposed to quiesce writes to a logfile when it never closes it?

Then add that to the process for rolling transaction log backups forward in MSSQL is dependent on the idea that you used the MS Backup process to back it up.  It seems to be completely unaware of file level backups of the database.

I’m at a loss here – any ideas?

I can’t even talk during the day because of the great sucking sound coming from our microsoft infrastructure.

From a storage end, it’s even harder, because natively Microsoft doesn’t have ANY tools to unmount a filesystem or quiesce a production volume so you can take a hardware based snapshot of it.

Of course they’ve introduced VSS, which is like saying that there is never any way but their way to clone a volume.

The main problem with VSS (besides it being a product of the limited minds at microsoft) is that it’s yet another stupid host-based application that requires system resources on the host when engaged.

VSS, and most other volume “Snapshot” providers, work in the same way.  The simplistic description is “Copy on first write.”

Let’s go over it step-by-step.

(more…)

I don’t know what most people know about TimeFinder, so I’ll start with a short introduction.

EMC Timefinder was developed to provide customers with a dynamic mirror they could use to try and cut some of the tediousness out of copying data, whether from one host to another or within the same host.

When I was working for EMC, Timefinder was still more or less in it’s infancy, and only came in one flavor.  (Now referred to as “TimeFinder/Mirror”)

A Timefinder volume is a volume that is essentially a dynamic mirror of a production volume.  Called a ‘BCV’ (for Business Continuance Volume) it was a straight 1:1 mirror of your production data.  (If you were running in a 2-Way-Mirrored configuration, the BCV essentially become a third mirror.)  At a point-in-time, you could split the third mirror off it’s production pair and make it available to another host.

The main benefits were obviously discovered in Backup.  You could split a BCV of your production data off, mount it to another host, and back it up to a locally attached tape library with zero network overhead.  Using this you could also use a single backup host to back-up copies of BCV’s from multiple production hosts.

Another good use for BCV’s was in development.  One story I like to tell was from I was an admin several years ago.  Developers liked to “break” their database on a friday afternoon, knowing that the restore from tape would take the better part of a day and in doing so they guaranteed they would make their tee-time.  With the advent of TimeFinder, I was able to tell them “Not a problem, I’ll have it back up in 20 minutes.”   The reason being that I could restore from the mirror almost instantly.

The main negatives for TF/Mirror are that in all cases, the initial synchronization has to complete before the data can be made available to the target system.  Now this is mitigated by the fact that after the initial relationship is established all further mirrors are incremental, meaning that only changed tracks are copied to the BCV volume, but it can still be a time consuming process.

Now EMC has come out with two new forms of TimeFinder in Symmetrix, very similar to the Clariion functionality.

TimeFinder/SNAP

  SNAP uses a process called “Copy On First Write”.   This uses a much smaller volume than the production volume as a “virtual device”  (Called a VDEV)   The VDEV serves as a list of pointers for each track in the volume.  Reads are serviced from the original volume until the track is changed.  When the track is changed the original data is copied to a cache area, and the pointer for this track in the VDEV device is changed to point to the cached original track.  In doing this the VDEV device will contain an exact copy of the production data as it was when the Snap session was activated.  When the data is no longer needed, you terminate the Snap session and the cached changes are discarded.

The data is available the instant the SNAP session is activated.

The downside to this is that all reads touch the production volumes.  In a heavily utilized system there can be a noticible impact.  Another negative is that SNAP sessions are limited to the amount of cache set aside.  A usual configuration is to set aside about 20% of the area used by production as “SnapCache”  This can then be used as needed.  If the SnapCache fills up, the Snap session ends and that is that.

TimeFinder/CLONE

  Clone uses another process, similar to SNAP, called “Copy On Access”.  Clone volumes are identically sized to the production volumes, which of course uses up more space, but provides for a more permanent home.  This provides the data permanance of TimeFinder Mirror, the speed of TimeFinder Snap, and the agility to move data from Standard volume to another standard volume. (Raid-1 production volumes to Raid-5 Development volumes is a good example)

What copy-on-access offers is the unique ability to use the volume before it’s actually finished mirroring.  When a clone session is first started, all the target volume contains are pointers to the source volume.  Every time a track is accessed, (read or write) it is copied to the target volume first.  (prior to any write operations)  If no options are selected this is the only time a track is copied.  If the -copy option is selected when the Clone session is created, a background copy of the production volume is started.  This will eventually result in a copy of the data that will persist after the clone session is terminated.  (when no option is specified, the data will disappear when the session is terminated)  There is also the option to copy (mirror) the entire production volume to the target volume before the session is activated.  This is called “Precopy” and is a close emulation to what is done using TimeFinder mirror without the limitations of having to use BCV’s as targets.

TF/Clone has to be the best of all worlds.  It gives you the flexibility of Snap with the data-resilience of Mirror, and the flexibility of being able to go from one volume to another without restrictions on what type of volume your target is.  (TimeFinder/Mirror requires the use of BCV volumes)

Timefinder is the production that gave me my introduction into EMC.  TimeFinder and SRDF are also the technologies I’ve implemented more often than any others in my work for (and with) EMC.

If you’ve got questions, feel free to post them.  You’re probably not the only ones.