Bulletproof storage
by Jesse on May.19, 2007, under General, Symmetrix
You know – I’ve worked with EMC hardware for a long time (in technology terms, eons.)
in my going on 10 years experience working with EMC hardware, I’ve seen ONE symm completely fail. an *OLD* Symmetrix 3530 failed during a 9 month project I was working to decomission 4 of them and migrate them to DMX1000′s.Â
The funny part is, it failed because two of the three power supplies went at the same time…..and still took almost 7 hours to go down completely. (This is where EMC dropped the ball, they were getting the alerts for some time and because it was registering as “AC Failure” assumed that it was environmental (customer power) and didn’t check it out properly.
At the end of the cycle, when the batteries in the Symm were almost dead, we finally took the hosts down gracefully and shut the Symm down gracefully to await the new power supplies. They arrived about an hour later.
I’ve seen Symms in a lot of states, and heard of more. Stories abound about a Symmetrix that was lying on it’s side in California after the Northridge earthquake, still passing data.Â
However, still more stories about New Orleans after Katrina that weren’t as good. No hardware, no matter how bulletproof, will work sitting in two feet of water. 
That’s what you get for putting a data center 25′ below sea-level in a coastal area.
May 22nd, 2007 on 12:11 am
I never understood anyone putting ANYTHING of monetary value in New Orleans. I can’t imagine it’s not a hell of a lot cheaper to just pay the lease on the fiber to have it somewhere *more reliable*, than to replace all that hardware.
May 22nd, 2007 on 1:11 pm
What I can’t believe is that insurance companies will underwrite a datacenter built in any unstable location.
When I worked at Intuit, ages ago, they actually relocated their corporate datacenter from the environmentally stable Tucson, Arizona to the side of a cliff overlooking a fault-line north of San Diego California.
Logic? I think not.
May 25th, 2007 on 11:28 pm
I worked for a company who had a roof collapse during a storm, pouring immense amounts of water onto all the kit in the computer room. In fact there was water pouring out of the 3 holes (3-pin sockets in the UK) in some of the regular power outlets on that were fixed to the columns that supported the roof. Yes your reading that right, there were waterfalls or rain water pouring out of live electricity outlets!
There was a huge IBM i890, two Compaq EVA5Ks, a bunch or HP/Compaq server and some Dell servers. Amazingly, the only issues were with a few of the Dell servers! The HP servers were picked up and tilted to pour the water out! When they were plugged back in they fired right back up and got on with the job.
May 26th, 2007 on 4:56 am
It’s funny, I did some work at Columbia University in New York, their datacenter is in a basement right next to the river. Their “water emergency system” consists of a cart with a bunch of rolls of 10mil plastic sheeting to cover the servers up with.
July 4th, 2007 on 1:17 am
Hello,
This is a query for Symmetrix SRDF pros…
I am getting this error on est/split on symmetrix device group..
======================================================
bash-2.05# symrdf -g crm-tibco-app_dg split -force -symforce
Execute an RDF ‘Split’ operation for device
group ‘crm-tibco-app_dg’ (y/[n]) ? y
An RDF ‘Split’ operation execution is
in progress for device group ‘crm-tibco-app_dg’. Please wait…
Unable to acquire the Symmetrix device lock
=========================================
Though i have a fair knowledge what this error means i am not able to find the solution. I have my important data replication going on , so would request you guys to suggest a solution to this issue.
Reg
~Dee
July 4th, 2007 on 1:34 am
Try this:
symdev -sid -lock list
You may find there are device specific locks on the drives in question. This can result from a command hanging or being interupted in mid stream.
Now my “official” response is to have you call EMC at 1-800-782-4362 (1-800-SVC-4EMC) and have them dial into the box and remove the lock, but if you are confident that you know when the locks were created, you can break them yourself.
**DISCLAIMER** The author is not responsible if, in attempting this, you bollocks your system up. Breaking devices or configuration locks can be a very dangerous practice, and should be left to EMC and their liability insurance. (Wheras if I get sued all you’ll get is debt)
symdev -sid -force release -lock ## (lock # will be identified in the “list” command)
Again, I can’t urge you enough to let EMC take care of this, because only they can tell what lock #’s are significant for different reasons, and can usually guarantee that you can break the lock without incident.