50Micron.com

When the triage guys @ EMC actually listened to the person calling in the problem and directed the call appropriately. (Just had one I specifically asked for PSE and they routed me to software for some unknown reason, now, three hours later, it’s been re-routed to PSE)

When the support specialist working a call would stay through the end of the problem, and didn’t give you the “Sorry my shift is over, please explain your problem to a new guy for 45 minutes before you do anything else productive”

Follow-up. It’s now 0100 the next morning. I’ve been working on this problem for 8 hours straight. The software guy who was originally assigned went home without turning the call over to someone else. I’ve gotten not a single call since before 9pm.

This is not support people. This is the opposite of support. I’ve since fixed the problem myself, without the help of the SAC/PSE folks. The sad part is if I had done this 8 hours ago I could have been home eating my corned-beef / cabbage and drinking my Guiness.

Totally missed St. Patrick’s day, the only religious holiday I actually observe.

Not happy.

Ok, for about 4 months I’ve had an SRDF problem that has been kicking my ass. 

SRDF over Ethernet presents a host of new problems, unique to Ethernet. 

First, and mostly – the RE adapters want to see *ALL* of the remote adapters in the RDF group.  That means if you have four RDF/Ethernet (RE) boards in the source box and four RE boards in the target, the way the Symm does the SRDF is that it expects EACH source RE to see *ALL* target RE’s.  It allows for a nice load balancing scenario, but doesn’t do much to allow for either dual paths or direct attach connections.

The second problem I found was an undocumented feature of Solutions Enabler 6.3.1 which caused problems querying SRDF devices, and more importantly, problems running any configuration change script against an RDF device.

For instance – the config change script as follows, took about 45 minutes before it timed out and died without error.

convert dev 0db to 2-way-mir ;

Now some will recognize this as a simple “de-configuration” of an SRDF volume.  But because the two symms have to be correctly communicating with each other for this to work, the script simply failed.

I went in to work this morning determined to kick this thing once and for all.

First off I picked up a couple of 29xx series Cisco switches, configured them with ports 1-4 in a trunk leaving the 4 optical ports on each switch for the Symmetrix connections.  This allowed me to put all four connections on each Symm into one subnet.

Then I realized that that’s NOT in fact how the symm was configured – the CE left the symm configured (granted – per my suggestion) so that each RE was in a subnet with only the corresponding RE on the remote side.  My understanding being that if there was no gateway it would not try to contact the other (target) RE’s.  (bad assumption)

So I did the one thing neither Customer-me or Consultant-me should ever do.  I did a binfile change on the Symm without contacting EMC and putting it through any form of change control.  It’s not a difficult thing to do a binfile change, provided you know the passwords to get into Symmwin and have a basic understanding of how a Symm functions. 

It’s just frowned upon by EMC because they don’t like the idea that they are not the gods of the universe that they purport themselves to be.  (for binfile changes not related to initial setup they also expect to be engaged professionally, for approximately $5k per change – if I were smart, and had the time to spare, I’d start offering my services at half that price)

After the binfile change all lights as far as SRDF went were green, but my config change was still hanging.  A brief email exchange with one of the SAC’s better engineers (I’ve worked with him before) pointed me very quickly to the Solutions Enabler upgrade.

The 6.3.2 upgrade did the trick, and is now in the process of being pushed out to all servers.