50Micron.com

Ok, that was too cutsey for such a classy establishment.

When you’re building a SAN, everything should play together, in the same SAN box if you will (with my apologies to QLogic.)

When you start putting in multiple stand-alone SAN islands you increase your maintenance overhead exponentially.  You also prevent the very thing that make a SAN a huge advantage over DAS.

Everything can see everything.

If you have a host and need to throw a certain type of storage at it, you can do that easily.  (and if it’s already cabled you can do it from your living-room)

However, if SAN A/B are connected to one group of hosts/storage, and SAN C/D are connected to a second group of storage, and SAN E (no redundancy) is connected to even a third, you run into a problem.

*NOW* if I want Host_A (Connected to SAN A/B) to see Storage_C (Connected to SAN C/D) I have to do much more than a simple zoning change.

In the end, this is where a few well placed ISL connections can come in handy.  VSAN them off so they don’t cause the fabrics to merge, create an IVR zone to route across them, and then presto.  Host_A can see Storage_C with a minimum of fuss.

Or maybe a *REAL* core-edge topology even.  Where you put a core switch with 24-port (fully subscribed) blades ISL’d to an edge switch, which maybe has the 4/4/40 configuration (4 8-Gig ports, 4 dedicated 4Gig ports, and 40 shared 4Gig ports)

And put one person in charge of it.  Preferably someone with a touch of OCD.

It’s funny that I got my start working with HP systems, first HP-MPE, then HP-UX, because I hate HP these days. HPUX is still the only operating system that can’t tolerate something as simple as a switch migration.

HPUX is still, to this day, dependant on the /dev/dsk/cxtxdx numbering in it’s volume groups.  Unlike all *REAL* Logical Volume Managers, HPUX doesn’t write any sort of PVID to the disk, doesn’t store the disk configuration for later use.

So what happens is, if the D_ID (Destination ID) of the port changes, HPUX will generate a new cXtXdX number, and all existing volume group configuration will be lost.

So what you have to do – gather the volume group configuration using ‘vgdisplay’ and ‘syminq’ before you start.   Deactivate the volume group and export it.

Once you’ve moved the cables, and rezoned, rescan the bus and run ‘insf -e’ to build the new special files.

Run the ‘syminq’ again and use the information you get there (new symdev –> cXtXdX number)  to rebuild and reimport the volume groups.  Reactivate the volume groups and ensure everything is present.

Reboot the system and all should be good.

What’s pretty pathetic is that even Windows can handle such a simple change, Solaris doesn’t care about the port #’s, AIX cares, but the simple process of removing and rebuilding the devices in the ODM fixes that, because the AIX LVM is dependant on the PVID of the disk, not the device number.

It just seems like something that would have been so simple to fix, so long ago….

Brocade bought SBS.

I don’t know how many of you happen to have looked at the resume I had posted – but I spent a couple of years at Strategic Business Systems (www.sbsplanet.com).

I’m not sure what Brocade is hoping to get out of this.  SBS doesn’t do sales, and doesn’t even really have any influence in the buying process.

SBS has been a pretty successful company – grown by leaps and bounds.  I would never go back to them because they wield their non-compete agreement like a battle-axe and use every opportunity as a chance to hook someone in.

The real problem is that Brocade as a switch manufacturer is on it’s way out.  From a 90% install base they really have nowhere to go but down, and Cisco is gaining very quickly.

I’m not a big fan of Brocade.  I have a brocade switch in my home SAN not because of any preference, but because they are cheap on Ebay.   Their ASIC’s are slow and their licensing is oppressive.

Does this make sense to anyone?

I was whiteboarding a switch migration today with one of the DCR people, and it occurred to me:

If you can visualize the data’s path through the system, your life gets a lot easier. 

If you can see the path the data is going to take through the San, I.E. from Switch1, Blade1, Port4 through an ISL to Switch2, to Switch 2, blade 2, port 16.  you put yourself in a better position to shorten that path.

First, cut the ISL’s.  Unless you can absolutly avoid it, don’t pass data down an ISL link.  Pushing data through an ISL does two things.  It creates a bottleneck where there may not be enough bandwidth to handle multiple hosts.

Second, keep the intra-switch hops to a minimum.  If you can do it, plug the storage and the higest performing hosts into the same grouping of ports.  Most switches use 4 port ASIC’s, and if you look at the switch you’ll see that the ports are grouped by ASIC.

However, the 32port Cisco blade is an exception, uses basically the same ASIC, but shares the bandwidth over 8 ports, so keeping that in mind it’s best to connect the lower performing hosts (Windows?) to those ports, and definately *DO NOT* connect ISL’s to the 32 port blades.

In response to this article on “EnterpriseStorageForum”:

 Synchronous SAN Sets Fibre Channel Distance Record

My Response:

True Synchronous transmission works over any distance – if you can live with the latency.   The problem is that most hosts operating systems can’t.  So different buffering schemes are cooked up to fool the host into thinking the write is complete on both sides when in fact it’s not.

Any time you get over about 30km the latency, that is the time it takes for the IO to be transmitted, acknowledged, and released, becomes about that of a normal unbuffered physical drive, about 20-30 ms.

Any further and you will start seeing slower and slower response times and eventually IO timeouts on the source hosts.

In order for a storage system to be truly “synchronous”, the array cannot acknowledge the I/O to the host until it’s received the write ACK from both the source, *AND* the target array.  If there is buffering going on between point-A and point-B, such as a cisco MDS with the buffer credits cranked up or a Nisshan IPS3300, it is not a truly synchronous replication, because the failure of the switch on the source (or target side) will cause the target array to have missed I/O’s that have already been acknowledged to the host as having been complete.

Sorry – but this test, as it appears to have been run was obviously designed by the various vendors to accentuate their hardware without showing the failures and flaws in the logic.  I’m sure I could walk in there and in about 5 minutes simulate a link failure that would have the remote site in an inconsistent state at worst, or having to roll back journaled IO’s at best.

Cisco has released version 3.0(2) of their firmware, and with this, comes, finally, an EMAIL-HOME feature for the switch.  It’s not perfect (because unlike the dial-home on the Symm, this is dependent on an external server or two, and doesn’t work if the network connection is down.

The long and the short of it is the Cisco Fabric Manager, which is installed on whatever host you’re going to manage the switch from (as it is with the current versions of the cisco SanOS) monitors the switch via SNMP and when an error condition is detected, sends an email to EMC with the appropriate information.

I want to say it’s about bloody time.  Cisco switches have been the red-headed stepchild of EMC for too long.  it’s about time they supported them with the fervor that they do the rest of their products.

The down-side?  You have to go from version 2.1(x) to 3.0(x) and I’ve not heard back from EMC at this time as to whether this is an online upgrade in anything shy of the MDS95xx switches.  (I’m using MDS9216′s, and an MDS9140, which I suspect need a full reboot to come up on the new firmware rev)

 

Ok, there is something I hate about Cisco- and it’s not really about Cisco, it’s about EMC’s complete failure to completely support their MDS series of switches.

First – When you buy an MDS 9xxx switch from Cisco, it comes with the ability to dial into Cisco when there is a problem, much like EMC does with the Symmetrix.  This feature is not available when you buy the switch from EMC.  (IE a Cisco switch with a $10,000 sticker on it that says “EMC”) you don’t get this ability.  In fact, you get no dial-home capability at all. 

I called in to the SAC, talked to our sales team, our former Project Manager who was (sort-of) handling the implentation, etc, and the long and the short of it is that in order to be notified when there is a problem with the switch, you have to set Control Center up to email you, and then you have to call EMC.  (I’ve called them so many times I know the number by heart)

Secondly, (and more to the point of the subject of this email) making a zoning change in ECC requires that you log into the switch and type “copy run start” at the prompt to save the configuration to the start-up.  I’ve known people to go months without doing this – only to find that every zoning change made in the last number of weeks is lost on the reboot of the switch.

And Third – when you do call EMC for support, they have to wait on Cisco to ship the parts.  So technically you’re not even really getting EMC support, you’re getting third party “ghost” support.

This seems to me to be inexcusable considering the number of MDS switches that EMC is putting on the market.

My advice – if you’re going to buy a Cisco fibrechannel switch, and I do recomend them, buy it from Cisco.  Until EMC decides to support them fully, I don’t think it’s worth the convienence of getting everything from one place given what you come up short on.

http://www.mcdata.com/about/news/releases/2006/0808c.html

Given the choice between brocade and McData I’ll usually choose McData.  Now that I don’t have a choice, I think I’ll stick with Cisco.

Truthfully Brocade needed McData.  McData has a solid director class switch, one that will withstand the tests of time and the rigors of a large-scale production datacenter.  Brocade never really has.  What they’ve had is a collection of blade-mounted edge switches linked together with a pseudo-fabric back-end.  Either way, it was a joke. 

I remember specifically a series of incidents down at a DoD client in Norfolk, VA.  Huge fabric.  4 64 port Brocade DS-12000B “directors”  (EMC wouldn’t even give it an “ED” designation usually reserved for “Enterprise Director” and instead gave it a “DS” designation for “Departmental Switch”.

Out of the four enclosures that were sent, one of them was dead on it’s face, and the rest of them underwent many many failures, blades dropping off-line, losing power, the whole nine yards.

The funny part is, I remember seeing this switch when it was in it’s infancy at the company’s headquarters in northern california.  It just didn’t look right.  The only thing I could see that they did correctly was to at least mount the swtich boards vertically, which allows for better heat disipation.

The one thing I really have to say for Brocade is that their command line interface is much easier to deal with that Mcdata’s ever was.  With common-sense command structures and a consistent format it was easy to get into a routine while doing zoning or creating aliases or whatnot.  I was never comfortable with Mcdata’s command line, mostly because EMC discouraged it’s use at just about every turn.  (Cisco’s command line is a bit weird, but if you’ve used IOS you’ll fall right in with it) 

That’s the great part about the Cisco, is that as an IT Manager, you can actually put the SAN equipment into the realm of the Network gurus without too much additional training.  (Save for zoning and theory and the like)  (That plus the fact that once the MDS switches were released, I literally don’t think I went on another engagement that was a new McData install)

I would like to invite comments on this – I’m curious as to what other people see as coming out of BrocData.  More speed?  Better reliability?  10G?   Or just a little more moderate competition for Cisco?

Jesse

It’s like religion with some people.  Cisco vs. Brocade.

As someone who is no longer affiliated with any storage manufacturer or vendor, I can finally voice a real opinion.

As a consultant, I’ve installed more than 100 switches of every size and flavor.  The most disasterous install was a job in Norfolk, VA, where a DS-12000B (brocade) ”director” had to be swapped out at the last minute for a defective backplane.  (Every time we plugged a blade into it the blade would fail – permanently)

If given the choice I will always prefer to go with the McData for just plain old brute force reliability.  I know of at least a dozen of the old ED-1032 switches that are still functioning in a production environment.  They last forever, are easy to manage, integrate into most of the emc packages seamlessly, and are great bang for the buck.

That being said, some of my best recent experiences have been with the new Cisco MDS series switches.  Now they take some getting used to, but once you set them up they require minimal management.

Cisco – with VSAN support and it’s internal routing capability (FCIP is a real possibility with the Cisco) is hands down one of the best switches on the market.  Using VSAN’s you can logically carve up a switch into multiple virtual switches, guaranteeing no cross-talk between certain ports (I believe each VSAN also has it’s own fibre name-server as well)  Of course if you’re zoning properly, with only a single initiator and single target in every zone, you don’t have to worry about this…

The downside of Cisco of course is that to use the command line, you have to be quite a bit more knowledgable about the Cisco IOS, as the fabric os follows most of the same command syntax.  (IE – to disable a port you type “shut” to enable it you type “no shut”)

The GUI is fairly intuitive on both, though Cisco requires a *LOT* of java and the management server has to install and run on the system you’re managing it from.  (unlike the brocade, which can be managed from any host with a basic java version installed.

The Cisco seems to have the best Interoperability – working well with most other Vendor’s hardware, I’ve yet to see something that won’t plug into a Cisco switch, including a brocade switch.

if you’re not planning on doing anything fancy and you just need a good reliable switch, I’ve got to say go with McData.  But if you want VSAN capability, FCIP, a switch that can route IP like the best routers in the world, buy a Cisco.

If you’re going to buy a Cisco switch, my only suggestion is that you buy the switch from Cisco, and not from EMC.  Cisco support (dial-home) is built into the swtich, but currently EMC has no method of remotely monitoring cisco switches.  (They suggest using ECC, which works, but puts someone between the switch and the EMC Software Assistance Center (SAC).  So if a switch or component fails, it’s up to you to both notice it and call it in before the clock even starts on the repair.  (the 4 hour response window is from the point when EMC is first notified of the issue).

Of course EMC’s sales force might disagree.