In all the work I did in both arenas I found that when multiple targets are put into a single zone, they will try to act like initiators and log-into each other. While I never found a specific problem with this behiavor (IE couldn’t pin a failure down and attribute it to the behavior) I did notice that even from a pure management standpoint, keeping them separate was logically the best practice. IE if two devices aren’t passing data directly between the two of them, they don’t belong in the same zone. I also noticed a slight (not really quantifiable, so it’s technically opinion) increase in stability when you do single-initiator, multi-target.

This came down to ‘single-pair zoning’ – which has always been my preference (and practice) but EMC’s official policy is ‘single initiator zoning’ which is slightly less restrictive. It’s a single HBA and multiple targets in the same zone.

The behiavor becomes apparent when you look at the login table on a Symm that is using Single-Initiator, Multi-Target zoning. (using ‘symmask -sid xxx -dir 8a -p 0 list logins’ for example) you will see multiple Symm ports logged into your target.

The only time you should see this is in the case of RDF over Fibrechannel, when you are actually intending to Pair the SRDF devices together.

Hope this answers your question.

See for me it’s easier, everything on the SAN has 4x paths (excluding the obvious tape, etc).

I have scripts that go out and check powermt periodically (ECC is fine and dandy, but I a bit of an old curmudgeon in my trusting department). We have the same host attatched to both a DMX (primary storage) and a Clariion (Oracle flashback), trying to write monitoring scripts to take into account nuances like that is rather annoying. Having a set policy of 4x paths for everything allows scripts to not worry about such things, else I’d have to add a rather significant ammount of logic in the scripts.

For me I don’t directly write symask or symconfigure scripts so whether it’s 1x device or 200x devices isn’t much different. I run things through a perl script I made to create the scripts (script to create the script) so I don’t have to worry about typos, etc and it can do a bit of very rudimentary checking as well. Since I don’t create the scripts directly, 2x paths vs 4x paths vs 32x paths doesn’t add any risk in typos, configuration mistakes, etc. Something could be said that the end scripts have 2x the number of characters in it, so it would take longer to actually submit and run on the DMX; but from my experience if I’m dropping a lun onto 4x instead of 2x FA’s it really doesn’t take much measurable time longer (but I don’t stare at the screen with a stopwatch waiting for symconfigure to complete the full 85x steps either).

These above for me fill the requirements of (to steal a bit of your verbage)

1) InsaneGeek abhors extra work and avoids it at all costs
2) If you script it right the first time you won’t need to do it (or type it) again

I’ve been a perl junky for years and years, so everything I do I try to make a script for, and the symcli is almost made directly for doing perl scripts. When scripting if I can reduce the amount of logic and exceptions I need to worry about I have less extra work to do and there is less chance for messups (and less spaghetti code).

i.e. to check the status of DMX & Clariion devices using a script

Using a requirment of Clariion has 4x paths and DMX has 2x paths
parse the output of powermt display
Determine which devices are Clariion
Determine which devices are DMX
Match Clariion devices to number of active paths
Match DMX devices to number of active paths
if Clariion device 4x is the expected number of paths else alert
if DMX device 2x is the expected number of paths else alert

or

Using a requirment of Clariion has 4x paths and DMX has 4x paths
parse the output of powermt display
Match device to number of active paths
If paths not equal to 4x alert

For me this is the cleanest, easiest, and simplest way to think of it. If all I know is a host is attatched to port BA, I know that it should have access on 7BA, 8BA, 9BA & 10BA. This is very easy for people to understand, everything on the san should have 4x paths, rather than these storage arrays should have 2x paths and these should have 4x paths and these are how you find out which is which (which is really a huge pickup). It isn’t that more is “better” so turn up the paths to “11″ (it only goes to ten), but that 4x paths just seems to be the magic number that works, from a simplicity, understandability, and availability perspective for general use. No special cases, no special rules, no special anything, just 4x; being the anal person as well being able to say “one shallt allways have four paths to thine EMC storage no matter the make” makes me happy.

IG – First off – this is not supposed to sound confrontational, and I’m sorry if it’s coming off that way. It’s always been my experience that this is how discussion works and how people learn, myself included. (I don’t pretend to know it all, I just try to make it a habit to know where to look)

As for my clever technical retort, It’s not a “you’re doing it wrong” it’s a “you’re doing more work than you need to”.

Two absolute truths in life:

SanGod abhors extra work and avoids it at all costs.
If you do it right the first time you won’t need to do it again

(I keep trying to tell my 11 year old the latter, he’s not buying it)

The skill lies in finding the balance between those two statements.

At heart, I’m lazy. I don’t want to do more than I have to. I have a friend/former co-worker who followed me into two different jobs and said he loved it, because by the time i was done, things ran more smoothly in fewer steps than they had when I got there.

Provided you’re running powerpath, the following setup:

HBA1 – Fabric1 – FA1

HBA2 – Fabric2 – FA2

(and if you find the need for extra bandwidth or redundancy)

HBA3 – Fabric1 – FA3

HBA4 – Fabric2 – FA4

(if you have four fabrics, make the obvious substitutions)

Is optimal. Fewest moving parts, Fewest places for configuration problems / mistakes to affect you, and you’re protected from the failure of any component. EMC’s internal infrastructure is robust enough to handle anything that the back-end can throw at it. And all you need is something to monitor PowerPath (which ECC does a pretty good job of) and you’re golden. Because the newer DMX’s use SFP’s and not fixed optics, you even have an easy recovery from a port failure. (an educated, experienced guess places 80% of port failures in the optics)

it also means your configuration change scripts and lun masking scripts are half as long, which when you’re presenting 100 devices at a time means something.

I dunno, I guess I see where you’re coming from in light of the ISL situation you’ve got, but it just doesn’t seem “clean” to me. I’m kind of anal that way I guess.

Your fan-out concern, is really the first and only legitimate quantifiable statement why zoning to two ports could be a problem. A DMX-3/4 has a fan-out ratio of 128:1 so even that argument while valid for a configuration with > 768 hosts going to a single DMX with FC SRDF (or >1024 hosts with gig-e SRDF); we are into levels that are semi silly as to whether it’s a concern or not for *general* usage statements (unless > 768 hosts going to a single DMX is a general deployment rule than the exception).

With that I’m not trying to aggressive, argumentative, jerk, etc about it, I’m very open to being convinced that 2x is better than 4x, that it’s the way of the future and my thinking is out-dated stupidity; but nobody has given me any reasonable data to change my opinion. I’ve given a number of examples that are still relevant today, while nobody has been able to show any reasonable technical counter arguments. People have made statements about managing, difficulty, and complexity, but nobody has made any quantifiable statements around why it’s so difficult/complex to manage. So nobody has shown a technical reason why it’s bad (except in a very extremely rare case with massive amount of hosts), nobody has been able to (or even attempted) quantify why it’s difficult. It’s your blog and I appreciate you maintaining a resonable dialog when you don’t have to with a random guy on the internet, but I don’t think anybody has shown me anything substantial other than “I said it, so that’s the way it should be done”.

I agree, some people are going to do it the way they’re going to do it and that is that. At least the advent of 4gbit FC means that when you trunk 4 cables together from core to edge, the odds of you overutilizing your trunk is minimal. maybe in a burst, but highly unlikely under regular load.

As a consultant I can only make suggestions. What the end-customer does from that point on is up to them. I just find it ironic that people pay consultants only to ignore their suggestions.

If there are multiple “shortest paths” it will simply round robin them.. So if you have two ISL’s and you connect Host-A, Host-B, Host-C, Host-D in order, A and C will be on one path, and B and D will be on the other.

This is especially problematic in clustered environments, because there is a 25% chance that both active nodes will end up utilizing one ISL, and the passive nodes will sit and exactly NOT utilize the second.

Trunking solves this problem, but requires that you burn more ports than you may like. Also Brocade’s trunking is limited to single-ASIC, so you have to put all 4 cables into one ASIC to get a trunk. Single ASIC failure results in a segmented fabric.

McData and Cisco switches both trunk independant of ASIC, which is a good thing, because it means you can spread ISL’s across blades and still get the performance.

I still maintain that pushing data across ISL’s is universally a bad idea, but understandably sometimes it can’t be avoided. When I was out at Disney they had the storage in the basement and the hosts one floor up, they used ISL’s between directors on each floor to hook everything together.

I guess the point is if you’re going to run to 4 FA ports, you’re gaining nothing by that unless you’re running 4 HBA’s. The FSPF calculation is still going to be based on the HBA path, and if both FA’s are in the same switch you’re gaining nothing, if the FA’s are in the different switch, you are potentially introducing extra, needless hops.

Best way is to devide the Symm up depending on the number of switches you have. If you have 2 switches, use the Low-FA’s to one, and the high-FA’s to the other. If you have four, devide the symm into quads, with Low/AB on one, Low/CD on the second, High/AB on the third, and High/CD on the fourth.

If you have three switches. God help you. There really is no easy way to balance across three swtiches. Lord knows, i’ve tried. (Don’t ask, though I can say it involved a government agency and leave it at that.

As far as doing one HBA to multiple FA’s, again, you gain nothing but added complexity and cost. If for the sake of argument you are following the 16:1 fan-in which I believe is still the recommended. Over 4 FA’s (with single-initiator/single-target – and reserving the D ports for SRDF) that’s 192 total hosts you can connect to this Symm. If you go 2:4, that cuts your number in half, or to 96.) Remember, the Fan-In refers to the number of HBA’s zoned to a single FA, not the number of real hosts.

So to get to the number of host ports you should have had before, you have to buy twice as many FA’s. Not a problem from EMC’s perspective, but a waste nonetheless. If the SAN is assembled correctly, with Dual fabrics that are connected properly, you gain absolutely nothing in the process.

i.e. you have
3x, 2GB ISL links between some Brocade switches
As an aggregate on the hosts you normally push ~3GB of ISL throughput across both fabrics
Theoretically I’m happy as a clam as you actually have 2x the throughput that you normally use (6GB of ISL’s, 3GB of actual traffic)

Problem comes in when the lovely path algorithm comes into play, this is the tricky part:
Host A: uses 10MB
Host B: uses 10MB
Host C: uses 140MB
Host D: uses 140MB
Host E-?: uses whatever

There is a possiblity that the Brocade has assigned hosts C & D to the same ISL port, as it’s calculated upon first seeing the port come up and stays stuck there until an “event” (failure, or otherwise) causes the switch to recalculate paths. Even with double the amount of ISL bandwidth than I use, I can still run into a problem (280MB of traffic will not fit into a 200MB ISL), heck I could quadruple my ISL bandwidth and still have this issue. Admittedly this is some old-school living, but I’ve felt the pain and as you said have “years of experience” and learned from it. Now you can buy channel bonding licenses for Brocade, etc to reduce this but I’ve lived the dream, had the experiences. Having ran Brocade switches for years and years, I’ve had to do the whole fabric goes down multiple times over the years (existing 1GB switches, want to plug in 2GB switches, need to change core fabric values so they match, major code revs). Don’t remember what code it was exactly but after going from 3.x to 4.x of Brocade firmware or 4.2 to 4.22 (been a number of years, kinda fuzzy). Any fabric event (non-major ones, just simply reboot a host, etc and have the hba login) and running HBA’s on completely different switches would be kicked off the fabric (2nd customer in the world to run into this lovely one). This required us to poweroff all the switches in the entire fabric simultaneously, one by one bring a switch online upgrade it, power it off until all the switches were complete and then power the entire fabric on.

You still have not really given me a reason that is a really bad reason to do it, that it causes major headaches in day to day management. I have lived through some very fun experiences, and I’ve found this works best for me. The above noted brocade ugly 2nd customer in the world bug, had a tier 1 array powercycle while in production due to vendor service engineer (not in a nice way either, committed cache writes to hosts were not flushed, but were instead lost… very, very bad day), arrays with clustered heads that instead of failing to the working head the working head shutdown losing *all* access to the storage (fix: powercycle both heads). We’ve found very interesting ways for DMX’s, Clariions, NetApps, Brocades, Cisco’s etc to fail while they were in production, things that aren’t supposed to *ever* happen do… I’ve truely lived those pains. Maybe having 4x paths to an HBA is not really absolutely “needed”, but if the only penalty is 5min setup time, that’s the only penalty? Why wouldn’t one be doing it? That’s really is the question, unless 5 minutes is really that much more important to you than a little more piece of mind? Not to be argumentative but Is it that much more painful, is it in a whitepaper that says this is against best practices, etc I’ve felt the goodness with no pain. What really is the hearburn that you experience from this, other than it takes 5 minutes longer to do.

After 4 months or so of work,, I just sent out a PO this afternoon to do *all* chassis switches in the different fabrics and upgrade *all* DMX & Clariion units to a number of DMX-4′s.