50Micron.com

Got into a scary situation this week.  Got called into help with a customer with a Mirrorview implementation.

Situation was:  Customer had Mirrorview/S set up within the existing switch environment, replication worked perfectly.

Then they reconfigured the switches to run FCIP so they could start replication to a remote site.  This is where things went badly.

First off – Cisco sets the Gig/E ports on the 9216i for jumbo-frames.  (MTU defaults to 2300)

This is a great idea for Fibrechannel replication, because a fibrechannel frame is 2114 bytes and this allows an entire FC frame to be sent within an ethernet packet.

Problem is that the default MTU on most network environments is 1500.  Now the *REAL* problem is that when  you first connect the GIg/E ports on the 9216i to a 6509 or other switch – it will at first appear to work perfectly…..until you try to pass data.

When you try to pass data across this link, the DF (Don’t Fragment) bit is set and the larger frames get dropped.  This causes an ISL connection between switches to flap, which causes no end of issues.  The fabrics will segment and re-join repeatedly until the first time you do anything that causes a reconfiguration, like updating the zoneset.  If you do that during a cycle where the ISL is going up and down, the vsan’s will fragment and stay fragmented because it will not be able to re-merge the fabrics.

So I come into this situation and the switches are so badly configured that it takes me a day just to get the ISL’s up and stable.  I set the MTU to 1500 on the switches, took the gig-e links down, and went to each switch and (carefully) deleted each vsan that didn’t belong on that switch.  (In addition to this being set up incorrectly, all three vsans merged when the swtiches were first connected due to the ISL’s being configured incorrectly)

Now the Clariion issue is still open.  A normal mirrorview configuration is as follows:

Source_SPAx –> Target_SPAx  (Where ‘x’ is the highest SP port #)

Source_SPBx –> Target_SPBx   (same here)

Now when the customer’s Clariion’s are zoned this way (in this case SPB3 to SPB1) nothing shows up in the Connectivity Status window.   But when I reverse the zoning, running SPA3 to SPB1, it shows up fine.  (Unfortunately Mirrorview doesn’t work in that configuration.

That’s where we stand.  A “simple 15 minute FCIP fix” is coming to the end of it’s third day.