Dual Channel Fiber - Necessary for SPEED or REDUNDANCY?

[JesusAli]

Hello Everyone,

For a Mac Pro 2008 with a 4GB/s Dual Channel Fiber Channel PCIe card (2 sfp's, 4 strands total), how exactly does it use the two Fiber Channel ports?

Does it concurrently send data out of both ports?
Or does it concurrently Send with one and Receive with the other?
Or does it only use one at a time and wait to Send or Receive?

Connecting to a 4GB/s SanBox 5600, what would it do to my overall Throughput to only use 1 channel on the card? Half it, two-thirds it? Or simply leave it un-redundant?

Only half the necessary SFP's were ordered and before I inform the admins about the additional $1500 cost I wanted to have some idea of how much performance I'd be arguing/advocating for.

(I realize I could go and unplug one right now, but I don't have time at the moment and I thought someone might know immediately off hand. Or could inform me of other concerns aside from just speed.)

Thank you all for any insight.

[Guru Evi]

Both. Depending on what you're doing a full 4-channel is not always needed to all stations. The recommendation is to have full 4-channel to the MDC's and 2-channel or 4-channel to the devices depending on their usage of the system.

As far as how it's used: usually round-robin unless otherwise configured. They are 4Gbps full duplex meaning 4Gbit up and 4Gbit down. They fail over (unless otherwise configured) if one of the cables goes missing.

If you have for example a system that streams video at 8Gbit/s (you might want to calculate in overhead, collisions etc.) and you have only 2 cables and one falls off, you'll start losing frames.

Also if you have lots of requests, depending on the length and type of the cable you won't be able to fill the theoretical 4Gbit/s on your cable because of the limitations of the speed of light and the inherent delays between all the devices, switches etc. so the 2 or 4 cables are used to interleave requests on route.

If you're just serving files I would say use 4 lines to your MDC and 2 lines to your machines. If you're streaming video, you should see your load and calculate out if you would benefit from the 2 extra lines.

[abstractrude]

the key here is what your doing. prores, uncompressed, dv ? what kind of data do you need? tell me what your doing, I bet your fine with just one 4gig link to the client.

i dont see any reason why you would do a full 4 channel to the controllers. thats a lot of precious real estate on your switches and its expensive!the worst part is you won't see any performance increase.

maybe if you were using fork or something on the controllers,but that would be a stretch and it would be a massive amount of data. think about it, 16gb a second!

the only data accessed on the controllers is journal data, unless your doing something with a mounted volume on a controller like copying files or doing admin of the filesystem. i usually do my admin of the actual files within the filesystem as root on the controller running the fsm, but thats a personal choice and i know I never need more than 4gigs a second...

do a standard dual channel card and send one port to your first switch and the second to another. follow the same on the backup and controller and your good. there are things that will go wrong with your client before you need to worry about fibre redundancy. if your fibre is down on the client because of the switch you have bigger fish to fry.

also during high numbers of requests your bottleneck will be metadata access times and the controllers ability to keep up with requests.not the fibre link to your clients. also it sounds like you only have 1 switch, without a second switch the redundancy is moot, your more likely to loose a switch than a fibre card or a port on a card. plus you have a backup controller already available if you lose something in the primary controller.

what are you doing with the application. its rare that a station needs more than a 4 gig link. the only time I have ever seen the need was for a customer doing 4 10bit uncompressed streams at once for playout at a tradeshow style performance.

[JesusAli]

Thanks for the thoughts guys.
But I am not certain we are all using the same terminology.

I'm calling a "Dual-Channel" Fiber Channel card, the standard, 2 SFP card. This would terminate 4 strands of Fiber Optic cable.
This card:
http://store.apple.com/us/product/MB842G/A?mco=NDc1ODc1NA

So a quad channel card would have 4 SFP ports, and 8 total strands of Fiber.
This card:
http://store.apple.com/us/product/MB843G/A?mco=NDcwMDMwOA


So my question is:
Can the SAN work at full potential with the Clients using only 1 4GB/s SFP (two Fiber strands)?

The biggest load we're looking to have students work with is probably 3 simultaneous streams of HD in 10-Bit ProRes.

Reading: 27.5 * 3 = 82.5MB/s


Since the MDCs are in the same room as the SanBox Fiber Channel Switches (we have 2), we are connecting the MDC's to the FC Switch with two Copper Apple SFP cables.

[abstractrude]

dont worry about the strands of fiber. its gonna be multimode. anyway you need one 4 or 2 gig link per machine. jesus we overshot that one.

also. if you only have one switch, you only need a single cable going to your controller. you wont see any advantage ti that second cable, just a wasted port.

[JesusAli]

Thanks abstract! You've been a generous and big help.

We do have 2 5600 switches.

I had originally been duplicating ports over the two switches. Fiber 01 to port 01 on the top switch, Fiber 02 to port 02 on the bottom switch, ect.

But then I read something about the "considerable latency of the ISL (switch interlink)" so that made me think I should keep things on one switch if I could help it.

I suppose then we would need the 15 extra SFPs to ensure Fiber channel redundancy... in case a switch went down. ...or to save $1500, I would just have to go and manually move all the SFPs if a switch went down...

Well that give perspective.

Thanks again!

[abstractrude]

honestly, those switches are pretty reliable. I have seen one die. The fan went out in a machine room that was already too hot and you know the rest of the story. do 2 cables from your controllers, one to each switch and your client can connect with a single cable. make sure you storage it also connected to each switch. you only need one cable from each controller. I know there are two ports on each raid processor/controller, dont be tempted you wont see any help in your workflow just wasted switch space:)

[addihetja]

There are two questions here: Redundancy and Speed

Each stream of video will only travel through a single path. Multiple streams will travel along all available paths. If you're doing DV it doesn't really matter. Neither will 10-bit SD. I haven't updated since 1.4 so 2.x may be different (I believe Promise targets behave a bit differently than Xserve RAIDs wrt multi-pathing)

As for redundancy I have have one installation where "they're always tripping over the cables" resulting in the client hanging. Having both ports plugged in would minimize this happening.

[JesusAli]

[abstractrude]

YOU ONLY NEED 1 4GIG SFP. I Promise. Your using ProRes.
If you want to put two just because, go for it.
4Gb SFP=500MB

[JesusAli]

"4Gb SFP=500MB"

aha! I hadn't thought it of that way. since the entire Throughput of my Xsan Volume isn't more than 500MB, I don't need the two connections... very interesting.

Thanks.