Sharing an XSAN volume with Mac clients

andym · partially protected Joined: 31 Jan 2007 Posts: 5

Greetings Gents,

We are currently sharing the contents of an Xsan volume with a bunch of Mac Mini clients running Final Cut Express. These machines use media files from a folder containing quite a large number of files (up to 20,000 files at some times). The current setup consists of using Samba on a Dual G5 XServe XSan client machine that is also capturing multiple DV streams onto the same folder of the SAN (I know, I know, not a good thing, but the whole system was hideously underspecced).

My question is, assuming I'm buying a new Xsan-powered Intel XServe to host my files to the network. What file protocol should I be using to get the best performance with my other Macs? AFP? NFS? SMB?

I'm also considering adding a dual network card and using link aggregration to try and improve network performance to the clients. Is it worth doing?

Cheers,

Andy M

messenger82 · Xsan Master Joined: 16 Jan 2007 Posts: 113

I'll assume all of your clients are on the same network.

We have a similar setup and tested all three, our topology puts the three cisco 10/100 switches with gigabyte interconnect between us and our farthest point. AFP was the highest performing with a consistent 4.0mbs benchmark. SMB was benching around 3.8 but was easier to operate on other platforms and more of our students and staff were familiar with it. That plus our web servers are Windows powered we opted for SMB. For comparison FTP was delivering about 5.5 mbs. We briefly explored NFS and while we experienced no problems first hand there were reports on the web with directory structure issues relating to Xserve systems so we steered away.

dan-o-matic · Xsan Master Joined: 28 Sep 2006 Posts: 117

Whatever way you fix it, network is never a complete solution for video editing. There is always way too much overhead in a whole bunch of protocol layers (I believe 7 in the OSI model) to have a constant data stream. In my experience you're going to have frames dropping in these kind of situations wether it is one machine connected to a Gbit ethernet or 20 machines.
As for which network protocol is most lightweigth, I believe it is NFS, but this doesn't have any authenthication methods, so you're then having open shareponints.

mikeo · partially protected Joined: 23 May 2006 Posts: 7

I am currently running a gigabit network with a dual G5 xserve sharing the XSAN volume to non-Xsan clients. I added a 4x1GB ethernet card and aggregated 5 of the ethernet ports to give a 5GB pipe. It is connected to the XSAN via 2GB fibre. Anyone know if you can aggregate the fibre connections?

If you use OSX 10.4 Server you can control NFS sharing access permissions. We currently use SMB (for windows clients), AFP (for non XSAN mac clients), and NFS for linux/IRIX.

Also, has anyone successfully setup a multi-xserve file server. I'd like to distribute the file-serving process among multiple xserve's. Any insight would be greatly appreciated.

andym · partially protected Joined: 31 Jan 2007 Posts: 5

Well, I got the new xserve, my first intel one, quite impressed with the performance. I also got a Small Tree Communications quad port ethernet adaptor but I've yet to commission that, so I'm currently serving off one of the built in ethernet ports.

I have to say the latency and throughput has certainly improved markedly. I'm still serving over SMB as it seems there is very little to choose between it, AFP and NFS. I'm sure once I've set up the trunk through the new card I should see even further improvements.

Also, in order to accomodate the new xserve I've finally rolled up to 1.4.1 (been holding back at 1.2 after reading some horror stories concerning 1.3) and it certainly seems more stable, I'd had problems with the metadata servers not failing over and general crappy performance/reliability of the admin interface.

Just got to get this damn B4M software working properly now!

Bryson · Could work for Apple Joined: 31 May 2005 Posts: 43

I've seen some pretty amazing numbers sharing a DAS fiber raid over a 4 port 1gig small tree card link aggregated. I'm assuming that xsan would be able to do well too. But that depends a ton on the overhead as Dan said.

You may find that for your situation DAS shared through the server may be faster, unless you have to have the flexibility that the xsan volume brings you.

I tested around 100MB per sec r/w from a 6 drive raid 5 SATA box connected via 4Gb fibre to a new Mac Pro running leopard server sharing via a 4 port aggregated small tree 1Gb card to 1 ethernet client yesterday. (of course that number is BS from the "Blackmagic speed test") (note the word "magic" in the name) Still, the internal SATA drive was showing 38MB per sec on the client machine. NOTE!!! That was EMPTY raid, no traffic on the LAN, to one client, downhill, in a windstorm, ya hear?

Video over the LAN is not the best idea. SCSI protocols, like fibre channel, stream data. Ethernet sends packets and isn't too concerned about when they arrive. But low res video is totally do-able on these new servers and switches. Just don't hit them too hard or you will learn a lot about what editors sound like when frames drop.

(hint: YHAAAGGGGGG &^&*&%%$%#!!!)

I do not let folks edit over ethernet willingly, but I've seen it done all over. Look at friggin' editshare. lol

I think a server and some heavy ethernet iron will knock a lot of these NAS's disguised as SAN's in the head.

I intend to do some testing next week on this new install just so I can answer client's questions.

pradeep.pg · Xsan Master Joined: 20 Sep 2007 Posts: 84

I have had mixed experience over filesharing. We have lots of Customers who have Xsan and they share the Xsan volume over the network and we have never heard 1 single complaint from them regarding this. And on the other hand we have 1 particular customer where everything is a mess. At this customer place we had shared the Xsan Volume over the ethernet using a Fileserver with AFP and SMB. (Xsan 1.4.1 + 10.4.10) 4-6 months everything was working fine, bt then AFP started conking off on regular basis. We tried all the tricks in the trade bt we could not trace out the cause. So we made them shift to NFS and then we get this permission issue where-in the WGM shows proper permissions bt the user with write access on the folder could not write onto the folder. On the same setup SMB has conked off once... it crashed and crashed and crashed until it made the server unresponsive making us to reboot the server. Crying or Very sad

Mikeo: u cant aggregate Fibre connections. On 10.4/10.5 u can have permissions, bt NFS does not use Authentication and mounts the share-point upfront without even checking which user is mounting the share-point. Hence less secure.
We have done file-serving process among multiple Xserve on a small scale. We shared the same share-points on different servers with different protocols. i.e. We ask all Macs on the network to connect to 192.168.8.5 where the AFP service is running, and we ask Windows users to connect to 192.168.8.12 where the SMB service is running. Both the services share the same share-point.

Ernesto Sanchez · Xsan Master Joined: 31 Jul 2006 Posts: 74

Pradeep,

Leopard now supports Kerberos for NFS mounts. Also "Leo" has an improved TCP stack, multi-threaded now. In general Leo increases file sharing speed. As Bryson stated, you can edit over ethernet you just have to be smart about it. One requirement is a "non-Best Buy" grade switch. The guys at Small Tree say you need a low latency switch that uses hardware, an ASIC or FPGA. An alternative to bonding 1GbE ports on the server end is using a 10GbE connection to the switch. More switched include 1 or 2 10GbE ports.

On the client side it's a good idea to use a Small Tree card, single or dual port (trunked.) on at least a multi-core machine. Even though Leopard greatly improves the TCP stack the Small Tree card still has advantages over the built-in ports, mainly their driver. A single GbE connection usually consumes one core so a faster machine is a must if editing is also to be done.

Ernesto Sanchez
ernesto.sanchez@mac.com

mufugger · partially protected Joined: 22 Oct 2007 Posts: 7

is anyone using xsan2 to reshare yet?

does apple "officially" support resharing now?

thinking about getting xsan2 and resharing via afp/smb...but it's a big leap costwise, and im not sure if it's worth the risk yet.

JonThompson · Xsan Master Joined: 06 Apr 2007 Posts: 123

I'm resharing, and have been on XSan 1 and now 2.

The one thing that hasn't been mentioned yet is that it takes _forever_ to do the spotlight index, and it will degrade the reshare performance.

XSan2 seems to be more picky about dns, as I've been finding nits with my configuration that I've needed to solve.

I have not found a good measure of how much cache to give the xsan volume on the reshare server. I just bumped mine from 10 to 50 MB, as I am resharing user folders (and have been). The comments that I have received in the last hour tell me that it was a good thing to do. I'm wondering if I should go higher.

I guess we have to get used to not having ACLs again until XSan 2.1, which better be out tomorrow Razz

.

rstasel · Xsan Master Joined: 03 Aug 2007 Posts: 120

cache on the clients, do you mean in the automount.plist file on the clients (/Library/Filesystems/Xsan/config/<volumename>/)?

JonThompson · Xsan Master Joined: 06 Apr 2007 Posts: 123

In XSan2 server admin, there is the ability to set a Directory Cache size, the number of client worker threads, and when the file access time is run.

The pdf that Apple has is vague in reference to the directory cache. They tell you to increase it if you have a lot of small files, but they don't say by how much.

anselan · JBOD Joined: 21 Jan 2008 Posts: 4

We're sharing just two clients over ethernet/AFP (our other suites are on Fibre), with storage on Xsan 2 including an Apple XSR and Promise VTRAK.

Seems to work fine in fairly low data-rate formats, including DV, HDV and even DVCPRO-HD (720p). We only use a single Ethernet cable out of the Xserve that's doing the resharing (Quad 2.8GHz) and single cable into each of the clients.

Works perfectly, doesn't even get sluggish.