XSAN3.1 / OSX10.9 sloooow write speed

dutchbird's picture

I'm trying to figure out why our XSAN performance is crawled to a halt.
We have an XSAN 3.1 running on OSX 10.9 with 2 MDC controllers and 4 clients (Mac Pro)
The XSAN volume is a raid6 volume of 22TB with a Raid1 2x2Tb for metadata on a 4Gbps Sanbox 5600 fiber switch.

Writespeeds are now almost 0 with BM speedtest utility while readspeed is decent at 230Mb
Volume is pretty full : was 1.8Tb free out of 22Tb capacity and is now 2.62Tb free (of 22Tb cap.)

small manual file copy with drag/drop seems to be OK.

Is this a question of defragmenting the volume ? or should I be looking into something else.
I see no strange mentioning in the loglist.
Tonight I will try this defrag tutorial from Promise Tech unless I find something else.

Any advice is appreciated :-) thx !

csanchez's picture

I would suspect you have some bad free space fragmentation. After you see slow write speeds writing a large file, check the extent count of the new file with:

snfsdefrag -c /path/to/file

This will not defragment a file. It will only report the number of extents. If you see 1 extent, the problem is not fragmentation.

If it's not fragmentation, I would check for hardware issues. I'd suggest looking for underperforming arrays or drives by monitoring write speeds in the RAID array (Promise VTrak shows this in the web interface) or by monitoring write speeds to Xsan LUNs from the client using iostat. Also check that you have WriteBack caching enabled. If your battery is recharging or has died, WriteBack caching can become disabled which will significantly decrease performance.

If you determine that fragmentation is a problem, you should consider backing up all data to another volume, reinitialize the volume, then copy your data back onto the volume. If you can, get rid of old, unused files when you do this to give yourself more free space. That is not a great solution, but it will resolve both file and free space fragmentation.

Whatever you do, please DO NOT run snfsdefrag -r. While that should reduce file fragmentation, it is virtually guaranteed to significantly increase free space fragmentation and lead to even worse write speeds.

abstractrude's picture

don't fill over 90%.

-Trevor Carlson