We're getting very strange, but very damaging, file corruption happening with ProRes MOV files residing on XSAN volumes. While the files can still be opened without issue, there are frames within the MOV that are being corrupted (skips, garbled images, etc.).
Unfortunately this seems to be happening very randomly and it doesn't look like it's a specific volume problem as I created a brand new volume and saw the problem rear its head within a week.
Every time we have shut down the system and brought it back online I have run cvsck -jv, cvfsck -nv, and if there are any issues (which has only happened once) cvfsck -wv. All the LUNs are online and I don't see any errors in the logs.
Apparently this is something that has been around for quite a long time as the owner of this company used to work for Apple in the iTunes unit back in 2006 and they saw this same issue. Looks like Apple isn't telling anyone about it and haven't ever addressed it, but I'd love to hear any ideas/solutions from the community.
This only seems to happen with ProRes MOV files. All MPG files are just fine.
Here's a rundown of our XSAN hardware:
(2) Mac Mini MDCs running Lion Server with 2xSSD drives in a RAID 1 mirror with a TrendNet 1GB USB ethernet adapter and a Promise SANLink
Brocade DCX-8518 Fiber Switch
(6) Proware 42-bay chassis with dual 2GB RAID controllers and dual 1100 watt power supplies
(20) ATTO Faststream 8550 SAS->Fiber RAID controllers
(40) 24-bay JBODs
Most all of the clients are running Mac Pro machines with Lion (a few are running Snow Leopard) and we've got 8 Windows 7 machines running StorNext FX client software.