MDC failover crashes clients with CVFS ASSERTION FAILED

xsanguy's picture

Simply failing over a volume from one MDC to the next results in approx. 1/2 to 1/4th of the clients crashing hard with CVFS ASSERTION FAILED panics.

Crashes seem to (but not always) involve clients actively transferring a lot of data at the time.

matx's picture

What's the setup?

Describe the software/hardware involved

wrstuden's picture

What assertion? There are many, and they each have a unique log message.

xsanguy's picture

Here's the complete crash report

[code]Interval Since Last Panic Report: 1967519 sec
Panics Since Last Report: 1
Anonymous UUID: 3248A8EA-D63B-47B2-B654-0E5879376CD7

Mon Oct 8 18:26:34 2012
panic(cpu 16 caller 0xffffff7f81eeef75): "CVFS ASSERTION FAILED: f_rwlck->rw_state & RW_WRITE line 1879 file lock.c"@nomad/md_debuglog.c:358
Backtrace (CPU 16), Frame : Return Address
0xffffff838eefbc40 : 0xffffff8000220792
0xffffff838eefbcc0 : 0xffffff7f81eeef75
0xffffff838eefbd10 : 0xffffff7f81ef87d0
0xffffff838eefbd60 : 0xffffff7f81ee0795
0xffffff838eefbda0 : 0xffffff7f81ee44b4
0xffffff838eefbdc0 : 0xffffff7f81f0be5e
0xffffff838eefbdf0 : 0xffffff7f81f20cc7
0xffffff838eefbe40 : 0xffffff7f81f20b73
0xffffff838eefbe70 : 0xffffff7f81f0ea2d
0xffffff838eefbee0 : 0xffffff7f81f0f8e7
0xffffff838eefbf40 : 0xffffff7f81ef53f3
0xffffff838eefbfb0 : 0xffffff8000820057
Kernel Extensions in backtrace:
com.apple.filesystems.acfs(457.7)[68FF12CD-B6D6-3D9A-9810-D99AEF0FFB63]@0xffffff7f81e6b000->0xffffff7f81fb3fff
dependency: com.apple.iokit.IOStorageFamily(1.7.1)[5DF14080-C47C-3393-9176-6A2CD3CFE1CB]@0xffffff7f80785000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
11E53

Kernel version:
Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64
Kernel UUID: A8ED611D-FB0F-3729-8392-E7A32C5E7D74
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 547895591426323
last loaded kext at 50342042120: com.apple.filesystems.smbfs 1.7.2 (addr 0xffffff7f8081b000, size 241664)
last unloaded kext at 172841006954: com.apple.driver.AppleIntel8254XEthernet 2.1.3b1 (addr 0xffffff7f80c76000, size 110592)
loaded kexts:
com.ATTO.driver.ATTOExpressSASRAID2 2.1.1
com.ATTO.driver.ATTOCelerityFC8 2.0.1
com.apple.filesystems.smbfs 1.7.2
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.filesystems.autofs 3.0
com.apple.driver.AudioAUUC 1.59
com.apple.driver.AppleHDA 2.2.0f3
com.apple.driver.AppleTyMCEDriver 1.0.2d2
com.apple.driver.AGPM 100.12.42
com.apple.driver.AppleMikeyHIDDriver 122
com.apple.driver.AppleMikeyDriver 2.2.0f3
com.apple.driver.AppleUpstreamUserClient 3.5.9
com.apple.driver.AppleMCCSControl 1.0.26
com.apple.kext.ATIFramebuffer 7.1.8
com.apple.iokit.IOUserEthernet 1.0.0d1
com.apple.iokit.IOBluetoothSerialManager 4.0.5f11
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AudioIPCDriver 1.2.2
com.apple.driver.ACPI_SMC_PlatformPlugin 5.0.0d0
com.apple.ATIRadeonX3000 7.1.8
com.apple.driver.AppleLPC 1.5.8
com.apple.driver.XsanFilter 404
com.apple.driver.BroadcomUSBBluetoothHCIController 4.0.5f11
com.apple.iokit.SCSITaskUserClient 3.2.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.BootCache 33
com.apple.driver.AppleFWOHCI 4.8.9
com.apple.driver.AppleRAIDCard 209
com.apple.iokit.IOAHCISerialATAPI 2.0.3
com.apple.driver.AppleUSBHub 4.5.0
com.apple.driver.AirPort.Brcm4331 530.4.20
com.apple.driver.Intel82574L 2.2.0b2
com.apple.driver.AppleAHCIPort 2.3.0
com.apple.driver.AppleUSBEHCI 4.5.8
com.apple.driver.AppleUSBUHCI 4.4.5
com.apple.driver.AppleEFINVRAM 1.5.0
com.apple.driver.AppleRTC 1.5
com.apple.driver.AppleHPET 1.6
com.apple.driver.AppleACPIButtons 1.5
com.apple.driver.AppleSMBIOS 1.8
com.apple.driver.AppleACPIEC 1.5
com.apple.driver.AppleAPIC 1.5
com.apple.driver.AppleIntelCPUPowerManagementClient 193.0.0
com.apple.nke.applicationfirewall 3.2.30
com.apple.security.quarantine 1.3
com.apple.driver.AppleIntelCPUPowerManagement 193.0.0
com.apple.filesystems.acfsctl 457.7
com.apple.filesystems.acfs 457.7
com.apple.kext.triggers 1.0
com.apple.driver.DspFuncLib 2.2.0f3
com.apple.driver.AppleSMBusController 1.0.10d0
com.apple.iokit.IOSurface 80.0.2
com.apple.iokit.IOSerialFamily 10.0.5
com.apple.driver.AppleHDAController 2.2.0f3
com.apple.iokit.IOHDAFamily 2.2.0f3
com.apple.iokit.IOAudioFamily 1.8.6fc17
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleSMBusPCI 1.0.10d0
com.apple.driver.ApplePolicyControl 3.0.16
com.apple.driver.AppleGraphicsControl 3.0.16
com.apple.driver.AppleSMC 3.1.3d8
com.apple.driver.IOPlatformPluginLegacy 5.0.0d0
com.apple.iokit.IONDRVSupport 2.3.2
com.apple.driver.IOPlatformPluginFamily 5.1.0d17
com.apple.kext.ATI5000Controller 7.1.8
com.apple.kext.ATISupport 7.1.8
com.apple.iokit.IOGraphicsFamily 2.3.2
com.apple.iokit.IOFireWireIP 2.2.4
com.apple.driver.AppleUSBBluetoothHCIController 4.0.5f11
com.apple.iokit.IOBluetoothFamily 4.0.5f11
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.2.0
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.7
com.apple.iokit.IOCDStorageFamily 1.7
com.apple.iokit.IOFireWireFamily 4.4.5
com.apple.iokit.IOSCSIBlockCommandsDevice 3.2.0
com.apple.iokit.IOUSBUserClient 4.5.8
com.apple.iokit.IO80211Family 420.3
com.apple.iokit.IONetworkingFamily 2.1
com.apple.iokit.IOAHCIFamily 2.0.8
com.apple.iokit.IOSCSIParallelFamily 2.5.1
com.apple.iokit.IOSCSIArchitectureModelFamily 3.2.0
com.apple.iokit.IOUSBFamily 4.5.8
com.apple.driver.AppleEFIRuntime 1.5.0
com.apple.iokit.IOHIDFamily 1.7.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 177.5
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 7
com.apple.driver.DiskImages 331.6
com.apple.iokit.IOStorageFamily 1.7.1
com.apple.driver.AppleKeyStore 28.18
com.apple.driver.AppleACPIPlatform 1.5
com.apple.iokit.IOPCIFamily 2.6.8
com.apple.iokit.IOACPIFamily 1.4
Model: MacPro5,1, BootROM MP51.007F.B03, 12 processors, 6-Core Intel Xeon, 2.66 GHz, 32 GB, SMC 1.39f11
Graphics: ATI Radeon HD 5770, ATI Radeon HD 5770, PCIe, 1024 MB
Memory Module: DIMM 1, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 2, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 3, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 4, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 5, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 6, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 7, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
Memory Module: DIMM 8, 4 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393142353237334348302D4348392020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8E), Broadcom BCM43xx 1.0 (5.106.198.4.20)
Bluetooth: Version 4.0.5f11, 2 service, 18 devices, 1 incoming serial ports
Network Service: Internet, Ethernet, en0
Network Service: Meta, Ethernet, en1
PCI Card: ATTO ExpressSAS R680, sppci_raid, Slot-3
PCI Card: Apple RAID Card, sppci_raid, Slot-4
PCI Card: ATI Radeon HD 5770, sppci_displaycontroller, Slot-1
PCI Card: ATTO Celerity FC-82EN, sppci_fibrechannel, Slot-2
PCI Card: ATTO Celerity FC-82EN, sppci_fibrechannel, Slot-2
Serial ATA Device: HL-DT-ST DVD-RW GH41N
Fibre Channel Device: SCSI Target Device @ 4
Fibre Channel Device: SCSI Target Device @ 7
Fibre Channel Device: SCSI Target Device @ 8
Fibre Channel Device: SCSI Target Device @ 13
Fibre Channel Device: SCSI Target Device @ 17
Fibre Channel Device: SCSI Target Device @ 18
Fibre Channel Device: SCSI Target Device @ 19
Fibre Channel Device: SCSI Target Device @ 20
Fibre Channel Device: SCSI Target Device @ 21
Fibre Channel Device: SCSI Target Device @ 22
Fibre Channel Device: SCSI Target Device @ 23
Fibre Channel Device: SCSI Target Device @ 24
Fibre Channel Device: SCSI Target Device @ 25
Fibre Channel Device: SCSI Target Device @ 26
Fibre Channel Device: SCSI Target Device @ 28
Fibre Channel Device: SCSI Target Device @ 29
Fibre Channel Device: SCSI Target Device @ 33
Fibre Channel Device: SCSI Target Device @ 34
Fibre Channel Device: SCSI Target Device @ 35
Fibre Channel Device: SCSI Target Device @ 36
Fibre Channel Device: SCSI Target Device @ 37
Fibre Channel Device: SCSI Target Device @ 38
Fibre Channel Device: SCSI Target Device @ 39
Fibre Channel Device: SCSI Target Device @ 40
Fibre Channel Device: SCSI Target Device @ 41
Fibre Channel Device: SCSI Target Device @ 42
Fibre Channel Device: SCSI Target Device @ 43
Fibre Channel Device: SCSI Target Device @ 44
Fibre Channel Device: SCSI Target Device @ 45
Fibre Channel Device: SCSI Target Device @ 46
Fibre Channel Device: SCSI Target Device @ 47
Fibre Channel Device: SCSI Target Device @ 48
Fibre Channel Device: SCSI Target Device @ 49
Fibre Channel Device: SCSI Target Device @ 50
Fibre Channel Device: SCSI Target Device @ 51
Fibre Channel Device: SCSI Target Device @ 52
Fibre Channel Device: SCSI Target Device @ 53
Fibre Channel Device: SCSI Target Device @ 54
Fibre Channel Device: SCSI Target Device @ 55
Fibre Channel Device: SCSI Target Device @ 56
Fibre Channel Device: SCSI Target Device @ 57
Fibre Channel Device: SCSI Target Device @ 58
Fibre Channel Device: SCSI Target Device @ 59
Fibre Channel Device: SCSI Target Device @ 60
Fibre Channel Device: SCSI Target Device @ 61
Fibre Channel Device: SCSI Target Device @ 62
Fibre Channel Device: SCSI Target Device @ 63
Fibre Channel Device: SCSI Target Device @ 64
Fibre Channel Device: SCSI Target Device @ 65
Fibre Channel Device: SCSI Target Device @ 66
Fibre Channel Device: SCSI Target Device @ 67
Fibre Channel Device: SCSI Target Device @ 68
Fibre Channel Device: SCSI Target Device @ 69
Fibre Channel Device: SCSI Target Device @ 70
Fibre Channel Device: SCSI Target Device @ 71
Fibre Channel Device: SCSI Target Device @ 4
Fibre Channel Device: SCSI Target Device @ 7
Fibre Channel Device: SCSI Target Device @ 8
Fibre Channel Device: SCSI Target Device @ 13
Fibre Channel Device: SCSI Target Device @ 17
Fibre Channel Device: SCSI Target Device @ 18
Fibre Channel Device: SCSI Target Device @ 19
Fibre Channel Device: SCSI Target Device @ 20
Fibre Channel Device: SCSI Target Device @ 21
Fibre Channel Device: SCSI Target Device @ 22
Fibre Channel Device: SCSI Target Device @ 23
Fibre Channel Device: SCSI Target Device @ 24
Fibre Channel Device: SCSI Target Device @ 25
Fibre Channel Device: SCSI Target Device @ 26
Fibre Channel Device: SCSI Target Device @ 27
Fibre Channel Device: SCSI Target Device @ 29
Fibre Channel Device: SCSI Target Device @ 30
Fibre Channel Device: SCSI Target Device @ 34
Fibre Channel Device: SCSI Target Device @ 35
Fibre Channel Device: SCSI Target Device @ 36
Fibre Channel Device: SCSI Target Device @ 37
Fibre Channel Device: SCSI Target Device @ 38
Fibre Channel Device: SCSI Target Device @ 39
Fibre Channel Device: SCSI Target Device @ 40
Fibre Channel Device: SCSI Target Device @ 41
Fibre Channel Device: SCSI Target Device @ 42
Fibre Channel Device: SCSI Target Device @ 43
Fibre Channel Device: SCSI Target Device @ 44
Fibre Channel Device: SCSI Target Device @ 45
Fibre Channel Device: SCSI Target Device @ 46
Fibre Channel Device: SCSI Target Device @ 47
Fibre Channel Device: SCSI Target Device @ 48
Fibre Channel Device: SCSI Target Device @ 49
Fibre Channel Device: SCSI Target Device @ 50
Fibre Channel Device: SCSI Target Device @ 51
Fibre Channel Device: SCSI Target Device @ 52
Fibre Channel Device: SCSI Target Device @ 53
Fibre Channel Device: SCSI Target Device @ 54
Fibre Channel Device: SCSI Target Device @ 55
Fibre Channel Device: SCSI Target Device @ 56
Fibre Channel Device: SCSI Target Device @ 57
Fibre Channel Device: SCSI Target Device @ 58
Fibre Channel Device: SCSI Target Device @ 59
Fibre Channel Device: SCSI Target Device @ 60
Fibre Channel Device: SCSI Target Device @ 61
Fibre Channel Device: SCSI Target Device @ 62
Fibre Channel Device: SCSI Target Device @ 63
Fibre Channel Device: SCSI Target Device @ 64
Fibre Channel Device: SCSI Target Device @ 65
Fibre Channel Device: SCSI Target Device @ 66
Fibre Channel Device: SCSI Target Device @ 67
Fibre Channel Device: SCSI Target Device @ 68
Fibre Channel Device: SCSI Target Device @ 69
Fibre Channel Device: SCSI Target Device @ 70
Fibre Channel Device: SCSI Target Device @ 71
USB Device: BRCM2046 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0x5a100000 / 2
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x8215, 0x5a110000 / 3
FireWire Device: built-in_hub, 800mbit_speed
/code/quote

xsanguy's picture

Any ideas? Half the clients crash whenever a volume moves form MDC1 to MDC2... exceptionally frustrating, and the only time I've ever seen this.

xsanguy's picture

upgraded MDCs from 10.7.3 on Mac Minis (overloaded) to 09 xserves and 10.8.2. No change.

wrstuden's picture

Unfortunately no shipping version of Xsan has a fix for this issue.

Do you have AppleCare? Note: if the mac in question was purchased recently-enough, you may have it from that purchase. Alternatively if you were a developer, you could probably ask about it on the Developer Forums.

xsanguy's picture

Can you go into more detail on the issue? I've only seen this on this particular (large but not the largest I admin) environment. It makes most of the benefits of Xsan meaningless.