MDCs both in standby, volume won't mount
Had some serious funkiness with our SAN volume. Lots of inode issues. I ran cvfsck -j, then -nv, then -wv, which seems to have cleaned up all the errors. After trying to start the volume again, it immediately went into a RPL_upgrade process. A couple times, that failed near the end (fsm process crashed), then the volume would fail over to the other MDC and start the who RPL process again. I saw a "completed" message at least three times now, but the volumes will still not mount correctly. Both MDCs have their fsm processes running in standby mode now, though the volume appears to be started and hosted by the master in Xsan Admin. Mounting the volume anywhere just causes another crash.
After running cvadmin activate on the master MDC, it eventually tells me:
Admin Tap Connection to FSM failed
This also kicks off another round of RPL upgrades.
I'm hoping I don't have to completely restore our backup, because that'll take days… Any suggestions great appreciated. Thank you!