MDCs both in standby, volume won't mount

Had some serious funkiness with our SAN volume. Lots of inode issues. I ran cvfsck -j, then -nv, then -wv, which seems to have cleaned up all the errors. After trying to start the volume again, it immediately went into a RPL_upgrade process. A couple times, that failed near the end (fsm process crashed), then the volume would fail over to the other MDC and start the who RPL process again. I saw a "completed" message at least three times now, but the volumes will still not mount correctly. Both MDCs have their fsm processes running in standby mode now, though the volume appears to be started and hosted by the master in Xsan Admin. Mounting the volume anywhere just causes another crash.

After running cvadmin activate on the master MDC, it eventually tells me:

Admin Tap Connection to FSM failed

This also kicks off another round of RPL upgrades.

I'm hoping I don't have to completely restore our backup, because that'll take days… Any suggestions great appreciated. Thank you!

Sorry for being too brief--I'm frazzled…

MDCs on 10.7.5, Xsan 2.3
All clients on 10.7.5, except for one (10.6.8).

DNS is solid. After the crash, I even added manual entries to the hosts file on each MDC to make sure they could see each other.

RAIDs report no LUN issues (ActiveStorage ActiveRAIDs).

After the RPL upgrade, which only completes to about 90% now, the fsm process crashes with a SIGABRT.

Sigh… Backups it is…