Xsan's cvfsck fall down, Xsan go boom...

Jimmay · Guest

Howdy, this is fragmented due to the state of my mind seeing 2TB of critical data disappear -

Basically, I was told to run a cvfsck based on an issue we were having with our san.

Ran once just fine read-only, though it had a few warnings/notice level log entries.

When we ran it "for real", it ran out of room trying to re-allocate all the orphaned inodes (something, apparently, that the read-only doesn't complain about?)... And now the SAN volume has about 50 .FOUNDXXXX type files at the root level, and nothing else (though Xsan admin and cvadmin report the same amount of usage as would be expected for the data we had listed - roughly 100GB free on a 2TB SAN).

So, I guess the question is, is this situation recoverable at all? I have backups that are a little old for my taste (loss of one business days' work for 170 people), and I'm wondering if there are any means to rebuild/recover the SAN enough to move the data off onto another SAN volume (which, ironically, just arrived to meet our growth needs).

Help?

NateLF · Guest

I had a similar issue where my volume disconnected and would no longer mount. Attempted a cvfsck and it would crash while trying to fix the inode problems. I have Apple working on it but it is taking longer than expected to fix the issue. I only have 200+ gig's of data lost and my backups failed to recover anything. Yuck!

Jimmay · Guest

Okay, so I just deleted the orphaned files, and re-ran the CVFSCK command 16 more times (each time deleting single orphaned files), and finally it found orphaned directories, which held ALL my data.

Does anyone have a clue about why it kept running "out of space" during the cvfsck? I had 100GB available.

eM · Guest

The problem is that the root inode cannot hold more then about 50 entries.

In old times fsck would create a directory "lost+found" and store orphans there. But cvfsck stores all orphaned inodes in the root inode.

My problem is, I cannot use the FS unless all orphans are cleaned, because of inconsistencies this will cause later. Unfortunately I have about 4500 orhpans waiting and one cvfsck takes 1 hour.