Fast cvfsck question

Andrew Allen's picture
Tags: 

We have three SANs that have started failing over very frequently after we updated to Mavericks and Xsan 3.1. After trying a great many things, I'm concerned that the SAN definitions might corrupted.

I've just learned about the cvfsck command in Xsan, and the steps to check if a volume has problems. I just have one question:

1) How long does this process take? The first two SANs are the ones we're concerned about. SAN1 is 42 TB and SAN2 is 36 TB. We're in the middle of Baseball Season and we're ingesting a lot of video of the Seattle Mariners every day. We need to find a time to run this test but I'm not sure how long it could take. Any approximate guesses?

thomasb's picture

What do you mean with "SAN definitions"?

Failovers do not necessarily damage the file system, but running a check doesn't hurt.

All Xsan environments running OS X 10.9 see issues with failovers, and it's a bug in OS X, that hopefully will be fixed soon. The best thing you can do in the meantime is to make sure you reboot the MDC that was in control of the volumes before the failover, to make sure it's ready for the next failover.

Last time I ran cvfsck on a 70TB Xsan volume where the majority of files were large video files, a read-only check took about 5-10 minutes. The file system needed repair, and the repair took about 10-15 minutes. The time required depends a lot on the amount of files on the volume.

What is the total capacity of your Xsan volumes? How much space is available?

Have you read Apple's articles about cvfsck? Read both carefully.

Xsan: How to repair the filesystem
http://support.apple.com/kb/ht1081

Xsan: File System Status reported by cvfsck does not indicate need for repair
http://support.apple.com/kb/TA24607

Info from "man cvfsck"

-j Execute journal recovery and then exit. Running journal recovery will ensure all operations have been  committed  to disk, and that the metadata state is up to date. It is recommended that cvfsck is run with the -j flag before any read-only checks or volume reports are run.

-n This  option allows a volume to be checked in a read-only mode. The modifications that would have happened are described but are not actually performed. A read-only volume check may display errors if there  are  journaled volume  transactions  which have not yet been committed. It is recommended that cvfsck is run with the -j flag before a read-only check is run.

-v Use verbose reporting methods.

NOTE: On  large  file  systems  cvfsck may requires 100s of megabytes or more of local system disk space for working files.

 

Andrew Allen's picture

Thanks very much for replying to this. Sorry for the late reply.

By "SAN definitions" I mean that we have made 3 different SANs in Xsan. Three different volumes, each an Infortrend RAID (one with 2 JBODs).

We sure are experiencing more failovers under 10.9.2. We have been rebooting the controllers whenever a failover has occurred. We've been relatively stable for the last few weeks however.

SAN1 has 14.61 TB free out of 42 TB total
SAN2 has 22.09 TB free out of 36 TB total
SAN3 has 42.87 TB free out of 72 TB total.

I will probably try running this tomorrow. Thank you for the articles about the command. I'll read them.

Andrew Allen's picture

Just an update. We did find some errors; I don't know how many but modifications need to be made. We'll probably do the repairs next week. Thanks for pointing me in the right direction ThomasB.

thomasb's picture

You're welcome. Good luck!