xsan 2.2.1 volume problem

[mahmoud]

Hello everyone,
yesterday we had a big problem in the building electricity which caused a power drop for about 1 hour . we have a xsan 2.2.1 system with two volumes :
Media ( two xserve raid )
Media_HD ( promise )
after the problem was solved we figure out that:
Media_hd is started and mounted on all the machines but Media volume is stopped and can not be started i tried to use the cvfsck -nv command with no luck it gives me the following output :


----------------------------------------------------------------
oscar:bin oscar$ sudo ./cvfsck -nv

BUILD INFO:
#!@$ Server Revision 3.5.0 Build 7443 Branch branches_35X (411.3)
#!@$ Built for Darwin 9.0 i386
#!@$ Created on Mon Dec 7 12:35:35 PST 2009

Xsan File Systems on host oscar.pro:

1) *Media
2) *Media_HD

The asterisk (*) denotes the file system is active.

Choose a file system by number (1-2) or zero (0) to exit -> 1


Created directory /tmp/cvfsck979a for temporary files.
Creating MetadataAndJournal allocation check file.
Creating Video allocation check file.
Creating Audio allocation check file.
Creating Other allocation check file.

** NOTE ** Read Only Check.
File system journal will not be recovered.
The results may be inconsistent and mis-leading.


Super Block information.
FS Created On : Mon May 31 16:01:48 2010
Inode Version : '2.5'
File System Status : Clean
Allocated Inodes : 119808
Free Inodes : 42839
FL Blocks : 18
Next Inode Chunk : 0x9b3
Metadump Seqno : 0
Restore Journal Seqno : 0
Windows Security Indx Inode : 0x7
Windows Security Data Inode : 0x8
Quota Database Inode : 0x9
ID Database Inode : 0xd
Client Write Opens Inode : 0xa

Stripe Group MetadataAndJournal ( 0) 0x746f60 blocks.
Stripe Group Video ( 1) 0x417f0e0 blocks.
Stripe Group Audio ( 2) 0x1d0fa70 blocks.
Stripe Group Other ( 3) 0x123fc80 blocks.

Building Inode Index Database 119808 (100%).

Verifying NT Security Descriptors
Found 239 NT Security Descriptors: all are good


Verifying Free List Extents.
Free List data block 0x11a80 not found in forward data block chain
Free List Extents are Corrupt! Free List will be rebuilt.

Scanning inodes *Fatal*: Can't find xattr inode 0x24bbb9 in database.
Clearing xattr chain for inode 0x25.
*Fatal*: Can't find xattr inode 0x24bbba in database.
Clearing xattr chain for inode 0x2a.
*Error*: XATTR ERROR detected for inode 0x2a (0x24bbba)
*Error*: *** RESETTING RPL DATA FOR FILE SYSTEM ***
*Fatal*: inode_read cannot find extender inode 0x80 in database!

*Error*: Cannot read inode 0x800000000047.
Scanning inodes 112640 ( 94%).
Extender inode 0x400810 is on the free list.

*Error*: Cannot read inode 0x28000003fe7c7.
Xattr inode flag not set in 0x402801.
Clearing xattr chain for inode 0x400bfe.
Extender inode 0x402800 is on the free list.

*Error*: Cannot read inode 0x2800000400bfe.
Scanning inodes 119808 (100%).

Sorting extent list for MetadataAndJournal pass 1/1
Updating bitmap for MetadataAndJournal extents 3310 ( 2%).
Sorting extent list for Video pass 1/1
Updating bitmap for extents Inode 0x2800000400bfa (base-0x2a2340 end-0x2a268f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x34a7d0 end-0x34ac5f sg-1 depth-1) marked doubly allocated!
Inode 0x800000000025 (base-0x37a2d0 end-0x37a3ff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x4f3010 end-0x4f31ff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x529040 end-0x5292af sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 10240 ( 7%). Inode 0x80000000003b (base-0xceb310 end-0xceb34d sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 30720 ( 22%). Inode 0x2800000400be4 (base-0x2780be0 end-0x2780dcf sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 40960 ( 30%). Inode 0x2800000400bd6 (base-0x2c476b0 end-0x2c47c7f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x2c59500 end-0x2c59778 sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x2eb4d60 end-0x2eb4faf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3325810 end-0x3325abf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x33af8d0 end-0x33afbff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x33ff090 end-0x33ff3bf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x34ab9a0 end-0x34abbcf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3506b00 end-0x3506e6f sg-1 depth-1) marked doubly allocated!
Inode 0x800000000025 (base-0x3561a43 end-0x3561a6f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3709100 end-0x370964f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x3713479 end-0x371380f sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 51200 ( 37%). Inode 0x80000000003b (base-0x374ab30 end-0x374ab6d sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x37904b0 end-0x379083f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x37efa30 end-0x37efdff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x37f5e50 end-0x37f607f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x381ad30 end-0x381b01f sg-1 depth-1) marked doubly allocated!
Inode 0x80000000003b (base-0x38cf1f0 end-0x38cf22d sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bc7 (base-0x3946830 end-0x3946b1f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x396c660 end-0x396c96f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x397fe10 end-0x39800ff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x39ab550 end-0x39ab7df sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3ac7050 end-0x3ac75df sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x3b4e040 end-0x3b4e28f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bc7 (base-0x3b987f0 end-0x3b98b1f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3ba5ad0 end-0x3ba5fff sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3bae5b0 end-0x3bae8bf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3bb67c0 end-0x3bb6b4f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3bcaa40 end-0x3bcadaf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3d1ff40 end-0x3d202ef sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3dad8e0 end-0x3daddcf sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x3db7660 end-0x3db792f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400be4 (base-0x3e11720 end-0x3e1192f sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bd6 (base-0x3e1b490 end-0x3e1b6df sg-1 depth-1) marked doubly allocated!
Inode 0x80000000003b (base-0x3f07875 end-0x3f078b2 sg-1 depth-1) marked doubly allocated!
Inode 0x2800000400bfa (base-0x3f22260 end-0x3f2254f sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 61440 ( 45%). Inode 0x2800000400be4 (base-0x40ab100 end-0x40ab3cf sg-1 depth-1) marked doubly allocated!
Updating bitmap for Video extents 62930 ( 46%).
Sorting extent list for Audio pass 1/1
Updating bitmap for Audio extents 106718 ( 78%).
Sorting extent list for Other pass 1/1
Updating bitmap for Other extents 135543 ( 99%).

Checking for dead inodes Clearing inode 0x800000000025.
Clearing extender inode 0x27.
*Warning*: Failed to get Inode 0x27, try to clear anyway
Clearing extender inode 0x28.
*Warning*: Failed to get Inode 0x28, try to clear anyway
Clearing Orphaned Stream inode 0x800000000026.
Clearing Orphaned Stream inode 0x80000000002b.
Clearing inode 0x80000000003b.
Clearing extender inode 0x3d.
*Warning*: Failed to get Inode 0x3d, try to clear anyway
Clearing extender inode 0x3e.
*Warning*: Failed to get Inode 0x3e, try to clear anyway
Clearing extender inode 0x3f.
*Warning*: Failed to get Inode 0x3f, try to clear anyway
Clearing extender inode 0x40.
*Warning*: Failed to get Inode 0x40, try to clear anyway
Clearing extender inode 0x41.
*Warning*: Failed to get Inode 0x41, try to clear anyway
Clearing inode 0x800000000047.
Clearing extender inode 0x49.
*Warning*: Failed to get Inode 0x49, try to clear anyway
Clearing extender inode 0x4a.
*Warning*: Failed to get Inode 0x4a, try to clear anyway
Clearing extender inode 0x4b.
*Warning*: Failed to get Inode 0x4b, try to clear anyway
Clearing extender inode 0x4c.
*Warning*: Failed to get Inode 0x4c, try to clear anyway
Clearing extender inode 0x4d.
*Warning*: Failed to get Inode 0x4d, try to clear anyway
Clearing extender inode 0x4e.
*Warning*: Failed to get Inode 0x4e, try to clear anyway
Clearing extender inode 0x4f.
*Warning*: Failed to get Inode 0x4f, try to clear anyway
Clearing extender inode 0x50.
*Warning*: Failed to get Inode 0x50, try to clear anyway
Clearing extender inode 0x51.
*Warning*: Failed to get Inode 0x51, try to clear anyway
Clearing extender inode 0x52.
*Warning*: Failed to get Inode 0x52, try to clear anyway
Clearing extender inode 0x53.
*Warning*: Failed to get Inode 0x53, try to clear anyway
Clearing extender inode 0x54.
*Warning*: Failed to get Inode 0x54, try to clear anyway
Clearing extender inode 0x55.
*Warning*: Failed to get Inode 0x55, try to clear anyway
Clearing extender inode 0x57.
*Warning*: Failed to get Inode 0x57, try to clear anyway
Clearing extender inode 0x58.
*Warning*: Failed to get Inode 0x58, try to clear anyway
Clearing extender inode 0x59.
*Warning*: Failed to get Inode 0x59, try to clear anyway
Clearing extender inode 0x5a.
*Warning*: Failed to get Inode 0x5a, try to clear anyway
Clearing extender inode 0x5b.
*Warning*: Failed to get Inode 0x5b, try to clear anyway
Clearing extender inode 0x5c.
*Warning*: Failed to get Inode 0x5c, try to clear anyway
Clearing extender inode 0x5d.
*Warning*: Failed to get Inode 0x5d, try to clear anyway
Clearing extender inode 0x5e.
*Warning*: Failed to get Inode 0x5e, try to clear anyway
Clearing extender inode 0x5f.
*Warning*: Failed to get Inode 0x5f, try to clear anyway
Clearing extender inode 0x60.
*Warning*: Failed to get Inode 0x60, try to clear anyway
Clearing extender inode 0x61.
*Warning*: Failed to get Inode 0x61, try to clear anyway
Clearing extender inode 0x62.
*Warning*: Failed to get Inode 0x62, try to clear anyway
Clearing extender inode 0x63.
*Warning*: Failed to get Inode 0x63, try to clear anyway
Clearing extender inode 0x64.
*Warning*: Failed to get Inode 0x64, try to clear anyway
Clearing extender inode 0x65.
*Warning*: Failed to get Inode 0x65, try to clear anyway
Clearing extender inode 0x66.
*Warning*: Failed to get Inode 0x66, try to clear anyway
Clearing extender inode 0x67.
*Warning*: Failed to get Inode 0x67, try to clear anyway
Clearing extender inode 0x68.
*Warning*: Failed to get Inode 0x68, try to clear anyway
Clearing extender inode 0x69.
*Warning*: Failed to get Inode 0x69, try to clear anyway
Clearing extender inode 0x6a.
*Warning*: Failed to get Inode 0x6a, try to clear anyway
Clearing extender inode 0x6b.
*Warning*: Failed to get Inode 0x6b, try to clear anyway
Clearing extender inode 0x6c.
*Warning*: Failed to get Inode 0x6c, try to clear anyway
Clearing extender inode 0x6d.
*Warning*: Failed to get Inode 0x6d, try to clear anyway
Clearing extender inode 0x6e.
*Warning*: Failed to get Inode 0x6e, try to clear anyway
Clearing extender inode 0x6f.
*Warning*: Failed to get Inode 0x6f, try to clear anyway
Clearing extender inode 0x70.
*Warning*: Failed to get Inode 0x70, try to clear anyway
Clearing extender inode 0x71.
*Warning*: Failed to get Inode 0x71, try to clear anyway
Clearing extender inode 0x72.
*Warning*: Failed to get Inode 0x72, try to clear anyway
Clearing extender inode 0x73.
*Warning*: Failed to get Inode 0x73, try to clear anyway
Clearing extender inode 0x74.
*Warning*: Failed to get Inode 0x74, try to clear anyway
Clearing extender inode 0x75.
*Warning*: Failed to get Inode 0x75, try to clear anyway
Clearing extender inode 0x76.
*Warning*: Failed to get Inode 0x76, try to clear anyway
Clearing extender inode 0x77.
*Warning*: Failed to get Inode 0x77, try to clear anyway
Clearing extender inode 0x78.
*Warning*: Failed to get Inode 0x78, try to clear anyway
Clearing extender inode 0x79.
*Warning*: Failed to get Inode 0x79, try to clear anyway
Clearing extender inode 0x7a.
*Warning*: Failed to get Inode 0x7a, try to clear anyway
Clearing extender inode 0x7b.
*Warning*: Failed to get Inode 0x7b, try to clear anyway
Clearing extender inode 0x7c.
*Warning*: Failed to get Inode 0x7c, try to clear anyway
Clearing extender inode 0x7d.
*Warning*: Failed to get Inode 0x7d, try to clear anyway
Clearing extender inode 0x7e.
*Warning*: Failed to get Inode 0x7e, try to clear anyway
Clearing extender inode 0x7f.
*Warning*: Failed to get Inode 0x7f, try to clear anyway
Clearing extender inode 0x80.
*Warning*: Failed to get Inode 0x80, try to clear anyway
Checking for dead inodes 20480 ( 17%).
Clearing inode 0x80000024bbba.
Checking for dead inodes 112640 ( 94%).
Clearing inode 0x28000003fe7c7.
Clearing extender inode 0x400810.
*Warning*: Failed to get Inode 0x400810, try to clear anyway
Clearing inode 0x2800000400bc7.
Clearing extender inode 0x400bc8.
*Warning*: Failed to get Inode 0x400bc8, try to clear anyway
Clearing extender inode 0x400bc9.
*Warning*: Failed to get Inode 0x400bc9, try to clear anyway
Clearing extender inode 0x400bca.
*Warning*: Failed to get Inode 0x400bca, try to clear anyway
Clearing inode 0x2800000400bd6.
Clearing extender inode 0x400bd7.
*Warning*: Failed to get Inode 0x400bd7, try to clear anyway
Clearing extender inode 0x400bd8.
*Warning*: Failed to get Inode 0x400bd8, try to clear anyway
Clearing extender inode 0x400bd9.
*Warning*: Failed to get Inode 0x400bd9, try to clear anyway
Clearing extender inode 0x400bda.
*Warning*: Failed to get Inode 0x400bda, try to clear anyway
Clearing inode 0x2800000400be4.
Clearing extender inode 0x400be5.
*Warning*: Failed to get Inode 0x400be5, try to clear anyway
Clearing extender inode 0x400be6.
*Warning*: Failed to get Inode 0x400be6, try to clear anyway
Clearing extender inode 0x400be7.
*Warning*: Failed to get Inode 0x400be7, try to clear anyway
Clearing extender inode 0x400be8.
*Warning*: Failed to get Inode 0x400be8, try to clear anyway
Clearing extender inode 0x400be9.
*Warning*: Failed to get Inode 0x400be9, try to clear anyway
Clearing extender inode 0x400bea.
*Warning*: Failed to get Inode 0x400bea, try to clear anyway
Clearing inode 0x2800000400bfa.
Clearing extender inode 0x400bfb.
*Warning*: Failed to get Inode 0x400bfb, try to clear anyway
Clearing extender inode 0x400bfc.
*Warning*: Failed to get Inode 0x400bfc, try to clear anyway
Clearing inode 0x2800000400bfe.
Clearing extender inode 0x400bff.
*Warning*: Failed to get Inode 0x400bff, try to clear anyway
Clearing extender inode 0x402800.
*Warning*: Failed to get Inode 0x402800, try to clear anyway
Clearing inode 0x2800000402801.
Checking for dead inodes 119808 (100%).

Checking directories 1050 ( 31%).
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.



----------------------------------
and so on with no end......


Does any one has a solution for this case?

[abstractrude]

did you actually try to repair the volume with a -wv? where is the end of the output. is it crashing? it should exit stdout still
does it ever say the volume needs to be repaired?

[mahmoud]

yes i tried to repair it with -wv but it also hang up after :

Checking directories 1050 ( 31%).
Found duplicate . entry (p=0x28000002a79c3). Removing entry.
Found duplicate .. entry (p=0x28000002a79c3). Removing entry.
and it stops for along time.

if i left it to take it time the terminal closed.

[abstractrude]

do you have a backup? although there may be some additional metadata hammering techniques. this doesn't look good. if you have support you may want to see if apple can help rebuild your metadata

[geoffcross]

I would recommend contacting ConSol they sorted us out last year and are the BEST... ravindrank@web.de

Cheers, Geoff Cross
Senior XSAN Systems Administrator
Leeds Institute of Molecular Medicine

[francisyo]

mahmoud,

I encounter this problem late last year, but i dont know if we are the same. Mine was also power interruption and when the power was back i automatically opens all the raids, servers and metadata. Just when i thought that everything is okay, thats the time i saw my xsan volume not mounted on clients. It has an X mark on each volume. I tried repairing it via nv and wv but it did not solve. I tried also restarting everything but it did not work also. The solution was, DNS. Try to check both your ethernet (public and private) especially the public. it should be well connected to your DNS.

I hope i contribute solution to your problem.

[mahmoud]

Thanks all for your help .
i really appreciate you efforts,i`m really grateful for you.

abstractrude:
yes we have a backup but ,this problem will affect the repetition of my work here as i don't know why it happened . so I'm trying to find a way to sort it out as some insignificant material are not on the backup volume , which means that we need to capture it again.


geoffcross:
thanks for the advice , they also have an article on their website that helps xsan disaster recovery but even this did not work out with our solution . the material it self was retrieved from the backup volume but you know we need to find a solution for it in case that it happens again.


francisyo:
yes , i checked every thing even the dns and the Ethernet cables . but in our case the other volume is still working and mounted on all the machines the problem is only on one volume which i had no luck with it using any kind of known solution ( retrieving the old xsan config. backup , restating the system , all xsan terminal commands...)

any way thanks again for all you help
i will continue trying to find a solution for this problem hoping to find a reasonable solution for it.