Volume corrupt after update from 1.2 to 1.4

[axello]

Hi
We have a serious problem after we converted our XSan from 1.2 to 1.4, and were hoping anyone here can help us.

It is an XSan 1.2 with 2 XServe RAIDs. We didn't install this ourselves, but it was done by a previous service provider.

We stopped the Xsan on both MDCs and shut down all the clients as well. Then we upgraded the MDC to 1.4. According to Apple's update docs it is possible to do this without going through 1.3 first, because we stopped the SAN. After the update of the MDC, we updated the backup-MDC as well, so that a fail-over wouldn't go back to older Xsan software.
We generally apply the rule "if it ain't broke, don't fix it". The main reason for us to update to 1.4 is that a problem was fixed resharing the San over NFS, which is what we do. We also thought we could always uninstall the update and go back to 1.2 if necessary, but that was not the case! Please read on.

Problem:
After the update the volume store doesn't come online, because a PANIC occurs. XSan 1.4 has a different LUN size in the store.cfg file. There is a discrepancy between the old 1.2 config file and the new 1.4 one. To prevent misunderstandings: the san volume is called 'store'.

First impression:
The 1.4 update changed the store.cfg, but then it didn't mount with a 'bad configuration' error. Why it changed the config file we have no idea.

Some lines from the system.log:

[aaron]

Wow. This one is beyond me. Anyone else?
_________________
Aaron Freimark
http://www.tekserve.com/vcard/af.vcf

[MattG]

Ok three thoughts...

1) The main cause of your woes is that your original integrator used slicing on your large RAID to get four LUNs whose sizes were similar, but not equal, to the LUN sizes of your 3.5TB RAID. When these slices were placed into a Storage Pool along with the non-sliced LUNs of the 3.5TB RAID, the 3.5TB LUNs got smaller to be exactly the same size as the slices. So, to be clear, the LUNs on the 3.5TB RAID have been truncated to match the size of the sliced LUNs. In fact, you even mention that you saw an "icon" (probably an arrow facing downwards) that indicated that the sector size was smaller than the physical size.

In essence, your integrator did you no favors using this technique. We stress in class that using any combination of sliced and unsliced LUNs to create a multiple LUN storage pool is more effort than it's worth.

2) No question that the LUNs were re-labeled in the update.

A full disclaimer: messing with LUN labels is the surest way of ensuring that the volume will never mount again.

But, in class we did practice once with unlabeling and relabeling LUNs with the exact same name and seeing that the LUNs' data still remained intact. However, we didn't re-specify sizes during that exercise.

3) I hope you have AppleCare, because I am almost certain that using the cvlabel command, with its myriad of switches, could bring about a forced sector size on those two LUNs that reflect the original truncated size, which would really allow you to run the volume in 1.2 or 1.4. I'm just not familiar of how to do it, and I think their tier 3 has probably done one or two of these in the past year.

So, once again, truncating LUNs to fit the size of others in the name of performance is a bad idea.

[axello]

So, it basically surmounts to : you should be glad it worked in the first place! I've also posted this to Apple, maybe they come up with something...

Who or what is this 'tier 3' you are talking about in regard to AppleCare?

Thanks

[axello]

To recap:
We have problems updating a 1.2 xsan system that was hacked to support truncated LUNs. the main storage pool contains 6 LUNS, 4x 931GB and 2x 935GB (from a new xserve raid and an older model xserve raid). The larger LUNs were truncated to match the smaller LUN size in the pool. This was visible in the Xsan Admin GUI with the greyed out total size and an arrow pointing to the usable size.

We ran some more tests:

- boot the MDC (all clients switched off)
now the volume runs (automatically)
Then we try to mount (read only) in the Xsan Admin GUI on the MDC, but this gives the error "configured sector count exceeds labeled capacity"

- We check cvlabel and the volume.cfg file:
cvlabel shows the smaller LUNs as using their full size (931GB), and the larger LUNS as using only 931GB out of the 935GB total. So that seems OK.
the volume config file shows a larger 935GB for the two bigger LUNs. This is strange as they are truncated. However:

When we adjust the store.cfg file to match the larger LUN's to their truncated size, a subsequent cvfsck (read only) gives many, many errors.

When we revert the store.cfg file to the original config on xsan 1.2 and run cvfsck it reports all is well and 0 errors. It seems that the larger size setting in the config file corresponds to the actual data on the volume.

The only problem we have is that in the Xsan Admin GUI the larger LUNs no longer show up as truncated 935GB, but as actual 931GB LUNs... which they aren't. And because of this the mount command fails...

[axello]

Update
We were able to recover the SAN, by redefining the labels with the original cvlabel configuration file. Some people convinced us that this would not cause irrepairable damage, so we took the plunge.

After that and restoring the old store config file, cvfsck gave no errors and the SAN mounted correctly. Then we updated to 1.3: no problems, the LUN labels stay the same, and the SAN mounts automagically.
We then upgraded to XSan 1.4. That also went well. So it seems something went wrong with the LUN labels during the immediate upgrade from 1.2 to 1.4, and it went well when we did it in smaller steps...

So in short:

Uninstall 1.4
Reinstall 1.1 + 1.2
restore old 1.2 config back
change cvlabels to original size
1.3 update + test: store mounts OK
1.4 update: also mounts OK.

Thanks for all who helped.

Case closed.

[aaron]

Congratulations! Let me know if you are in the market for a good backup solution.
_________________
Aaron Freimark
http://www.tekserve.com/vcard/af.vcf

[axello]

javascript:emoticon('')

They now bought a DLT2 tape archive solution from Quantum. (Note: I explicitely don't say: backup solution).

This is mainly to free up space on the san.