Expanding Volume Woes

messenger82's picture

We expanded one of our volumes today and had some interesting results, I'm hoping someone might have seen this and can offer a resolution,

Config
We expanded our VTRAK storage by stacking on an expansion chassis. We formatted the volume with Apple's preconfigured scripts, all logs show that went fine and the jobs are completed. We have a one 2TB LUN and pool for metadata and two 10 TB LUNs in a single pool for video. We added two additional 10 TB LUNs to the existing 20 TB pool for a total of 40TB. We're running XSAN 2.2.

Problem
We ran the expansion via XSAN to add the new LUNs to the current pool, all indications are that it went fine. The new pool and volume shows up as 40TB as you would expect. However, we do not have any more free space. We started with 3.75TB of free space, even though we expanded to 40TB we still have 3.75TB of free space according to XSAN admin, Finder, and cvadmin all indicate that we now have doubled the capacity of our volume but have not added any free space.

Attempted resolutions:
Log files indicate no problems. We have incrementally rebooted everything and also performed a full shutdown.

Any ideas where our free space is?

singlemalt's picture

You did a bandwidth expansion by adding storage to an existing storage pool. I suspect your
volume was already severely fragmented and doing a bandwidth expansion has made it worse.
See
http://support.apple.com/kb/TS3041
You should also run cvfsck -f and see how bad your free space fragmentation is (I suspect it's going to be appallingly bad). You can run cvfsck -f on a running and mounted volume.
You should seriously consider backing up all needed data, then destroy and re-create the volume then
restore the data.

messenger82's picture

Singlemalt, thanks for the suggestion. I had the following return. What about snfsdefrag? I've heard mixed things about it and never had a volume in this much peril so I've had no reason to run it before. Will that repair things?

-- Free Block Fragmentation Analysis - Stripe Group "MetadataAndJournal" --

Pct. (sum) Chunk Size Chunk Count

<1% ( <1%) 8977 1
99% ( 99%) 121684891 1

Total blocks 122022368 Allocated blocks 328500 Free blocks 121693868 (99%)

-- Free Block Fragmentation Analysis - Stripe Group "Video" --

Pct. (sum) Chunk Size Chunk Count

<1% ( <1%) 1 1
<1% ( <1%) 3 27
<1% ( <1%) 4 1294
<1% ( <1%) 5 2480
<1% ( <1%) 6 3375
<1% ( <1%) 7 5
<1% ( <1%) 8 9
<1% ( <1%) 9 5
<1% ( <1%) 10 3
<1% ( <1%) 11 3
<1% ( <1%) 12 1
<1% ( <1%) 13 10
<1% ( <1%) 14 17
<1% ( <1%) 15 45
<1% ( <1%) 16 61
<1% ( <1%) 17 44
<1% ( <1%) 18 29
<1% ( <1%) 19 24
<1% ( <1%) 20 14
<1% ( <1%) 21 24
<1% ( <1%) 22 12
<1% ( <1%) 23 34
<1% ( <1%) 24 18
<1% ( <1%) 25 15
<1% ( <1%) 26 9
<1% ( <1%) 27 2
<1% ( <1%) 28 6
<1% ( <1%) 29 3
<1% ( <1%) 30 2
45% ( 45%) 32 34893613
<1% ( 45%) 35 11
<1% ( 45%) 36 479
<1% ( 45%) 37 1154
<1% ( 45%) 38 2132
<1% ( 45%) 39 1
<1% ( 45%) 40 9
<1% ( 45%) 41 3
<1% ( 45%) 42 2
<1% ( 45%) 43 6
<1% ( 45%) 45 2
<1% ( 45%) 46 12
<1% ( 45%) 47 42
<1% ( 45%) 48 19556
<1% ( 45%) 49 1174
<1% ( 45%) 50 564
<1% ( 45%) 51 378
<1% ( 45%) 52 498
<1% ( 45%) 53 392
<1% ( 45%) 54 267
<1% ( 45%) 55 501
<1% ( 45%) 56 948
<1% ( 45%) 57 658
<1% ( 45%) 58 512
<1% ( 45%) 59 420
<1% ( 45%) 60 401
<1% ( 45%) 61 619
<1% ( 45%) 62 275
<1% ( 45%) 63 178
<1% ( 45%) 64 8624
<1% ( 45%) 65 125
<1% ( 45%) 66 1
<1% ( 45%) 67 1
<1% ( 45%) 68 1
<1% ( 45%) 72 1
<1% ( 45%) 75 1
<1% ( 45%) 77 1
<1% ( 45%) 80 1
<1% ( 45%) 96 8523
<1% ( 45%) 97 101
<1% ( 45%) 98 1
<1% ( 45%) 99 2
<1% ( 45%) 104 1
<1% ( 45%) 106 1
<1% ( 45%) 107 1
<1% ( 45%) 108 1
<1% ( 45%) 111 2
<1% ( 45%) 112 25
<1% ( 45%) 117 1
<1% ( 45%) 120 1
<1% ( 45%) 126 1
<1% ( 45%) 127 10
<1% ( 45%) 128 14
<1% ( 45%) 129 5
<1% ( 45%) 130 1
<1% ( 45%) 131 1
<1% ( 45%) 134 2
<1% ( 45%) 144 4
<1% ( 45%) 145 2
<1% ( 45%) 160 8
<1% ( 45%) 161 9
<1% ( 45%) 162 1
<1% ( 45%) 165 1
<1% ( 45%) 166 1
<1% ( 45%) 176 358
<1% ( 45%) 177 6
<1% ( 45%) 192 1
<1% ( 45%) 195 1
<1% ( 45%) 198 1
<1% ( 45%) 204 1
<1% ( 45%) 205 1
<1% ( 45%) 206 119
<1% ( 45%) 207 152
<1% ( 45%) 208 6
<1% ( 45%) 209 38
<1% ( 45%) 210 17
<1% ( 45%) 211 31
<1% ( 45%) 212 27
<1% ( 45%) 213 13
<1% ( 45%) 214 15
<1% ( 45%) 224 3
<1% ( 45%) 232 1
<1% ( 45%) 238 98
<1% ( 45%) 239 169
<1% ( 45%) 240 835
<1% ( 45%) 241 62
<1% ( 45%) 242 36
<1% ( 45%) 243 48
<1% ( 45%) 244 59
<1% ( 45%) 245 62
<1% ( 45%) 246 47
<1% ( 45%) 256 1
<1% ( 45%) 257 1
<1% ( 45%) 258 3
<1% ( 45%) 271 1
<1% ( 45%) 272 2
<1% ( 45%) 275 1
<1% ( 45%) 288 1
<1% ( 45%) 304 458
<1% ( 45%) 305 1
<1% ( 45%) 306 1
<1% ( 45%) 307 1
<1% ( 45%) 308 1
<1% ( 45%) 309 2
<1% ( 45%) 310 5
<1% ( 45%) 311 2
<1% ( 45%) 312 3
<1% ( 45%) 320 8
<1% ( 45%) 352 4
<1% ( 45%) 368 4
<1% ( 45%) 400 1
<1% ( 45%) 416 2
<1% ( 45%) 432 2
<1% ( 45%) 448 1
<1% ( 45%) 449 1
<1% ( 45%) 464 1
<1% ( 45%) 480 1
<1% ( 45%) 496 6
<1% ( 45%) 504 2
<1% ( 45%) 512 5
<1% ( 45%) 528 2
<1% ( 45%) 544 3
<1% ( 45%) 545 1
<1% ( 45%) 560 2
<1% ( 45%) 576 1
<1% ( 45%) 608 1
<1% ( 45%) 611 1
<1% ( 45%) 672 1
<1% ( 45%) 749 1
<1% ( 45%) 752 2
<1% ( 45%) 768 2
<1% ( 45%) 880 2
<1% ( 45%) 928 1
<1% ( 45%) 1008 1
<1% ( 45%) 1016 2
<1% ( 45%) 1136 1
<1% ( 45%) 1264 1
<1% ( 45%) 1392 1
<1% ( 45%) 1508 1
<1% ( 45%) 1712 1
<1% ( 45%) 1810 1
<1% ( 45%) 1914 1
<1% ( 45%) 1968 1
<1% ( 45%) 2016 1
<1% ( 45%) 2096 1
<1% ( 45%) 2144 1
<1% ( 45%) 3040 1
<1% ( 45%) 3952 1
<1% ( 45%) 5296 1
<1% ( 45%) 5728 1
<1% ( 45%) 7216 2
<1% ( 45%) 8368 1
<1% ( 45%) 10016 1
<1% ( 45%) 11296 1
<1% ( 45%) 15072 1
<1% ( 45%) 15088 2
<1% ( 45%) 17040 1
<1% ( 45%) 18416 1
<1% ( 45%) 20352 1
<1% ( 45%) 21152 1
<1% ( 45%) 21392 1
<1% ( 45%) 23152 1
<1% ( 45%) 23312 1
<1% ( 45%) 23568 1
<1% ( 45%) 24320 1
<1% ( 45%) 25104 1
<1% ( 45%) 25136 1
<1% ( 45%) 25200 5
<1% ( 45%) 25904 1
<1% ( 45%) 25952 1
<1% ( 45%) 25968 1
<1% ( 45%) 26032 1
<1% ( 45%) 26080 1
<1% ( 45%) 26096 1
<1% ( 45%) 26496 1
<1% ( 45%) 27824 2
<1% ( 45%) 27840 1
<1% ( 45%) 27872 1
<1% ( 45%) 27888 3
<1% ( 45%) 27904 1
<1% ( 45%) 27936 2
<1% ( 45%) 27952 1
<1% ( 45%) 27968 1
<1% ( 45%) 29008 1
<1% ( 45%) 29120 1
<1% ( 45%) 29168 2
<1% ( 45%) 29232 1
<1% ( 45%) 29296 1
<1% ( 45%) 29408 1
<1% ( 45%) 29616 2
<1% ( 45%) 30608 1
<1% ( 45%) 31008 1
<1% ( 45%) 31024 1
<1% ( 45%) 31664 1
<1% ( 45%) 33856 1
<1% ( 45%) 35456 1
<1% ( 45%) 36656 1
<1% ( 45%) 37120 1
<1% ( 45%) 37232 2
<1% ( 45%) 37424 1
<1% ( 45%) 37440 1
<1% ( 45%) 37472 3
<1% ( 45%) 37504 1
<1% ( 45%) 38139 1
<1% ( 45%) 40016 1
<1% ( 45%) 41808 1
<1% ( 45%) 42224 1
<1% ( 45%) 46016 3
<1% ( 45%) 46048 4
<1% ( 45%) 46976 1
<1% ( 45%) 48000 1
<1% ( 45%) 48032 1
<1% ( 45%) 49472 1
<1% ( 45%) 50288 1
<1% ( 45%) 50352 1
<1% ( 46%) 53248 14
<1% ( 46%) 53280 19
<1% ( 46%) 57344 1
<1% ( 46%) 57376 2
<1% ( 46%) 57456 1
<1% ( 46%) 57584 1
<1% ( 46%) 58464 1
<1% ( 46%) 58720 1
<1% ( 46%) 58944 1
<1% ( 46%) 59024 1
<1% ( 46%) 59056 1
<1% ( 46%) 59088 2
<1% ( 46%) 59136 1
<1% ( 46%) 59232 1
<1% ( 46%) 59696 1
<1% ( 46%) 60592 1
<1% ( 46%) 61168 1
<1% ( 46%) 61200 1
<1% ( 46%) 61216 1
<1% ( 46%) 61296 1
<1% ( 46%) 61440 8
<1% ( 46%) 61472 9
<1% ( 46%) 62560 2
<1% ( 46%) 62576 1
<1% ( 46%) 63664 2
<1% ( 46%) 64112 1
<1% ( 46%) 66528 2
<1% ( 46%) 67856 1
<1% ( 46%) 67888 1
<1% ( 46%) 67920 1
<1% ( 46%) 72048 1
<1% ( 46%) 76176 1
<1% ( 46%) 78256 1
<1% ( 46%) 79872 3
<1% ( 46%) 79904 10
<1% ( 46%) 82016 1
<1% ( 46%) 86016 3
<1% ( 46%) 86048 2
<1% ( 46%) 87088 1
<1% ( 46%) 87280 1
<1% ( 46%) 87472 1
<1% ( 46%) 87536 1
<1% ( 46%) 87600 1
<1% ( 46%) 87664 1
<1% ( 46%) 87728 1
<1% ( 46%) 87792 1
<1% ( 46%) 87856 1
<1% ( 46%) 87920 1
<1% ( 46%) 87984 1
<1% ( 46%) 88112 1
<1% ( 46%) 88240 1
<1% ( 46%) 88304 1
<1% ( 46%) 88368 1
<1% ( 46%) 88432 1
<1% ( 46%) 88496 1
<1% ( 46%) 88560 1
<1% ( 46%) 88624 1
<1% ( 46%) 88688 1
<1% ( 46%) 88752 1
<1% ( 46%) 88816 1
<1% ( 46%) 88880 1
<1% ( 46%) 88944 1
<1% ( 46%) 89008 1
<1% ( 46%) 89072 1
<1% ( 46%) 89136 1
<1% ( 46%) 89200 1
<1% ( 46%) 89264 1
<1% ( 46%) 89328 1
<1% ( 46%) 89392 1
<1% ( 46%) 89456 1
<1% ( 46%) 90112 2
<1% ( 46%) 90144 7
<1% ( 46%) 91504 1
<1% ( 46%) 92144 1
<1% ( 46%) 92272 1
<1% ( 46%) 93616 1
<1% ( 46%) 93680 1
<1% ( 46%) 93744 1
<1% ( 46%) 93808 1
<1% ( 46%) 93872 1
<1% ( 46%) 93936 1
<1% ( 46%) 94000 1
<1% ( 46%) 94064 1
<1% ( 46%) 94128 2
<1% ( 46%) 94192 1
<1% ( 46%) 94208 22
<1% ( 46%) 94240 28
<1% ( 46%) 97328 1
<1% ( 46%) 97968 1
<1% ( 46%) 98304 1
<1% ( 46%) 98309 1
<1% ( 46%) 98336 1
<1% ( 46%) 102400 1
<1% ( 46%) 102437 1
<1% ( 46%) 103728 1
<1% ( 46%) 106032 1
<1% ( 46%) 106496 3
<1% ( 46%) 109568 1
<1% ( 46%) 110576 1
<1% ( 46%) 110592 3
<1% ( 46%) 114176 1
<1% ( 46%) 114224 1
<1% ( 46%) 114688 1
<1% ( 46%) 115568 1
<1% ( 46%) 120240 1
<1% ( 46%) 122912 1
<1% ( 46%) 123936 1
<1% ( 46%) 124208 1
<1% ( 46%) 127473 1
<1% ( 46%) 131072 1
<1% ( 46%) 135168 1
<1% ( 46%) 135200 1
<1% ( 46%) 139296 1
<1% ( 46%) 143392 1
<1% ( 46%) 145952 1
<1% ( 46%) 147456 1
<1% ( 46%) 147488 3
<1% ( 46%) 151584 1
<1% ( 46%) 159776 1
<1% ( 46%) 160432 1
<1% ( 46%) 163840 1
<1% ( 46%) 164848 1
<1% ( 46%) 164976 1
<1% ( 46%) 165040 1
<1% ( 46%) 168944 1
<1% ( 46%) 172016 1
<1% ( 46%) 172032 1
<1% ( 46%) 172064 2
<1% ( 46%) 172416 1
<1% ( 46%) 172800 1
<1% ( 46%) 173184 1
<1% ( 46%) 173600 1
<1% ( 46%) 182176 1
<1% ( 46%) 182560 1
<1% ( 46%) 184352 1
<1% ( 46%) 184688 1
<1% ( 46%) 188416 1
<1% ( 46%) 196608 1
<1% ( 46%) 196640 1
<1% ( 46%) 200741 1
<1% ( 47%) 203808 1
<1% ( 47%) 204800 1
<1% ( 47%) 217088 1
<1% ( 47%) 221184 1
<1% ( 47%) 225280 2
<1% ( 47%) 225318 1
<1% ( 47%) 227104 1
<1% ( 47%) 228912 1
<1% ( 47%) 229376 1
<1% ( 47%) 229408 1
<1% ( 47%) 230896 1
<1% ( 47%) 237568 1
<1% ( 47%) 237600 4
<1% ( 47%) 241664 3
<1% ( 47%) 241696 3
<1% ( 47%) 244419 1
<1% ( 47%) 245760 5
<1% ( 47%) 245765 1
<1% ( 47%) 245792 1
<1% ( 47%) 246512 1
<1% ( 47%) 249856 2
<1% ( 47%) 249888 2
<1% ( 47%) 253952 1
<1% ( 47%) 253989 1
<1% ( 47%) 272112 1
<1% ( 47%) 282624 1
<1% ( 47%) 286752 2
<1% ( 47%) 294944 1
<1% ( 47%) 299040 1
<1% ( 47%) 307232 1
<1% ( 47%) 326512 1
<1% ( 47%) 326704 1
<1% ( 47%) 326896 1
<1% ( 47%) 327088 1
<1% ( 47%) 327280 1
<1% ( 47%) 327472 1
<1% ( 47%) 327712 1
<1% ( 47%) 328224 1
<1% ( 47%) 328480 1
<1% ( 47%) 328704 1
<1% ( 47%) 342320 1
<1% ( 47%) 350704 1
<1% ( 47%) 352288 1
<1% ( 47%) 354992 1
<1% ( 47%) 368640 2
<1% ( 47%) 372742 1
<1% ( 47%) 389152 1
<1% ( 47%) 393216 1
<1% ( 47%) 401408 1
<1% ( 47%) 405504 1
<1% ( 47%) 408432 1
<1% ( 47%) 417792 1
<1% ( 47%) 421888 2
<1% ( 47%) 421920 1
<1% ( 47%) 426016 1
<1% ( 47%) 430080 1
<1% ( 47%) 438272 1
<1% ( 47%) 442368 1
<1% ( 47%) 444336 1
<1% ( 47%) 458752 1
<1% ( 47%) 465968 1
<1% ( 47%) 474352 1
<1% ( 48%) 476064 1
<1% ( 48%) 497776 1
<1% ( 48%) 499712 1
<1% ( 48%) 515696 1
<1% ( 48%) 536368 1
<1% ( 48%) 536560 1
<1% ( 48%) 544800 1
<1% ( 48%) 549040 1
<1% ( 48%) 549232 1
<1% ( 48%) 549424 1
<1% ( 48%) 549616 1
<1% ( 48%) 551600 1
<1% ( 48%) 553248 1
<1% ( 48%) 570096 1
<1% ( 48%) 577536 1
<1% ( 48%) 593920 1
<1% ( 48%) 634880 1
<1% ( 48%) 638832 1
<1% ( 48%) 643072 1
<1% ( 48%) 723840 1
<1% ( 48%) 729584 1
<1% ( 48%) 736288 1
<1% ( 48%) 756512 1
<1% ( 48%) 757792 1
<1% ( 48%) 792608 1
<1% ( 48%) 824192 1
<1% ( 48%) 894752 1
<1% ( 48%) 894880 1
<1% ( 48%) 895008 1
<1% ( 48%) 895136 1
<1% ( 48%) 895232 1
<1% ( 48%) 895360 1
<1% ( 48%) 895488 1
<1% ( 48%) 895616 1
<1% ( 48%) 898336 1
<1% ( 49%) 898464 1
<1% ( 49%) 898560 1
<1% ( 49%) 898688 1
<1% ( 49%) 906240 1
<1% ( 49%) 906368 1
<1% ( 49%) 993584 1
<1% ( 49%) 1160304 1
<1% ( 49%) 1252016 1
<1% ( 49%) 1334048 1
<1% ( 49%) 1351680 1
<1% ( 49%) 1385128 1
<1% ( 49%) 1403376 1
<1% ( 49%) 1453280 1
<1% ( 49%) 1497248 1
<1% ( 49%) 1497478 1
<1% ( 49%) 1497632 1
<1% ( 49%) 1497728 1
<1% ( 49%) 1501472 1
<1% ( 50%) 3563404 1
<1% ( 50%) 5303042 1
<1% ( 50%) 10757892 1
3% ( 54%) 82999856 1

Total blocks 2441403712 Allocated blocks 1120940828 Free blocks 1320462884 (54%)

File System Read-Only Check completed successfully.

singlemalt's picture

Well it's not the worst cvfsck -f I've seen. The problem is that snfsdefrag doesn't coalesce free space into
large contigous blocks. And snfsdefrag will only defragment a file if there is enough contigous
free space to re-write the whole file into. Ideally you would another pretty empty storage pool
to re-write the files to via snfsdefrag. But since there is only one storage pool and it's free space
is pretty fragmented itself, snfsdefrag might be of limited value. Ultimately it's going to come down
to how big the files are and if there is enough chunks of free space to fit them into. I would say it
can't hurt but I would make very sure all critical data is backed up ( which you are doing anyway, right?).

Were you able to set
ABMFreeLimit Yes
in the volume config file? Did the volume now show the missing free space?
If you are running Xsan 2.2 ( as per the kbase article) then this may at least make the
free space available. If all you deal with is small files then this may be OK. But if
it's mostly large files that really need to not be fragmented, i.e. video, then it may not help much.

messenger82's picture

We just had a downtime window tonight and I tried the ABMFreeLimit KB suggestion and it worked. It didn't show the free space right away though. We ran cvfsck -wv which showed the volume as clean. We attempted a full shut down, all MDC's and storage and brought everything back up. When we did the additional free space appeared. I definitely owe you a beer.