Tracking metadata volume usage

andy · partially protected Joined: 20 Mar 2009 Posts: 9

Today we ran in to an issue where our meta volume almost completely filled up (blame image sequences)...

I'm now wondering if there's a way to track meta volume usage easily - I can't seem to see anything about it on t'interwebs.

If there's a shell script - or a way to get it out via the shell - I'd very much like to know, so we can start logging exactly what's going on.

I'm guessing that it's not going to output the information we need over SNMP...

Any ideas?

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 860

i find that impossible to believe.

code:
cvadmin
select "number or name of FS"
show
stripe group 0 is metadata.

How big is your metadata LUN?

Stripe Group 0 [MetadataAndJournal] Status:Up,MetaData,Journal,Exclusive
Total Blocks:7630336 (465.72 GB) Reserved:0 (0.00 B) Free:7492068 (457.28 GB) (98%)
MultiPath Method:Rotate
Primary Stripe [MetadataAndJournal] Read:Enabled Write:Enabled

andy · partially protected Joined: 20 Mar 2009 Posts: 9

There's the possibility I just wasn't entering the right search terms then Embarassed

Thanks! From that I've used this to spit the data out:

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 860

I would move to larger metadata lun. there are instructions on this site on how to move your metadata.

jtownsend · Xsan Master Joined: 24 Feb 2008 Posts: 74

You can also add an additional metadata storage pool to the volume.

cedge318 · Could work for Apple Joined: 12 Jan 2009 Posts: 47

The only time I've seen a metadata LUN fill up has been due to corruption. You can add another metadata LUN, but if there is corruption then it too will fill fairly quickly. Does cvfcsk show as dirty or any other signs of corruption?

Sidd · JBOD Joined: 19 May 2009 Posts: 2

Hello all. I'm working with Andy on this issue.

We ran cvfsck (read only) and this came back clean. We are currently copying off all data off the XSAN to nearline storage before making any changes to metadata. We have a pair of larger 300GB SAS drives in a raid1 configuration ready for the migration or adding an additional metdata LUN. Copying the 40TB of data off is taking some time.

The comment that Cedge318 made about only having seen this issue once before is interesting - as since I've been working with this xsan setup the meta data has always been between 50% and 75% full, depending on how much data we've had on the system (betwen 50% and 75% keeping this under 80% on the video LUN as advised) An XSAN sysadmin I spoke to recently said this should rarely go above 3%. What's the consensus on how metadata should behave or does it depend on how the XSAN is configured in the first place?

Tim Burton · Xsan Master Joined: 04 Aug 2006 Posts: 260

Heya Andy n' Sidd

Talking percentages is all obviously relative. Most people have MD sitting on large SATA disks, of the same verity they are using for the data LUNs. If the MD lun is sitting on a 73GB high speed disk, then when compared to a 1TB disk 3% becomes 42% comparatively.

I think a dry run defrag would tell you how many extents the files have, high levels here obviously gobbles up MD space. Has the volume gotten really full at any point, fragmenting data whilst writing?

T

soward · Could work for Apple Joined: 30 Dec 2007 Posts: 42

how many files total do you have on your volume? Looking at one volume with a bit north of 3 million files, looks like it's using a bit under 4G of the MD partition. (xsan 2.2)

Sidd · JBOD Joined: 19 May 2009 Posts: 2

Thanks Tim. So the 300GB disk should make huge difference hopefully bring the metadata down to just under 25% full, which should see us through for a while, unless we're suffering from the type of corruption that Cedge318 mentioned. If that is the case I expect we'll be in similar position in the near future. About 5 months ago we had a one close call where the volume went passed 80% by approximately 3%. However since then we've been keeping a much closer eye on this and it has not happened again, not on the Video LUN anyway. That project has since been archived off of the volume.

Hi Soward, thanks for that. I'm running a find to determine that number now, hopefully that will finish in a couple of hours.

cedge318 · Could work for Apple Joined: 12 Jan 2009 Posts: 47

To answer the question about how much data, we usually see about 1 to 2 gigs per 500,000 to 1,000,000 files. It can change as you reduce or increase block sizes and stripe breadths but that's about the average. Therefore, seeing even smaller disks fill up tends to point to something being off. I did see an environment where the iNode expansion min had been customized that caused a lot more files, but we were still talking about less than 50 Gigs for a 200TB volume with about 2,000,000 files (if memory serves)...