Finder crashes after xsan finder copy, Apple is stumped

[abstractrude]

i totally believe him. are you using native extended attributes? i have about 70 hollywood movies at prores 422 hq in our SAN. they are usually about 125-175 gigs. Haven't been able to reproduce those issues...what are your metadata latency times like. can you give us a cvsummary.

Typical large prores file found on the big SAN.
Macgruber.mov: 8 extent(s)
137G Aug 6 15:33 Macgruber.mov

Have you thought about doing a perl script or even bash that does nightly extent scrubbing?? so doesnt solve the problem but it would be a great workaround.

[lucasnap]

Latency is between 250-280 us. So that's OK.
Extended Attributes are turned off, you never know when they come with the question if they can hook up a Linux machine for this and that.

What do you mean by cvsummary? Do you mean a cvgather?

Does anyone have any news on the Apple bugreport?

Our extent counts are up to 6000, but even more files are holy. This is because of the way they work. They record a show with 3 HD camera's simultaneously ingesting on the Xsan Volume.
Our workaround is a big bummer: The ingest Mac's record on an extra internal disk and after that the file is copied (one by one) to the Xsan Volume.

I've played with snfsdefrag, but because of the hanging and instability of machines when only touching the corrupt file i'm not so keen on scripting on this subject. I rather see an update.

[abstractrude]

cvsummary is in the volume log. Also, how much free space do you usually have on your Xsan? cvadmin show for us? im trying to figure out why are you getting such high extent counts on your files.

[lucasnap]

These are the summary's:

[Karepi]

Wanted to add a me too, to this situation. Currently experiencing slow browsing just as described in this thread. Sometimes it will become unstuck but most of the times a client reboot will be necessary

So right now the possible causes are:

Cables.
Fragmentation.

Any other option that we might be missing?

Have a good day.

[jfw]

I have a client with these symptoms too.

I saw it on a directory without any files with extents problems, but perhaps they hit a file with the problem without noticing it before navigating to the directory that wouldn't show up at all.

Apparently once the Finder gets in this state, you can still do an "ls" in Terminal to see the contents of the directory. But you have to reboot to get the Finder to work again (for a while anyway).

I'd like to get an update from the original poster, dmastroluca, who hasn't posted since 3 weeks ago when he said (http://www.xsanity.com/forum/viewtopic.php?p=53703#53703) that essentially defragmenting the whole volume (by erasing it and recopying the files) didn't help.

[lucasnap]

Nice to know there are more sites with this problem. Maybe we can try to figure out some more.

Do you guys use very large files as well? (around 100GB and up)
And how do these files get on the Xsan? Ingest, copy, FCP, FCSvr?

We have seen the same extreme amounts of extends on files who got picked up by FCSvr (Watch folder > copy and delete) so we need to make a work around on that as well.

[Karepi]

Very large files as well.
We get the files on the SAN through ingest.
And have some scripts moving files around from volume to volume.

Running snfsdefrag will get stuck while defragging, would seem like a latency problem the thing is if you test the whole fiber setup and network setup it seems ok.

If the Volume is clean there are no problems. It starts when the volume is around 50%.

Doing ls through the terminal works sometimes, but i bet if you do ls -ls it will get stuck.

My question is, is it a client-side problem or a server-side problem? Meaning what if the mdc's are hosted by a leopard instead of a snow leopard?

[lucasnap]

If you run Leopard on the MDC you also need to run Leopard (or lower) on the clients.. So it certainly is a good question but it is a rhetorical one.

The MDC makes too many extends, that is the problem. So the solution will be either at the server (to make less) or at the client (read more).

[lucasnap]

I posted a bugreport at radar.apple.com:

[AL]

Hello, I'm administering a system that's having similar problems. There's separate post about our issue (http://www.xsanity.com/forum/viewtopic.php?t=10603) but I think I should continue posting here.

We don't think we have any hardware issues but big problems with Finder when browsing the file system. I've been to contact with Apple's Xsan support and currently waiting information from their engineers of the collected log files. I suspect that the problem is caused by some the following or a combination of them: high fragmentation of the volume, high number of files and used inodes or/and volume settings.

[mcjunkie]

Today Apple released Mac OS X 10.6.5.
There is an Xsan related note in update description. (http://support.apple.com/kb/HT4249)

Xsan
Improves reliability of fibre channel connections, resolving a potential Xsan volume availability problem.

What do you think about this?
I hope it may fix issue we've experienced long time.

Thanks.

[lucasnap]

Doesn't sound like what we need... I was hoping more in the line of "improved reliability memory allocation/fragmentation".

I've seen this in the Pro App 2010-02 update:
Final Cut Pro "Several memory fragmentation issues have been resolved."

That sounds about right. Did anyone test this already?

[Karepi]

Still haven't tested 10.6.5. Anyone out there that has? So what's the current status of everybody? Over here still same old same old. Would moving to full fledged Stornext help?

[lucasnap]

As promised I decreased the InodeExpandMin settings to 2 and 4 and interesting enough it did change the behavior of the volume. (default is 8 )

First with "2" the volume became pretty unresponsive, especially with writhing big files.

With the "4" setting I exported 2 150GB QT files with QT 7 Pro on two different Xsan clients. The result was extents of 7000 and 14000!!! That must be a record! But unfortunately still all the Finder's crashed trying to do something with the files.

Next test: 10.6.5 (I'm at the test environment at the end of the month, so we will have to wait).