FCSvr not up to the task?

[bildon]

So here is my dilemma... I have an XSAN with three volumes of about 80TB of content. Most of this content is DVCProHD content with a mix of some stills and the occasional group of P2 Camera masters. My client is insisting that their ENTIRE production SAN is cataloged with FCSvr, yes, all 80TB. They will be integrating FCSvr with a MassTech system and therefor will be doing the majority of their archiving using FCSvr as the funnel (ahem bottleneck) into MassTech.

The server is pretty beefy, Dual-Quad 2.93 w/ 24GB RAM. It is also setup as its own Compressor Cluster with 6 services.

So far so good, nothing to crazy here except for the fact that we need to catalog 80TB asap. That's where I'm running into problems and everything falls on its face.

I've spent quite a bit of time making Schedules to run a few Full Scan Responses for a chunk of one Volume, say about 5,000 files in total, mostly DVCProHD assets. When the responses are run and I check the jobs i see the windows just explode in activity, my max running jobs (25) is filled almost immediately, and consequently, the remaining several thousand jobs go to a "WAIT" state. Assets will begin to be cataloged, thumbs will be created, compressor's job queue will fill up and everything will be chugging along quite nicely. After about 30-60minutes though I notice that the "Active Memory" is completely full and Jobs start to FAIL. The error, consistently, is: ERROR: E_RESTART Server shutdown or restarted unexpectedly.

So, any ideas what's going on here? From the behavior it looks as though it is triggering these as background tasks (even though they are not set to "background analyze" and the memory is filling up...

I'm at a loss right now and could really really use some help! Thanks so much guys!

Cliffs: FCSvr can't handle processing ~5,000 new assets at once through a full scan and it crashes the server...

[ogminlo]

When I rolled out FCSvr 1.1 in 2008, we had about 3,000 DV assets to do at once. I rounded up all the surplus Macs I could and made a Qmaster cluster to offload all the proxy encoding. No CPUs on the FCSvr host did any encoding work. I had 16GB RAM in the FCSvr host.

Maybe having the cluster off of the FCSvr host creates enough of an abstraction to save FCSvr from burning itself out? You're on a newer version so it could be a different story now. Worth a shot though.

[bildon]

Looks like I'll end up doing something similar. Decided to not enable clip proxies at all and, in the interest of time, only focus on cataloging the assets themselves with only thumbnails and posterframes enabled. When all is said and done there will most likely be over 1,000,000 assets in this system and the majority of the proxies will probably never be viewed. We'll deal with non-proxy assets on a case by case basis and only with new assets.

I do wish I had 20 brand new Xserves to build a cluster with though!

[blauer]

I'd love to know how interfacing with MASSSTORE worked. We have one that I want to use with FCsvr for archiving. But I don't get how to get the metadata to transfer. Like your system we will have 1,000,000 assets very quickly. The last thing I want to do is log those assets twice. Any Insights would be helpful.