The ever expanding Proxy Bundle

[Toby@ITV]

Hi there-

We have a major concern down the pike for us. We are about 6 months into what I would describe as a "full on" FCS usage. It's no longer an experiment, but it's something our facility of 8 editors, 2 compressionists, 3 gfx guys and 10 producers are using everyday.

We have an assistant editor who on average ingests 50 XDCAM HD discs into FCS every month. Sometimes it's less, sometimes it's alot more.

Currently our Proxy Bundle is tipping the scales at 1TB. Several months ago I reduced our proxy bitrate to approx 1800 (h.264) and that I imagine has helped considerably, but we really can't go much lower quality-wise. At our current rate of work and proxy expansion, I see the proxy bundle needing it's own RAID or JBOD within the next 1.5 years. This strikes me as absolutely nuts. A small company like ours that happens to do "big company" volume simply cannot keep up with that sort of rapid expansion. Apparently the word is that FCS does not replace proxies, it's merely adds more. I did a test one day by reanalyzing the same proxy over and over again to see if the bundle would stay the same size, but no...after every re-analyze, the thing grew and grew. I wish there was a way to manage these versions under the hood.

-Are there any rumblings from Apple about a way to manage proxies in the future?

-Is there any way to do so now?

-COULD I dispose of the proxy bundle and take a long weekend to re-analyze the thousands of HD clips in there into a new lower res proxy? (Like I said, I just changed the h264 bandwitdth into something more manageable...but 90% of the proxies were encoded considerably higher).

In regards to a re-do of the proxies, I've heard two conflicting opinions. One person has suggested I simply toss the bundle and re-analyze. (It would take days, but if it saved me 500 gigs of expensive Raid 6, it would be worth it). ANOTHER expert has told me NO, I cannot just toss the bundle and start recreating new ones...I'd need to begin with an empty bundle in it's place. Would changing the proxy device location in FCS prefs recreate an empty bundle for me?

What do you guys think? I opened the proxy bundle today to see the contents, but it was pretty indecipherable as to locating duplicates.

-Toby

[MattG]

While I'd be reticent to give you on an opinion on what you should do, I can shed a little more light on the Proxy bundle and its function.

It's a bundle only for the reason that a bundle is somewhat protected from further digging into via the Finder by the average Joe. Really, it's just a staggeringly large number of folders and subfolders, eventually with single files in each folder. The folder names correspond to an ascending counter in Final Cut Server that assigns these names to hidden fields for each asset. This is how Final Cut Server "finds" the derivative representations of its assets. Remember, the Proxies bundle not only contains H264 proxies, but also thumbs and poster frames as well.

Given this, if you simply blow away the bundle, and recreate another one by simply naming a folder "Proxies.bundle" in the same place*, that's all FCSvr will need to "start again" with its process of storing more proxies and thumbs and poster frames. The counters won't reset, and all links to what FCSvr thinks used to be in the old bundle will of course be broken. However, reanalyzing each asset will get you new derivative representations.

Please let us know if you've found another way, but the biggest problem of all is that of reanalyzing all your assets. The best practice here is to select them as an admin user and select Regenerate All Proxies from the contextual menu. Now in 1.5.X, you could set your search return to the maximum 2,500 assets, select the first asset in a search, go to the last page, shift-click the last, and ask for 2,500 assets' proxies to be remade. But I can almost predict a crash of the fcsvr_stored process if you attempt that.

So, let me offer the last sane idea, which is simply to relocate the current Proxies bundle to another location, like a newly puchased DAS hanging off the FCSvr. All you need to do is ditto the bundle to the new location, and then in the Device pane of the Admin window in FCSvr, just edit the path of the device. As soon as you save, FCSvr will source the proxies, etc. from the new location and write new ones there as well. Through the magic of relative pathing, everything is still accessible.

In all the classes I teach on FCSvr, I stress the importance of planning out a relatively permanent place for the proxies, but also mention the relative ease by which you can move them in the future. Yes, it was always the intention of the engineers for the bundle to grow and grow, especially since accessing proxies of assets long archived was a critical feature that most shops want to have.

*If you actually do this, it's important that the bundle is owned and has rw permissions for the same admin user that you used to install FCSvr.

[Toby@ITV]

Matt-

Thank you for responding. I'm going to share this letter with our FCS integrator and come up with a game plan.

Do you you personally have any clue why Apple chooses to keep creating new proxies instead of blowing over old ones? We were informed by Apple that they create "transient" proxies....every time any asset moves in or out of server that new proxies are being created. There could theoretically be 100 proxies for the same clip depending how many times it's been checked in and out.


Also, using this logic:

So, let me offer the last sane idea, which is simply to relocate the current Proxies bundle to another location, like a newly puchased DAS hanging off the FCSvr. All you need to do is ditto the bundle to the new location, and then in the Device pane of the Admin window in FCSvr, just edit the path of the device. As soon as you save, FCSvr will source the proxies, etc. from the new location and write new ones there as well. Through the magic of relative pathing, everything is still accessible.

....could I move the current proxy bundle to a new location for safe keeping (we currently have them mirror to a RAID 0 G-RAID for backup), then simply DELETE the bundle in it's current location, place a new "proxies.bundle" folder there and then begin a well paced process of re-generating the proxies? (You are right though, doing 2500 at once would burn our building down).

Thanks again Matt....it's really cool to have you answer this!

-Toby

[JonE]

[JSamuel]

[abstractrude]

i always dedicated a raid 5 lun for my proxy bundles. usually in xsan integrated environments i would have the customer purchase the half full promise raid. two drives for metadata for the san volume, a hot spare and the other 5 drives as a lun for the proxies for FCS. works pretty well. you can use lum affinties on the raid controller to make sure your proxy traffic doesnt affect san performance.

when saying ditto he means use the ditto command to copy the data. its ussually the best way to copy mac os x data. its also nice that it keeps your modification times and permissions intact and all extra filesystem metadata too. i used to symlink the fcs database other places but it turns out i didnt see a performance increase.

[JonE]

Hi JSamuel and Abstractrude...

Thanks for your replies. I might be over-thinking this - so sorry if that's the case.

So once the proxy.bundle has been ditto'd to the new location, i can tell FCSvr to run the proxies from this new location and then safely delete bundle from the old location?

Thanks,

Jon

[JSamuel]