Data Archive for XSAN

AlanS's picture

Forum-

I am looking for a solution to archive data stored on XSAN. I understand Retrospect is a capable application for storing large amounts of content and supported on Tiger.

An AIT tape drive attached via SCSI to a client is an option, but I am interested in what is in the field and operational. I want to remain on an Apple solution, and avoid the use of Storenext.

Has anyone used or seen the Quantum SDLT product line?

What popular archive solutions are you seeing in the field?

Thanks in advance-
AlanS

MattG's picture

For backup, I find it essential to have a fibre-based solution, independent of the machine that is doing the backup.

We these systems, we are talking about Terabytes upon Terabytes of storage: a single tape drive will not do, unless you want to spend your evenings/weekends sticking tapes into a drive.

I find the LTO3 technology intriguing and promising: 400GB uncompressed storage per tape, at about 250GB per hour. Put two of these into a library, and have them directly connected to the Fibre Switch, and you're in business.

Retrospect is not long for the Mac OS, IMHO. So, I would look at software from Bakbone, Tolis or ATempo in order to drive Fibre based tape drives.

MattG

aaron's picture

We're currently evaluating NetVault Bakbone and Atempo Time Navigator. I'll post some info when I have more experience with the product.

Aaron Freimark
CTO, Tekserve

AlanS's picture

Thank you for the direction. I will look at the solutions and vendors you are refering to.

Thanks again for the assistance.

-AlanS

smcculley's picture

We are running BakBone NetVault on our XSAN for backup, and it works like a dream. Well, at least not a nightmare like it would be with a single AIT...

We have a 6 Drive LTO3 library and have been getting amazing speeds over the fiber chanel network. I highly recomment it!

-Scott

matx's picture

We're using BakBone's Netvault with an ADIC Scalar 24 and an LTO3 fibre drive. I'd love to have another drive. Two drives would be nice. We're seeing average 100GB/hour. We've had it for a year, and I never thought 24 tapes slots would be insufficient, but our last projects needed 25 new tapes to back it all up. 400GB is great, but now I wish there were bigger tapes! When you're tape swapping like the old mac floppy swapping days, then it aint fun.

matx's picture

We're also using the simple and excellent Tolis Tape Tools. We need to read TAR tapes on occasion, but Netvault does not have this functionality (Atempo Time Navigator apparently does). It's very simple to disable the Netvault service, and jump into Terminal and pipe tar through the Tolis Tape Tools. Works well. Super cheap too. Wow.

brett's picture

If you are wanting to perform backups to an Xraid then of course you will need to go to tape as well... therein the idea of Disk to Disk to Tape. You will want to upgrade your single drive AIT to somthing more, say a tape library and some backup software that is compatible. If you are using Xsan, your choices are somewhat narrowed: Atempo (TIME NAVIGATOR), Bakbone (NETVAULT), TOLIS GROUP (BRU) and a few others. In any case your going to have to spend some money to get the job done so be prepared to invest. EMC's Retrospect and Xsan just don't mix, its not like the good ol' days since EMC has taken over that product. The windows version of Retropect is miles ahead of the mac version but that is off topic.

If I were to recommend tape media to go with:(not in order) it all depends on your capacity/growth needs.

LTO-3
SAIT
AIT5

for the library, if you got the money choose the fiber channel option otherwise you'll have to have a scsi card installed in your backup server that is direct attached to your library. In that case you would want to have your backup machine as a client on your Xsan so that you wouldn't be having to backup via ethernet but directly via the PCI-X or PCI-e bus of your Backup server.

In my situation I am using Netvault, Qualstar 2 drive SAIT library (scsi), Xsan 1.4, ATTO UL4S Scsi card, and backing up about 10TB currently with a growth without purchasing any additional hardware up to 24TB.

msudawg's picture

We have a 16Tb XSAN and have been using Time Navigator(TiNa) running on the backup metadata controller to manage tape backups to an ADIC Scalar i500 library with 2 LTO3 drives and 82 slots. TiNa has been a solid backup solution and I am able to backup @14Tb of data over the weekend. In addition, their support is top-notch and they have a tight relationship with the Apple developers to ensure full XSAN compatibility. TiNa has strong VTL, disk to disk to tape capabilities, as well as, an archive module that I am planning to implement soon. Spend the time and money to look at all of the options and buy more backup than you need today as you will need it tomorrow.

jkufahl's picture

I do not believe tolis bru will preserve permissions off a backed up san volume.

jelloknee's picture

Why go to tape?

- Tape formats change every 2 years
- You cannot find diddly on tape when you really need to.

If you are archiving multiple TBs then tape/optical is not always so appropriate given that the majority of problems occur at the restore and not the backup. It is even less appropriate in creative workgroup environments where access to all data both current and legacy is not simply desirable but absolutely a requirement.

Just my two cents.. expecting to be flamed :twisted:

As for applications I would stay away from Retrospect and Atempo, Tolis is OK but how long will it be before Time Machine makes it to the network?

ogminlo's picture

jelloknee wrote:
Why go to tape?

- Tape formats change every 2 years
- You cannot find diddly on tape when you really need to.

If you are archiving multiple TBs then tape/optical is not always so appropriate given that the majority of problems occur at the restore and not the backup. It is even less appropriate in creative workgroup environments where access to all data both current and legacy is not simply desirable but absolutely a requirement.

Just my two cents.. expecting to be flamed :twisted:

As for applications I would stay away from Retrospect and Atempo, Tolis is OK but how long will it be before Time Machine makes it to the network?/quote

I couldn't disagree more. See my post [url=http://www.xsanity.com/forum/viewtopic.php?t=397&postdays=0&postorder=as.../url.

Do you plan to just expand your disk forever and ever? HSM is the way to protect your SAN and Xsan's big brother StorNext has one that mixes in nicely. You have to plan it right, but it is very powerful and actually works.

LTO-4 is out now and offers excellent price and performance. Avoid anything with DLT or AIT in its name. LTO-4 drives can read LTO-3 tapes, so you can migrate formats and still read your old material.

jelloknee's picture

ogminlo wrote:
jelloknee wrote:
Why go to tape?

- Tape formats change every 2 years
- You cannot find diddly on tape when you really need to.

If you are archiving multiple TBs then tape/optical is not always so appropriate given that the majority of problems occur at the restore and not the backup. It is even less appropriate in creative workgroup environments where access to all data both current and legacy is not simply desirable but absolutely a requirement.

Just my two cents.. expecting to be flamed :twisted:

As for applications I would stay away from Retrospect and Atempo, Tolis is OK but how long will it be before Time Machine makes it to the network?/quote

I couldn't disagree more. See my post [url=http://www.xsanity.com/forum/viewtopic.php?t=397&postdays=0&postorder=as.../url.

Do you plan to just expand your disk forever and ever?
/quote

Yep. Data on tape drastically reduces the ability to take future value from that data or provide that data when required. The whole problem does not reside around archiving the data off to a medium it is the restore. I am sure there are plenty of people on this forum who have restore horror stories.

Many companies are also mandated to keep years worth of content available 24/7.

Disk technology is providing faster bigger drives for less. Within two or three years holographic or solid state disks will also drive down the power consumption costs.

Quote:
HSM is the way to protect your SAN and Xsan's big brother StorNext has one that mixes in nicely. You have to plan it right, but it is very powerful and actually works.

LTO-4 is out now and offers excellent price and performance. Avoid anything with DLT or AIT in its name. LTO-4 drives can read LTO-3 tapes, so you can migrate formats and still read your old material./quote

HSM or multiple tiers of infrastructure are inherently costly in terms of employing the expertise to keep them running.

Disk based archives that manage data redundancy and automate capacity growth and manage failure remove the priesthood from the process.

ogminlo's picture

jelloknee wrote:

Yep. Data on tape drastically reduces the ability to take future value from that data or provide that data when required. The whole problem does not reside around archiving the data off to a medium it is the restore. I am sure there are plenty of people on this forum who have restore horror stories.

Many companies are also mandated to keep years worth of content available 24/7. /quote

Data on LTO in a Scalar under StorNext Storage Manager is available with no human interaction beyond the user's request of the filesystem. I'm sorry if your experience with tape has been poor, but I assure you my data moves back and forth from LTO-3 all day, every day.

Quote:
Disk technology is providing faster bigger drives for less. Within two or three years holographic or solid state disks will also drive down the power consumption costs./quote

I look forward to the day when optical and holographic are ready to step up, but they have a long way to go and there is at least one more depreciation cycle between today and the day they are even remotely competitive with 2005's LTO-3 on price/performance. Compare the specs of the existing [url=http://www.inphase-technologies.com/products/default.asp?tnn=3]tapestry/url format to that of [url=http://en.wikipedia.org/wiki/Linear_Tape-Open#Generations]LTO-4/url.

Maybe you have an unlimited supply of NOC space, power, and HVAC, but I do not. Within the next decade tape will be hopelessly outcalssed by better formats like optical, but for today and the next 3-5 years, LTO is a very safe investment.

Quote:
HSM or multiple tiers of infrastructure are inherently costly in terms of employing the expertise to keep them running.

Disk based archives that manage data redundancy and automate capacity growth and manage failure remove the priesthood from the process./quote

I have had my share of headaches, but in my environment I need to be able to cycle old data off of expensive disk and onto something less costly to hold. Nearline tape is ideal for my situation. I expect to switch to optical some day, but the known roadmap says it will be a while before I do.

You are welcome to work with Isilon or the like, and I wish you the best of luck running it with no support staff. I simply do not agree that it is practical to eternally expand your disk to accommodate your eternally growing data. Eventually you run into environmental constraints that make single tier storage impossible to expand further. I suppose you can keep faith in Moore's law to keep pace with your data creation, but I'm not betting my job on it.

Ernesto Sanchez's picture

Totally agree with ogminlo. Power and cooling are major obstacles to continuously expanding 1st tier storage to meet storage requirements. Did I forget to mention the cost of very fast FC RAID arrays. It is a positive development that we are moving beyond the basics of just implementing a SAN solution. Now we are focusing on asset management and tiered storage. This is great. Our small community, "Video IT", is maturing and learning from the larger IT community.

Best,

Senior SAN Engineer - The DR Group
O - 323.896.3416
C - 949.690.3680
ernesto@thedrgroup.com

Moggs's picture

Ernesto Sanchez wrote:
Totally agree with ogminlo. Power and cooling are major obstacles to continuously expanding 1st tier storage to meet storage requirements. Did I forget to mention the cost of very fast FC RAID arrays. It is a positive development that we are moving beyond the basics of just implementing a SAN solution. Now we are focusing on asset management and tiered storage. This is great. Our small community, "Video IT", is maturing and learning from the larger IT community.
/quote

"Tier 1" SAN, spinning those disks at 15,000 revs per sec, is always going to be a little power hungry; but is different than "tier 2" storage for post-production (archive) data; which can be a lot more gentle on the accelerator, and accordingly burn a lot less gas...

Power and cooling is certainly an interesting question but perhaps it isn't as big a deal as some think?

Against power utilising storage:
- rising power costs

For:
- storage capacity doubles each (approx) 12 months, thus, ipso facto, if your storage requirement doubles each year, and you switch in double capacity drives every year then your power utilisation stays the same. In reality, you'll probably switch out hardware every 3-5 years, but even so, the sums are the same
- as much as power costs are going up, "greener" drive/solutions are coming in; and (dare I say, but this is of course an educated guess) will probably significantly outpace rising power costs

Thus, my argument being that making a disk-based archive (or similar) is consumes power, but that thoughts of it being a growing burden on the business are, arguably, overstated. Furthermore, if you want to have a fast post-production archive it might be the greener to go to an online archive solution now, rather than to buy a tape based solution which, if it doesn't give the functionality required, is going to end up on the scrap heap anyway.

Tape may seem like the best solution for some people, but as often is the case with cost and/or green arguments, there are a lot more factors to be taken into account than just the headline "today" figures.

Bryson's picture

I agree... LTO has been trouble free for us generally as well. The restore times aren't bad and even with a single drive (which I don't recommend) you can get by if you're not creating more than a few GB a day.

(a 400GB tape can last a few days/weeks in a small shop so you swap less than you might think)

Autoloaders are under $10k now (I've seen them under $5k).

I agree with the "green team" on here. Most admins aren't paying the light bill on a shop when they say, "let's add spinning drives for backup!"

I am looking into a plan to put the "most active" couple of TB on some spinning drives for "just in case" situations.

Please someone, give us tape virtualization for small shops! I want to drop a tape in and have it searchable and browse-able as a volume. (no, I don't want a Quantum "A" series lol) I want the ability to replace video tape with LTO in a paradigm that my video guys can understand for small backups and archiving video to tape.

I bring it up to companies and they all act like tape is dead. Video will revive it, I believe, as video tape decks die. The guys on this board alone are creating petabytes a year of data that can't spin forever... at least I don't wanna pay the power bills if it does.

Ernesto Sanchez's picture

Bryson, I am looking for the same kind of solution. On that does not force my clients to learn a backup application. I feel this is the main reason creative shops do not use their tape libraries. It seems that Final Cut Server along with a HSM solution such as Quantum's Storage Management Solution fits the bill. It allows for data to be moved to tape libraries without end-user interaction. The client only interacts with FCS to access assets. I am in the process of setting up this solution in our lab and will report what I find.

Cheers,
Ernesto Sanchez
Senior SAN Engineer - The DR Group
O - 323.896.3416
C - 949.690.3680
ernesto@thedrgroup.com

Bryson's picture

Thanks man, I'll look a little deeper into my developer pool too. FCS might help us if they get it rollin.

I'm gonna show your post to some of the software guys who claimed "tape is dead" lol.

peace

mark raudonis's picture

It's important to note that "back up" means different things to different people. There's a huge difference between "archive" and just "back up".

If I had to summarize all of the solutions that I've explored to date, it's that they're all too complex and too expensive for our needs. Sure, some people need absolute mission critical back up and intstantaneous restore, but in my world (video post production), that isn't critical. Often times the original master tapes are available as a "source of last resort" in the event of a total drive meltdown. So, I already have a "back up"... it's just too slow to restore in a crunch. So, we're really treading the fine line between convenience and necessity.

Also, as the recent fire at Universal Studios illustrated, you can't just assume that the vault (or your entire building) is secure. So, I'm exploring the concept of "fire insurance" back up. Ideally, this is a spindle based solution that physically lives in another building. It doesn't have to be fast, have raid redundancy, or even be elegant. It just has to be cheap and BIG! (75 terabytes) I only need it to "survive" for the duration of a project's post production schedule ( 6 months), and then it gets written over.

This kind of manual system isn't gonna excite any of the code monkeys on this board, but it will get the job done. Until Google opens up it's "G Drive" concept to this scale of data warehousing, I'm stuck in the "sneakernet" world.

Mark

Ernesto Sanchez's picture

Mark,

I found the following solution when I was researching offsite backup for disaster recovery. StorageDNA, http://www.storagedna.com , provides asynchrous backup. It is designed for post-production environments. I have not implemented it but I like the concept. I talked to them at NAB and they sound like the understand the needs of video post-production.

ogminlo's picture

I know I've been an LTO guy on this thread, but an alternative worth examining is StorNext 3.X's new storage disk de-duplication capability. They don't play it up much on their [url=http://www.quantum.com/Products/Software/StorNext/Index.aspx]product page/url, but one of the new features Quantum added to Storage Manager for version 3 was a software de-duplication engine. I haven't had my SAN analyzed yet to see if my video data is a good candidate for de-dupe, but if it is and the ratio is right, I would absolutely do a mezzanine disk tier before my Scalar. I figure the de-dupe disk would be nearline and the Scalar could be smaller than it otherwise would be and could be used for vaulting the oldest data that still has value. For some of you guys, this might be the trick that makes it practical to have nothing but disk.

I plan to have Quantum analyze my SAN to see what my de-dupe ratio is. I'll post results afterward.

ogminlo's picture

Ernesto Sanchez wrote:
Bryson, I am looking for the same kind of solution. On that does not force my clients to learn a backup application. I feel this is the main reason creative shops do not use their tape libraries. It seems that Final Cut Server along with a HSM solution such as Quantum's Storage Management Solution fits the bill. It allows for data to be moved to tape libraries without end-user interaction. The client only interacts with FCS to access assets. I am in the process of setting up this solution in our lab and will report what I find.

Cheers,
Ernesto Sanchez
/quote

Ernesto,

After testing FCSvr with Storage Manager I found that it is surprisingly easy to make FCSvr drive the HSM. Just set up the SNSM policies to manage your Archive device with immediate store and truncation and let Search Expired responses filter what goes to that Archive device. When a truncated file is requested by a FCSvr restore command, FCSvr waits patiently while SNSM automatically goes and retrieves the file blocks from tape and then copies the data back to the original asset position. The two systems don't really know about each other, but with these complimentary policies, they work well together. Best off all, the end user can recall a primary representation from SNSM with one click on the little file cabinet icon in FCSvr.

fritz.'s picture

Funny, I was discussing the possibility of FCSvr and Storage Manager yesterday. I haven't had anyone bite at it yet but it's comforting to know that it can be done!

MattG's picture

It's a lovely combo:

Two issues to consider:

1) SNSM is !(&*ing expensive.

2) The Restore Feature in FCSvr [i]deletes/i the file from the Archive device upon successful transfer of the file back to its original location. Hopefully, in a future version, Final Cut Server will be able to account for an archived representation regardless of whether the Primary Representation is there or not.

ogminlo's picture

MattG wrote:
It's a lovely combo:

Two issues to consider:

1) SNSM is !(&*ing expensive.

2) The Restore Feature in FCSvr [i]deletes/i the file from the Archive device upon successful transfer of the file back to its original location. Hopefully, in a future version, Final Cut Server will be able to account for an archived representation regardless of whether the Primary Representation is there or not./quote

1) Depends on what you compare it to. Compared to keeping DVCam masters of our finished content, it is about 40% cheaper for us to keep files in an SNSM managed Scalar. That includes the cost of the LTO tape. Also, for very large archives (several 100s of TBs) the cost of licensing SNSM drops significantly.

2) Indeed, but this just means you have to stay on top of your fragmentation cleanup in SNSM. Annoying, but workable.

vidiot's picture

Chalk me up for SNSM and FCSvr too. We've had the 2 running together well for about a month now. It's fantastic that I don't have to go hunting and loading of files and tapes. The users just click and restore as needed. Brilliant!

To MattG's #2 point, this is really a problem. Once you've restored an asset, FCS deletes the file from the Archive Device, in this case SNSM. Which means your asset is now once again unprotected. For us, that is usually a P2 MXF camera original file.

Sure it still lives on tape in a manner of speaking, unless you clean up all the tapes.

I've asked Apple for a quote from their Professional Services team to write me a different Archive process that does *not* remove the file from the Asset device.

Ernesto Sanchez's picture

Well gents, I'm glad to see there are real deployments of FCSvr and SNMS. I'm now in the shoes of the client. I need to implement an archive solution for in the next 6-12 months. I really want to see a demonstration of the this solution. Anyone know anyone at Quantum who could set this up in the LA area?

Cheers,
Ernesto
Toybox Entertainment

om_nick's picture

There are also plugins that allow seamless archiving to a disk based archive.

[b]Benefits: /b

- Assets AND metadata protected.
- Archived assets remain protected upon restore.
- Entire Productions and associated metadata can be archived with a single click.
- Scripts provided to backup FCSvr DB and proxies.

[b]To the subject of disk based archiving:/b

- 2TB disks now available using same or slightly less electricity than 1TB
- New disks spin down when not in use. Green and getting greener.
- Your data is available 24/7, when you need it.
- some clustered systems automate the most error prone and labourious tasks, are self-healing, self managing. < One admin per Pb+. Whats the manual resource count for the same amount of tape?
- Metadata kept with the assets not on a separate DB on a separate Tape.

The reality is that some customers are asking for disk only, some are asking to disk nearline/tape mix. Enabling customer choice is good which is why we have created a similar integration with CatDV.

Contact me to chat or find out more.

(I have edited this post to respect the non-PR bull policy Xsantiy is known for, thanks Nick)

bjr2's picture

Hi All! Me again!

So, our FCS system is running smoothly. I finally got approval to tier our SAN system. So we're ordering a G-Tech XL 16tb FC (quad) for editing, demoting our current XServe Raid SAN for broadcast and hosting, and I'm currently mulling over a quote for a Quantum LTO4 Autoloader for tape backup.

So, I've read in this thread that some are using the Quantum Autoloaders for their backup. My only question is if there's any possibility of designating the Quantum as an archive device in FCS so that the rest of the editors don't have to worry about running some script and can just archive assets within the FCS client.