SSD/Xyratex/Xsan questions

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

OK so let me give a little background to my issue before I dive into what I am thinking.

Currently I am sitting on a couple of high end NAS systems using NFS as mounts to my linux cluster that is running in front of it. I have 15 servers in the cluster. The volume behind the cluster is Exanet Exastor. It is currently a 110ish TB volume running of top of 7200 RPM sata drives in IBM DS4700 series enclosure.

I have a second setup with the same configuration minus a couple cluster nodes and is about 10tb less in disk.

I have a problem with i/o operations per second. The speed the data comes in and out of the system isn't the problem I am having at all. The problem I have is that I am having to navigate huge collections of block history and my block history is approximately 450 million files in what is 65TB stored on the one system right now.

The purpose of the block traversals is to verify the data on the system is correct and that I can rebuild the data from the blocks. Right now I can't accomplish this, I have had NetApps, EMC, HDS/BlueArc, Exanet look into the issue and the problem we all keep coming back to in our testing is that the i/o operations per second of the storage system is the culprit. Obviously sata has to go. Discover my max store size is 30tb, with my customers growing at the rate they are growing it means I have to cap my systems at 20TB and then let them organically grow to 30tb = Gross

Next I attempt to use an HDS Blue Arc system with high end AMS 15k SAS under it. Through testing and all of the above leg work I find myself able to achieve 50TB but still no where near what I need to be.

Obvious things that impede i/o operations per second.
NFS
NAS front ends
SATA seek time
SAS seek time (albeit better)
4gb fiber... not so much but could still be better

Now I am trying to figure out the proper way to go about this since I obviously have issue to resolve.

I am now going to lay out what I think is my solution but I want to verify with some people who have used at least part of this product chain I am thinking about using to see what they have to say.

• Xsan meta data cluster (2 servers maybe)
• 8gb FC interconnect
• Quantum's linux Xsan client for my front end nodes
• Xyratex 24-drive 2u SFF(2.5") disk array enclosures running two raid 6's per enclosure (12 drives per raid 6) (32 enclosures)
• Intel X25-M 160gb SSD Drives (768 Drives)

Another thing I am interested in is if I would be able to use my sata/NAS system as a backup point using block level replication or file level replication through xsan.

What I am wondering is will xsan improve my i/o ops as well as my mb/s transfer rates. And as I am a complete xsan nub will I be able to create a single volume from all of those arrays using Xsan.

I am wondering if Xsan will reduce one of my bottle neck points to allow the SSD to actually be useful for me. I know SSD will solve a great deal of my Random access i/o problems that I am sitting with now.

So what do you guys think?

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 860

let me bring this all in. ill get back to you tomorrow. also. what type of files are on your SAN

mw10dot1 · Xsan Master Joined: 06 Dec 2006 Posts: 106

Why are you thinking of xsan vs stornext for the controllers?

Michael

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 860

i dont think xsan is the solution IMHO

maybe one of the SUN distrubuted filesystems or other enterprise stuff. with the kinda of hardwarre you mention you can afford a small test. build a small xsan and see if it performs to spec. if it does build out to the size you need it to be. 100TB is about 16 Vtraks with hot spares and all the overhead. also stornext controllers are the way to go, because you will probably want HSM and stuff. plus on linux you will probably be more comfortable than OS X.

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 860

i think lustre is a better datacenter FS. stornext is too slow for millions of small files. They one way stornext would work for is if you broke your metadata up logically into tiers for faster access. I can tell you right now that you would be playing with fs block size. I just havent had the kind of performance with small files on Xsan that i want. Image sequences are always faster locally than on Xsan. I have never seen it otherwise. That being said, xsan is a great product but a test is really in order here. are you in los angeles? if so you can come by and check mine out.

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

matx · Xsan Master Joined: 15 Apr 2005 Posts: 378

dbruhn, i know someone that has a xyratex 5404 E that you can have... I built an Xsan volume from it, and it's being used as a backupSAN volume. the performance was horrible and could not be used for anything but cvcp or rsync.

abstractrude, Xsan rocks for image sequences. I've built a many render farms with an Xsan backend, from lowly Xserve RAIDs to Promise VTraks, (and maybe soon Active Storage XRAID). They all rocked. Of course.

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

soward · Could work for Apple Joined: 30 Dec 2007 Posts: 42

This certainly sounds like an interesting problem. I'd need to know a lot more about your software and exactly how it does what it does.

If you have something on the order of 10M files/Terra as indicated and do a lot of very small updates to the data behind them, then Xsan probably will decrease your overall throughput.

In one install I have about 3.5M files/T and it performs somewhat below HFS+ at intensive operations (like rsync, e.g.), but in this case having the FS available across multiple machines with no SPOF, was desireable and for normal use the XSan is well fast enough.

I've thought of using SSDs for the MD component of Xsan, and I think it could help, but then so would raid-10s of 8 + disks each.

As you indicated enough modern disks can overrun many raid controllers (espcially at R5), And likewise for a lot of small I/O bits, 8 2G FC channels will probably be faster than 2 8G FC Channels, and they could talk simultaneously to 8 targets, each of which could be a raid controller. Ca

Is it possible to modify your software and/or put a layer between your software and the FS? Can you monitor the SAN and see what the usage is?

I don't know much about the workings of the exanet, I assume it's a proprietary backend CFS, but what version of NFS does it use? Are you mounting with noatime (or is atime important?)?

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

Yeah the software is kind of out of my control at this point its an MSP over the wire backup solution called Asigra.

Will the xsan decrease my throughput as much as a NAS system with NFS, does? Thats the benchmark for me in this situation, will it be better than what I have, and will that betterment allow me to grow my system size.

Exanet is a NAS Storage Platform similar to Blue Arch or Netapps. It in all reality is just a file server with a bunch of disk behind it. It allows me to see my 110TB of arrays as a single file system. I am running a fiber backend to the storage servers themselves. My current front end systems are running 1gb ethernet to the NAS.

I currently don't have a traditional SAN, I have a NAS with a SAN behind it. The NAS system lets me see its usage as it is right now, the problem I am having is every measurement is worried about throughput which I don't have a problem with but IOPS which don't show up on my SAN as being that intensive. In fact to my NAS it just views it as wait times for the disk to give it what it needs.

AS much as I hate the company infortrend apparently has a dual controller 8gb FC array that might fit my needs. Or I am looking at storage servers of sorts with Infiniband to put behind this system.

I am unsure of the NFS version I could do a little checking but that might take a bit for me to get to. I am mounting using noatime at the moment.

I hope that answered all of your question

soward · Could work for Apple Joined: 30 Dec 2007 Posts: 42

I'm familiar with the basic operating principle of the exanet, but not familiar with the exact implementation. With, say, a NetApp depending on the model/version it used various different OSes (all called DataONTAP but really quite different), backend filesystems, and resharing implementations. And has various tools to monitor performance and see potential bottlenecks.

If you look at the statistics at the switch level on both your Gig-E switch and your FC switch, it might give you an idea of where a performance issue might be -- that is if the PPS on the FC net is low, and the pps or total throughput on the gig-e net is near max then maybe LAGing two e-net ports will provide some benefit in your current layout, etc.

Xsan too is somewhat forcused on throughput more than IOPs. All metadata (and journal) operations are funneled through a single machine (the MDC) over a private ethernet network. The metadata for each filesystem is stored on a particular set of LUNs, distinct from where the actual data is stored. This does give you the opportunity to use a small amount of very-high-performance storage (say a raid-10 of SSDs) to store the MD on , and commodity discs to store the data on, potentially increasing the number of IOPs. It will, though, have more latency for MD operations because of the whole contact-the-controller overhead.

There are, of course, various other optimizations which can be made to the MD system, but one would need to profile it in use with the target application to see what, if any, benefit could be obtained.

with Xsan 2 there is 'multi-san' which basically just means that you can host one or more volumes on a MD controller, where with xsan 1 you hosted _all_ volumes on a single MD controller -- the other MD conrtoller(s) were just for failover in case the primary went offline. So you could run, say 5 MD controllers and host 5 volumes.

If you care less about fail-over and more about performance, the MD LUNs could be direct-attache SSDs or even a memory-based-filesystem.

dbruhn · fully protected Joined: 23 Apr 2009 Posts: 12

The current Exanet system is 5 drive, 2tb luns. Each IBM DS4700 has three luns, with the 16th disk used as a spare. Each DS4700 has 5 EXP810 expander trays, which are set up in the same fashion. These 18 Luns make up a storage group. Exanet uses the storage group as a logical grouping of physical disk since they are connected with a single redundant controller. I am currently sharing the global file system out as a NFS mount and have AFP and CIFS turned off.

The arrays are all hooked up with dual path fiber all the way through to the Exanet front end nodes. From the nodes I have foundry x448's set up in layer 2 mode to the front end servers for my system.

In all the monitoring we have done we are not saturating the FC or the copper. I have changed every configuration in the system possible. The two people who held my position before me had the same problem (actually they created the problem) They had HDS in here with the Blue Arc guys trying to fix it. Blue Arc doubled our IOPS going from SATA to FCdisk but they were shooting for an 8/10 X improvement. Which they missed the mark on.

A big part of the reason I am looking at XSan/StoreNext is due to the pricing that it brings to the table. Apple sells XSan at $999 a pop for my system I would be looking at $12k StoreNext sells at $3k a pop so I would need about $36k to do what I need to with that. Exanet/NetApps/Blue Arc are all on per TB pricing models and the prices are insane in comparison. Exanet for instance for 60TB is $400k + NetApps charges as much for their standalone disk agnostic version as they do for their package deal with disk 50TB of that is almost $500K

So what I am really trying to do in the long run is offset the cost of these NAS packages, and fix the root of my problem which is IOPS. Here are some of the benchmarks that are driving my decision making.
• Reliable Operation - How long can I run without down time, does it compare to my current NAS solution.
• Price - Does the solution allow me to offset the price of the SSD
• Proof of concept - Can I prove this works
• How much time is going to be spent maintaining it?

To the MDC piece that you brought up. A NAS not only filters all of the metadata and journaling information through that controller but it also filter all of that data transfer through that controller. So my instincts are telling me that the MDC wont be my bottleneck.

The killers I have here are I need a 100TB+ volume that I can mount on my linux machines, and I need IOPS that are 4/5 time better than a SAS system with a NAS front end can give me.

I have been leaning to a pure StoreNext environment over the last week due to Quantums confidence in StoreNext accomplishing what I need it to accomplish.

This solution is looking better and better in contrast to what I currently have going on with my NAS systems. I really appreciate all of the feedback I have received from this forum.