User Functions
Don't have an account yet? Sign up as a New User
Who's Online
Guest Users: 11
|
| View previous topic :: View next topic |
| Author |
Message |
JesusAli
Xsan Master
Joined: 25 Jul 2008
Posts: 151
|
 Posted: Wed Jul 22, 2009 1:30 pm Post subject: VTrak E+J - 200/332 MB/second Read/Write - Public Metadata |
 |
|
Hey Everyone,
What do you think of 200MB/s Write and 332MB/s Read for an E+J setup? Do you think I can do better?
ALL the dirty details are here below. Please let me know any Command Line Programs or other techniques I can try to test and improve things.
• dedicated MDN is on the way,
• is the "dd" command line program appropriate to test with?
* * *
I am working through things with the IT department at the place I work at and am awaiting the installation of a Cisco 1000BT switch for Metadata, but in the meantime, I have installed the Xsan using the 10/100BT public network for Metadata.
We are running two ethernet connections from each MDC and Client.
All IP's are static.
i.e. eth00 Public = 172.x.x.50, eth01 MD = 172.x.x.51, ect.
All Macs have two 4GB Fiber connections (4 strands total).
Vtrak E controller has latest firmware and all four FC ports connected with 4GB copper Apple cables.
All plugged into a SanBox 5600 and showing up as 4GB.
Bullet Proof Zoning and RSCN suppression is on, Device scan on (based on recent discussion that it doesn't hurt.)
VTrak E+J configured using Apple KB / Promise recommended configuration script:
http://support.apple.com/kb/HT1162
All disks are 750GB SATA.
DATA 1 through DATA 4 are 6 disk RAID 5's grouped into the Main VIDEO Pool.
In Xsan Admin I setup two Folders in the Root of the Xsan Volume:
"Capture and Footage" - Affinity Tagged to the Data1 through Data4 VIDEO Pool.
"Not Video" - Affinity Tagged to the SCRATCH Pool.
I believe I left the Xsan Volume Block Allocation Size at 16KB and Round Robin.
But I set the Stripe Breadth for the VIDEO Pool to 256K
I had previously tested the default (32k?), 64k and decided to skip 128k for now. But for those previous tests, I had only used AJA System Test and the old Xsan Tuner.
But yesterday, I saw Xsanity Forum member Lotte's post containing the following Command Line Program:
| Code: | dd if=/dev/zero of=/PATH/TO/YOUR/VOLUME/testfile bs=8192k count=1000
(sends 8MB 1000 times to make an 8GB file)
dd of=/dev/null if=/PATH/TO/YOUR/VOLUME/testfile bs=8192k count=1000
(reads back that file, but only after you write it) |
So today I tested my Volume using those commands.
Running the programs from my Mac Pro Client02 produced:
113436622 bytes/sec (108 MegaBytes/sec) when Writing
to the Root level of the Xsan Volume.
159238744 bytes/sec (151 MegaBytes/sec) when Reading back.
210502792 bytes/sec (200 MB/s) when Writing
to the Affinity'ed "Capture and Footage" folder.
349120310 bytes/sec (332 MB/s) when Reading back.
and
114347325 bytes/sec (109 MB/s) when Writing
to the Affinity'ed "Not Video" folder.
159348495 bytes/sec (151 MB/s) when Reading back.
* * *
Based on these numbers, I think the root level of the disk and the SCRATCH Pool are the same place. Hmm...
Do Affinity Tags PREVENT non tagged data from being written to a LUN in a Tagged Pool or do to they ONLY ENSURE that Tagged Data is Routed to the matching LUN? Hmm... |
|
| Back to top |
|
 |
JesusAli
Xsan Master
Joined: 25 Jul 2008
Posts: 151
|
| Posted: Wed Jul 22, 2009 4:23 pm Post subject: |
|
|
I started thinking about making the script more real world, so I changed the numbers to represent Writing and Reading an hour of ProRes422 HQ (10-bit) footage. I think I got the math right...
27.5MB/s = 27.5*1024k = 28160k
Since that's the rate per second, I set it to write that value 60 times to get a minute and then multiplied it by another 60 (3600 total) to get an hour. This made a 97GB file which seems about right.
Here's the code for 1 Hour of HD ProRes422 10-bit:
| Code: | | dd if=/dev/zero of=/Volumes/256k_SAN/Capture\ and\ Footage/testfile_HourofProRes bs=28160k count=3600 |
My Results:
Terminal reported "3600+0 Records In and 3600+0 Records Out" (so no dropped frames?)
245031040 bytes/sec (233 MB/s) when Writing
to the Affinity'ed "Capture and Footage" folder.
362479467 bytes/sec (345 MB/s) when Reading back.
So then I tried to simulate Writing and Reading back 8 minutes of 3 simultaneous streams of HD ProRes422 10-bit.
(I know in normal Capture you'd only ever Write 1 stream at a time, but I had to put the file on the SAN before I could Read it off, so I just included the Write times...)
| Code: | dd if=/dev/zero of=/Volumes/256k_SAN/Capture\ and\ Footage/testfile_8Minutesof3ProResStreams bs=84480k count=480
and
dd of=/dev/null if=/Volumes/256k_SAN/Capture\ and\ Footage/testfile_8Minutesof3ProResStreams bs=84480k count=480 |
248431591 bytes/sec (236 MB/s) when Writing
to the Affinity'ed "Capture and Footage" folder.
366458650 bytes/sec (349 MB/s) when Reading back.
So I guess my revised numbers are 236MB Write and 349MB Read. |
|
| Back to top |
|
|
MattG
Xsan Master
Joined: 15 Apr 2005
Posts: 456
|
| Posted: Wed Jul 22, 2009 6:42 pm Post subject: |
|
|
I'm wondering why you were using those Stripe Breadths. The formula is:
Block Size X Stripe Breadth X # of LUNs in the Storage Pool = 1MB
With a Block Size of 16K, VIDEO Storage Pool should have a Stripe Breadth of 16, and the SCRATCH Storage Pool should have a Stripe Breadth of 64.
Next, like many other who have posted about not finding good performance from their system, the tests you are doing to the system are not real world. They provide a raw I/O perspective, but don't factor in the data allocation and buffering that occur during I/O operations more suited to video ingest and playback. This is what the AJA System Test is for. Alas, the AJA System test cannot be directed to a specific folder. So, you're better off not assigning an affinity to the VIDEO Storage Pool so that you can run the AJA tool. Then, make sure to use the maximum setting on the AJA tool to simulate the maximum stream of data. This will provide a more realistic test.
Lastly, the performance issues surrounding a single E/J combo, with metadata traffic co-mingling with regular data I/O, will always take its toll on your I/O.
With that in mind, these numbers are not too shabby. Are you sure that all the LUNs are finished building? |
|
| Back to top |
|
|
JesusAli
Xsan Master
Joined: 25 Jul 2008
Posts: 151
|
| Posted: Wed Jul 22, 2009 8:57 pm Post subject: |
|
|
| Quote: | MattG said:
I'm wondering why you were using those Stripe Breadths (SBs).
The formula is: Block Size X Stripe Breadth X # of LUNs in the Storage Pool = 1MB |
I went over 1MB on purpose. I had read some entries here about people experimenting with larger SBs when dealing with HD Video. I mean, aren't small SBs best suited to small files (metadata)? Isn't there a benefit to reading/writing larger chunks of Video data at a time?
Regardless, I'm not sure I was factoring in the # of LUNs, I thought it was simply Stripe Breadth x Block Size that was supposed to equal 1MB.
But let me work through your numbers...
1MB = 1,048,576 bytes,
1048576 bytes / 4 LUNs in the VIDEO Pool = 262144 bytes (256k),
256k/16 = 16k
Yup. And since there is only 1 LUN in the SCRATCH Pool its SB should be 64k. Yup, that definitely matches your numbers.
Since the Metadata Pool is also only 1 LUN, should it ALSO be set to a 64k SB? It seems counter-intuitive that the MD and SCRATCH would benefit from the same characteristics. 
In terms of Final Cut Pro workflows, do you use the 1 LUN SCRATCH Pool for the CAPTURE SCRATCH folder?
I was planning to provide a Folder labeled "NOT VIDEO" Affinity'ed to the SCRATCH Pool (3 disk LUN) and recommend that students use it for their "Support" files like photographs, Photoshop Comps and maybe even Audio files. What do you think about that? |
|
| Back to top |
|
|
JesusAli
Xsan Master
Joined: 25 Jul 2008
Posts: 151
|
| Posted: Wed Jul 22, 2009 9:47 pm Post subject: |
|
|
| Quote: | MattG said:
...these numbers are not too shabby. Are you sure that all the LUNs are finished building? |
99% sure. According to these Comments I left, I ran the current Promise Configs 3 weeks ago.
But I did start working with the disks immediately. I remember some status messages some days later in the Promise WebPAM PROe that I think said something about the RAID status being "cleared" or "finished".
Is there a place I can check now where it says "Finished" or is the absence of an "In Progress" Status Message the only indication?
* * *
What kind of speed increases does anybody think will be evidenced when we move the Metadata network to its own 1000BT switch? (right now MD is on the same network as everything in the school and only connected at 100BT.
Anybody have any numbers they wanna guess?
| Quote: | MattG said:
Lastly, the performance issues surrounding a single E/J combo, with metadata traffic co-mingling with regular data I/O, will always take its toll on your I/O. |
We were originally going to implement the Xsan with 3 Apple Xserve RAIDs. I might be able to ask real sweetly and get one back.
Do you think it would integrate well with the Promise E+J?
Can the Metadata be running at 2GB/s when all the Video Data is running at 4GB/s?
— How would you populate the 14 drive Apple RAID for Metadata?
-- Still only use 3 disks on one side? (It might help if I can tell IT they still have 7 disks in the Lower Controller to work with.)
-- If I used 3 disks in the Upper Controller for Metadata, would unrelated activity on the lower controller slow down the Metadata I/O?
-- If I only used 1 Controller, would I only use 1 Fiber connection? Or can the RAID send Upper Controller Data out of both Fiber Ports?
-- OR, is the recommendation to span the MD LUN across the Controllers?
This is really juicy stuff. I find it very interesting. Thanks MattG and thanks for anyone with further insight and input. |
|
| Back to top |
|
|
addihetja
Xsan Master
Joined: 20 Dec 2006
Posts: 109
|
| Posted: Thu Jul 23, 2009 3:48 am Post subject: |
|
|
| JesusAli wrote: |
Can the Metadata be running at 2GB/s when all the Video Data is running at 4GB/s? |
Sure
| Quote: |
— How would you populate the 14 drive Apple RAID for Metadata?
-- Still only use 3 disks on one side? (It might help if I can tell IT they still have 7 disks in the Lower Controller to work with.)
|
Yes. MD performs better with a two drive RAID1 than with a multiple drive RAID5.
| Quote: |
-- If I used 3 disks in the Upper Controller for Metadata, would unrelated activity on the lower controller slow down the Metadata I/O? |
No. The controllers are totally separate from each other. |
|
| Back to top |
|
|
staze
Been around the blocks
Joined: 15 Oct 2007
Posts: 25
|
| Posted: Thu Jul 23, 2009 5:28 pm Post subject: |
|
|
| JesusAli wrote: | | What kind of speed increases does anybody think will be evidenced when we move the Metadata network to its own 1000BT switch? (right now MD is on the same network as everything in the school and only connected at 100BT. |
My impression of how MD works is that you probably won't see much in the way of improvement with the tests you're doing because the MD is only going to be queried on initial write/read operations (where to put the file, or where a file is located). Since you're doing big single transfers, that's not a lot of MD traffic. If you were writing/reading a directory of a few thousand/million files, you'd see a huge difference since gigabit has a much lower latency than 100mb.
All that said, you commented that IT will be putting in a cisco switch for the MD network. Something that sophisticated really isn't needed, or in some cases, wanted. The dumber the switch the less it looks at the packets, the less latency is introduced. I personally use, and I know others do as well, a dumb 24 port Netgear switch (I believe a JGS524), that works just fine. The only problem I have with them is no redundant power, but given the price, you could buy a dozen of these switches for the price of 1 Cisco. =/ |
|
| Back to top |
|
|
addihetja
Xsan Master
Joined: 20 Dec 2006
Posts: 109
|
| Posted: Thu Jul 23, 2009 6:10 pm Post subject: |
|
|
| I concur. I had a chat with one forum member at the Xsanity meeting at WWDC '08 and he told me that he had to create very special "Xsan switch config" for Cisco switches at companies that "only use Cisco" because the default config of the switch introduces a lot of latency a Cnet switch doesn't |
|
| Back to top |
|
|
JesusAli
Xsan Master
Joined: 25 Jul 2008
Posts: 151
|
| Posted: Fri Jul 24, 2009 6:00 pm Post subject: |
|
|
ugh.
@ addihetja: do you happen to remember which Forum Member this was?
I remember speaking with my IT Director on a separate matter, some time ago, and he mentioned something along the lines that we "are a Cisco Only" house. |
|
| Back to top |
|
|
abstractrude
Xsan Master
Joined: 13 Mar 2008
Posts: 865
|
| Posted: Sat Jul 25, 2009 5:31 pm Post subject: |
|
|
whats wrong with cisco or managed switches. the problem is people dont know what the hell they are doing, thats all. latency is key here. i dont know how it happened but somewhere in xsan history people started thinking managed switches are bad for metadata networks. its ok they run 99% of the internet including forture 500 networks, but get to xsan metadata nope not good.
same happened with device scan on qlogic stuff! no evidence or information to back it up, just say it because well just because ?
| Quote: | | The dumber the switch the less it looks at the packets, the less latency is introduced. |
really? what does this even mean? |
|
| Back to top |
|
|
addihetja
Xsan Master
Joined: 20 Dec 2006
Posts: 109
|
| Posted: Sun Jul 26, 2009 9:15 am Post subject: |
|
|
I don't recall his name but I think he was from The Netherlands. He told me he had the numbers to prove it and his argument made good sense.
Managed switches have a lot of _throughput_ if they have a big backplane throughput is not really something we're concerned with in Xsan. We're looking for low latency. Latency is governed by the wire speed and what the switch does with the data. If it does a lot of work with the data (routing being a good example) the latency _will_ go up.
His argument is that you _can_ use managed switches but you should strip the config of stuff you don't need for Xsan (VLAN's f.x.). He decided that he'd rather buy two for 1/10th the price.
The "qlogic stuff" came from Apple. We only propagated the misinformation. |
|
| Back to top |
|
|
abstractrude
Xsan Master
Joined: 13 Mar 2008
Posts: 865
|
| Posted: Sun Jul 26, 2009 1:00 pm Post subject: |
|
|
guess where the "dumb" ethernet switch stuff came from... apple. I have the slides from WWDC to prove it.
the fact is, cisco and other high end network products can be configured to do many things, there is much confusion about what these configurations do. for example, vlans. remember everything on every cisco switch ever is in a vlan. that being said you can use cisco just fine. Also the cisco catalyst switches,especially the chasis based switches will blow any 70 dollar switch out of the water. to the parent, get a cisco guy into your shop and figure it out. use cvsummary to figure out what your latency times are.
the fact is, if you are in a large environment your not going to put business critical infrastructure on switches that people have in their homes. small xsan's you should use small switches. but if someone is in a cisco shop running enterprise hardware, you shouldn't say dont use cisco. remember datacenters that need even lower latency because of failover and database transactions run cisco.
the fact is, you can actually use your cisco infrastructure to produce higher speeds and lower latency because you can use the switch to block certain traffic and use QoS to make sure metadata is top priority. I have used QoS to move metadata between routers. works no problem.
this should be what apple says "use a dumb switch it will make life easier, make sure you have a professional implement xsan metadata networks when using enterprise hardware."
i respect your opinion, but i promise you I have seen xsan work on complicated enterprise networks on catalyst 6500 hardware with clients all over the facility.
http://www.cisco.com/en/US/products/hw/switches/ps708/
Last edited by abstractrude on Sun Jul 26, 2009 1:10 pm; edited 1 time in total |
|
| Back to top |
|
|
addihetja
Xsan Master
Joined: 20 Dec 2006
Posts: 109
|
| Posted: Sun Jul 26, 2009 1:06 pm Post subject: |
|
|
| abstractrude wrote: |
the fact is, you can actually use your cisco infrastructure to produce higher speeds and lower latency because you can use the switch to block certain traffic and use QoS to make sure metadata is top priority. I have used QoS to move metadata between routers. works no problem.
|
And I have seen "Cisco shops" have no end of trouble with exactly that thing because the didn't know what they were doing. In the end a dumb switch solved their problem. |
|
| Back to top |
|
|
ACSA
Xsan Master
Joined: 28 Jan 2007
Posts: 104
|
| Posted: Sun Jul 26, 2009 1:07 pm Post subject: |
|
|
| abstractrude wrote: | | whats wrong with cisco or managed switches. the problem is people dont know what the hell they are doing, thats all. latency is key here. i dont know how it happened but somewhere in xsan history people started thinking managed switches are bad for metadata networks. its ok they run 99% of the internet including forture 500 networks, but get to xsan metadata nope not good. |
Nothing is wrong with using managed switches.
But if the metadata network is connected to the core switch of your LAN network. And you already have several VLAN's active, i.e.: A server and printer VLAN, a couple of VLAN's for the clients at several locations.
Like on a campus network.
Had the experience to see a metadata LAN which was in reality a separate VLAN where the clients and servers where in fact connected to several different physical switches which in turn where connected via one r more 1 GB ethernet uplink cable.
The number of VLAN's being trunked via those uplink was on average between the 4 and 14.....
So you can see that any metadata request that was sent via this VLAN, had to share with a number of other VLAN's. So the latency went up big time...
A simple ping test proved us that a lot of packets where dropped. Roughly around 20 to 25% of the packets where lost.... And on a metadata network... not really something you would like to see...
Using a VLAN for the metadata is going to give you an increase in latency. An sich is the increase in latency on your campus LAN a problem. since if a packet is dropped for whatever reason, it will be sent again. And in a "normal"
So using a managed switch is not a problem, getting the right configuration for that switch is something else....
So the easiest solution is getting the "dummest" switch you could get... And that is probably the reason why a lot of people go for the unmanaged switch.
Knowing what a network is, and how it works are two separate things.
In my experience I've seen a number of times several high experienced Mac and Windows system engineers who do not even know what a VLAN is. But they are responsible for the network..... 
@ JesusAli & addihetja: I'm probably the guy... I do remember talking to another forum member, also forgot his name of handle, regarding this issue @WWDC'08
Greetings from the Netherlands.
Arnold |
|
| Back to top |
|
|
ACSA
Xsan Master
Joined: 28 Jan 2007
Posts: 104
|
| Posted: Sun Jul 26, 2009 1:25 pm Post subject: |
|
|
| abstractrude wrote: | guess where the "dumb" ethernet switch stuff came from... apple. I have the slides from WWDC to prove it.
the fact is, you can actually use your cisco infrastructure to produce higher speeds and lower latency because you can use the switch to block certain traffic and use QoS to make sure metadata is top priority. I have used QoS to move metadata between routers. works no problem.
this should be what apple says "use a dumb switch it will make life easier, make sure you have a professional implement xsan metadata networks when using enterprise hardware."
i respect your opinion, but i promise you I have seen xsan work on complicated enterprise networks on catalyst 6500 hardware with clients all over the facility.
http://www.cisco.com/en/US/products/hw/switches/ps708/ |
Yeah, I know what you mean. And I concur. And QoS is absolutly the solution... But we see more and more integration of VOIP on the same network. And you have to decide which service take priority... And on a integrated Voice and Data network, I can tell you which service will take priority.... Voice..... Since it effects all the users.... ( dropped packets means dropped speech in conversations..... )
And not every IT manager can afford an 6500...... 
greetings
Arnold |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
Xsanity Forum
Forum Search
Forum Help
Log in to check messages
Xsanity Articles
