Isilon Ingest Problem

[undercover]

Ok, I know this is an XSAN forum and this is an isilon question, but maybe you guys can help me out (there is an XSAN in the equation, but it's not the problem).

We're doing HD video ingest using a Kona card in an XServe directly to an NFS share on our Isilon. When we open the video, the aspect ratio is off (it somehow gets set to some weird resolution like 939x636 with a different pixel width). To make things weirder, when we ingest directly to the XServe raid, the video is just fine! If we ingest from XServe and copy across, it is also fine. A further test of ingesting to an NFS share on another XServe was ok as well. It's only when we ingest directly to the Isilon that our video gets whacked. The video itself plays fine, it's just in a weird format, which makes me think it's a metadata problem...

Ideas?

[JesusAli]

This sounds like something to bring to the attention of Kona.

But I agree that MetaData flags in the video could be to blame. Without scouring bit by bit through the hex code, a good place to start may be opening up both clips in Quicktime Pro with the "Get Info" Heads Up Display activated (CMD+I).

To dig deeper, press CMD+J on each clip and examine how video tracks are being written and defined.

This info could be passed along to Kona for analysis. They are pretty interested in solving technical problems for customers.

Good luck and if you can figure it out, please let us know!

[undercover]

Update:

It works fine via SMB to Isilon. Something is wrong with the way it handles the file using NFS in Isilon (it worked fine via NFS to a mac).

Spent some time trying to figure out if the ._ files were the problem, didn't really come to a conclusion. Is there a way to disable ._ writes in 10.6.2? I can only find instructions for 10.5...

Learning a bunch about Isilon... week 2 as interim SysAdmin.

[undercover]

I'll post the file info on the distorted files tomorrow... it's significantly different when using NFS vs SMB with all the settings the same...

[d4corp]

Hi undercover,

I've had fun issues with Isilon… One of them was that TimeCode track wasn't added to QT file ingested by FCP ONLY when recording on the Isilon with NFS.

In fact the answer is pretty obvious. All the DNS, NTP and Open Directory recommendations for Xsan applies to Isilon. I would buy a Mac mini and setup a proper server to deal with this.

Otherwise, just make sure that your isilon export is setup the right way. Final Cut Pro handles NFS differently that what you could expect. Double-check the authentifications. I solved all my issues by making sure that every user has the same ID (501) and that the id/gid on the nfs share is right.

Don't bother with Kona, I'm almost sure this has nothing to do with it.

[undercover]

Good tips -- But I'm not sure how dns, ntp, or OD would affect the metadata from a capture. OD maybe, with the uid/gid mappings, but still, how would that affect metadata? If I can read/write, what more permission/authentication do I need to bother with?

[undercover]

NFS Mount Script on the Client:
mount -t nfs -o rw,hard,tcp,vers=3,nolock 10.100.7.61 :ifs/Isilon_Production_Live_Ingest/Ingest_01 /Volumes/Ingest_01

QT Get Info:
Format DVCPRO HD 1080i60, 1280 x 1080 (1888 x 1062)... Current Size: 906x339 pixels (Actual)

FCPro Format:
Frame Size 614 x 345
Pixel Aspect Square
Field Dominance None

QT Get Info on correct file:
Format DVCPRO HD 1080i60, 1280x1080 (1888 x 1062)... Current Size: 1024x576

FCPro format on correct file:
Frame Size 1280 x 1080
Pixel Aspect HD (1280x1080)
Field Dominance: Uppder (Odd)

So you see they are significantly different...

We did a live test via the SMB share last night with a single HD source into Isilon... dropped a few hundred frames over the course of 4 hours, but the video is in the right format... progress I guess.

[bildon]

I had an issue where Audio and Video were becoming unsyncronized and followed a Isilon FCP technical bet practices whitepaper to solve the issue.

Here is the section on Client tuning on a Mac.

The following recommendations apply to changes on the Mac client, unless otherwise noted. Client Network Tuning
The following settings affect the behavior of the Mac OS network stack whether running SMB or NFS protocols:

1. 9000 MTU: Increasing network MTU to 9000 (Jumbo Frames) reduces the overhead associated with IP packet fragmentation. This requires network cards and switches that support 9000 MTU setting.

NOTE: the Isilon OneFS network profile will need to be set to 9000 MTU as well. This should be done during a maintenance window since the cluster will need to be rebooted for this setting to take place.
To check if the Macintosh can handle 9000 MTU go to the System Preferences under utilities, select Network, then select the Ethernet option and check if the system is capable of 9000 MTU. If 9000 MTU is grayed out, a third party network card that can support 9000 MTU needs to be installed. A suggested vendor for these network cards would be Small Tree systems (http://www.small-tree.com/mp_all_copper.html).

2. Delayed Acks: The Mac OS 10 network stack uses the ‘delayed acks’ feature to send TCP acks only when every other TCP packet is received, or after a certain timeout has elapsed if no additional data was received. The motivation is to reduce the number of TCP acks traversing the network, but this behavior is incompatible with Isilon OneFS network stack that uses Nagle’s algorithm to optimize TCP traffic by delaying the sending of TCP data until a full TCP packet assembled or all other outgoing packets have been acknowledged by the client. As a result network traffic may stall, while each side is waiting for a signal from the other, until the timeout period has elapsed. To avoid these deadlock conditions, Mac OS clients can be configured to not use the ‘delayed ack’ feature. To cancel ‘delayed ack’, edit the /etc/sysctl.conf file adding the following line:

net.inet.tcp.delayed_ack=0

3. TCP Memory Buffers: It is possible that TCP memory buffer settings on the Mac clients are adding network latency. Increasing these settings will help as long as there are no lost packets, otherwise significant latency problems may arise. A reasonable value for TCP buffer size setting is somewhere around the amount of port buffering on the intermediate switches and cluster. This setting can go up to 1MB on both the cluster and the Mac client, if the intermediate switches support it, or 32k or so for lower end switches. On the Isilon cluster the default setting is 128k for
both send and receive buffers. It may help to start off by changing those setting on the Mac client to the same value:

# sysctl -w net.inet.tcp.recvspace=131072
# sysctl -w net.inet.tcp.sendspace=131072

To make this setting permanent, add everything after the "-w" to /etc/sysctl-macosxserver.conf (XServe) or /etc/sysctl.conf.

Client NFS Performance Optimizations
The following recommendations apply to specifically to NFS:

1. NFS Mounts: Avoid using the Mac Finder to mount NFS exports. The finder adds another layer of overhead. Instead use a Mac terminal session to create mount points manually. Use the following mount command to mount ‘/ifs’ or any other NFS export.

mount_nfs -3 -T -l -r 32768 -w 32768 $ip_address:/ifs /Volumes/Isilon

The cluster may also be mounted via Automounter to reconnect to the cluster automatically after a reboot. Here is a link to a page that describes how to use the Automounter:

http://mactechnotes.blogspot.com/2005/08/mac-os-x-as-nfs-client_31.html

2. NFSIOD Threads: Another thing that can affect MacOS NFS performance is the number of nfsiod threads. These are user space threads that provide the interface between the NFS kernel module and the user space application sending and receiving data over NFS. Adding more threads means that more NFS transactions can execute concurrently (including read-ahead and write-behind operations). To increase this value, edit the file /System -> Library -> StartupItems -> NFS -> NFS, and change the line that reads:

nfsiod -n 4. Replace the 4 with 8 or more, followed by manually killing and re-running nfsiod by running:

‘killall nfsiod; nfsiod –n 8’

[d4corp]

I was assuming that you were already following Isilon osx best practices. This will fine tune your system. I didn't see a difference after setting the sysctl, but they must have good reasons to recommend them

About the NTP, DNS, OD…, they are not directly related to your problem. But they have to be there.

About your problem, now…

You're right, it seems that if you can read/write, there'll be no problem. But for a reason that I don't understand, Final Cut Pro deals the metadata differently… 

In my case there was no TimeCode track written at all, after putting the right uid/gid (try with a new user, it will take you no more than 15mn), everything was back to normal. I didn't setup an OD as there's only two workstations working on the Isilon. Having the UID+GID right should be enough. Consider as well removing ACL's to test.

Other stuff to watch out :

1. Disable DS_Store
defaults write com.apple.desktopservices DSDontWriteNetworkStores true
sudo cp ~/Library/Preferences/com.apple.desktopservices.plist /Library/Preferences

> http://support.apple.com/kb/HT1629
> http://www.macosxhints.com/article.php?story=2005070300463515
> http://support.apple.com/kb/TA21373 ("Get Info" comments not propaged to share)

2. remove .DS_Store & .Trashes

First check:
find "$@" \( -name ".DS_Store" -or -name ".Trashes" -or -name "._*" \) -exec echo "{}"

Then do :
find "$@" \( -name ".DS_Store" -or -name ".Trashes" -or -name "._*" \) -exec rm -rf "{}" \; -prune

3. Disable ACLs if not needed
sudo fsaclctl -p /Volumes/ifs2/ifs -d

> man fsaclctl

4. Make sure your export is in streaming mode


[b]5. Check your 10GbE switch config[/5]

Ex. for my Switch Fujitsu 10G XG700-CX4
http://www.google.com/url?q=http%3A%2F%2Fwww.fujitsu.com%2Fdownloads%2FCOMP%2Fffna%2Fethernet%2Fxg700_user-guide.pdf&sa=D&sntz=1&usg=AFrqEzdkzprWA6dMifLHm32ms3rYDNPJAw [PDF]

[undercover]

I'll be looking into all that in detail early next week, thank you for the instruction. As I wasn't the one who originally implemented this system, it will be a good opportunity for me to familiarize myself with it. I believe most of this was done (It is set to MTU 9000 for example), but some of it has not.

Also, we have hours of footage from a recent event that were recorded into this bogus format. Any tips on salvaging it? Assuming it's a metadata problem, is there any way to edit the metadata?

[d4corp]

I'm not saying your issue will be resolved this way, but there's great chances.

What was happening is that final cut pro was writing the file with osx uid/gid, then the isilon changed it to nobody:nobody, and final cut pro got confused with it while inserting metadata back in the file.

[undercover]

That sounds very similar to what we have going on, so I am hopeful. Also just to clarify, we are not capturing with FCP, we are using some AJA utility. We are opening it in either QT or FCP. I attempted to open one of the files using VLC, and it appeared to be displaying correctly... So that gives me hope that we might be able to recover the footage.

I'll let you guys know what I find out. Trying hard not to work on the weekend

[d4corp]

Hehe, not working on weekends is too hard for me

Just send me your email so I can send you back isilon's best practices PDF.

[undercover]

Slowly grinding through this, a couple questions/comments:

In snow leopard to turn of .ds_store the file is com.apple.com.desktopservices

Number 2. up there is different for us. The find command doesn't work quite as you posted it. I have basically recreated the first one:

find -f /Volumes/Ingest_01/ "$@" \( -name ".DS_Store" -or -name ".Trashes" -or -name "._*" \)

Can I somehow pipe that into rm -rf? Or is there a better way to do that?

Also I notice that the guide suggests MTU 8244, is there a reason you run MTU 9000? Also your switch is set to 9216, is there a reason for that? We haven't gotten to the switch yet...

[undercover]

Current testing:
With your mount script, DS_Store turned off, MTU set correctly on the client, delayed acks off, still getting the problem.

Tomorrow we will look at the uid/gid (my hunch) and the switch MTU etc.

Questions:
1. What do you mean by making sure the export is in streaming mode? Is that for FCP? We're capturing using AJA, but in our previous testing with FCP we had the same issue. *EDIT* Found this on Isilon, NVM.

2. NFSIOD -- I didn't find that file, is it somewhere else in SL?

*EDIT* -->

UID/GID hunting:

Client UID:
uid=505(vtop) gid=20(staff) groups=20(staff),404(com.apple.sharepoint.group.3),402(com.apple.sharepoint.group.1),61(localaccounts),12(everyone),403(com.apple.sharepoint.group.2)

Isilon UID:
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

Do I just change one of those to be the same as the other? Which would you recommend changing?