"OpHangLimitSecs exceeded VOP-Setattr 183.18 secs"

[MaXi-XCeL]

Hi Guys,

My XSAN setup is driving me nuts... My MDC's keep crashing on the following error message:

[ACSA]

Hi can you give us more info than you have provided us sofar?


What OS are the MDC's running?
What version is the XSAN?
What is the firmware versions of the RAID's?
What versions are on the Clients?
Is the Meta Data lun seperate from the Data LUN?

etc

Then we perhaps now what to do.

[MaXi-XCeL]

MDC's: Mac OS X Server 10.4.10
XSAN: Version 1.4.1
XRAID FIRMWARE: 1.5/1.50f

Metadata is separate from the data luns.

XRAID 1:
RAID 1 Metadata LUN
RAID 5 Data LUN

XRAID 2:
RAID 5 Data LUN
RAID 5 Data LUN

XRAID 3:
RAID 5 Data LUN
RAID 5 Data LUN

StudioWorkDisc (13.65 TB)
-> metadatapool
- XSERVE1-LUN1-METADATA 465,73 GB
-> storagepool
- XSERVE1-LUN2-RAID5 2,73 TB
-> Any
- XSERVE2-LUN1-RAID5 2,73 TB
- XSERVE2-LUN2-RAID5 2,73 TB
-> Last
- XSERVE2-LUN1-RAID5 2,73 TB
- XSERVE2-LUN2-RAID5 2,73 TB

I know my choice of config and storagepools degrade the way XSAN should be used but that has some historical issues.

[donald]

From the log it seems you have trouble with metadata.

with cvlabel -l -s you can check if the lun for metadata is available.
from cvadmin you can check the storagepool (aka stripegroups) with the command show long.

[ipott]

Does the ophanglimit message appear BEFORE the fsm crashed message or after the fsm crashed message?

do you have filesharing active for the volume on more than one node?

[jordan]

Maxi,

have you found a fix for this one?

[BenT]

We have recently started seeing this same behaviour:
XSAN 1.4.1
25 FC-connected users
24TB volume approx 92% full
OSX 10.4.9
MDC is Intel XServe with 2GB RAM
All clients and servers dual-FC connected
FSM process on MDC sits at very high CPU usage, when it gets to 100% usage the FSM process dies with the error:

0x180ac00 (**FATAL**) PANIC: /Library/Filesystems/Xsan/bin/fsm "OpHangLimitSecs exceeded VOP-VopLookupV4 183.31 secs Conn[71] Thread-0x1881200 Pqueue-0x405018 Workp-0x959f618 MsgQ-0x959f608 Msg-0x959f664 now 0x43e4a1e3e8923 started 0x43e4a13517fd5 limit 180 secs." file queues.c, line 612

Anyone else found a solution to this problem?
We are going to increase MDC RAM, upgrade the 10.4.10 and XSAN 1.4.2 and then maybe try an MDC with faster CPUs

[drocamor]

Hey gang,

I'm working with romannumeral5 on his Xsan. We see the "OpHangLimitSecs exceeded" message after the san has already frozen for the users. (approximately 180 seconds later...)

We've escalated up through Applecare and they tell us that the message means that the MDC is unable to write to the metadata LUN for this period of time. When this happens fsm panics and attempts to shut itself down. In our situation the process does not exit and a manual failover must be forced.

We have seen some high latency on the fibre channel and a handful of IO errors in the system.log on the hosting MDC every once in a while. We've replaced the fiber cable, switch port, SFPs, RAID controller, and one disk in the metadata LUN while trying to fix this. Our next steps are to replace the second disk in the LUN and then move the disks and controllers to a new XServe RAID.

We upgraded to 1.4.2 in an attempt to resolve the issue but this just brought us to a situation where the manual failover does not work reliably. We were unable to fail the volume over when just the two MDCs were online and being unable to stop production outside of our downtime window we brought the volume back up without knowing if we would be able to failover. There was a panic yesterday and we were able to failover. I attribute this to bad luck when we were testing. We've got a ticket in Applecare open for this as well as one for the original panics.

BenT, I would hope that you're not having a similar experience to ours. Does your volume failover after the panic? I would check all the fiber connections look for errors on the Xserve RAID that hosts your metadata LUN. You should also probably have more than 2 GB of RAM in your MDC because of the size of your volume. Your volume is also mad full. In my experience this just makes any Xsan problem worse.

If anyone has any other ideas about things to try please let me know.

Thanks,
Dave

[ipott]

we have the same issue.. fsm is going up to 350% CPU and then crashing.
The failover hangs most of the time. Sometimes we have to failover several times.

We opened a ticket at applecare 6!!! months ago. Till now they did not come up with any solution, telling us we are the only customer with this kind of problem.

We will throw the XSAN out in the next weeks.

[ipott]

our MDC has 4 cores. 16 Gb RAM and the filesystem is filled up to 75%.
10.4.11 and XSAN 1.4.2

[drocamor]

ipott,

Sorry to hear you are having issues. How many clients do you have? What are they doing? How large is your volume?

[ipott]

Hi,

the volume is 17 TB. 5 Xsan clients and smb + afp filesharing over 2 of the clients for the renderfarm.

We tried a lot for solving the issue, but currently even a simple "find" command crashes the volume after some minutes.

[drocamor]

What kind of renderfarm software is accessing the Xsan over the AFP and SMB share?

I wonder if that could be an issue, we are resharing to several machines that are transcoders and others that do some heavy searching and indexing of certain folders.

Is everyone else having this problem also resharing the volume over AFP, SMB, or NFS?

[ipott]

it's a royal render renderfarm. with about 25 nodes. but I think that is not the problem, because even if filesharing is turned off, I can crash the volume by typing a simple "find" command. Or by doing an snfsdefrag -r on the volume.

we are currently thinking about tuning Volume parameters like buffercache and inodecachesize, threadpoolsize. ADICs documentation says these parameters are increasing metadata performance...

we also have had 3 weeks without crash some month ago and people were rendering a lot on the volume. That was after we installed more memory.

we are trying some things at this stage and I will post the results here.

[drocamor]

To help me track down my own problems and maybe help you out too, do you think you could send me some of the system.logs, cvlogs, and volume configuration for your xsan?

I'm looking for possible issues with my volume involving latency and disk io. If you could send me those files I'd like to compare them to my own. Would that be alright? Maybe we can help each other.

Thanks,
Dave