Random Disk I/O Errors

[Skaffen]

Hi all,

Has anyone seen any problems with 10.6.2 and Xsan and seemingly random disk I/O errors? The setup is with a Promise Vtrack E-Class/J-Class setup and a Qlogic 5602 switch. Neither the switch nor the Promise show any errors in their event logs, but every now and then the Xserve just comes up with a "disk0 i/o error", with the actual disk in question rotating between any of the Promise LUNs.

There's no other errors in the Console at the time of these disk i/o errors, they just appear on their own, and obviously with no errors on the switch/RAID it doesn't look like the fabric itself is dropping at any point.

Thanks.

EDIT: Just realised, the only thing I changed this week that may have set these off was turning device scan off on the initiator ports. I'd left that on on all ports, but checking Apple's recommendations I turned it off. After further checking I think that I was probably right the first time with having that on, although I can't believe that having it off could actually cause this kind of issue?

The other thing is, is it worth doing a hard reset on the switch in case that it's got an issue? Clutching at straws a bit here, but all the usual suspects check out fine.

[Skaffen]

Well I turned Device Scan back on for the Initiator (Xserve's) ports and I've not had an I/O error for two days now. Not saying that's fixed it, should be able to confirm that tomorrow, but it's looking like an odd coincidence.

How on earth would that cause a problem? I/O Streamguard is manually set, so it's not like that is set to auto and causing problems with Device Scan turned off.

[Skaffen]

Sadly, they've come back, I'm getting these again:

[lotte]

You may want to check for FC cabling and /or the Gbics. Also have a look at the errors of the Lun´s Switch Port in the stats tab.

Does this appear on several Clients or just one?

Lotte

[Skaffen]

The cabling has been changed recently from Apple SFP copper cabling to Finisar transceivers, and the errors have remained the same, so I'm pretty sure it's not that.

There are a few errors listed in the port stats, none for AL Init Error or Flow Errors, but there are a few Decode Errors listed. Is there a level of acceptable errors for Decode Errors or should there basically be none listed?

[Skaffen]

Looks like it could be one of the controllers in the Vtrack failing silently. Noticed after the last round of problems that all the LUNs affected had their affinity set to Controller 2. Changed the affinities to point to Controller 1 for all LUNs and I've not had it crop up since. A little bit annoying that there's absolutely nothing in the logs on the Promise, but fingers crossed this has actually isolated the issue.

[lotte]

The latest firmware should have a fix for that, also make sure, both controllers are connected with Ethernet.


Lotte

[Skaffen]

Hi Lotte,

Both controllers are running the latest firmware (10.06.2270.00).

I'm not sure if the second controller has ethernet connected but the config hasn't changed for 4 months or so and this problem has only just cropped up over the past two weeks really. What would cause I/O problems if the management port on that controller was just unplugged?

Thanks.

[vicpache]

Skaffen,

The Vtrak E-Class WebPam Pro-E allows the user to save the Subsystem report.
This report contains a wealth of information that will help Promise Support staff understand the problem.

From the report we can check Fiber Channel front end ports statistics, internal components status and error counters for just about every component.
It also allow us to correlate which initiators are connected to what target port and correlate the LUN reporting the IO error to the LUN on the storage (very useful for large SAN deployments).
It offers way much more....

Please email me the following:

- http://kb.promise.com/KnowledgebaseArticle10095.aspx?Keywords=save+subsystem
- Primary and secondary MDC system profilers

See my contact information below. I am confident will get this straightened out.
--
Best Regards,


Victor Pacheco
Manager, Field Application Engineering & Support
Promise Technology
580 Cottonwood Drive
Milpitas, Ca 95035
Office (408) 228-1441
Mobil (408) 202-6808
Technical Support - (408) 228-1400
http://www.promise.com

[Skaffen]

Thanks Victor, have sent you a PM. Not sure if you can help directly as we're UK based. If you PM me back with an email address I can send any details over though if you think you can help.