Panic in XSan and Clients

[rogerbg]

Hello,

I posted this earlier this week in someone else's similar thread and received no response. So, I thought I'd move it out to its own thread...

Our Hardware:
1 XServe 10.4.11 performing DHCP/DNS
1 XServe 10.4.11 performing MDC
1 XRaid 3 arrays using raid 5: 1.3TB/MDJ 2.04TB/User Data 4.09TB/User Data
1 Qlogic Fiber Switch (16 port)
1 Linksys Ethernet Switch (16 port)

We are able to get everything up and running. The issue is the stability of the XSan itself. Every 3 or 4 hours, it stalls. When it stalls, we have to restart the clients and the MDC SXerve. In fact, the clients display the 4 language "you must restart this machine" message. After restarting everything, the XSan is perfectly fine. I don't have to remount anything, re-authenticate anything...it's like nothing happened. Everything is fine for the next 3 to 4 hours and it happens again.

During a stall I've tried accessing the XSan via cvadmin, but it won't let me select the proper FSS. It times out. After typing select, cvadmin responds with Tap timed out...

Here's a portion of the XSan log when a stall happens:
Nov 26 11:46:09 server2 kernel[0]: FusionMPT: Notification = 1 (Log Data) for SCSI Domain = 2
Nov 26 11:46:09 server2 kernel[0]: FusionMPT: No log data type available, data = 0x07610400
Nov 26 15:09:41 server2 fsm[289]: Xsan FSS 'R1V1[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm "OpHangLimitSecs exceeded VOP-Setattr 183.34 secs Conn[6] Thread-0x186e600 Pqueue-0x4049d8 Workp-0x188bc18 MsgQ-0x188bc08 Msg-0x188bc64 now 0x43fda8b74223c started 0x43fda808692f7 limit 180 secs. " file queues.c, line 612
Nov 26 15:09:41 server2 fsm[289]: PANIC: /Library/Filesystems/Xsan/bin/fsm "OpHangLimitSecs exceeded VOP-Setattr 183.34 secs Conn[6] Thread-0x186e600 Pqueue-0x4049d8 Workp-0x188bc18 MsgQ-0x188bc08 Msg-0x188bc64 now 0x43fda8b74223c started 0x43fda808692f7 limit 180 secs.\n" file queues.c, line 612\n
Nov 26 15:09:41 server2 fsm[289]: Xsan FSS 'R1V1[0]': PANIC: aborting threads now.
Nov 26 15:09:50 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:09:50 server2 servermgrd: xsan: [45/3ADA70] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:09:50 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_clients_for_fsmvol_named(R1V1): Could not connect: Connect to FSM failed - Connection refused
Nov 26 15:09:50 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_computer_properties(R1V1): No client list for volume
Nov 26 15:10:51 server2 servermgrd: xsan: [45/3BD1B0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:10:51 server2 servermgrd: xsan: [45/3A0860] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:10:51 server2 servermgrd: xsan: [45/3BD1B0] ERROR: get_clients_for_fsmvol_named(R1V1): Could not connect: Connect to FSM failed - Connection refused
Nov 26 15:10:51 server2 servermgrd: xsan: [45/3BD1B0] ERROR: get_computer_properties(R1V1): No client list for volume
Nov 26 15:11:51 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Nov 26 15:11:51 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 2
Nov 26 15:11:51 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Nov 26 15:11:51 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 2
Nov 26 15:11:52 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:11:52 server2 servermgrd: xsan: [45/39B8D0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:12:17 server2 servermgrd: xsan: [45/3BA980] ERROR: get_remote_properties(R1V1): Could not reach 192.168.1.22:311
Nov 26 15:12:17 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_clients_for_fsmvol_named(R1V1): Could not connect: Connect to FSM failed - Connection refused
Nov 26 15:12:17 server2 servermgrd: xsan: [45/39DDC0] ERROR: get_computer_properties(R1V1): No client list for volume
Nov 26 15:13:17 server2 servermgrd: xsan: [45/348030] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:13:17 server2 servermgrd: xsan: [45/3B3820] ERROR: get_fsmvol_at_index: Could not connect to FSM because Connect to FSM failed - Connection refused
Nov 26 15:13:31 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Nov 26 15:13:31 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 2
Nov 26 15:13:33 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Nov 26 15:13:33 server2 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 2
Nov 26 15:13:43 server2 servermgrd: xsan: [45/3CD0F0] ERROR: get_remote_properties(R1V1): Could not reach 192.168.1.22:311
Nov 26 15:13:43 server2 servermgrd: xsan: [45/348030] ERROR: get_clients_for_fsmvol_named(R1V1): Could not connect: Connect to FSM failed - Connection refused
Nov 26 15:13:43 server2 servermgrd: xsan: [45/348030] ERROR: get_computer_properties(R1V1): No client list for volume

Anyone have any suggestions?

Thanks,
roberbg

[colbru]

[rogerbg]

Hello,

We had other issues when we ran 10.4.10. In the course of trying to solve all of our problems, we upgraded to 10.4.11. All other issues have been solved (not by the OS upgrade alone) except for this stability issue.

One other piece of information...if I prevent the clients from going into sleep mode, the XSan runs longer. I configured the clients to not go into sleep mode yesterday afternoon and this morning everything was still running. This morning, If I refreshed the XSan admin gui, it constantly displayed the proper up time. All was well until I tried to place a small txt file onto the volume from one of the clients. This initiated the crash/stall. I'm at a loss for what is happening. We do have an Apple ticket created. We are awaiting their feedback.

Thanks,
Roger

[dernsber]

We've been having a similar error with our XSAN, getting the FusionMPT errors left and right and lastnight one of our MDCs crashed, i believe from a hung processor, here's the log:

Dec 4 16:58:45 sidhq-vid-mdc-00 servermgrd: [18729] exception in get_and_lock_context: *** -[NSConcreteFileHandle readDataOfLength:]: Bad file descriptor\n
Dec 4 16:58:45 sidhq-vid-mdc-00 servermgrd: [18729] exception in set_and_unlock_context: *** -[NSConcreteFileHandle truncateFileAtOffset:]: Bad file descriptor\n
Dec 4 17:02:36 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 17:02:36 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 17:02:36 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 17:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 17:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 17:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 17:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 18:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 18:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 18:09:55 sidhq-vid-mdc-00 dashboardadivsoryd[26369]: fetch-parsers: unable to fetch data from read stream
Dec 4 18:09:55 sidhq-vid-mdc-00 dashboardadivsoryd[26369]: fetch: unable to fetch data from read stream while fetch advisory list
Dec 4 18:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 18:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 19:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 19:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 19:32:31 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 19:32:31 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 19:32:31 sidhq-vid-mdc-00 kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 19:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 19:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 20:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 20:09:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 20:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no name available via DNS for 192.9.200.53
Dec 4 20:39:31 sidhq-vid-mdc-00 servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair
Dec 4 21:09:03 localhost kernel[0]: standard timeslicing quantum is 10000 us
Dec 4 21:09:03 localhost mDNSResponder-108.4 (Jan 8 2007 20: 36:00)[36]: starting
Dec 4 21:09:03 localhost kernel[0]: vm_page_bootstrap: 508902 free pages
Dec 4 21:09:03 localhost memberd[47]: memberd starting up
Dec 4 21:09:03 localhost kernel[0]: mig_table_max_displ = 70
Dec 4 21:09:03 localhost kernel[0]: 96 prelinked modules
Dec 4 21:09:03 localhost lookupd[51]: lookupd (version 369.5) starting - Tue Dec 4 21:09:03 2007
Dec 4 21:09:03 localhost kernel[0]: Copyright (c) 1982, 1986, 1989, 1991, 1993
Dec 4 21:09:03 localhost DirectoryService[52]: Launched version 2.1 (v353.6)
Dec 4 21:09:03 localhost kernel[0]: The Regents of the University of California. All rights reserved.
Dec 4 21:09:03 localhost kernel[0]: using 5242 buffer headers and 4096 cluster IO buffer headers
Dec 4 21:09:03 localhost kernel[0]: DART enabled
Dec 4 21:09:03 localhost kernel[0]: FireWire (OHCI) Apple ID 42 built-in now active, GUID 000a95ff fea6bed4; max speed s800.
Dec 4 21:09:03 localhost kernel[0]: Security auditing service present
Dec 4 21:09:03 localhost kernel[0]: BSM auditing present
Dec 4 21:09:03 localhost kernel[0]: disabled
Dec 4 21:09:03 localhost kernel[0]: rooting via boot-uuid from /chosen: 59015F37-B9E5-39E0-9072-D78B91059606
Dec 4 21:09:03 localhost kernel[0]: Waiting on <dict ID="0"><key>IOProviderClass</key><string ID="1">IOResources</string><key>IOResourceMatch</key><string ID="2">boot-uuid-media</string></dict>
Dec 4 21:09:03 localhost kernel[0]: Got boot device = IOService:/MacRISC4PE/ht@0,f2000000/AppleMacRiscHT/pci@7/IOPCI2PCIBridge/k2-sata-root@C/AppleK2SATARoot/k2-sata@0/AppleK2SATA/ATADeviceNub@0/IOATABlockStorageDriver/IOATABlockStorageDevice/IOBlockStorageDriver/ST3160023AS Media/IOApplePartitionScheme/Apple_RAID_OfflineV2_Untitled_2@4/AppleRAIDMember/AppleRAIDMirrorSet/Untitled RAID Set 1@0
Dec 4 21:09:03 localhost kernel[0]: BSD root: disk2, major 14, minor 8
Dec 4 21:09:03 localhost kernel[0]: jnl: replay_journal: from: 3657728 to: 2451968 (joffset 0x4aa000)
Dec 4 21:09:03 localhost kernel[0]: HFS: Removed 41 orphaned unlinked files
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 0
Dec 4 21:09:03 localhost kernel[0]: Jettisoning kernel linker.
Dec 4 21:09:03 localhost kernel[0]: Resetting IOCatalogue.
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 0
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 7
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 7
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 7
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 7
Dec 4 21:09:03 localhost kernel[0]: Matching service count = 7
Dec 4 21:09:03 localhost kernel[0]: IOPlatformControl::registerDriver Control Driver AppleSlewClock did not supply target-value, using default
Dec 4 21:09:04 localhost watchdogtimerd: Automatic reboot timer enabled.\n


We're Running 10.4.9 for the MDCs, and clients are 10.4.10, I'm mainly looking for an explanation as to what the FusionMPT: Notification error stands for.

Any ideas?
Thanks!

[dernsber]

Was talking with a coworker, got an idea.

What version of XSAN are you using?

I'm using 1.4.1, wondering if you're still getting it with 1.4.2, or if you're using 1.4.1 yet.

[Tim Burton]

Have you tried using single initiator zoning?

Create a zone for each client and MDC and just put that one client and the storage into each zone. Might help stop mass kernel panics.

Obviously this isn't a solution, however it might just limit the hang to one or two clients. Damage limitation.

T

[rogerbg]

Our setup...

XServe 1 is for DHCP/DNS and is running 10.4.11

XServe 2 is for XSan MetaDataController and is running 10.4.11

XRaid is firmware 1.5.1f using RAID 5 and has about 6-7 TB space

LUNS
1.3TB for MD
2.0TB for User Data
4.0TB for User Data

QLogic 5600 Fiber Switch is firmware 6.8 and has 16 ports and No Zones

Linksys Ethernet Switch and has 16 ports

A single G5 PPC as the client running 10.4.11

All equipment is running XSan 1.4.2

After Apple parsed the data I sent to them, they asked that I do the following:
1. Use static IP addresses on everything
2. Replace the ethernet switch (was using a 3 year old Netgear)
3. Make sure fiber switch ports are properly set to Target or Initiator
4. RAID ports on switch: set I/O stream to Auto
5. RAID ports on switch: set Device Scan to Enable
6. Switch ports connecting to Apple Fibre Cards: set I/O to Enable
7. Switch ports connecting to Apple Fibre Cards: set Device Scan to Disable

After making all of those adjustments, I restarted everything in the following order:
XRaid
Fiber Switch
Ethernet Switch
XServe running XSan as MetaDataController
XServe running DHCP and DNS
Client

After all of that, our XSan ran for 17 hours and 30 minutes. At that point, we saw the same issue as I originally posted. The only change was that I only had to restart the XServe running XSan MDC. I did not have to restart the client this time.

Hope that info helps,
Roger

[aaron]

[rogerbg]

Thank you for the information. I will try your suggestion of setting up zones. I will use your "Bulletproof Zoning" article as a guide.

Because I'm learning as I go, could someone please explain how zoning my fabric will solve the timeout/hang issue we're experiencing?

I will post my zone setup in a few.

Thanks,
RBG

[Tim Burton]

It will stop one machine having a 'tizzy' (technical term) knocking over the rest on the fabric. This is because the machines are separated from talking to each other.

T

[rogerbg]

Thank you for that. It makes sense...

Currently we are only running one client for testing purposes. Therefore having zones makes no sense. BUT, when this is setup and running, we will have 6-7 clients. Therefore, zoning makes sense. So...I am setting up zones. I will post my zone map in a moment.

Quick question though...each device has two fiber cables...should I place both WWN #'s for a device within its own zone? Our XServe #1 doesn't have XSan installed and it's never going to connect to XSan. It's only for DHCP/DNS. Do I need to add the XRAID alias to the XServe #1 zone?

Thanks,
RBG

[rogerbg]

Zone Map

ZoneSet: ORPHAN ZONE SET

ZoneSet: ZoneByWWN

-Zone: JohnDoe
--WWN:xxxxxx
--WWN:xxxxxx
--Alias: XRAIDONE

-Zone: Xserve1DHCPDNS
--WWN:xxxxxx
--WWN:xxxxxx
--Alias: XRAIDONE

-Zone: Xserve2MDC
--WWN:xxxxxx
--WWN:xxxxxx
--Alias: XRAIDONE

-Alias: XRAIDONE
--WWN:xxxxxx
--WWN:xxxxxx

Does that look correct? As stated in my previous post...each device has two fiber cables, so there are two WWN for each device. Should I remove one of them?

Thanks,
RBG

[aaron]

That seems right to me.
_________________
Aaron Freimark
http://www.tekserve.com/vcard/af.vcf

[rogerbg]

To be honest, I never implemented the zones...at least not yet. I was awaiting word from apple support on their next suggestion before I implemented the zones. Just in case zoning would conflict with their suggestions. While waiting...I got the idea of just starting over.

I reformatted both servers. I removed XSAN from the test client. I reinstalled the OS on the servers. I reinstalled XSAN on everything. I DID NOT update XSAN to 1.4.2. Our original test client computer has been mounted for over 24 hours without issue. I have added two more clients for testing purposes. One has been mounted for 4 hours and the other has been up for 2 hours. The logs are clean.

Now I need to research a proper storage/naming convention...and I also need to setup my zones now that it's all working and I'm adding clients.

Thanks to everyone for the help. It's appreciated...
RBG