Reboot/Startup = No SAN

vidiot's picture

We have a very odd situation happening. Of the 45 Xsan clients we have, about 7 of them fail to mount any of the volumes after a restart or a shutdown/boot. Listed in the /Volumes folder is just one of the SAN volumes as a folder with a Red slash as though it is attempting to mount. Removing this folder has no affect.

Some background...
We're using Stornext 3.1.3 MDCs. Xsan is running on at least 45 Mac clients all running Xsan 2.1.1 and 10.5.x. Nothing was changed or modified in about 1.5 years and ran flawlessly until about 3 weeks ago. Now it is a roll of the dice with about half of the Mac clients if they'll be successful on reboot with mounting any SAN volume. It may take several hours worth of reboots to get the SAN to mount and you hope you don't need to restart for any reason or you might be right back where you started.

I've even gone so far as to uninstall the Xsan software and do a full reinstall. Sometimes on first reboot all is well, but quickly the problem returns. All clients can see all the LUNs under System Profile as well as cvlabel -l.

Any ideas what could be going wrong?

Here's a snippet from nssbg.out. Note that one of our MDCs is down (10.48.19.21), but that wouldn't cause the SAN issue:

[code][0213 17:20:59] 0xb0185000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:21:08] 0xb0185000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:21:17] 0xb0185000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:21:19] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:21:29] 0xb0185000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:23:52] 0xa0325720 (debug) PortMapper: Unable to open /Library/Filesystems/Xsan/debug/verbose, using default debug flags, errno = 2
[0213 17:23:52] 0xa0325720 INFO StorNext PortMapper (FSMPM) starting.
[0213 17:23:53] 0xa0325720 INFO NSS: Primary Name Server is '10.48.19.21' (10.48.19.21)
[0213 17:23:53] 0xb0103000 INFO NSS: Establish Coordinator GetHostByName of '10.48.19.21' complete: 10.48.19.21
[0213 17:23:53] 0xa0325720 INFO NSS: Secondary #1 Name Server is '10.48.19.22' (10.48.19.22)
[0213 17:23:53] 0xb0185000 INFO NSS: Establish Coordinator GetHostByName of '10.48.19.22' complete: 10.48.19.22
[0213 17:23:53] 0xa0325720 NOTICE Portmapper: ComputerInfo: computer_name = "Robert's Mac Pro", hostname = "Roberts-Mac-Pro"
[0213 17:23:53] 0xa0325720 (debug) No fsports file - port range enforcement disabled.
[0213 17:23:53] 0xa0325720 (debug) PortMapper: using UDP SO_RCVBUF = 1048576
[0213 17:23:53] 0xa0325720 (debug) PortMapper: using UDP SO_SNDBUF = 65536
[0213 17:23:54] 0xa0325720 INFO NSS: Listening on UDP port 49153
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID16LEFT on device: /dev/rdisk4 (blk 0xe00000a raw 0xe00000a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052701000000D52BF968' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID15RIGHT on device: /dev/rdisk5 (blk 0xe00000b raw 0xe00000b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052201000000D52C0CC0' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID16RIGHT on device: /dev/rdisk6 (blk 0xe00000c raw 0xe00000c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002051501000000D52BE265' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID20LEFT on device: /dev/rdisk7 (blk 0xe00000d raw 0xe00000d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002046A01000000D52BAD55' Size: 490190848 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID6LEFT on device: /dev/rdisk8 (blk 0xe00000e raw 0xe00000e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002045201000000D52AFE09' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID19LEFT on device: /dev/rdisk9 (blk 0xe00000f raw 0xe00000f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002044D01000000D52B2785' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID21LEFT on device: /dev/rdisk10 (blk 0xe000010 raw 0xe000010) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002042301000000D52B65B5' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID20RIGHT_SNFSHA on device: /dev/rdisk11 (blk 0xe000011 raw 0xe000011) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002040F01000000D52C2F35' Size: 976713728 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID17RIGHT on device: /dev/rdisk12 (blk 0xe000012 raw 0xe000012) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202F501000000D52BC06B' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID17LEFT on device: /dev/rdisk13 (blk 0xe000013 raw 0xe000013) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202D401000000D52BD767' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID14RIGHT on device: /dev/rdisk14 (blk 0xe000014 raw 0xe000014) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000201F101000000D52C456B' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID14LEFT on device: /dev/rdisk15 (blk 0xe000015 raw 0xe000015) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002019B01000000D52C5C78' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID19RIGHT on device: /dev/rdisk16 (blk 0xe000016 raw 0xe000016) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001FBC501000000D52B102E' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID18RIGHT on device: /dev/rdisk17 (blk 0xe000017 raw 0xe000017) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F7D401000000D52B4774' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID18LEFT on device: /dev/rdisk18 (blk 0xe000018 raw 0xe000018) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F6F001000000D52B87FC' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID13RIGHT on device: /dev/rdisk19 (blk 0xe000019 raw 0xe000019) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E10501000000D52B095E' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID12LEFT on device: /dev/rdisk20 (blk 0xe00001a raw 0xe00001a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F801000000D52B03A5' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID11LEFT on device: /dev/rdisk21 (blk 0xe00001b raw 0xe00001b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F101000000D52AF32F' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID13LEFT on device: /dev/rdisk22 (blk 0xe00001c raw 0xe00001c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0E701000000D52B2059' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID12RIGHT on device: /dev/rdisk23 (blk 0xe00001d raw 0xe00001d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0DD01000000D52AED74' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID10RIGHT on device: /dev/rdisk24 (blk 0xe00001e raw 0xe00001e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E08A01000000D52BA014' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID11RIGHT on device: /dev/rdisk25 (blk 0xe00001f raw 0xe00001f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E06801000000D52ADBE2' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID10LEFT on device: /dev/rdisk26 (blk 0xe000020 raw 0xe000020) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E00E01000000D52BB71C' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID7RIGHT on device: /dev/rdisk27 (blk 0xe000021 raw 0xe000021) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A2A101000000D52AD5DC' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID7LEFT on device: /dev/rdisk28 (blk 0xe000022 raw 0xe000022) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A28701000000D52AED58' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID21RIGHT on device: /dev/rdisk29 (blk 0xe000023 raw 0xe000023) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A0A201000000D52B4EFC' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID6RIGHT on device: /dev/rdisk30 (blk 0xe000024 raw 0xe000024) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A08E01000000D52AEC09' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID5RIGHT on device: /dev/rdisk31 (blk 0xe000025 raw 0xe000025) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A07501000000D52C043F' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID9LEFT on device: /dev/rdisk32 (blk 0xe000026 raw 0xe000026) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A05601000000D52B0F6C' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID9RIGHT on device: /dev/rdisk33 (blk 0xe000027 raw 0xe000027) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04A01000000D52AF7FF' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID5LEFT on device: /dev/rdisk34 (blk 0xe000028 raw 0xe000028) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04101000000D52C17CE' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID8RIGHT on device: /dev/rdisk35 (blk 0xe000029 raw 0xe000029) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A01201000000D52B1731' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID4LEFT on device: /dev/rdisk36 (blk 0xe00002a raw 0xe00002a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A00801000000D52C0F9F' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID4RIGHT on device: /dev/rdisk37 (blk 0xe00002b raw 0xe00002b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FB601000000D52BFB5C' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID8LEFT on device: /dev/rdisk38 (blk 0xe00002c raw 0xe00002c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FAB01000000D52B2EBA' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID3LEFT on device: /dev/rdisk39 (blk 0xe00002d raw 0xe00002d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001664401000000D52B7BC1' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID3RIGHT on device: /dev/rdisk40 (blk 0xe00002e raw 0xe00002e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165F501000000D52B7895' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID2RIGHT on device: /dev/rdisk41 (blk 0xe00002f raw 0xe00002f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165AD01000000D52B3D88' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID2LEFT on device: /dev/rdisk42 (blk 0xe000030 raw 0xe000030) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165A001000000D52B4FAD' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID1LEFT on device: /dev/rdisk43 (blk 0xe000031 raw 0xe000031) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000014F2C01000000D6CD1CCD' Size: 490190848 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID1RIGHT1 on device: /dev/rdisk44 (blk 0xe000032 raw 0xe000032) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001436501000000D52B995D' Size: 490190848 Sector Size: 512
[0213 17:23:57] 0xa0325720 NOTICE PortMapper: CVFS Volume RAID15LEFT on device: /dev/rdisk3 (blk 0xe000009 raw 0xe000009) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002053701000000D52C2411' Size: 4883789791 Sector Size: 512
[0213 17:23:57] 0xa0325720 INFO PortMapper: No fsmlist file - No File System Services launched.
[0213 17:23:57] 0xa0325720 INFO PortMapper: name=Roberts-Mac-Pro.local IP/Port=192.168.62.200/49156 DbgLvl=0
scan interval=1800
[0213 17:23:57] 0xb040f000 ERR Disk arb runloop starting
[0213 17:23:58] 0xa0325720 (debug) NSS: Name Server '10.48.19.21' (10.48.19.21) port unknown, unable to send message.
[0213 17:23:58] 0xa0325720 (debug) NSS: Name Server '10.48.19.22' (10.48.19.22) port unknown, unable to send message.
[0213 17:24:02] 0xb038d000 INFO Starting Disk rescan
[0213 17:24:02] 0xb038d000 INFO Disk rescan delay completed
[0213 17:24:03] 0xa0325720 (debug) IP address list changed
[0213 17:24:03] 0xb0185000 INFO NSS: Name Server '10.48.19.22' (10.48.19.22) port is 32768, revision is 0x0101.
[0213 17:24:03] 0xa0325720 (debug) Dropping 10.48.19.22 coordinator 0 for new 32768
[0213 17:24:03] 0xa0325720 INFO PortMapper: Added mapping from id 10.48.19.22 to addr 10.48.19.22
[0213 17:24:03] 0xa0325720 (debug) NSS: Coordinator 10.48.19.22 id is 10.48.19.22
[0213 17:24:03] 0xa0325720 INFO PortMapper: Creating sync file
[0213 17:24:03] 0xb038d000 INFO Disk rescan found 42 disks
[0213 17:24:03] 0xb038d000 INFO Starting Disk rescan
[0213 17:24:04] 0xa0325720 NOTICE PortMapper: Local FSD client is registered, on port 49156.
[0213 17:24:14] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:24:23] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.1 seconds.
[0213 17:24:32] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:24:41] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:24:50] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:24:53] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:25:02] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:11] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:20] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:23] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:25:32] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:25:53] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:26:03] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:03] 0xb038d000 INFO Disk rescan delay completed
[0213 17:26:07] 0xb038d000 INFO Disk rescan found 42 disks
[0213 17:26:10] 0xb038d000 INFO Starting Disk rescan
[0213 17:26:12] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:21] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:26:33] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:26:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:27:03] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:12] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:21] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:27:33] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:27:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:28:03] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:07] 0xb038d000 INFO Disk rescan delay completed
[0213 17:28:09] 0xb038d000 INFO Disk rescan found 42 disks
[0213 17:28:12] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:13] 0xb038d000 INFO Starting Disk rescan
[0213 17:28:21] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:28:33] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:28:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:29:03] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:12] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:21] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:29:33] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:29:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:30:03] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:09] 0xb038d000 INFO Disk rescan delay completed
[0213 17:30:11] 0xb038d000 INFO Disk rescan found 42 disks
[0213 17:30:12] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:21] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.4 seconds.
[0213 17:30:34] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:43] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:52] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:30:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:31:04] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:13] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:22] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:31:34] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:43] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:52] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:31:54] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:32:04] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:32:13] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:32:22] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:32:24] 0xa0325720 (debug) Name Server 10.48.19.21 heartbeat absent for 30.0 seconds.
[0213 17:32:33] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:32:42] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
[0213 17:32:51] 0xb0103000 NOTICE NSS: Cannot acquire port for 10.48.19.21
/code

Any help, ideas, anything, would be greatly appreciated. Thanks!!

lotte's picture

I assume your fsnameserver file on the Macs is looking like:

10.48.19.21
10.48.19.22

remove the first entry while this machine is down. Even more interesting is the fsmlist file on machine 10.48.19.22.

This should look like

10.48.19.21 YourSanVolumeName 0
10.48.19.22 YourSanVolumeName 1

Can you confirm that? And can you confirm ( I assume you are running linux on the mdc) that the portmap service is running?

Lotte

vidiot's picture

My fsnameserver file looked like this:

[code]10.48.19.21
10.48.19.22/code

I removed 10.48.19.21 from the list and have this result:

[code][0214 07:45:19] 0xa0338830 (debug) PortMapper: Unable to open /Library/Filesystems/Xsan/debug/verbose, using default debug flags, errno = 2
[0214 07:45:19] 0xa0338830 INFO StorNext PortMapper (FSMPM) starting.
[0214 07:45:19] 0xa0338830 INFO NSS: Primary Name Server is '10.48.19.22' (10.48.19.22)
[0214 07:45:19] 0xf0103000 INFO NSS: Establish Coordinator GetHostByName of '10.48.19.22' complete: 10.48.19.22
[0214 07:45:19] 0xf0103000 INFO NSS: Name Server '10.48.19.22' (10.48.19.22) port is 32768, revision is 0x0102.
[0214 07:45:19] 0xa0338830 NOTICE Portmapper: ComputerInfo: computer_name = "Gateway Server", hostname = "Gateway-Server2"
[0214 07:45:19] 0xa0338830 (debug) No fsports file - port range enforcement disabled.
[0214 07:45:19] 0xa0338830 (debug) PortMapper: using UDP SO_RCVBUF = 1048576
[0214 07:45:19] 0xa0338830 (debug) PortMapper: using UDP SO_SNDBUF = 65536
[0214 07:45:19] 0xa0338830 INFO NSS: Listening on UDP port 49153
[0214 07:45:19] 0xa0338830 WARNING PortMapper: No CVFS Disk Volumes are accessible.
[0214 07:45:19] 0xa0338830 INFO PortMapper: No fsmlist file - No File System Services launched.
[0214 07:45:19] 0xa0338830 INFO PortMapper: name=GatewayServer.abcvideopost.local IP/Port=10.48.19.50/49153 DbgLvl=0
scan interval=1800
[0214 07:45:19] 0xf038d000 ERR Disk arb runloop starting
[0214 07:45:19] 0xa0338830 (debug) Dropping 10.48.19.22 coordinator 0 for new 32768
[0214 07:45:19] 0xa0338830 INFO PortMapper: Added mapping from id 10.48.19.22 to addr 10.48.19.22
[0214 07:45:19] 0xa0338830 (debug) NSS: Coordinator 10.48.19.22 id is 10.48.19.22
[0214 07:45:19] 0xa0338830 (debug) NSS: Computing nss_coord_sum
[0214 07:45:19] 0xa0338830 ERR NSS: Coordinator 10.48.19.22 (id 10.48.19.22) coordinator list mismatch
[0214 07:45:19] 0xa0338830 INFO PortMapper: Creating sync file
[0214 07:45:19] 0xf030b000 INFO Starting Disk rescan
[0214 07:45:19] 0xf030b000 INFO Disk rescan delay completed
[0214 07:45:19] 0xf030b000 INFO Disk rescan found 0 disks
[0214 07:45:20] 0xa0338830 NOTICE PortMapper: Local FSD client is registered, on port 49153.
[0214 07:45:33] 0xf030b000 INFO Starting Disk rescan
[0214 07:45:39] 0xa0338830 (debug) IP address list changed
[0214 07:47:19] 0xf030b000 INFO Disk rescan delay completed
[0214 07:47:20] 0xf030b000 INFO Disk rescan found 42 disks
/code

Then I changed the fsnameserver to this:
[code]10.48.19.22 DataDrive
/code
And got this result:

[code][0214 07:52:49] 0xa0338830 (debug) PortMapper: Unable to open /Library/Filesystems/Xsan/debug/verbose, using default debug flags, errno = 2
[0214 07:52:49] 0xa0338830 INFO StorNext PortMapper (FSMPM) starting.
[0214 07:52:49] 0xf0103000 INFO NSS: Establish Coordinator GetHostByName of '10.48.19.22' complete: 10.48.19.22
[0214 07:52:49] 0xa0338830 INFO NSS: Primary Name Server is '10.48.19.22' (10.48.19.22)
[0214 07:52:49] 0xf0103000 INFO NSS: Name Server '10.48.19.22' (10.48.19.22) port is 32768, revision is 0x0102.
[0214 07:52:49] 0xa0338830 NOTICE Portmapper: ComputerInfo: computer_name = "Gateway Server", hostname = "Gateway-Server2"
[0214 07:52:49] 0xa0338830 (debug) No fsports file - port range enforcement disabled.
[0214 07:52:49] 0xa0338830 (debug) PortMapper: using UDP SO_RCVBUF = 1048576
[0214 07:52:49] 0xa0338830 (debug) PortMapper: using UDP SO_SNDBUF = 65536
[0214 07:52:49] 0xa0338830 INFO NSS: Listening on UDP port 49153
[0214 07:52:49] 0xa0338830 WARNING PortMapper: No CVFS Disk Volumes are accessible.
[0214 07:52:49] 0xa0338830 INFO PortMapper: No fsmlist file - No File System Services launched.
[0214 07:52:49] 0xa0338830 INFO PortMapper: name=GatewayServer.abcvideopost.local IP/Port=10.48.19.50/49153 DbgLvl=0
scan interval=1800
[0214 07:52:49] 0xf038d000 ERR Disk arb runloop starting
[0214 07:52:50] 0xa0338830 (debug) Dropping 10.48.19.22 coordinator 0 for new 32768
[0214 07:52:50] 0xa0338830 INFO PortMapper: Added mapping from id 10.48.19.22 to addr 10.48.19.22
[0214 07:52:50] 0xa0338830 (debug) NSS: Coordinator 10.48.19.22 id is 10.48.19.22
[0214 07:52:50] 0xa0338830 (debug) NSS: Computing nss_coord_sum
[0214 07:52:50] 0xa0338830 ERR NSS: Coordinator 10.48.19.22 (id 10.48.19.22) coordinator list mismatch
[0214 07:52:50] 0xa0338830 INFO PortMapper: Creating sync file
[0214 07:52:50] 0xf030b000 INFO Starting Disk rescan
[0214 07:52:50] 0xf030b000 INFO Disk rescan delay completed
[0214 07:52:50] 0xf030b000 INFO Disk rescan found 0 disks
[0214 07:52:50] 0xa0338830 NOTICE PortMapper: Local FSD client is registered, on port 49153.
[0214 07:52:54] 0xf030b000 INFO Starting Disk rescan
[0214 07:53:10] 0xa0338830 (debug) IP address list changed
/code

Why would the SANVolumes be listed in fsnameserver? I haven't needed that in the past. And if I have 3 Volumes do I make 6 entries with each MDC listed for each volume?

Yes the MDCs are linux. Portmapper service should be running because 40 other clients have no issues. But how would i verify Portmapper specifically? Does that appear to be the issue?

Thanks!

lotte's picture

Hi vidiot, we have a sort of missunderstanding, I wanted you to have a look at the linux fsmlist file, not to add the name of the Filesystem to the fsnameserver file of a client!

Can you post the output of /var/log/system.log during mounting and also post /Library/Filesystems/Xsan/config/automount.plist

Portmapper on linux can be watched with "service portmap status"

Lotte

vidiot's picture

Ha, ok that makes more sense.

The Linux MDC fsmlist:

[code]iTX
Soapnet
xSAN
SNFS_HA
DataDrive/code

system.log:

[code]Feb 14 07:51:57 GatewayServer shutdown[257]: SHUTDOWN_TIME: 1266162717 393964
Feb 14 07:51:57 GatewayServer com.apple.loginwindow[52]: Shutdown NOW!
Feb 14 07:51:57 GatewayServer com.apple.loginwindow[52]: System shutdown time has arrived^G^G
Feb 14 07:51:57 GatewayServer mDNSResponder mDNSResponder-176.3 (Sep 30 2008 16:59:41)[31]: stopping
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: Stopping IP Failover services
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: Disabling Network Address Translation
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: net.inet.ip.forwarding: 0 -> 0
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: ipfw: rule 10 does not exist
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: cat: /var/run/failoverd.pid: No such file or directory
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: cat: /var/run/heartbeatd.pid: No such file or directory
Feb 14 07:51:57 GatewayServer com.apple.SystemStarter[45]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Feb 14 07:51:57 GatewayServer SystemStarter[45]: IP Failover (265) did not complete successfully
Feb 14 07:51:57 GatewayServer SystemStarter[45]: The following StartupItems failed to properly start:
Feb 14 07:51:57 GatewayServer SystemStarter[45]: /System/Library/StartupItems/IPFailover
Feb 14 07:51:57 GatewayServer SystemStarter[45]: - execution of Startup script failed
Feb 14 07:51:57 GatewayServer servermgrd[48]: dnssd_clientstub read_all(15) failed 0/28 0
Feb 14 07:52:41 localhost kernel[0]: Darwin Kernel Version 9.6.0: Mon Nov 24 17:39:01 PST 2008; root:xnu-1228.9.59~1/RELEASE_PPC
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12300 allow tcp from any to any established
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12301 allow tcp from any to any out
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12302 allow tcp from any to any dst-port 22
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12302 allow udp from any to any dst-port 22
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12303 allow udp from any to any out keep-state
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12304 allow tcp from any to any dst-port 53 out keep-state
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12304 allow udp from any to any dst-port 53 out keep-state
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12305 allow udp from any to any in frag
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12306 allow tcp from any to any dst-port 311
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12307 allow tcp from any to any dst-port 625
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12308 allow udp from any to any dst-port 626
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12309 allow icmp from any to any icmptypes 8
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12310 allow icmp from any to any icmptypes 0
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 12311 allow igmp from any to any
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01000 allow ip from any to any via lo0
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01010 deny ip from any to 127.0.0.0/8
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01020 deny ip from 224.0.0.0/4 to any in
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01030 deny tcp from any to 224.0.0.0/4 in
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 65534 deny ip from any to any
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: net.inet.ip.fw.enable: 1 -> 0
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 00001 allow
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: udp from any to any 626
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01000 allow
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: ipv6 from any to
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: any via lo0
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 01100 allow
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: ipv6 from any to
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: ff02::/16
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: 65000 deny
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: ipv6 from any to any
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: net.inet6.ip6.fw.enable: 1 -> 0
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: net.inet.tcp.delayed_ack: 3 -> 2
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: /etc/rc.server: line 55: logger: command not found
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: kern.maxproc: 532 -> 2500
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: kern.ipc.somaxconn: 128 -> 2500
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: kern.maxnbuf: 16384 -> 21000
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: kern.maxvnodes: 58368 -> 120000
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: kern.maxprocperuid: 266 -> 1000
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: launch_msg(): Socket is not connected
Feb 14 07:52:38 localhost com.apple.launchctl.System[2]: Bug: launchctl.c:1414 (23642):9: fwexec(rcserver_tool, true) != -1
Feb 14 07:52:41 localhost com.apple.launchctl.System[2]: launchctl: Please convert the following to launchd: /etc/mach_init.d/dashboardadvisoryd.plist
Feb 14 07:52:41 localhost com.apple.launchd[1] (com.apple.blued): Unknown key for boolean: EnableTransactions
Feb 14 07:52:41 localhost com.apple.launchd[1] (org.cups.cupsd): Unknown key: SHAuthorizationRight
Feb 14 07:52:41 localhost com.apple.launchd[1] (org.ntp.ntpd): Unknown key: SHAuthorizationRight
Feb 14 07:52:41 localhost DirectoryService[26]: Launched version 5.6 (v514.24)
Feb 14 07:52:41 localhost kernel[0]: standard timeslicing quantum is 10000 us
Feb 14 07:52:41 localhost kernel[0]: vm_page_bootstrap: 1020907 free pages and 27669 wired pages
Feb 14 07:52:41 localhost kernel[0]: mig_table_max_displ = 79
Feb 14 07:52:41 localhost kernel[0]: 97 prelinked modules
Feb 14 07:52:41 localhost kernel[0]: Loading security extension com.apple.security.TMSafetyNet
Feb 14 07:52:41 localhost kernel[0]: calling mpo_policy_init for TMSafetyNet
Feb 14 07:52:41 localhost kernel[0]: Security policy loaded: Safety net for Time Machine (TMSafetyNet)
Feb 14 07:52:41 localhost kernel[0]: Loading security extension com.apple.nke.applicationfirewall
Feb 14 07:52:41 localhost kernel[0]: Loading security extension com.apple.security.seatbelt
Feb 14 07:52:41 localhost kernel[0]: calling mpo_policy_init for mb
Feb 14 07:52:41 localhost kernel[0]: Seatbelt MACF policy initialized
Feb 14 07:52:41 localhost kernel[0]: Security policy loaded: Seatbelt Policy (mb)
Feb 14 07:52:41 localhost kernel[0]: Copyright (c) 1982, 1986, 1989, 1991, 1993
Feb 14 07:52:41 localhost kernel[0]: The Regents of the University of California. All rights reserved.
Feb 14 07:52:41 localhost kernel[0]: MAC Framework successfully initialized
Feb 14 07:52:41 localhost kernel[0]: using 16384 buffer headers and 4096 cluster IO buffer headers
Feb 14 07:52:41 localhost kernel[0]: DART enabled
Feb 14 07:52:41 localhost kernel[0]: Enabling ECC Error Notifications
Feb 14 07:52:41 localhost kernel[0]: FireWire (OHCI) Apple ID 42 PCI now active, GUID 001124fffe44eef0; max speed s800.
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 10 (Event Change) for SCSI Domain = 0
Feb 14 07:52:41 localhost kernel[0]: mbinit: done
Feb 14 07:52:41 localhost kernel[0]: Security auditing service present
Feb 14 07:52:41 localhost kernel[0]: BSM auditing present
Feb 14 07:52:41 localhost kernel[0]: rooting via boot-uuid from /chosen: FE5A640C-C0C2-3D89-93CF-8381922A2F6F
Feb 14 07:52:41 localhost kernel[0]: Waiting on IOProviderClassIOResourcesIOResourceMatchboot-uuid-media
Feb 14 07:52:41 localhost kernel[0]: Got boot device = IOService:/MacRISC4PE/ht@0,f2000000/AppleMacRiscHT/pci@7/IOPCI2PCIBridge/k2-sata-root@C/AppleK2SATARoot/k2-sata@0/AppleK2SATA/ATADeviceNub@0/AppleATADiskDriver/IOATABlockStorageDevice/IOBlockStorageDriver/Hitachi HDS725050KLA360 Hitachi HDS725050KLA360/IOApplePartitionScheme/Untitled@3
Feb 14 07:52:41 localhost kernel[0]: BSD root: disk0s3, major 14, minor 2
Feb 14 07:52:41 localhost kernel[0]: IPv6 packet filtering initialized, default to accept, logging disabled
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 10 (Event Change) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 7 (Link Status Change) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionFC: Link is down for SCSI Domain = 1.
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 5 (External Bus Reset) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: External Bus Reset for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 8 (Loop State Change) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionFC: Loop Initialization Packet for SCSI Domain = 1, fLIPCount = 0.
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 7 (Link Status Change) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: FusionFC: Link is active for SCSI Domain = 1.
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:41: --- last message repeated 1 time ---
Feb 14 07:52:41 localhost kernel[0]: Jettisoning kernel linker.
Feb 14 07:52:41 localhost kernel[0]: Resetting IOCatalogue.
Feb 14 07:52:41 localhost kernel[0]: BCM5701Enet: Ethernet address 00:0d:93:9d:d8:4a
Feb 14 07:52:41 localhost kernel[0]: BCM5701Enet: Ethernet address 00:0d:93:9d:d8:4b
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:41 localhost mDNSResponder mDNSResponder-176.3 (Sep 30 2008 16:59:41)[31]: starting
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: Matching service count = 0
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:41 localhost kernel[0]: Matching service count = 9
Feb 14 07:52:41: --- last message repeated 4 times ---
Feb 14 07:52:41 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:42: --- last message repeated 21 times ---
Feb 14 07:52:42 localhost kextd[25]: 419 cached, 0 uncached personalities to catalog
Feb 14 07:52:42 localhost kernel[0]: AppleRS232Serial: 604f4020 80013020 chip base, virtual, physical
Feb 14 07:52:42 localhost kernel[0]: IOPlatformControl::registerDriver Control Driver AppleSlewClock did not supply target-value, using default
Feb 14 07:52:43 localhost com.apple.launchd[1] (com.openssh.sshd): Unknown key: SHAuthorizationRight
Feb 14 07:52:43 localhost fseventsd[58]: bumping event counter to: 0x63657341783e7b64 (current 0x0) from log file '63657341783e735f'
Feb 14 07:52:44 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:44 localhost watchdogtimerd[43]: Automatic reboot timer enabled.
Feb 14 07:52:44 localhost kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:52:45 GatewayServer configd[29]: setting hostname to "GatewayServer.abcvideopost.local"
Feb 14 07:52:45 GatewayServer bootlog[66]: BOOT_TIME: 1266162757 0
Feb 14 07:52:45 GatewayServer com.apple.HeadlessStartup[57]: 65:6e:30:00:0d:93
Feb 14 07:52:46 GatewayServer org.net-snmp.snmpd[40]: found the platform plugin: RackMac3_1_PlatformPlugin
Feb 14 07:52:46 GatewayServer getty[41]: getty: unknown gettytab entry 'serial.57600'
Feb 14 07:52:47 GatewayServer emond[60]: SetUpLogs: uid = 0 gid = 0
Feb 14 07:52:47 GatewayServer emond[60]: SetUpLogs: opening /Library/Logs/EventMonitor/EventMonitor.error.log
Feb 14 07:52:47 GatewayServer rpc.statd[47]: statd.notify - no notifications needed
Feb 14 07:52:47 GatewayServer /System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow[52]: Login Window Application Started -- Threaded auth
Feb 14 07:52:48 GatewayServer kernel[0]: AppleBCM5701Ethernet - en1 link active, 1000-Mbit, full duplex, symmetric flow control enabled
Feb 14 07:52:48 GatewayServer xsand[42]: kern.coredump: 1 -> 1
Feb 14 07:52:48 GatewayServer xsand[42]: kern.corefile: '/cores/core.%P' -> '/cores/core.%N.%P'
Feb 14 07:52:48 GatewayServer xsand[42]: kern.ipc.maxsockbuf: 8388608 -> 16777216
Feb 14 07:52:48 GatewayServer com.apple.xsan[42]: kextload serialization lock busy; sleeping (89 retries left)
Feb 14 07:52:49 GatewayServer com.apple.xsan[42]: kextload: extension /System/Library/Extensions/acfs.kext is already loaded
Feb 14 07:52:49 GatewayServer com.apple.xsan[42]: kextload: extension /System/Library/Extensions/acfsctl.kext is already loaded
Feb 14 07:52:49 GatewayServer kernel[0]: Xsan Client Revision 3.1.0 Build 2 (339.24) Built for Darwin 9.0 ppc Created on Mon Oct 6 15:14:05 PDT 2008
Feb 14 07:52:49 GatewayServer org.ntp.ntpd[39]: Error : nodename nor servname provided, or not known
Feb 14 07:52:49 GatewayServer ntpdate[93]: can't find host time.apple.com
Feb 14 07:52:49 GatewayServer ntpdate[93]: no servers can be used, exiting
Feb 14 07:52:49 GatewayServer fsmpm[94]: Portmapper: ComputerInfo: computer_name = "Gateway Server", hostname = "Gateway-Server2"
Feb 14 07:52:49 GatewayServer fsmpm[94]: PortMapper: No CVFS Disk Volumes are accessible.
Feb 14 07:52:49 GatewayServer fsmpm[94]: Disk arb runloop starting
Feb 14 07:52:50 GatewayServer kernel[0]: FusionMPT: Notification = 6 (Rescan) for SCSI Domain = 1
Feb 14 07:52:50 GatewayServer fsmpm[94]: NSS: Coordinator 10.48.19.22 (id 10.48.19.22) coordinator list mismatch
Feb 14 07:52:50 GatewayServer fsmpm[94]: PortMapper: Local FSD client is registered, on port 49153.
Feb 14 07:52:51 GatewayServer kernel[0]: serialnumberd 109 FS_WRITE_DATA SBF /dev/dtracehelper 13 (seatbelt)
Feb 14 07:52:51 GatewayServer kernel[0]: serialnumberd 109 FS_READ_DATA SBF /dev/autofs_nowait 13 (seatbelt)
Feb 14 07:52:51 GatewayServer kernel[0]: serialnumberd 109 FS_READ_DATA SBF /usr/sbin 13 (seatbelt)
Feb 14 07:52:53 GatewayServer kextd[25]: writing kernel link data to /var/run/mach.sym
Feb 14 07:52:54 GatewayServer servermgrd[48]: servermgr_calendar: created default calendar virtual host
Feb 14 07:52:54 GatewayServer servermgrd[48]: servermgr_ipfilter:ipfw config:Notice:Flushed IPv4 rules
Feb 14 07:52:54 GatewayServer servermgrd[48]: servermgr_ipfilter:ipfw config:Notice:Flushed IPv6 rules
Feb 14 07:52:56 GatewayServer ARDAgent [124]: ********ARDAgent Launched********
Feb 14 07:52:56 GatewayServer ARDAgent [124]: ********ARDAgent Ready********
Feb 14 07:52:57 GatewayServer loginwindow[52]: Login Window Started Security Agent
Feb 14 07:53:05 GatewayServer kernel[0]: AppleBCM5701Ethernet - en0 link active, 100-Mbit, full duplex, flow control disabled
Feb 14 07:53:11 GatewayServer kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:53:15 GatewayServer kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:53:44 GatewayServer AppleVNCServer[128]: CGSCreateKeyboardEvent is obsolete; please use CGSCreateKeyboardEventOfLength
Feb 14 07:53:44 GatewayServer com.apple.RemoteDesktop.agent[124]: Sun Feb 14 07:53:44 GatewayServer.abcvideopost.local AppleVNCServer[128] : CGSCreateKeyboardEvent is obsolete; please use CGSCreateKeyboardEventOfLength
Feb 14 07:53:53 GatewayServer authorizationhost[130]: MechanismInvoke 0x12c130 retainCount 2
Feb 14 07:53:53 GatewayServer SecurityAgent[131]: MechanismInvoke 0x101680 retainCount 1
Feb 14 07:53:54 GatewayServer SecurityAgent[131]: NSSecureTextFieldCell detected a field editor ((null)) that is not a NSTextView subclass designed to work with the cell. Ignoring...
Feb 14 07:53:54 GatewayServer SecurityAgent[131]: NSExceptionHandler has recorded the following exception:\nNSRangeException -- *** -[NSCFArray objectAtIndex:]: index (0) beyond bounds (0)\nStack trace: 0x39dbc 0x901614ec 0x90c3acd0 0x90c3ad08 0x95209b7c 0x71168 0x5c74c 0x6f578 0x63fb4 0x68b08 0x77a88 0xe068 0x14200 0x13f64 0xdb58 0x95241274 0x90bab5fc 0x90bcd7f4 0x941f2bc8 0x941f29ec 0x941f282c 0x95f75728 0x95f750e0 0x95f6ed9c 0x11e14 0x2db0
Feb 14 07:53:54 GatewayServer loginwindow[52]: Login Window - Returned from Security Agent
Feb 14 07:53:54 GatewayServer SecurityAgent[131]: MechanismDestroy 0x101680 retainCount 1
Feb 14 07:53:54 GatewayServer authorizationhost[130]: MechanismDestroy 0x12c130 retainCount 2
Feb 14 07:53:54 GatewayServer loginwindow[52]: USER_PROCESS: 52 console
Feb 14 07:53:54 GatewayServer com.apple.launchd[1] (com.apple.UserEventAgent-LoginWindow[125]): Exited: Terminated
Feb 14 07:53:55 GatewayServer ARDAgent [160]: ********ARDAgent Launched********
Feb 14 07:53:55 GatewayServer mDNSResponder[31]: Client application registered 2 identical instances of service Gateway\032Server._net-assistant._udp.local. port 3283.
Feb 14 07:53:55 GatewayServer ARDAgent [160]: ********ARDAgent Ready********
Feb 14 07:53:58 GatewayServer kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:53:58 GatewayServer ARDAgent [160]: Exiting because bind error is not EADDRINUSE.
Feb 14 07:53:58 GatewayServer com.apple.launchd[151] (com.apple.RemoteDesktop.agent[160]): Stray process with PGID equal to this dead job: PID 165 PPID 1 AppleVNCServer
Feb 14 07:53:58 GatewayServer com.apple.launchd[151] (com.apple.RemoteDesktop.agent): Throttling respawn: Will start in 7 seconds
Feb 14 07:53:59 GatewayServer /System/Library/CoreServices/coreservicesd[75]: SFLSharePointsEntry::CreateDSRecord: dsCreateRecordAndOpen(iTX) returned -14135
Feb 14 07:54:03 GatewayServer SystemUIServer[175]: \n MenuCracker\n see http://sourceforge.net/projects/menucracker\n MenuCracker is now loaded. Ready to accept new menus. Ignore the failure message that follow.
Feb 14 07:54:03 GatewayServer SystemUIServer[175]: failed to load Menu Extra: NSBundle (loaded)
Feb 14 07:54:04 GatewayServer SystemUIServer[175]: MenuCracker: Loading 'MenuMeterCPUExtra'.
Feb 14 07:54:04 GatewayServer SystemUIServer[175]: MenuMeterCPU loaded.
Feb 14 07:54:04 GatewayServer SystemUIServer[175]: MenuCracker: Loading 'MenuMeterNetExtra'.
Feb 14 07:54:04 GatewayServer SystemUIServer[175]: MenuMeterNet loaded.
Feb 14 07:54:05 GatewayServer ARDAgent [195]: ********ARDAgent Launched********
Feb 14 07:54:05 GatewayServer ARDAgent [195]: ********ARDAgent Ready********
Feb 14 07:54:06 GatewayServer ARDAgent [195]: ServerNotificationReplyHandler: serverEntryRef is NULL
Feb 14 07:54:13 GatewayServer AppleVNCServer[196]: CGSCreateKeyboardEvent is obsolete; please use CGSCreateKeyboardEventOfLength
Feb 14 07:54:13 GatewayServer com.apple.RemoteDesktop.agent[195]: Sun Feb 14 07:54:13 GatewayServer.abcvideopost.local AppleVNCServer[196] : CGSCreateKeyboardEvent is obsolete; please use CGSCreateKeyboardEventOfLength
Feb 14 07:54:20 GatewayServer kernel[0]: FusionMPT: Notification = 9 (Logout) for SCSI Domain = 1
Feb 14 07:56:03 GatewayServer Unknown[31]: Client application bug: DNSServiceResolve(ABCcom\0323rd\032Floor._airport._tcp.local.) active for over two minutes. This places considerable burden on the network.
Feb 14 07:56:03 GatewayServer Unknown[31]: Client application bug: DNSServiceResolve(ABCcom\0322nd\032Floor._airport._tcp.local.) active for over two minutes. This places considerable burden on the network.
Feb 14 07:58:45 GatewayServer ntpd[39]: time reset -0.185629 s
Feb 14 10:11:38 GatewayServer login[905]: USER_PROCESS: 905 ttys000
Feb 14 10:12:41 GatewayServer login[905]: DEAD_PROCESS: 905 ttys000/code

automount:

[code]<?xml version="1.0" encoding="UTF-8"?>

DataDrive

AutoMount
rw
MountOptions

Soapnet

AutoMount
no
MountOptions

iTX

AutoMount
no
MountOptions

xSAN

AutoMount
no
MountOptions

/code

How do I watch via "service portmap status"??

Thank you!

lotte's picture

So, your fsmlist file is wrong, it should look like for the primary mdc :

iTX 10.48.19.21 0
Soapnet 10.48.19.21 0
xSAN 10.48.19.21 0
SNFS_HA 10.48.19.21 0
DataDrive 10.48.19.21 0

and for the secondary mdc:

iTX 10.48.19.22 1
Soapnet 10.48.19.22 1
xSAN 10.48.19.22 1
SNFS_HA 10.48.19.22 1
DataDrive 10.48.19.22 1

Extract from /usr/cvfs/examples/fsmlist:

  1. is the name of the FSM file system. This must match
  2. the configuration's file name, (.cfg).
  3. This name is Storage Area Network (SAN) wide and must be
  4. unique across all the interconnected SAN machines.

#

  1. [.] is required if the field is specified. It takes the
  2. place of a deprecated parameter and is required for compatibility
  3. with old fsmlist files.

#

  1. [] is a number assigned to FSS services to give more
  2. determinism to which service can take over a failed service.
  3. The lower the value the higher the priority. If there is no priority
  4. assigned to a service, it defaults to priority 0.
  5. FSS Services with equal priority will not have deterministic fail
  6. over characteristics.

#

So you may also use a dot instead of the ip as I suggest!

Also I wonder about:

Feb 14 07:52:48 GatewayServer com.apple.xsan[42]: kextload serialization lock busy; sleeping (89 retries left)

and

Feb 14 07:52:49 GatewayServer fsmpm[94]: PortMapper: No CVFS Disk Volumes are accessible.

What´s the output of "cvlabel -l" run in terminal as root?

But the main thing why not mounting could be the firewall you´re using, give it a try and disable it, reboot again and let us know what happens.

From your automount.plist file I see that you only want to use the DataDrive Volume on that particular host. Otherwise replace

AutoMount
no

with

AutoMount
rw

for the Volumes you want to mount...

"service portmap status" should print the running or not running status of that service (Linux only). What linux are you using, what´s the output of "more /etc/*elease*"

Lotte

vidiot's picture

Ok I'll look at modifying the fsmlist. Honestly, this is how Quantum set it up so I'm a little hesitant to modify anything on the MDC unless of course that is the culprit.

Output of cvlabel -l from the client:

[code]Last login: Sun Feb 14 15:56:45 on console
GatewayServer:~ macadmin$ sudo cvlabel -l
Password:
/dev/rdisk42 [APPLE Xserve RAID 1.51] acfs "RAID1RIGHT1" Sectors: 490190848. SectorSize: 512. Maximum sectors: 490207199.
/dev/rdisk2 [APPLE Xserve RAID 1.51] acfs "RAID16LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk3 [APPLE Xserve RAID 1.51] acfs "RAID16RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk4 [APPLE Xserve RAID 1.51] acfs "RAID15RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk5 [APPLE Xserve RAID 1.51] acfs "RAID20LEFT" Sectors: 490190848. SectorSize: 512. Maximum sectors: 490207199.
/dev/rdisk6 [APPLE Xserve RAID 1.51] acfs "RAID6LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk7 [APPLE Xserve RAID 1.51] acfs "RAID19LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk8 [APPLE Xserve RAID 1.51] acfs "RAID20RIGHT_SNFSHA" Sectors: 976713728. SectorSize: 512. Maximum sectors: 976730079.
/dev/rdisk9 [APPLE Xserve RAID 1.51] acfs "RAID17RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk10 [APPLE Xserve RAID 1.51] acfs "RAID21LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk11 [APPLE Xserve RAID 1.51] acfs "RAID14RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk12 [APPLE Xserve RAID 1.51] acfs "RAID17LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk13 [APPLE Xserve RAID 1.51] acfs "RAID19RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk14 [APPLE Xserve RAID 1.51] acfs "RAID18LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk15 [APPLE Xserve RAID 1.51] acfs "RAID14LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk16 [APPLE Xserve RAID 1.51] acfs "RAID13RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk17 [APPLE Xserve RAID 1.51] acfs "RAID12LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk18 [APPLE Xserve RAID 1.51] acfs "RAID18RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk19 [APPLE Xserve RAID 1.51] acfs "RAID11LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk20 [APPLE Xserve RAID 1.51] acfs "RAID13LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk21 [APPLE Xserve RAID 1.51] acfs "RAID10RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk22 [APPLE Xserve RAID 1.51] acfs "RAID12RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk23 [APPLE Xserve RAID 1.51] acfs "RAID21RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk24 [APPLE Xserve RAID 1.51] acfs "RAID6RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk25 [APPLE Xserve RAID 1.51] acfs "RAID5RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk26 [APPLE Xserve RAID 1.51] acfs "RAID10LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk27 [APPLE Xserve RAID 1.51] acfs "RAID7RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk28 [APPLE Xserve RAID 1.51] acfs "RAID7LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk29 [APPLE Xserve RAID 1.51] acfs "RAID9RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk30 [APPLE Xserve RAID 1.51] acfs "RAID9LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk31 [APPLE Xserve RAID 1.51] acfs "RAID11RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk32 [APPLE Xserve RAID 1.51] acfs "RAID5LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk33 [APPLE Xserve RAID 1.51] acfs "RAID8RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk34 [APPLE Xserve RAID 1.51] acfs "RAID4LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk35 [APPLE Xserve RAID 1.51] acfs "RAID4RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk36 [APPLE Xserve RAID 1.51] acfs "RAID8LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk37 [APPLE Xserve RAID 1.51] acfs "RAID3LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk38 [APPLE Xserve RAID 1.51] acfs "RAID3RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk39 [APPLE Xserve RAID 1.51] acfs "RAID2RIGHT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk40 [APPLE Xserve RAID 1.51] acfs "RAID2LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
/dev/rdisk41 [APPLE Xserve RAID 1.51] acfs "RAID1LEFT" Sectors: 490190848. SectorSize: 512. Maximum sectors: 490207199.
/dev/rdisk1 [APPLE Xserve RAID 1.51] acfs "RAID15LEFT" Sectors: 4883789791. SectorSize: 512. Maximum sectors: 4883789791.
GatewayServer:~ macadmin$
/code

I don't see any firewall running, at least not in the GUI of this Mac. It is Mac OS X server 10.5.6. It isn't listed under System Preferences/Security like a standard Mac. So I ran this command to disable firewall and restarted the computer: [code]sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 0/code

After a restart, same issue persists.

Yes the automount is the way we want it. Only 1 volume is supposed to mount currently.

Linux: Red Hat Enterprise Linux Server release 5.1 (Tikanga)

Thanks!

lotte's picture

Your fsmlist file correct for a single server without failover, but not if you have two mdc´s controlling the san.
So it seems that all disks can be seen, can you print again the output of /var/log/system.log while running these commands:

launchctl unload /System/Library/LaunchDeamons/com.apple.xsan.plist

and then

launchctl load /System/Library/LaunchDeamons/com.apple.xsan.plist

hop to come closer....

Lotte<

vidiot's picture

Right now the other MDC is down, either related or unrelated to this problem. So the fsmlst is correct for the moment.

I ran the commands, here is the output:
[code]Feb 15 11:11:32 GatewayServer com.apple.launchd[158] (0x1090a0.Locum[6241]): Exited: Terminated
Feb 15 11:12:43 GatewayServer sudo[6259]: macadmin : TTY=ttys000 ; PWD=/Users/macadmin ; USER=root ; COMMAND=/bin/launchctl unload /System/Library/LaunchDeamons/com.apple.xsan.plist
Feb 15 11:12:55 GatewayServer sudo[6268]: macadmin : TTY=ttys000 ; PWD=/Users/macadmin ; USER=root ; COMMAND=/bin/launchctl load /System/Library/LaunchDeamons/com.apple.xsan.plist
Feb 15 11:13:35 GatewayServer sudo[6271]: macadmin : TTY=ttys000 ; PWD=/Users/macadmin ; USER=root ; COMMAND=/usr/sbin/cvlabel -l
Feb 15 11:14:26 GatewayServer sudo[6278]: macadmin : TTY=ttys000 ; PWD=/Users/macadmin ; USER=root ; COMMAND=/bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xsan.plist
Feb 15 11:18:26 GatewayServer com.apple.launchd[1] (com.apple.xsan[42]): Exit timeout elapsed (240 seconds). Killing.
Feb 15 11:18:26 GatewayServer com.apple.launchd[1] (com.apple.xsan[42]): Stray process with PGID equal to this dead job: PID 95 PPID 94 mount_acfs
Feb 15 11:18:26 GatewayServer com.apple.launchd[1] (com.apple.xsan[42]): Stray process with PGID equal to this dead job: PID 94 PPID 1 mount
Feb 15 11:18:26 GatewayServer com.apple.launchd[1] (com.apple.xsan[42]): Stray process with PGID equal to this dead job: PID 92 PPID 1 fsmpm
Feb 15 11:18:26 GatewayServer com.apple.launchd[1] (com.apple.xsan[42]): Exited: Killed
Feb 15 11:18:26 GatewayServer kernel[0]: Reconnecting to local portmapper on host '127.0.0.1'
Feb 15 11:18:26 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:20:06 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:21:46 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:23:26 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:25:06 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:26:46 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:28:26 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 11:28:26 GatewayServer sudo[6353]: macadmin : TTY=ttys000 ; PWD=/Users/macadmin ; USER=root ; COMMAND=/bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xsan.plist
Feb 15 11:28:26 GatewayServer xsand[6355]: kern.coredump: 1 -> 1
Feb 15 11:28:26 GatewayServer xsand[6355]: kern.corefile: '/cores/core.%N.%P' -> '/cores/core.%N.%P'
Feb 15 11:28:26 GatewayServer xsand[6355]: kern.ipc.maxsockbuf: 16777216 -> 16777216
Feb 15 11:28:26 GatewayServer com.apple.xsan[6355]: kextload: extension /System/Library/Extensions/acfs.kext is already loaded
Feb 15 11:28:27 GatewayServer com.apple.xsan[6355]: kextload: extension /System/Library/Extensions/acfsctl.kext is already loaded
Feb 15 11:28:27 GatewayServer fsmpm[6363]: Portmapper: ComputerInfo: computer_name = "Gateway Server", hostname = "Gateway-Server2"
Feb 15 11:28:27 GatewayServer com.apple.launchd[1] (com.apple.seatbelt.compilerd[6365]): Bug: launchd_core_logic.c:6703 (23714):0: mspolicy_new(target_j, target_service, flags & BOOTSTRAP_ALLOW_LOOKUP, flags & BOOTSTRAP_PER_PID_SERVICE, false)
Feb 15 11:28:29: --- last message repeated 1 time ---
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID1LEFT on device: /dev/rdisk42 (blk 0xe00002c raw 0xe00002c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000014F2C01000000D6CD1CCD' Size: 490190848 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID16LEFT on device: /dev/rdisk2 (blk 0xe000004 raw 0xe000004) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052701000000D52BF968' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID16RIGHT on device: /dev/rdisk3 (blk 0xe000005 raw 0xe000005) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002051501000000D52BE265' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID15RIGHT on device: /dev/rdisk4 (blk 0xe000006 raw 0xe000006) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052201000000D52C0CC0' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID19LEFT on device: /dev/rdisk5 (blk 0xe000007 raw 0xe000007) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002044D01000000D52B2785' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID20RIGHT_SNFSHA on device: /dev/rdisk6 (blk 0xe000008 raw 0xe000008) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002040F01000000D52C2F35' Size: 976713728 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID6LEFT on device: /dev/rdisk7 (blk 0xe000009 raw 0xe000009) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002045201000000D52AFE09' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID20LEFT on device: /dev/rdisk8 (blk 0xe00000a raw 0xe00000a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002046A01000000D52BAD55' Size: 490190848 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID17LEFT on device: /dev/rdisk9 (blk 0xe00000b raw 0xe00000b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202D401000000D52BD767' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID21LEFT on device: /dev/rdisk10 (blk 0xe00000c raw 0xe00000c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002042301000000D52B65B5' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID17RIGHT on device: /dev/rdisk11 (blk 0xe00000d raw 0xe00000d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202F501000000D52BC06B' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID14RIGHT on device: /dev/rdisk12 (blk 0xe00000e raw 0xe00000e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000201F101000000D52C456B' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer kernel[0]: Local portmapper OK
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID14LEFT on device: /dev/rdisk13 (blk 0xe00000f raw 0xe00000f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002019B01000000D52C5C78' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID19RIGHT on device: /dev/rdisk14 (blk 0xe000010 raw 0xe000010) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001FBC501000000D52B102E' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID18RIGHT on device: /dev/rdisk15 (blk 0xe000011 raw 0xe000011) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F7D401000000D52B4774' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID18LEFT on device: /dev/rdisk16 (blk 0xe000012 raw 0xe000012) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F6F001000000D52B87FC' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID12LEFT on device: /dev/rdisk17 (blk 0xe000013 raw 0xe000013) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F801000000D52B03A5' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID11LEFT on device: /dev/rdisk18 (blk 0xe000014 raw 0xe000014) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F101000000D52AF32F' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID13LEFT on device: /dev/rdisk19 (blk 0xe000015 raw 0xe000015) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0E701000000D52B2059' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID13RIGHT on device: /dev/rdisk20 (blk 0xe000016 raw 0xe000016) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E10501000000D52B095E' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID11RIGHT on device: /dev/rdisk21 (blk 0xe000017 raw 0xe000017) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E06801000000D52ADBE2' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID10LEFT on device: /dev/rdisk22 (blk 0xe000018 raw 0xe000018) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E00E01000000D52BB71C' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID7RIGHT on device: /dev/rdisk23 (blk 0xe000019 raw 0xe000019) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A2A101000000D52AD5DC' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID12RIGHT on device: /dev/rdisk24 (blk 0xe00001a raw 0xe00001a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0DD01000000D52AED74' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID7LEFT on device: /dev/rdisk25 (blk 0xe00001b raw 0xe00001b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A28701000000D52AED58' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID10RIGHT on device: /dev/rdisk26 (blk 0xe00001c raw 0xe00001c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E08A01000000D52BA014' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID5LEFT on device: /dev/rdisk27 (blk 0xe00001d raw 0xe00001d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04101000000D52C17CE' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID6RIGHT on device: /dev/rdisk28 (blk 0xe00001e raw 0xe00001e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A08E01000000D52AEC09' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID5RIGHT on device: /dev/rdisk29 (blk 0xe00001f raw 0xe00001f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A07501000000D52C043F' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID9RIGHT on device: /dev/rdisk30 (blk 0xe000020 raw 0xe000020) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04A01000000D52AF7FF' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID21RIGHT on device: /dev/rdisk31 (blk 0xe000021 raw 0xe000021) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A0A201000000D52B4EFC' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID8RIGHT on device: /dev/rdisk32 (blk 0xe000022 raw 0xe000022) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A01201000000D52B1731' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID4LEFT on device: /dev/rdisk33 (blk 0xe000023 raw 0xe000023) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A00801000000D52C0F9F' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID4RIGHT on device: /dev/rdisk34 (blk 0xe000024 raw 0xe000024) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FB601000000D52BFB5C' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID9LEFT on device: /dev/rdisk35 (blk 0xe000025 raw 0xe000025) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A05601000000D52B0F6C' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID8LEFT on device: /dev/rdisk36 (blk 0xe000026 raw 0xe000026) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FAB01000000D52B2EBA' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID3LEFT on device: /dev/rdisk37 (blk 0xe000027 raw 0xe000027) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001664401000000D52B7BC1' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID1RIGHT1 on device: /dev/rdisk38 (blk 0xe000028 raw 0xe000028) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001436501000000D52B995D' Size: 490190848 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID3RIGHT on device: /dev/rdisk39 (blk 0xe000029 raw 0xe000029) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165F501000000D52B7895' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID2RIGHT on device: /dev/rdisk40 (blk 0xe00002a raw 0xe00002a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165AD01000000D52B3D88' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID2LEFT on device: /dev/rdisk41 (blk 0xe00002b raw 0xe00002b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165A001000000D52B4FAD' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: CVFS Volume RAID15LEFT on device: /dev/rdisk1 (blk 0xe000003 raw 0xe000003) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002053701000000D52C2411' Size: 4883789791 Sector Size: 512
Feb 15 11:28:29 GatewayServer fsmpm[6363]: Disk arb runloop starting
Feb 15 11:28:29 GatewayServer fsmpm[6363]: PortMapper: Local FSD client is registered, on port 55462.
Feb 15 11:28:29 GatewayServer fsmpm[6363]: NSS: Coordinator 192.168.60.22 (id 10.48.19.22) coordinator list mismatch
Feb 15 11:28:30 GatewayServer xsand[6355]: mkdir '/Volumes/DataDrive': File exists
Feb 15 11:29:00: --- last message repeated 2 times ---
Feb 15 11:29:11 GatewayServer xsand[6355]: mkdir '/Volumes/DataDrive': File exists
Feb 15 11:29:17 GatewayServer login[6372]: USER_PROCESS: 6372 ttys001
Feb 15 11:29:18 GatewayServer login[6372]: DEAD_PROCESS: 6372 ttys001
Feb 15 11:29:26 GatewayServer xsand[6355]: mkdir '/Volumes/DataDrive': File exists
Feb 15 11:29:31 GatewayServer com.apple.launchd[158] (0x1090e0.Locum[6382]): Exited: Terminated
/code

lotte's picture

Hi Vidiot, I still assume that it has something to do with the existing /Volume/DataDrive, can you do:

launchctl unload /System/Library/LaunchDeamons/com.apple.xsan.plist

then, when finished:

ls -al /Volumes/DataDrive/

and if /Volumes/DataDrive/ should exists and is [b]completly empty/b and only then!!! do a

rm -r /Volumes/DataDrive/

then again start

launchctl load /System/Library/LaunchDeamons/com.apple.xsan.plist

and post the output again...

Lotte

vidiot's picture

After stopping Xsan, this is the output of ls -al

[code]GatewayServer:log macadmin$ sudo ls -al /Volumes/DataDrive/
total 0
d--x--x--x 2 root admin 68 Feb 15 11:29 .
drwxrwxrwt@ 5 root admin 170 Feb 15 11:29 ..
/code

Would you say it is safe to assume DataDrive is an empty folder?

Thanks!

lotte's picture

Funny is, it´s neither read or writeable... it just has the execution flag... This on the other hand could prevent from reading it´s content... If it´s possible shutdown the machine, remove the fibrecables and/or the metadata network ethernet cable, power on again and then remove the directory... shutdown again, plug the cables back in and let´s see what happens...

Lotte

vidiot's picture

I'm not at the office today, I'm doing this remotely so I don't have physical access until Wednesday.

I can certainly disable the metadata network port, reboot, remove the directory then reboot again. But I wouldn't have access to the fibre cable physically to pull it out.

lotte's picture

That should be fine, again do

launchctl unload /System/Library/LaunchDeamons/com.apple.xsan.plist

then deactivate the metadatanetwork just to make sure...
remove the /Volumes/DataDrive directory, enable the metadatanetwork and

launchctl load /System/Library/LaunchDeamons/com.apple.xsan.plist

Lotte

vidiot's picture

Ok I've done this, re-enabled the metadata network and started Xsan again:

[code]Feb 15 13:49:54 GatewayServer sudo[7194]: macadmin : TTY=ttys000 ; PWD=/private/var/log ; USER=root ; COMMAND=/bin/rm -r /Volumes/DataDrive/
Feb 15 13:50:40 GatewayServer kernel[0]: AppleBCM5701Ethernet: 0 4 setupCopperPhy - link is down
Feb 15 13:50:42 GatewayServer kernel[0]: AppleBCM5701Ethernet - en1 link active, 1000-Mbit, full duplex, symmetric flow control enabled
Feb 15 13:50:44 GatewayServer ARDAgent [198]: ServerNotificationReplyHandler: serverEntryRef is NULL
Feb 15 13:50:44 GatewayServer SCHelper[7222]: no command
Feb 15 13:50:44 GatewayServer com.apple.launchd[158] ([0x0-0x36036].com.apple.systempreferences[7200]): Stray process with PGID equal to this dead job: PID 7222 PPID 1 SCHelper
Feb 15 13:50:55 GatewayServer kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Feb 15 13:50:58 GatewayServer sudo[7239]: macadmin : TTY=ttys000 ; PWD=/private/var/log ; USER=root ; COMMAND=/bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xsan.plist
Feb 15 13:50:58 GatewayServer xsand[7240]: kern.coredump: 1 -> 1
Feb 15 13:50:58 GatewayServer xsand[7240]: kern.corefile: '/cores/core.%N.%P' -> '/cores/core.%N.%P'
Feb 15 13:50:58 GatewayServer xsand[7240]: kern.ipc.maxsockbuf: 16777216 -> 16777216
Feb 15 13:50:58 GatewayServer com.apple.xsan[7240]: kextload: extension /System/Library/Extensions/acfs.kext is already loaded
Feb 15 13:50:58 GatewayServer com.apple.xsan[7240]: kextload: extension /System/Library/Extensions/acfsctl.kext is already loaded
Feb 15 13:50:58 GatewayServer fsmpm[7243]: Portmapper: ComputerInfo: computer_name = "Gateway Server", hostname = "Gateway-Server2"
Feb 15 13:50:58 GatewayServer com.apple.launchd[1] (com.apple.seatbelt.compilerd[7245]): Bug: launchd_core_logic.c:6703 (23714):0: mspolicy_new(target_j, target_service, flags & BOOTSTRAP_ALLOW_LOOKUP, flags & BOOTSTRAP_PER_PID_SERVICE, false)
Feb 15 13:51:00: --- last message repeated 1 time ---
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID1LEFT on device: /dev/rdisk42 (blk 0xe00002c raw 0xe00002c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000014F2C01000000D6CD1CCD' Size: 490190848 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID16LEFT on device: /dev/rdisk2 (blk 0xe000004 raw 0xe000004) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052701000000D52BF968' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID16RIGHT on device: /dev/rdisk3 (blk 0xe000005 raw 0xe000005) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002051501000000D52BE265' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID15RIGHT on device: /dev/rdisk4 (blk 0xe000006 raw 0xe000006) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002052201000000D52C0CC0' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID19LEFT on device: /dev/rdisk5 (blk 0xe000007 raw 0xe000007) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002044D01000000D52B2785' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID20RIGHT_SNFSHA on device: /dev/rdisk6 (blk 0xe000008 raw 0xe000008) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002040F01000000D52C2F35' Size: 976713728 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID6LEFT on device: /dev/rdisk7 (blk 0xe000009 raw 0xe000009) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002045201000000D52AFE09' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID20LEFT on device: /dev/rdisk8 (blk 0xe00000a raw 0xe00000a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002046A01000000D52BAD55' Size: 490190848 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID17LEFT on device: /dev/rdisk9 (blk 0xe00000b raw 0xe00000b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202D401000000D52BD767' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID21LEFT on device: /dev/rdisk10 (blk 0xe00000c raw 0xe00000c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002042301000000D52B65B5' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID17RIGHT on device: /dev/rdisk11 (blk 0xe00000d raw 0xe00000d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000202F501000000D52BC06B' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID14RIGHT on device: /dev/rdisk12 (blk 0xe00000e raw 0xe00000e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000201F101000000D52C456B' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID14LEFT on device: /dev/rdisk13 (blk 0xe00000f raw 0xe00000f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002019B01000000D52C5C78' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID19RIGHT on device: /dev/rdisk14 (blk 0xe000010 raw 0xe000010) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001FBC501000000D52B102E' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID18RIGHT on device: /dev/rdisk15 (blk 0xe000011 raw 0xe000011) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F7D401000000D52B4774' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID18LEFT on device: /dev/rdisk16 (blk 0xe000012 raw 0xe000012) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001F6F001000000D52B87FC' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID12LEFT on device: /dev/rdisk17 (blk 0xe000013 raw 0xe000013) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F801000000D52B03A5' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID11LEFT on device: /dev/rdisk18 (blk 0xe000014 raw 0xe000014) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0F101000000D52AF32F' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID13LEFT on device: /dev/rdisk19 (blk 0xe000015 raw 0xe000015) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0E701000000D52B2059' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID13RIGHT on device: /dev/rdisk20 (blk 0xe000016 raw 0xe000016) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E10501000000D52B095E' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID11RIGHT on device: /dev/rdisk21 (blk 0xe000017 raw 0xe000017) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E06801000000D52ADBE2' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID10LEFT on device: /dev/rdisk22 (blk 0xe000018 raw 0xe000018) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E00E01000000D52BB71C' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID7RIGHT on device: /dev/rdisk23 (blk 0xe000019 raw 0xe000019) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A2A101000000D52AD5DC' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID12RIGHT on device: /dev/rdisk24 (blk 0xe00001a raw 0xe00001a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E0DD01000000D52AED74' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID7LEFT on device: /dev/rdisk25 (blk 0xe00001b raw 0xe00001b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A28701000000D52AED58' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer kernel[0]: Local portmapper OK
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID10RIGHT on device: /dev/rdisk26 (blk 0xe00001c raw 0xe00001c) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001E08A01000000D52BA014' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID5LEFT on device: /dev/rdisk27 (blk 0xe00001d raw 0xe00001d) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04101000000D52C17CE' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID6RIGHT on device: /dev/rdisk28 (blk 0xe00001e raw 0xe00001e) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A08E01000000D52AEC09' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID5RIGHT on device: /dev/rdisk29 (blk 0xe00001f raw 0xe00001f) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A07501000000D52C043F' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID9RIGHT on device: /dev/rdisk30 (blk 0xe000020 raw 0xe000020) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A04A01000000D52AF7FF' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID21RIGHT on device: /dev/rdisk31 (blk 0xe000021 raw 0xe000021) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A0A201000000D52B4EFC' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID8RIGHT on device: /dev/rdisk32 (blk 0xe000022 raw 0xe000022) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A01201000000D52B1731' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID4LEFT on device: /dev/rdisk33 (blk 0xe000023 raw 0xe000023) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A00801000000D52C0F9F' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID4RIGHT on device: /dev/rdisk34 (blk 0xe000024 raw 0xe000024) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FB601000000D52BFB5C' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID9LEFT on device: /dev/rdisk35 (blk 0xe000025 raw 0xe000025) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001A05601000000D52B0F6C' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID8LEFT on device: /dev/rdisk36 (blk 0xe000026 raw 0xe000026) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '6000393000019FAB01000000D52B2EBA' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID3LEFT on device: /dev/rdisk37 (blk 0xe000027 raw 0xe000027) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001664401000000D52B7BC1' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID1RIGHT1 on device: /dev/rdisk38 (blk 0xe000028 raw 0xe000028) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300001436501000000D52B995D' Size: 490190848 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID3RIGHT on device: /dev/rdisk39 (blk 0xe000029 raw 0xe000029) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165F501000000D52B7895' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID2RIGHT on device: /dev/rdisk40 (blk 0xe00002a raw 0xe00002a) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165AD01000000D52B3D88' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID2LEFT on device: /dev/rdisk41 (blk 0xe00002b raw 0xe00002b) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '60003930000165A001000000D52B4FAD' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: CVFS Volume RAID15LEFT on device: /dev/rdisk1 (blk 0xe000003 raw 0xe000003) con: 2 lun: 0 state: 0xf4 inquiry [APPLE Xserve RAID 1.51] controller # 'default' serial # '600039300002053701000000D52C2411' Size: 4883789791 Sector Size: 512
Feb 15 13:51:00 GatewayServer fsmpm[7243]: Disk arb runloop starting
Feb 15 13:51:00 GatewayServer fsmpm[7243]: PortMapper: Local FSD client is registered, on port 58160.
Feb 15 13:51:00 GatewayServer fsmpm[7243]: NSS: Coordinator 192.168.60.22 (id 10.48.19.22) coordinator list mismatch/code

And again, I the executable only folder "DataDrive" has returned.

lotte's picture

This will become the longest forum message ever... :evil:

Can you send me a personal message with your skype id or email adress...

Lotte

d4corp's picture

I don't know about how to resolve your issue, and I don't know Xsan 1,

But your fsmlist is correct. Check manual page
http://creative-storage.info/Xsan-2.2.1-man/man4/fsmlist.4.html

on my system, there's only one entry which is
[code]SANVolume . 0/code

This config file should be on only on the MDCs.
/code

vidiot's picture

So with the help of Lotte, we discovered the issue. I got hit hard with the same serial number bug as reported here: [url]http://www.xsanity.com/forum/viewtopic.php?t=6663&sid=02a0033bbd4c91cf7e.../url

Even though I'm running Xsan 2.1.1, I upgraded over 20 of our Xserves to OS X Server 10.5.8 v1.1. That caused all my other Macs to fail to run the serial number daemon thinking it was under attack:

[code]Wed Feb 17 08:49:04 2010: LOGNOTE: --- Serial Number Support Daemon Started.
Wed Feb 17 08:49:04 2010: LOGNOTE: SN_UnRegister() returned 1
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder15.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder14.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder13.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder17.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encodingmaster.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder16.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = Encoder-8.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder19.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = temp.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = Encoder-9.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = Neal-Mac-Pro.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = compressor.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = Encoder-10.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder18.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder19.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder20.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder22.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encodingmaster.local]
Wed Feb 17 08:49:11 2010: LOGERR: RESPONSE with BAD replyhash! [tag = xsan, rhostname = encoder23.local]
Wed Feb 17 08:49:13 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:16 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:20 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:23 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:26 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:29 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:32 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:35 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:38 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:41 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:44 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:47 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:50 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:53 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:56 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:49:59 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:50:02 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:50:05 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:50:08 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:50:11 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]
Wed Feb 17 08:50:14 2010: LOGWARN: : Too many unexpected messages received. Could be a SPAM attack. Sleeping for 3 seconds. [C]/code

The LOGWARN message continues forever...

So while I know what the problem is now (thanks Lotte!!!), I don't yet have a solution. The only temporary solution I have is to shut down all the 10.5.8 v1.1 servers, boot up all the Macs that have issue, then boot the 10.5.8 v1.1 servers. This at least gets me past the Serial Daemon.

The above is hardly a permanent solution of course. Once I have a permanent solution I'll report back.

Thanks!

polemanewru's picture

We experiencing this problem too. Downgrade clients/mdc's to 10.5.7 resolves the issue - but the cause remains undiagnosed.

dido_'s picture

polemanewru wrote:
We experiencing this problem too. Downgrade clients/mdc's to 10.5.7 resolves the issue - but the cause remains undiagnosed./quote

Hi

I have similar problem and:
- i have downgrade most (15 of 18) MacOSX Server to 10.5.7 (3 are 10.5.8)
- MacPro (with MacOSX) can be 10.5.7 and 10.5.8 (but not 10.6.x)

... still try to find solution :/

abstractrude's picture

i have been running into this issue as well. restarting the clients usually fixes the issue. but my serial number logs are full of the same errors. well when i say fixing i mean the volume will mount.

-Trevor Carlson
THUMBWAR

dido_'s picture

abstractrude wrote:
i have been running into this issue as well. restarting the clients usually fixes the issue. but my serial number logs are full of the same errors. well when i say fixing i mean the volume will mount./quote

You don`t need to restart clients, just kill serialnumberd process and wait until it`s start (and try to automonut), if not mount kill it again until its mount.

If You have over 10 MacOSX Server (10.5.8 or 10.6) and MacOSX 10.6 they "attack" serialnumberd very fast and demon going sleep to fast (and cannot register seriall for xsan - and mount volume)

Best Regards

abstractrude's picture

yeah but for an editor, trust me its easier to say restart. do we know what port it attacks on? is it 626. Maybe i can put some access lists on the switch to prevent the talk... has anybody looked at that as a solution?

-Trevor Carlson
THUMBWAR

vidiot's picture

We were finally able to upgrade some clients to 10.6.3 and Xsan 2.2.1 (previously 10.5.8 and Xsan 2.1.1). The issue appears to now be resolved. We can reboot at will.

Thanks again Lotte!

cedge318's picture

Just curious, did 10.5.8 v1.1 not fix this as?

morphenine's picture

10.5.8 v1.1 was SUPPOSED to have fixed the serial number problem. It did not fix it completely. Something in 10.5.8 introduced this bug that seems to affect all servers 10.4.x-10.6.x in this way.

In our environment we have seen this bug randomly pop up for several hours and then disappear. We've also seen it show up and stay for several weeks.

A couple of workarounds are available, but as of yet, there is no FIX (despite apple's claims of v1.1)
One of the workarounds we used involved unplugging the main ethernet port during startup.

Another involved blocking port 626, which is tricky; The serial number daemon adds rule #1 to the ipfw to allow all traffic on 626. You have to remove that rule, add a deny rule, and kill the daemon so it restarts and doesn't end up in an infinite spamming loop.

All in all this is a pain in the patoot! And I hope Apple gets on the ball, although their solution will probably just be: "upgrade to snow leopard."