Xsan not starting after adding storage

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

I added storage to my Xsan this am and now I cant get it started...I get this error in terminal:

State: BLOCKED (configuration file problem) 2012-07-21 09:44:49
Last Admin: START 2012-07-21 09:44:48
Last Termination: exit(19) 2012-07-21 09:44:49
Launches 5, core dumps 0, flags <CONFIGURED,AUTO-START>

FSS 'XSAN01' start unsuccessful. Check event log for more details.

Any advice would be greatly appreciated.

ogminlo · Xsan Master Joined: 29 May 2008 Posts: 149

We need more detail. By "added storage" do you mean plugging in new LUNs or successfully configuring them as a new storage pool? What hardware are you using and what version of Xsan? What do your logs say when you try to start?

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

I knew you'd need more info, I was just pretty frustrated at the time and since I backed up all the config files I just went back to my old configuration and everything works fine....just no additional storage.
We are running Xsan 2.2.3. We have 6 total storage pools..4 DATA, 1 MISC, and 1 Metadata. All the hardware is composed of Promise Vtrak E & J chassis ranging from 750GB to 2TB in size. Last year I created the 4th pool and added it myself with no issues. It is a Promise E, and I was trying to add the J to that this year. In Xsan Admin it was adding the J as a 5th separate DATA pool. It shows up in Xsan admin correctly as DATA15 and DATA16 just as I labeled them (I divided the vtrak in 2 adding 2 global spares, as is the setup of all the other E & J drives). I try to add the storage in Admin and it goes through the montions that its stopping adding and restarting, but it just doesn't start. I went into terminal ran cvadmin and tried to start it and got the error message included in my earlier post. the event log doesn't reveal much because it picks up after I reverted back to my old configuration. I plan on trying once again tomorrow AM and I'll post the results of the event log tomorrow if i'm unsuccessful.

Thank You

singlemalt · Xsan Master Joined: 27 Feb 2009 Posts: 109

How many active ethernet ports are there on the mdcs? I saw some thing similar a couple of years ago where there was three or four ( can't remember
how many) active ports. One for public, one for meta data, and another one
for monitoring or file services or some such. We could not add another pool
until that third/fourth e.net port was deactivated.

Might not be a bad idea to remove all clients and controllers from the SAN
accept for one meta data controller. See if it will expand then. Then add 'em
back afterwards.

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

Update...Now when I try and add the LUNs to the existing data pool I get LESS storage than I currently have...The DATA pool is currently 84 TB and when I add the 2 additional LUNs (which are 24 TB total) it drops down to 78 TB with 108 TB in a smaller red font with a line through it next to the 78 TB.

Also there are only 2 active ports on each MDC...standard setup, nothing unusual.

singlemalt · Xsan Master Joined: 27 Feb 2009 Posts: 109

Normally you seeing the red font line through the luns storage because
the existing luns are smaller than the luns you're adding. Just like a RAID array,
Storage pools can only have luns of the same size. So if a lun(s) is bigger than some other
lun, the larger lun gets truncated to the size of the smallest lun in the array.

Probably not a good idea to add storage to existing pools. See this thread from earlier this month.
http://www.xsanity.com/forum/viewtopic.php?t=16593

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

But I'm chaining J storage on to an existing E chassis thats configured exactly the same as the E. They both contain 2TB drives...why would I LOSE storage?

Thanks for the responses, I appreciate it.

singlemalt · Xsan Master Joined: 27 Feb 2009 Posts: 109

Post a screen cap.

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

So I went back into Xsan Admin and tried to go it again for the screengrabs, and now it seems to work fine...

But I cant continue the process because everyone is working. I'll have to attempt to add later this week.

Thank you again.

huntson · Could work for Apple Joined: 27 Mar 2012 Posts: 41

is it possible the new raid is not configured properly and doesn't have the same raid level?

abstractrude · Xsan Master Joined: 13 Mar 2008 Posts: 864

you did a bandwidth expansion and hosed your SAN. Start over, you have a backup though so no worries.

hint. in your new san use a smaller number of luns in each pool. I like 4 per pool.

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

So here's the system log after I added storage and the system couldn't restart...

8 07:06:49: --- last message repeated 2 times ---
Aug 8 07:06:49 mlbpmdc1 org.clamav.freshclam[52]: WARNING: Incremental update failed, trying to download daily.cvd
Aug 8 07:06:49 mlbpmdc1 org.clamav.freshclam[52]: WARNING: Can't download daily.cvd from database.clamav.net
Aug 8 07:06:49 mlbpmdc1 org.clamav.freshclam[52]: Trying again in 5 secs...
Aug 8 07:06:54 mlbpmdc1 org.clamav.freshclam[52]: ClamAV update process started at Wed Aug 8 07:06:54 2012
Aug 8 07:06:54 mlbpmdc1 org.clamav.freshclam[52]: main.cld is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
Aug 8 07:06:54 mlbpmdc1 org.clamav.freshclam[52]: WARNING: Current functionality level = 58, recommended = 60
Aug 8 07:06:54 mlbpmdc1 org.clamav.freshclam[52]: Please check if ClamAV tools are linked against the proper version of libclamav
Aug 8 07:06:54 mlbpmdc1 org.clamav.freshclam[52]: DON'T PANIC! Read http://www.clamav.net/support/faq
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: WARNING: getpatch: Can't download daily-15194.cdiff from database.clamav.net
Aug 8 07:06:55: --- last message repeated 1 time ---
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: ERROR: getpatch: Can't download daily-15194.cdiff from database.clamav.net
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: WARNING: Incremental update failed, trying to download daily.cvd
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: ERROR: Can't download daily.cvd from database.clamav.net
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: Giving up on database.clamav.net...
Aug 8 07:06:55 mlbpmdc1 org.clamav.freshclam[52]: Update failed. Your network may be down or none of the mirrors listed in /private/etc/freshclam.conf is working. Check http://www.clamav.net/support/mirror-problem for possible reasons.
Aug 8 07:14:40 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:14:40 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (1)
Aug 8 07:14:41 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:14:41 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (2)
Aug 8 07:14:42 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:14:42 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (3)
Aug 8 07:14:43 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:14:43 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (4)
Aug 8 07:14:44 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:14:44 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' exhausted its retry attempts
Aug 8 07:14:44 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' returned: 16
Aug 8 07:14:44 mlbpmdc1 servermgrd[63]: xsan: [63/1851330] ERROR: -[SANFilesystem unmountVolumeNamed:andForget:]: unmount of 'PRODXSAN01' failed: Unable to unmount volume `PRODXSAN01'
Aug 8 07:16:25 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:16:25 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (1)
Aug 8 07:16:26 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:16:26 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (2)
Aug 8 07:16:27 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:16:27 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (3)
Aug 8 07:16:28 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:16:28 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' needs to be retried (4)
Aug 8 07:16:29 mlbpmdc1 xsand[55]: Kernel denied unmount request
Aug 8 07:16:29 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' exhausted its retry attempts
Aug 8 07:16:29 mlbpmdc1 xsand[55]: Unmount of 'PRODXSAN01' returned: 16
Aug 8 07:16:29 mlbpmdc1 servermgrd[63]: xsan: [63/1851330] ERROR: -[SANFilesystem unmountVolumeNamed:andForget:]: unmount of 'PRODXSAN01' failed: Unable to unmount volume `PRODXSAN01'
Aug 8 07:17:23 mlbpmdc1 KernelEventAgent[70]: tid 00000000 received event(s) VQ_NOTRESP (1)
Aug 8 07:17:23 mlbpmdc1 KernelEventAgent[70]: tid 00000000 type 'acfs', mounted on '/Volumes/PRODXSAN01', from '/dev/disk22', not responding
Aug 8 07:17:23 mlbpmdc1 KernelEventAgent[70]: tid 00000000 found 1 filesystem(s) with problem(s)
Aug 8 07:17:24 mlbpmdc1 fsmpm[124]: PortMapper: Initiating activation vote for FSS 'PRODXSAN01'.
Aug 8 07:17:25 mlbpmdc1 fsmpm[124]: PortMapper: Stopping FSS 'PRODXSAN01'
Aug 8 07:17:25 mlbpmdc1 fsmpm[124]: PortMapper: FSS 'PRODXSAN01' has been stopped.
Aug 8 07:17:27 mlbpmdc1 com.apple.xsan[55]: xsan:perfDispatchMicroseconds = 1267648
Aug 8 07:17:27 mlbpmdc1 com.apple.xsan[55]: xsan:perfFunctionMicroseconds = 1267937
Aug 8 07:17:38 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-1" 89021805 free blocks in 292121 fragments inserted.
Aug 8 07:17:38 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-1" 1545849 free blocks in 150392 fragments ignored.
Aug 8 07:17:39 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-2" 91310322 free blocks in 365195 fragments inserted.
Aug 8 07:17:39 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-2" 3778200 free blocks in 307652 fragments ignored.
Aug 8 07:17:39 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Misc" 227210867 free blocks in 235691 fragments inserted.
Aug 8 07:17:39 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Misc" 188537 free blocks in 19481 fragments ignored.
Aug 8 07:17:41 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-3" 89595391 free blocks in 395555 fragments inserted.
Aug 8 07:17:41 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-3" 13833364 free blocks in 529418 fragments ignored.
Aug 8 07:17:41 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-4" 90637397 free blocks in 371945 fragments inserted.
Aug 8 07:17:41 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': FSM Alloc: Stripe Group "Data-4" 2202706 free blocks in 201348 fragments ignored.
Aug 8 07:17:41 mlbpmdc1 fsm[47081]: Xsan FSS 'PRODXSAN01[0]': Windows Security has been turned off in config file but clients have been requested to enforce ACLs. Windows Security remains in effect.
Aug 8 07:18:10 mlbpmdc1 servermgrd[63]: xsan: [63/2112C0] ERROR: get_fsm_process_stats(PRODXSAN01): Unable to find pid of fsm
Aug 8 07:18:30 mlbpmdc1 servermgrd[63]: xsan: [63/18C1F20] ERROR: -[XsanPlugin(SetVolumeState) setVolumeStateWithRequest:]: fsm didn't exit for volume PRODXSAN01
Aug 8 07:18:30 mlbpmdc1 fsmpm[124]: PortMapper: utility 'PRODXSAN01' excl:1 (pid 54909) type:6 registered.
Aug 8 07:18:35 mlbpmdc1 servermgrd[63]: Got error -9806 for SSLHandshake remote address is 172.21.203.100:15352
Aug 8 07:18:35 mlbpmdc1 servermgrd[63]: Exception in threadListen: Socket: Connect failed
Aug 8 07:18:35 mlbpmdc1 fsmpm[124]: PortMapper: Starting FSS service 'PRODXSAN01[0]' on mlbpmdc1.central.mlb.org.
Aug 8 07:18:35 mlbpmdc1 servermgrd[63]: Got error -9806 for SSLHandshake remote address is 172.21.203.100:16888
Aug 8 07:18:35 mlbpmdc1 servermgrd[63]: Exception in threadListen: Socket: Connect failed
Aug 8 07:18:35 mlbpmdc1 fsm[54910]: Xsan FSS 'PRODXSAN01[0]': Disk DATA15 in stripe group Data-5 has invalid index
Aug 8 07:18:35 mlbpmdc1 fsm[54910]: Xsan FSS 'PRODXSAN01[0]': Disk DATA16 in stripe group Data-5 has invalid index
Aug 8 07:18:35 mlbpmdc1 fsm[54910]: Xsan FSS 'PRODXSAN01[0]': Stripe Group Data-5 is missing a disk for index 0
Aug 8 07:18:35 mlbpmdc1 fsm[54910]: Xsan FSS 'PRODXSAN01[0]': Stripe Group Data-5 is missing a disk for index 1
Aug 8 07:18:35 mlbpmdc1 fsm[54910]: Xsan FSS 'PRODXSAN01[0]': ^Configuration parsing detected 0 error.
Aug 8 07:19:14 mlbpmdc1 fsmpm[124]: PortMapper: Initiating activation vote for FSS 'PRODXSAN01'.
Aug 8 07:19:44: --- last message repeated 34 times ---
Aug 8 07:19:44 mlbpmdc1 fsmpm[124]: PortMapper: Initiating activation vote for FSS 'PRODXSAN01'.
Aug 8 07:19:54: --- last message repeated 9 times ---
Aug 8 07:19:54 mlbpmdc1 servermgrd[63]: xsan: [63/67F4E0] ERROR: activate_volume_by_host(PRODXSAN01): Waited for activation but it never happened
Aug 8 07:19:55 mlbpmdc1 servermgrd[63]: xsan: [63/2112C0] ERROR: get_fsm_process_stats(PRODXSAN01): Unable to find pid of fsm
Aug 8 07:19:59 mlbpmdc1 servermgrd[63]: Got error -9806 for SSLHandshake remote address is 172.21.203.100:22776
Aug 8 07:19:59 mlbpmdc1 servermgrd[63]: Exception in threadListen: Socket: Connect failed
Aug 8 07:19:59 mlbpmdc1 fsmpm[124]: PortMapper: Starting FSS service 'PRODXSAN01[0]' on mlbpmdc1.central.mlb.org.
Aug 8 07:19:59 mlbpmdc1 servermgrd[63]: Got error -9806 for SSLHandshake remote address is 172.21.203.100:24824
Aug 8 07:19:59 mlbpmdc1 servermgrd[63]: Exception in threadListen: Socket: Connect failed
Aug 8 07:19:59 mlbpmdc1 fsm[54974]: Xsan FSS 'PRODXSAN01[0]': Disk DATA15 in stripe group Data-5 has invalid index
Aug 8 07:19:59 mlbpmdc1 fsm[54974]: Xsan FSS 'PRODXSAN01[0]': Disk DATA16 in stripe group Data-5 has invalid index
Aug 8 07:19:59 mlbpmdc1 fsm[54974]: Xsan FSS 'PRODXSAN01[0]': Stripe Group Data-5 is missing a disk for index 0
Aug 8 07:19:59 mlbpmdc1 fsm[54974]: Xsan FSS 'PRODXSAN01[0]': Stripe Group Data-5 is missing a disk for index 1
Aug 8 07:19:59 mlbpmdc1 fsm[54974]: Xsan FSS 'PRODXSAN01[0]': ^Configuration parsing detected 0 error.
Aug 8 07:20:38 mlbpmdc1 fsmpm[124]: PortMapper: Initiating activation vote for FSS 'PRODXSAN01'.
Aug 8 07:21:08: --- last message repeated 33 times ---
Aug 8 07:21:08 mlbpmdc1 fsmpm[124]: PortMapper: Initiating activation vote for FSS 'PRODXSAN01'.
Aug 8 07:21:19: --- last message repeated 10 times ---
Aug 8 07:21:19 mlbpmdc1 servermgrd[63]: xsan: [63/1E75F20] ERROR: activate_volume_by_host(PRODXSAN01): Waited for activation but it never happened
Aug 8 07:21:19 mlbpmdc1 servermgrd[63]: xsan: [63/2112C0] ERROR: get_fsm_process_stats(PRODXSAN01): Unable to find pid of fsm
Aug 8 07:22:10 mlbpmdc1 servermgrd[63]: xsan: [63/2112C0] ERROR: get_fsm_process_stats(PRODXSAN01): Unable to find pid of fsm
Aug 8 07:23:10 mlbpmdc1 servermgrd[63]: xsan: [63/2112C0] ERROR: get_fsm_process_stats(PRODXSAN01): Unable to find pid of fsm

singlemalt · Xsan Master Joined: 27 Feb 2009 Posts: 109

Yeah. This-> "<some number> free blocks in <some other number> fragments ignored." Is bad.
Your volumes free space is fragmented into swiss cheese.
Setting ABM free limit Yes in the config file may help. http://support.apple.com/kb/TS3041
But fsm reporting this
"…Disk DATA15 in stripe group Data-5 has invalid index
… Disk DATA16 in stripe group Data-5 has invalid index
…Stripe Group Data-5 is missing a disk for index 0 "
Maybe fatal.
If you don't have a backup you may be able to use the old .cfg file and get the volume
back up like it was before the expansion. Then back it up and start over.

cmonico · fully protected Joined: 07 Jul 2009 Posts: 10

yup...thats my plan. I already got it started using the old config files. I'm not sure it was ever set up correctly in the first place....I just added to the misery.
Thanks for all the advice, I appreciate it.