Xsan volume won't stop and shows unmounted on mdc

[memblin]

I was about to go in and stop one of our volumes to destroy it and rebuild it
with some new settings. It sat at the stopping status screen for quite some
time before it responded with a failed to unmount volume from its metadata
controller.

Setup is

Two 10.5 Xserve metadata controllers (mdc0 and mdc1)
Two volumes, one volume was being hosted by each controller (Data and Data1)

Data1 is currently in production and working just fine. Data is the original
volume before we upgraded.

Data says hosted by mdc0, Data1 says hosted by mdc1

I unmounted all the mounts except the metadata controllers Data to
prepare for stopping and destroying the volume. The mdc0 controller says
that the Data volume is not mounted but when in Xsan admin it shows that
it is mounted and it can't stop it because it can't unmount the volume from
mdc0.

Has anyone been able to get past something like this? Only thing I can
think of is maybe rebooting mdc0 and hoping the Data volume fails over to
mdc1 so I can stop it there after mdc0 has rebooted and remounted the
volume properly. That just sounded like it might cause some issues. I don't
need any of the data on the Data volume if that helps.

Thanks in advance,
-Mem

[MattG]

Sounds like you have spotlight enabled and it might be in the middle of an index operation. You can always stop the volume via the CLI although it won't be elegant. But since you are rebuilding anyway, it's not that important.

sudo cvadmin
(authenticate)
> stop <volumename>
> quit

[memblin]

Appreciate it MattG, I'll try that Thursday when we get a maintenance in case
something goes sideways and decides to bring the other volume down. heh

In the mean time this is what I found digging deeper. Wondering what could
cause this kind of thing to happen.

mdata0 is mdc0 referred to above

The Data volume does not show mounted on the desktop or in a mount list
but xsanctl says Hey you're mounted yay!

mdata0:~ admin$ sudo xsanctl list
Data1 - /Volumes/Data1
Data - /Volumes/Data
mdata0:~ admin$ sudo xsanctl mount Data
mount command failed: Volume `Data' is already mounted (error code: 17)
mdata0:~ admin$ sudo xsanctl unmount Data
unmount command failed: Unable to unmount volume `Data' (error code: 60)


I have only spent a few minutes Googling for those error codes to see
if I can't figure out how this server got confused.

[cedge318]

If you tried to do an unmount using:
xsanctl unmount Data

And got those errors then chances are there's something open and it can't close out gracefully to unmount. If so, then the next step is to see what has open files that is thwarting your efforts. This could be done with lsof or try making sure any app that could have an open file on the volume is closed/quit (especially Final Cut Pro).

[memblin]

That's the problem, the volume isn't actually mounted.

xsanctl thinks it is mounted
xsan admin thinks it is mounted

But running a mount from the terminal only shows the Data1
volume mounted and in finder only the Data1 volume shows up.

If I do a ls /Volumes I see both Data1 and Data but if I try to do
an ls in the /Volumes/Data directory the whole shell locks up. I
can't even do an ls -alF on the /Volumes directory without locking
up the shell.

Planning to attempt the cvadmin stop on thursday night and
if that doesn't work I'm just going to reboot the MD server and
hope for the best since we're destroying that volume anyway.
That is unless someone else comes through with other info.

[memblin]

The cvadmin stop seems to have worked for the most part. Now both controllers show the Data volume as stopped. The Xsan Admin GUI is still all kinds of off kilter though.

Does anyone know if Xsan Admin remote information on each metadata
controller can get out of sync? If so can you tell me how to check this
and how to fix it? heh

[singlemalt]

Try closing the Xsanadmin window, don't just quit xsanadmin. Confirm that you want to disconnect. The quit Xsanadmin. run
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.servermgrd.plist (this is all on one line)
Then run
sudo launchctl load /System/Library/LaunchDaemons/com.apple.servermgrd.plist (this is all on one line)

Then relaunch Xsanadmin and reconnect via "connect to an existing San". You may need to do it on anything
running Xsanadmin.

Just out of curiosity, if you look in /Volumes via the Finder (go menu, go to folder) do you see any Folders?
Sometimes what happens is that at boot, something, i.e. an afp connection is attempted to a share on a volume
that hasn't mounted yet (like an Xsan volume), the system will go ahead and create a folder in /Volumes by
the same name. Later when the volume does try and mount, it can't because there's already an object by that
name in /Volumes. So it will append a <space> 1 to the name. So if the Xsan volume was named Data and the above scenario happens, you can end up with a FOLDER called Data and a Volume named Data 1.
Hilarity ensues.

[memblin]

Singlemalt, I'd seen the servermgrd suggestion in another forum but we're running 2 volumes currently and i wasn't sure what it would do to the volume
that is currently functioning properly. I'll do some digging in to that and post the results if it seems safe.

Thanks for the response.

[wentzr]

hey all.. typically error code 17 in xsanctl means you're trying to mount a volume that's already mounted, but it more directly seems to come up when the volume's mountpoint is already created in /Volumes/

the catch is that you can have a mountpoint in /Volumes/ but not have the volume mounted to it... in some situations (after a crash, hard reset, etc) a "ghosted" folder may exist in your volumes folder. just rm it, then try xsanctl mount volumename again...

eg if my volume label is "xsanity", it's not mounted and xsanctl mount xsanity pulls up error code 17, cd to /Volumes, then do an ls... chances are you'll see xsanity in the list, meaning it's mountpoint exists. just rm -R xsanity, then xsanctl mount xsanity again and you should be golden.

-Rob

[MattG]

Unless of course the volume is in the process of mounting, in which case you rm -r solution will begin to erase the entire contents of the volume.

There are two "safer" techniques:

• In the Finder, Use Command-Shift-G to go to the /Volumes folder, then drag the ghost mount to the trash. The volume will probably mount within 10 seconds, at which point you can safely empty the trash.

• Use rmdir <volumename> which will only remove the root of the volume itself, and not delete internal folders/files.

[ravi]