Xsan

AnthonyRS's picture

Xsan MDC crashes when mounting volume with malloc error

Xsan 1.4.2 MDC 10.5.8 freezes and crashes when trying to mount the Volume on the MDC. the only error that gets reported is a malloc error.

This is an old Xsan 1.4.2 that can only mount on the MDC so they can backup the data so we can stop using the system. now in the middle of backing up this happens.
By the way this system has been running as a general use area and not really in proper use and not critical so it has only 1 MDC. But it turns out some people have put data on it they shouldnt have.

I can start and activate the Volume without any problems but the problem comes in when i want to mount the volume
The following is what the logs report when i run "sudo mount -t acfs DataSan /Volumes/DataSan" to mount the volume.

Aug 19 15:27:13 xserve03 kernel[0]: CVFS 'DataSan': Buffer Cache blksize 4096, #blocks min 256 max 8192
Aug 19 15:27:13 xserve03 kernel[0]: CVFS 'DataSan': request reserved space 0x2333333
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label MD_Lun con 2 lun 0 unit 0.0 state 0xf4 dev 0xe00000a rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN08 con 2 lun 0 unit 1.0 state 0xf4 dev 0xe000004 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN07 con 1 lun 0 unit 2.0 state 0xf4 dev 0xe00000b rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN06 con 2 lun 0 unit 3.0 state 0xf4 dev 0xe00000c rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN05 con 1 lun 0 unit 4.0 state 0xf4 dev 0xe000003 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN04 con 2 lun 0 unit 5.0 state 0xf4 dev 0xe000006 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN03 con 1 lun 0 unit 6.0 state 0xf4 dev 0xe000005 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN02 con 1 lun 0 unit 7.0 state 0xf4 dev 0xe000007 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label RLUN01 con 2 lun 0 unit 8.0 state 0xf4 dev 0xe000008 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label Alt_Lun03 con 1 lun 0 unit 9.0 state 0xf4 dev 0xe000009 rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label Alt_Lun02 con 1 lun 0 unit 10.0 state 0xf4 dev 0xe00000d rawname
Aug 19 15:27:13 xserve03 kernel[0]: MdOpenDisks: label Alt_lun01 con 2 lun 0 unit 11.0 state 0xf4 dev 0xe00000e rawname
Aug 19 15:27:13 xserve03 kernel[0]: CVFS 'DataSan': FsBlk size 16384, bits 14, mask 0x3fff
Aug 19 15:27:13 xserve03 kernel[0]: CVFS 'DataSan': Sector size 512, bits 9, mask 0x1ff
Aug 19 15:27:13 xserve03 kernel[0]: Mounted filesystem DataSan to FSM on host 192.168.101.24
Aug 19 15:27:13 xserve03 kernel[0]: Using v1 readdir for 'DataSan'
Aug 19 15:27:13 xserve03 kernel[0]: Xsan: enabling extended security on volume "DataSan"
Aug 19 15:27:13 xserve03 fsmpm[177]: PortMapper: Mount Event for /Volumes/DataSan on /dev/disk16
Aug 19 15:27:13 xserve03 com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 2 seconds
Aug 19 15:27:13 xserve03 fsmpm[177]: fsmpm(177,0xa0847720) malloc: *** error for object 0x809a00: non-page-aligned, non-allocated pointer being freed\n*** set a breakpoint in malloc_error_break to debug
Aug 19 15:27:22 xserve03 com.apple.launchd[1] (org.openldap.slapd[336]): Exited with exit code: 1
Aug 19 15:27:22 xserve03 com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 3 seconds
Aug 19 15:27:33 xserve03 com.apple.launchd[1] (org.openldap.slapd[337]): Exited with exit code: 1
Aug 19 15:27:33 xserve03 com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 3 seconds
Aug 19 15:27:39 xserve03 ReportCrash[339]: Formulating crash report for process fsmpm[177]
Aug 19 15:27:39 xserve03 kernel[0]: Reconnecting to local portmapper on host '127.0.0.1'
Aug 19 15:27:39 xserve03 kernel[0]: Fsmportmapper on host 127.0.0.1 not responding, retrying...
Aug 19 15:27:39 xserve03 KernelEventAgent[94]: tid 00000000 received VQ_NOTRESP event (1)
Aug 19 15:27:39 xserve03 KernelEventAgent[94]: tid 00000000 type 'acfs', mounted on '/Volumes/DataSan', from '/dev/disk16', not responding
Aug 19 15:27:40 xserve03 ReportCrash[339]: Saved crashreport to /Library/Logs/CrashReporter/fsmpm_2014-08-19-152738_xserve03-2.crash using uid: 0 gid: 0, euid: 0 egid: 0
Aug 19 15:27:41 xserve03 kernel[0]: Reconnecting to FSS 'DataSan'
Aug 19 15:27:41 xserve03 com.apple.launchd[1] (org.openldap.slapd[338]): Exited with exit code: 1
Aug 19 15:27:41 xserve03 com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 5 seconds
Aug 19 15:27:43 xserve03 sshd[292]: DEAD_PROCESS: 307 ttys001
Aug 19 15:27:50 xserve03 com.apple.launchd[1] (org.openldap.slapd[342]): Exited with exit code: 1
Aug 19 15:27:50 xserve03 com.apple.launchd[1] (org.openldap.slapd): Throttling respawn: Will start in 7 seconds

I have run permissions repairs
run cvfsck -nv, cvfsck -wv and cvfsck -j
but none of these report any INode errors or any problems with the volume.

There isn't really another machine to setup as an Xsan client that will work with this version of Xsan so some help to get the MDC up and running would be great

Regards,
Anthony

Apple Knowledge Base's picture

Xsan: Compatibility of SAN clients with Xsan and StorNext controllers (Apple KB)

Learn which versions of StorNext can be used in the same SAN with Xsan controllers, and which versions of Xsan can be used in the same SAN with StorNext controllers.

Read more: http://support.apple.com/kb/HT1517

Nolf's picture

XSAN Controller

Tags: 

Good afternoon. Faced with the problem of constant forking Comptroller xsan. That is 2 minutes ready, 5 minutes Offline. The logs at the moment, the following occurs:
servermgrd [910] xsan: [910/2112D0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Admin Tap Connection to FSM failed: [errno 60]: Operation timed out
servermgrd [910] xsan: [910/6033400] ERROR: get_fsmvol_at_index: Could not connect to FSM because Admin Tap Connection to FSM failed: [errno 60]: Operation timed out

Volume mount only on the controller when the controller when you try to get online to connect to the client the following error:
kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
postfix / master [3029] master exit time has arrived
kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
com.apple.xsan [2774] mount_acfs: Operation timed out [2774] mount of volume 'prod_f' failed (exit code = 22)
servermgrd [53] xsan: [53/3199F60] ERROR: - [SANFilesystem mountVolumeNamed: writable: withOptions:]: mount of 'prod_f' failed: Unable to mount volume `prod_f '

Please help sort out the problem and solve it. Thanks in advance!!!

percisely's picture

Poor SMB reshare write performance

Forums: 

Our 10.8.5 Xsan 3 SAN is reshared by an Xserve via AFP and SMB. Currently AFP performance is as good as 100MB/sec for read and write but SMB performance on Windows clients is terrible, with only about 40MB/sec read and <10MB/sec write.

Is anyone else resharing via SMB to Windows clients? How is your performance?

Some additional notes:
-Not isolated to our reshare server. I setup a test SMB reshare off the backup MDC with the same results.
-OS X SMB clients perform better than Windows - as good as 60-70MB/sec read and write.
-Definitely Xsan related - shares off the boot volume SMB performance is fine.

abstractrude's picture

OS X 10.10 Yosemite and Xsan

Tags: 

OS X Server and Xsan live on!

pro_ject's picture

Fibre channel card no network settings

hi guys,

im totally new to the whole xsan and fibre channel business. I just did a fresh install of snow leopard on a mac pro 3,1.

The fibre channel card gets recognized and even shows up in the sytem preferences of the mac under "Fibre Channel" with both channels of which one is connected indicated by the green light.

But under network settings no adapter shows up in which i could configure a ip or subnet etc.
even the disk utility shows me the vtrak 12TB storage which runs on the Xserve

i figure it must be some kind of driver issue or additional software im missing.

hope u guys can help me

josh

undercover's picture

Achievement Unlocked: Upgrade!

I finally did it.

After looking forward / fearing the upgrade for years, I finally migrated off my old XServes to Mac Minis, and moved from Mac OS 10.5 to 10.9.

The move had been long delayed for lack of scheduled down time, lack of funding for new servers, and a little bit of fear of things going wrong.

The background story.

My clients (6 of them) were all running 10.6, even though the servers were 10.5. I almost never had trouble with XSAN itself, except the occasional ACL issue or snfsdefrag necessary when disks got near full. We had been planning to upgrade for a long time, but for one reason or another it just never happened. Even lately, I had planned to upgrade to new Mac Pros for clients when they were released, but I think we will change our short term plans to work with four new iMacs and four of our old Mac Pros.

Of course adding new clients required an XSAN upgrade, which also required new servers since I was on first-gen XServes. So Mac Mini + xMac chassis + old fiber cards were the plan, whenever we were able to get the funds set aside.

Things get scary.

My util server failed about a year back. It really wasn't doing much at the time so it wasn't a great loss. But then my mdc2 failed. It was always the more stable mdc. Sometimes XSAN admin would not work well on mdc1. Also mdc2 was my OD master. So I had to promote mdc1 which was the replica.

Fortunately, mdc1 continued to work fine.

Time to prepare.

So we purchased the Mac Mini Servers back in October, but we were in the middle of our busy projects, so I could not upgrade until January at the earliest. I got the team to stay on top of backups in case things went sour and to prepare for the migration. I also made a time machine backup of my mdc, exported lunlabels, and copied xsan config folder.

Should be good to go? Am I forgetting anything?

It's time.

Shut it all down. Disconnect clients, shut down volumes, etc. Connect mini, plug in mdc1 time machine backup, import data. Go through setup.

Got hung up on network config. Apparently it locks if you have two network ports connected. Unplug one, reboot, do it again, replug network. No big deal. Oh wait, fiber zoning. Was using an old card, had to re-alias and re-zone it.

After all that, one of my XSAN volumes decided to mount! Even before I launched XSAN admin. Launch XSAN admin, start second volume, things look good!

Do it again.

First problem - these minis shipped with 10.8. Shut down volumes, download Mavericks, do another time machine backup, install. Good to go.

And again.

Turn on new mdc2, upgrade to Mavericks, change fiber zoning, add second mdc in XSAN admin. No problems.

Oh wait, where's Open Directory? 

For some reason my OD did not make it through the upgrade to 10.8. No big deal, as my OD pretty much consists of one group. Recreated it on mdc1, setup replica on mdc2. [note I plan to move primary to util server].

Re-add the clients!

Booted up clients. Some of them somehow popped up one of the volumes before I re-added them in XSAN admin. Weird. Added them in XSAN admin, and despite complaining that their XSAN versions were older, everything worked fine.  Turned spotlight on for both volumes. Everything is good! Woohoo!

This was not nearly as bad as I thought it was going to be!

Still to do:

  • Setup util server for SMB shares.
  • Upgrade clients to 10.9 (some will stay at 10.6 because they are old)
  • Add new iMacs when they arrive
  • Change IP to new subnet
  • Change to new Active Directory domain
  • Figure out if I can now remove lunzero from my NexSAN satabeast. I think Abs told me something about ALUA in 10.7+, will have to go look at old notes.
abstractrude's picture

ProRes White Paper Updated

Apple has updated their ProRes white paper with information on bandwidth for 4K and 5K video streams. You find the whitepaper here.

aaron's picture

Migrate Metadata From Xsan to StorNext

Charles Edge writes about the snmetadump command on his blog Krypted:

http://krypted.com/mac-security/migrate-metadata-from-xsan-to-stornext/

metadreamer's picture

Xsan vs GigE NAS?

Forums: 

We're looking at upgrading our Xsan (and all of its clients to new Mac Pros) as it's a 5 year old install and locked to 10.6.8.

I know that some places are marketing "video workflow" optimized NAS solutions over 10 gig E as an alternative. I assume these boxes just use standard AFP and file level storage.

Does anyone have a comparison of using one of those vs. Xsan? Seems like a higher theoretical bandwidth vs 4 Gb fibre (at least that's what our current switch and cards can handle)... But how does Final Cut Pro X and/or Premiere work with a file level storage system vs. Xsan? I assumed that Xsan would offer maximum compatibility for collaborating on files but I'm not sure if any changes to Final Cut X or these newer NAS boxes mitigated any issues.

Subscribe to RSS - Xsan