MDC1 Kernel Panic (hwmond)

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

Good morning !

A nice email was waiting for me this morning :
"Le volume SANLaboutique sur le SAN SANXXX est à nouveau en ligne et fonctionne sur mdc2.xxx.pub." which means that the SAN is now available on mdc2.

After investigating a little bit, it turns out that mdc1 had a kernel panic earlier, related to hwmond.

Background :

Running apps :

Kernel panic log :

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

Paniced this morning at 04:23:48… interesting…

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

After the third consecutive crash in a week, I contacted AppleCare Support. Having no Xsan Applecare Contract, they gave me three choices :
- Buy AppleCare Xsan
- Remove Xsan from Metadata 1
- Stop calling them

I chose the second choice. I've uninstalled Xsan from MDC1, promoting MACPRO6 to metadata 2. A week later, MDC1 was still up and running.

What's more interesting is what was waiting for me this morning :

lotte · Xsan Master Joined: 11 Dec 2008 Posts: 190

Have you tried to check the Xsan Filesystem?

Lotte

memblin · Been around the blocks Joined: 22 Apr 2009 Posts: 20

We've been dealing with this issue for over a year.

Our setup is currently:

2 x Intel MDC 2 ( OSX 10.5.8 )
2 x Promise VTrak E610f for storage
XSan 2.2

1 Data volume
1 Metadata volume

We use this data volume to store small graphics files and print job
information then pull those same assets from the XSan when running the
print jobs on our presses.

In the spring of 2008 before I took this job an apple certified tech came out
to do the Xsan install. I started working here in July the same year, and then
in December of 2008 we started getting these zalloc / kalloc panics.

We worked with AppleCare for almost four months trying to resolve the
issue. We did the standard upgrades as we were told. We were told to
upgrade the memory in the MDC servers since we only had 4gig in each. We
handled that (12gig each now) and still had the trouble.

Around March of 2009, one of the AppleCare folks told us to try forcing the
maxvnodes setting to 90000. Here's a copy / paste from that email.

Make note, we never actually did this. The problem just quit happening
one day and it fell off the radar until it started happening again this past
December and has continued, last panic was yesterday.

*snip*
Chris,
After reviewing this and also doing some internal testing I wanted to make you aware of something.
In the last email I sent the change I had you make is ideal if the MDC always only has 4 GB of RAM which your
server does, or at least did at the time. If you decide to add more RAM to the MDC's you'll want to edit the /etc/rc.server
file differently.
Basically we want to ensure that:
kern.maxnodes=90000 which the change I sent to you does IF the server has 4GB of RAM. If you later add more RAM or even if you don't but think you might later you would want to change the file thusly:

What we want to do is comment out the whole block of code accept for
sysctl -w kern.maxproc=2500
sysctl -w kern.ipc.somaxconn=2500
sysctl -w kern.maxvnodes=90000
*/snip*

I called Apple back again today, referenced our old case number. They said
they have engineers working on this right now already. Apparently a number
of people are having this trouble. My net searches turn up a few other
reports of the same behavoir, often times on MDCs that are running volumes
used for smaller randomly accessed files instead of the larger files that
everyone says XSan does best with.

The tech I spoke with told me that they have been handing out that
maxvnodes 'fix' and having fair results. He sent me a different email with
modified directions that I'm pasting below.

*snip*
Subject: Steps to resolve zalloc/kalloc panic in Xsan on Mac OS X Server 10.5

1. While logged in as root, use your preferred text editor to add the following line to the end of the /etc/rc.server file:
2.
3. sysctl -w kern.maxvnodes=90000
4. Repeat step 1 on all Mac OS X Server v10.5 servers in the Xsan deployment.
5. Reboot the standby Metadata Controllers and Server clients.
6. Failover the volume to a standby Metadata Controller. To do this, open Xsan Admin, click Volumes in the left sidebar, select the volume(s), then choose "Force failover" from the Action Menu in the lower left corner.
7.Reboot the originally-active Metadata Controller.
8.Failover the volume back to the originally-active Metadata Controller, if desired, using the process in step 4 above.
*/snip*

That said, I don't work for Apple. I'm not sure what possible bad things could
happen by monkeying with this setting. They couldn't tell me if I should
update the 90,000 number to something larger since we added an additional
8gig of RAM. I do have an email in to AppleCare about it though referencing
the original ticket when I couldn't find even a tiny whisper about this on the
net.

I did go ahead and set the setting on the current backup MDC and got it
all rebooted. I made the settings change on the primary MDC but haven't
rebooted it yet. I figure I'll fail it over tonight or just wait for it to panic and
fail itself over. I'll try to remember to post back in a couple of weeks when
we figure out if this is a true fix.

*edited for misspeling Promise*

rstasel · Xsan Master Joined: 03 Aug 2007 Posts: 120

as a note, on my Xsan 2.2.1 (and one 2.1.1) clients, running both 10.5 and 10.6, the setting for maxvnodes is higher than 90k.

On the machines with ≥8GB ram, it's set to 150k. On the clients with 4GB of ram, it's 120k. Even the PPC with 4GB of ram, running Xsan 2.1.1 is at 120k.

So something is odd here that your systems would have below a 90k maxvnode setting.

Only thing I can think is, are your MDC's Xserves? I know there are some tweeks Apple makes to some settings in the case of Xserves (I think because they figure no one is using them as desktop clients, they can increase some settings beyond the point you'd want if you were using the machine for "work")? In my case, all of my Xsan clients, and MDCs are Xserves (One G5, 5x 2006 Intel Xserves, and 2x 2009 Intel Xserves).

memblin · Been around the blocks Joined: 22 Apr 2009 Posts: 20

We are running both MDCs on Xserve (intel arch) hardware and OS X 10.5.8

Our metadata controllers have been automatically configuring the maxvnodes
setting to 120000 at 4GB, when we upgraded to 12GB RAM it started setting
them to 150000 just like yours. The kernel panics stopped happening in
March of 2009 so we never got around to trying the 'fix' provided by Apple.

It was the guys in XSan Enterprise Support up at Apple that recommended
dropping the setting manually to 90,000 using the /etc/rc.server file. It didn't
make much sense to me either so we put off the maintenance a bit, and the
kernel panics just stopped so I never made the change.

In December of 2009 when we started having the same kernel panics again
we engaged Apple support again and they recommended the same thing. I
asked him why I would want to lower it and he said that's just how they'd
been handling the kernel panic problem that is apparent in some Xsan
installations running a combination of XSan 2 and MacOS X 10.5.x. I went
ahead and modified both of our MDCs on their recommendation and have
not seen any trouble yet. Will post again with my results if it does anything
other than resolve the issue with the kernel panics. heh

Going back over the post it looks like d4corp is running 10.6.2 so this is
probably not applicable for their current problems.

I did email Apple Support to verify that with 12Gigs of RAM 90000 is still the
*sweet spot* number. Here is that response from them.

*snip*
If you are still running 10.5.8 you would want to leave it at 90000 due to
the panic issue with Xsan and 10.5.x. If you have upgraded to 10.6 that
issue has been addressed and maxvnodes would be set automatically by the
OS. Here is an article that you can use for reference:

Mac OS X Server v10.6: Understanding process limits
http://support.apple.com/kb/HT3854
*/snip*

The original post makes me wonder if this has actually been fixed in 10.6 or
if they just haven't had enough people report in with the problem yet. I'm
hoping for fixed, my boss won't accept the "This is why we recommend
having two metadata controllers instead of just one." that Apple tried to give
us the first time we called on the issue back in 2008. *grin*

rstasel · Xsan Master Joined: 03 Aug 2007 Posts: 120

well, it IS a valid reason to have more than a single MDC. =)

Okay, that makes more sense. Though, I haven't seen any problems with Xsan 2.2.1 or 2.1.1 on my 10.5 machines, though, neither are MDCs.

Certainly let us know. 10.6 has been, so far, a wonderful upgrade in our environment. AFP performance is worlds better. And Xsan seems to run better too, even without turning on Native Extended Attributes (which I probably won't do until Summer).

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

Great ! This post is finally living !

I've had two more failures this morning. More info on that asap.

I think the maxvnodes sysctl definitively makes sense. IMHO, it looks like spotlight indexing floods fsevents…

How much RAM do you have on the machine(s) that kernel panics ?
Do you have Spotlight enabled ?
Do you have a lot of files ? (especially small files, such as graphics or home folders ?)

memblin · Been around the blocks Joined: 22 Apr 2009 Posts: 20

We have 12gigs of RAM in both MDCs and are on Xserve hardware.

This is from our hwmond.log for our primary, the secondary has the same
setup.

*/snip*
Server:
Host : <removed>
Model : Xserve2,1
Uptime : 1 minutes
OS version : Mac OS X Server 10.5.8 (9L34)
Processor : 1 x 2800 MHz
Memory : 12288 MB
BootROM : XS21.88Z.006C.B01.0712212323

Memory:
Memory Slot "BRANCH 0 CHANNEL 0/DIMM 1" : 2048MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 0 CHANNEL 1/DIMM 2" : 2048MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 1 CHANNEL 0/DIMM 3" : 2048MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 1 CHANNEL 1/DIMM 4" : 2048MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 0 CHANNEL 0/DIMM 5" : 1024MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 0 CHANNEL 1/DIMM 6" : 1024MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 1 CHANNEL 0/DIMM 7" : 1024MB, ECC DDR2 FB-DIMM, 800 MHz
Memory Slot "BRANCH 1 CHANNEL 1/DIMM 8" : 1024MB, ECC DDR2 FB-DIMM, 800 MHz
*/snip*

That is our memory setup on both MDCs.

We had Spotlight enabled until we got some bad tech support from Promise
where a tech miss-spoke and told us to just pull one of the controllers. After
some corruption do to that it broke our Spotlight and we haven't been able
to take the time to re-index 16TB of data.

We have a TON (over 8,000 files in some directories) of tiny files (web
graphics, jlts, and pre-press PDFs) on the single volume currently hosted
by our MDCs.

Last night was our maintenance night, I was able to go ahead and force a
failover to the MDC that was already modified according to Apples direction
to have maxvnodes=90000. I made the change on the primary after the
failover and rebooted it so we're running with the 'fix' in place now.

Do be careful if you try that setting in /etc/rc.server. It was recommended to
us by Apple since our MDCs are in the 10.5.x family and some Xsan installs
using 10.5.x utilizing their XSan to store many smaller files have been
randomly popping these kernel panics. They say this has been taken care of
in 10.6.x and there shouldn't be a problem because 10.6 adjust the
maxvnodes automatically depending on installed memory.

From reading the rc.server file in 10.5.8, you can see that 10.5 adjusted it
as well. I'll keep the threat informed of what we run into.

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

Thanks for your reply…

That doesn't reassure me…

My MDC are 10.6.2 with 2GB of RAM. I have 699.401 files on the Xsan Volume… Spotlight is activated, but not working as I'm having a permission issue.

rstasel · Xsan Master Joined: 03 Aug 2007 Posts: 120

d4corp:

If you do indeed mean that your MDCs have 2GB of ram, this is against the minimum requirements of Xsan, which require 2GB of RAM for all Xsan clients, plus an ADDITIONAL 2GB on the MDCs for each volume hosted. So your MDCs should have a minimum of 4GB of ram (assuming 1 volume hosted). I can certainly see that causing issues...

Hardware Requirements: http://support.apple.com/kb/HT3927

These have been the memory requirements since Xsan 2.0 was released.

Both my MDCs have 8GB of ram, and my SAN has approximately 6M files on it. You can easily tell this by selecting the SAN volume in Disk Utility, which will then show "number of files".

Spotlight is disabled for our setup. But, as I said, our MDCs are 10.6. Previously we were running 10.5.8 on our MDCs, and had not seen this error. Previously one MDC was Intel, the other was PPC. Then both were Intel, still running 10.5.8 with Xsan 2.1.1. We only updated to 10.6.2 and 2.2.1 when both were available around Christmas.

d4corp · Could work for Apple Joined: 07 Nov 2008 Posts: 50

Thanks, you're right about the RAM, I misinterpreted the manual. Thanks for the link to the kbase.

As I told you, I'll put 8GB in each MDC, it should resolve the issue.

I'll keep you informed

Thanks

rstasel · Xsan Master Joined: 03 Aug 2007 Posts: 120

d4corp,

Great! Let us know. I imagine upping the ram will resolve the issue. But, I'm just guessing.

memblin · Been around the blocks Joined: 22 Apr 2009 Posts: 20

Just a quick follow up on what I had been told by Xsan Support to do to fix our issues with the kernel panics. Forcing the maxvnodes to 90000 did nothing for us we had another random kernel panic today same gig. Zalloc garbage collection goes crazy. I'm back on the phone with them again, figure they'll make us upgrade a few things then tell us again they don't know why it's happening. heh