HELP... please! Volume won't start!

MyrddynX's picture

Hi all!

I'm hoping someone can offer some insights here. After a month of working with no problem, our new XSAN 2 died a couple of hours ago. When I try to restart the volume, It seems to start, but then dies after about 5-10 seconds. What concerns me most are the words 'Fatal' and 'Panic' in the Volume logs! Any help would be MUCH appreciated!

Cheers

Mark M. Pinder

here are the logs:
Volume Log:

[0514 18:57:01] 0xa077ffa0 (Info) Server Revision 3.1.0 Build 2 (339.12)
[0514 18:57:01] 0xa077ffa0 (Info) Built for Darwin 9.0 ppc
[0514 18:57:01] 0xa077ffa0 (Info) Created on Tue Feb 19 17:29:50 PST 2008
[0514 18:57:01] 0xa077ffa0 (Info) Built in /SourceCache/XsanFS/XsanFS-339.12
[0514 18:57:01] 0xa077ffa0 (Info)
Configuration:
DiskTypes-12
Disks-12
StripeGroups-2
ForceStripeAlignment-1
MaxConnections-139
ThreadPoolSize-256
StripeAlignSize-256
FsBlockSize-16384
BufferCacheSize-32M
InodeCacheSize-8192
RestoreJournal-Disabled
RestoreJournalDir-None
[0514 18:57:01] 0xa077ffa0 (Info) Self (mdc1.artisan) IP address is 192.168.1.48.
[0514 18:57:01.232116] 0xa077ffa0 (Debug) No fsports file - port range enforcement disabled.
[0514 18:57:01] 0xa077ffa0 (Info) Listening on TCP socket mdc1.artisan:49586
[0514 18:57:01] 0xa077ffa0 (Info) Node [0] [mdc1.artisan:49586] File System Manager Login.
[0514 18:57:01] 0xa077ffa0 (Info) Service standing by on host 'mdc1.artisan:49586'.
[0514 18:57:02.193443] 0xa077ffa0 (Debug) Standby service - NSS ping from 10.0.1.24:49706.
[0514 18:57:02.193565] 0xa077ffa0 (Debug) Standby service - NSS ping from 10.0.1.25:49765.
[0514 18:57:02.194402] 0xa077ffa0 (Debug) FOUsurpCheck: read ARB info (pass 1): host (192.168.1.48:49452) conns 0 age 1210816329.00 secs his delta 0.00 secs my delta 0.00 secs.
[0514 18:57:02.194409] 0xa077ffa0 (Debug) FOUsurpCheck: polling ARB block to check for active peer (pass 1).
[0514 18:57:03.194902] 0xa077ffa0 (Debug) FOUsurpCheck: read ARB info (pass 2): host (192.168.1.48:49452) conns 0 age 1210816329.00 secs his delta 0.00 secs my delta 1.00 secs.
[0514 18:57:03.194920] 0xa077ffa0 (Debug) FOUsurpCheck: ARB is already mine.
[0514 18:57:03] 0xa077ffa0 (Info) Branding Arbitration Block (attempt 1) votes 2.
[0514 18:57:05.196682] 0xa077ffa0 (Debug) Cannot find fail over script /Library/Filesystems/Xsan/bin/cvfail.mdc1.artisan - looking for generic script.
[0514 18:57:05] 0xa077ffa0 (Info) Launching fail over script ["/Library/Filesystems/Xsan/bin/cvfail" mdc1.artisan 49586 ArtiSAN]
[0514 18:57:05.207328] 0xa077ffa0 (Debug) Starting journal log recovery.
[0514 18:57:05.366145] 0xa077ffa0 (Debug) Completed journal log recovery.
[0514 18:57:05.366572] 0xa077ffa0 (Debug) Inode_init_post_activation: FsStatus 0xd07, Brl_ResyncState 1
[0514 18:57:05] 0xb8e2f000 (Info) FSM Alloc: Loading Stripe Group "MetadataAndJournal". 372.59 GB.
[0514 18:57:05] 0xb8eb1000 (Info) FSM Alloc: Loading Stripe Group "Data". 45.03 TB.
[0514 18:57:05] 0xb8e2f000 (Info) FSM Alloc: Stripe Group "MetadataAndJournal" active.
[0514 18:57:11] 0xb8eb1000 (Info) FSM Alloc: free blocks 2632263800 with 0 blocks currently reserved for client delayed buffers.Reserved blocks may change with client activity.
[0514 18:57:11] 0xb8eb1000 (Info) FSM Alloc: Stripe Group "Data" active.
[0514 18:57:11] 0xa077ffa0 (Warning) Windows Security has been turned off in config file but clients have been requested to enforce ACLs. Windows Security remains in effect.
[0514 18:57:11] 0xa077ffa0 (Info) File system 'ArtiSAN' requires UTF8-NFC file names
[0514 18:57:11] 0xa077ffa0 (Info) File System Service 'ArtiSAN[0]' now active on host 'mdc1.artisan:49586'.
[0514 18:57:11] 0xb8dad000 (**FATAL**) PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(eblk) failed!" file alloc.c, line 478
[0514 18:57:11.798339] 0xa077ffa0 (Debug) Listener_thread: flushing journal.
[0514 18:57:11.798355] 0xa077ffa0 (Debug) Listener_thread: journal flush complete.
[0514 18:57:11.798409] 0xa077ffa0 (Debug) FSM memory SUMMARY resident size 51MB.
[0514 18:57:11.798416] 0xa077ffa0 (Debug) FSM I/O SUMMARY writes total/17.
[0514 18:57:11.798421] 0xa077ffa0 (Debug) FSM I/O SUMMARY writes journal/0 sb/0 buf/0 abm/0.
[0514 18:57:11.798430] 0xa077ffa0 (Debug) FSM I/O SUMMARY writes inode/0 ganged/0 (0.00%).
[0514 18:57:11.798435] 0xa077ffa0 (Debug) FSM wait SUMMARY inode pool expand waits/0.
[0514 18:57:11.798439] 0xa077ffa0 (Debug) FSM wait SUMMARY journal waits/0.
[0514 18:57:11.798444] 0xa077ffa0 (Debug) FSM wait SUMMARY journal bytes used avg/286720 max/327680.
[0514 18:57:11.798448] 0xa077ffa0 (Debug) FSM wait SUMMARY free buffer waits/0.
[0514 18:57:11.798453] 0xa077ffa0 (Debug) FSM wait SUMMARY free inode waits/0.
[0514 18:57:11.798459] 0xa077ffa0 (Debug) FSM wait SUMMARY revokes/0 avg/0 min/0 max/0.
[0514 18:57:11.798471] 0xa077ffa0 (Debug) FSM threads SUMMARY max busy hi-prio/0 lo-prio/0.
[0514 18:57:11.798476] 0xa077ffa0 (Debug) FSM threads SUMMARY max busy dmig/0 events/0.
[0514 18:57:11.798481] 0xa077ffa0 (Debug) FSM msg queue SUMMARY hi-prio now/0 min/0 max/0.
[0514 18:57:11.798490] 0xa077ffa0 (Debug) FSM msg queue SUMMARY lo-prio now/0 min/0 max/0.
[0514 18:57:11.798496] 0xa077ffa0 (Debug) FSM msg queue SUMMARY dmig now/0 min/0 max/0.
[0514 18:57:11.798500] 0xa077ffa0 (Debug) FSM msg queue SUMMARY events now/0 min/0 max/0.
[0514 18:57:11.798509] 0xa077ffa0 (Debug) FSM cache SUMMARY inode lookups/12 misses/8 hits/33.33%.
[0514 18:57:11.798514] 0xa077ffa0 (Debug) FSM cache SUMMARY free incore inodes now/8191 min/8190 max/8192.
[0514 18:57:11.798520] 0xa077ffa0 (Debug) FSM cache SUMMARY buffer lookups/21 misses/12 hits/42.86%.
[0514 18:57:11.798525] 0xa077ffa0 (Debug) FSM cache SUMMARY free buffers now/2048 min/1 max/2048.
[0514 18:57:11.798530] 0xa077ffa0 (Debug) FSM cache SUMMARY attrs now/0 min/0 max/0.
[0514 18:57:11.798535] 0xa077ffa0 (Debug) FSM extent SUMMARY extent lookups/6 misses/0 hits/100.00%.
[0514 18:57:11.798541] 0xa077ffa0 (Debug) FSM extent SUMMARY hint tries/0 misses/0 hits/0.00%.
[0514 18:57:11.798578] 0xa077ffa0 (Debug) SG SUMMARY MetadataAndJournal space total/372.59 GB free/371.57 GB (99.73%)
[0514 18:57:11.798588] 0xa077ffa0 (Debug) SG SUMMARY MetadataAndJournal space minfree/371.57 GB (99.73%) maxfree/371.57 GB (99.73%)
[0514 18:57:11.798595] 0xa077ffa0 (Debug) SG SUMMARY MetadataAndJournal alloc extent cnt/0 avgsize/0.00 B.
[0514 18:57:11.798600] 0xa077ffa0 (Debug) SG SUMMARY 2 btree free space fragments 2 splay tree fragments
[0514 18:57:11.798608] 0xa077ffa0 (Debug) SG SUMMARY Data space total/45.03 TB free/39.22 TB (87.11%)
[0514 18:57:11.798616] 0xa077ffa0 (Debug) SG SUMMARY Data space minfree/39.22 TB (87.11%) maxfree/39.22 TB (87.11%)
[0514 18:57:11.798622] 0xa077ffa0 (Debug) SG SUMMARY Data alloc extent cnt/0 avgsize/0.00 B.
[0514 18:57:11.798627] 0xa077ffa0 (Debug) SG SUMMARY 6052 btree free space fragments 12102 splay tree fragments
[0514 18:57:11.798656] 0xa077ffa0 (Debug) PIO Read SUMMARY 1-MotherlyLove cnt/234 maxq/3.
[0514 18:57:11.798663] 0xa077ffa0 (Debug) PIO Read SUMMARY 1-MotherlyLove avg/22484 min/303 max/49034.
[0514 18:57:11.798670] 0xa077ffa0 (Debug) PIO Read SUMMARY 1-MotherlyLove sysavg/11173 sysmin/250 sysmax/36948.
[0514 18:57:11.798677] 0xa077ffa0 (Debug) PIO Read SUMMARY 1-MotherlyLove avglen/1719241 minlen/512 maxlen/4194304.
[0514 18:57:11.798683] 0xa077ffa0 (Debug) PIO Write SUMMARY 1-MotherlyLove cnt/17 maxq/1.
[0514 18:57:11.798689] 0xa077ffa0 (Debug) PIO Write SUMMARY 1-MotherlyLove avg/340 min/296 max/657.
[0514 18:57:11.798705] 0xa077ffa0 (Debug) PIO Write SUMMARY 1-MotherlyLove sysavg/301 sysmin/282 sysmax/378.
[0514 18:57:11.798712] 0xa077ffa0 (Debug) PIO Write SUMMARY 1-MotherlyLove avglen/2379 minlen/512 maxlen/16384.
[0514 18:57:11] 0xb8dad000 (**FATAL**) PANIC: aborting threads now.
Logger_thread: sleeps/23 signals/0 flushes/7 writes/7 switches 0
Logger_thread: logged/73 clean/73 toss/0 signalled/0 toss_message/0
Logger_thread: waited/0 awakened/0

And FSM log:

[0514 18:51:48] 0xa077ffa0 INFO PortMapper: Deleting stale mapping for id 192.168.1.17
[0514 18:51:48] 0xa077ffa0 INFO PortMapper: Deleting stale mapping for id 10.0.1.15
[0514 18:51:58] 0xb038d000 NOTICE PortMapper: Starting FSS service 'ArtiSAN[0]' on mdc1.artisan.
[0514 18:51:58] 0xa077ffa0 (debug) FSS 'ArtiSAN' STOPPED (corefile limit exceeded) -> LAUNCHED, next event in 60s
[0514 18:51:58] 0xa077ffa0 (debug) FSS 'ArtiSAN' LAUNCHED -> REGISTERED
[0514 18:51:58] 0xa077ffa0 NOTICE PortMapper: FSS 'ArtiSAN'[0] (pid 245) at port 49452 is registered.
[0514 18:51:58] 0xa077ffa0 INFO NSS: Standby FSS 'ArtiSAN[0]' at 192.168.1.48:49452 (pid 245) - registered.
[0514 18:51:58] 0xa077ffa0 (debug) Heartbeat from 192.168.1.48 updating LOCAL ArtiSAN to 192.168.1.48:11457
[0514 18:51:59] 0xa077ffa0 NOTICE PortMapper: Initiating activation vote for FSS 'ArtiSAN'.
[0514 18:51:59] 0xa077ffa0 (debug) Initiate_nss_vote for FSS ArtiSAN
[0514 18:51:59] 0xa077ffa0 (debug) NSS: sending message (type 2) to Name Server '10.0.1.48' (10.0.1.48:49154).
[0514 18:51:59] 0xa077ffa0 INFO NSS: election initiated by 10.0.1.48:49154 (id 192.168.1.48) - admin request.
[0514 18:51:59] 0xa077ffa0 (debug) NSS_VOTE2 to 10.0.1.48:49154
[0514 18:51:59] 0xa077ffa0 INFO NSS: Starting vote for FSS ArtiSAN using 2 voting members: 10.0.1.25, 10.0.1.24.
[0514 18:51:59] 0xa077ffa0 NOTICE PortMapper: Initiating activation vote for FSS 'ArtiSAN'.
[0514 18:51:59] 0xa077ffa0 (debug) Initiate_nss_vote for FSS ArtiSAN
[0514 18:51:59] 0xa077ffa0 (debug) NSS: sending message (type 2) to Name Server '10.0.1.48' (10.0.1.48:49154).
[0514 18:51:59] 0xa077ffa0 INFO NSS: Vote call for FSS ArtiSAN is inhibited - vote dis-allowed.
[0514 18:51:59] 0xa077ffa0 (debug) tally_member:FSS/ArtiSAN COUNTED 1 vote for 10.0.1.48:49452 from member 10.0.1.25:49174
[0514 18:51:59] 0xa077ffa0 (debug) tally_member:FSS/ArtiSAN COUNTED 1 vote for 10.0.1.48:49452 from member 10.0.1.24:49174
[0514 18:51:59] 0xa077ffa0 (debug) Elect FSS[0] 10.0.1.48:49452 svc_pri/0x0 votes/2 wght/0x1d4546fdd
[0514 18:51:59] 0xa077ffa0 (debug) Elect FSS winner so far 10.0.1.48:49452 svc_pri/00 votes/2 wght/0x1d4546fdd
[0514 18:51:59] 0xa077ffa0 INFO NSS: Vote selected FSS 'ArtiSAN[0]' at 10.0.1.48:49452 (pid 245) - attempting activation.
[0514 18:51:59] 0xa077ffa0 (debug) set_fss_active: sending to 10.0.1.48:49154 (id 192.168.1.48)
[0514 18:51:59] 0xa077ffa0 (debug) NSS: FSS activation initiated by coordinator 10.0.1.48:49154 (id 192.168.1.48)
[0514 18:52:00] 0xa077ffa0 (debug) PORTMAP_INQUIRY_NA_V2
[0514 18:52:00] 0xa077ffa0 (debug) find_fsm: start
[0514 18:52:09] 0xa077ffa0 (debug) IP address list changed
[0514 18:52:12] 0xa077ffa0 (debug) PORTMAP_INQUIRY_V2
[0514 18:52:12] 0xa077ffa0 (debug) find_fsm: start
[0514 18:52:12] 0xa077ffa0 (debug) find_fsm: calling Nss_add_mount ArtiSAN
[0514 18:52:12] 0xa077ffa0 (debug) find_fsm fsm ArtiSAN ipaddr 10.0.1.48 port 0xc12c TestLink failed: getsockopt(SO_ERROR) returned error 61 [errno 61]: Connection refused
[0514 18:52:13] 0xa077ffa0 (debug) IP address list changed
[0514 18:52:17] 0xa077ffa0 NOTICE PortMapper: FSS 'ArtiSAN' disconnected.
[0514 18:52:17] 0xa077ffa0 NOTICE PortMapper: kicking diskscan_thread -1338986496.
[0514 18:52:17] 0xa077ffa0 (debug) FSS 'ArtiSAN' REGISTERED -> DYING, next event in 60s
[0514 18:52:17] 0xb030b000 INFO Starting Disk rescan
[0514 18:52:17] 0xb030b000 INFO Disk rescan delay completed
[0514 18:52:17] 0xb030b000 INFO Disk rescan found 12 disks
[0514 18:52:17] 0xb038d000 ERR Portmapper: FSS 'ArtiSAN' (pid 245) exited on signal 6
[0514 18:52:17] 0xb038d000 ERR FSS 'ArtiSAN' appears unstable (2 failures in 60 minutes or less), halting restarts
[0514 18:52:17] 0xb038d000 (debug) FSS 'ArtiSAN' DYING -> STOPPED (corefile limit exceeded)
[0514 18:52:17] 0xa077ffa0 INFO NSS: Active FSS 'ArtiSAN[0]' at 192.168.1.48:49452 (pid 245) - dropped.
[0514 18:52:17] 0xa077ffa0 (debug) NSS: removing vote inhibitor for FSS 'ArtiSAN'.
[0514 18:57:01] 0xb038d000 NOTICE PortMapper: Starting FSS service 'ArtiSAN[0]' on mdc1.artisan.
[0514 18:57:01] 0xa077ffa0 (debug) FSS 'ArtiSAN' STOPPED (corefile limit exceeded) -> LAUNCHED, next event in 60s
[0514 18:57:01] 0xa077ffa0 (debug) FSS 'ArtiSAN' LAUNCHED -> REGISTERED
[0514 18:57:01] 0xa077ffa0 NOTICE PortMapper: FSS 'ArtiSAN'[0] (pid 286) at port 49586 is registered.
[0514 18:57:01] 0xa077ffa0 INFO NSS: Standby FSS 'ArtiSAN[0]' at 192.168.1.48:49586 (pid 286) - registered.
[0514 18:57:01] 0xa077ffa0 (debug) Heartbeat from 192.168.1.48 updating LOCAL ArtiSAN to 192.168.1.48:45761
[0514 18:57:02] 0xa077ffa0 NOTICE PortMapper: Initiating activation vote for FSS 'ArtiSAN'.
[0514 18:57:02] 0xa077ffa0 (debug) Initiate_nss_vote for FSS ArtiSAN
[0514 18:57:02] 0xa077ffa0 (debug) NSS: sending message (type 2) to Name Server '10.0.1.48' (10.0.1.48:49154).
[0514 18:57:02] 0xa077ffa0 INFO NSS: election initiated by 10.0.1.48:49154 (id 192.168.1.48) - admin request.
[0514 18:57:02] 0xa077ffa0 (debug) NSS_VOTE2 to 10.0.1.48:49154
[0514 18:57:02] 0xa077ffa0 INFO NSS: Starting vote for FSS ArtiSAN using 2 voting members: 10.0.1.25, 10.0.1.24.
[0514 18:57:02] 0xa077ffa0 NOTICE PortMapper: Initiating activation vote for FSS 'ArtiSAN'.
[0514 18:57:02] 0xa077ffa0 (debug) Initiate_nss_vote for FSS ArtiSAN
[0514 18:57:02] 0xa077ffa0 (debug) NSS: sending message (type 2) to Name Server '10.0.1.48' (10.0.1.48:49154).
[0514 18:57:02] 0xa077ffa0 INFO NSS: Vote call for FSS ArtiSAN is inhibited - vote dis-allowed.
[0514 18:57:02] 0xa077ffa0 (debug) tally_member:FSS/ArtiSAN COUNTED 1 vote for 10.0.1.48:49586 from member 10.0.1.24:49174
[0514 18:57:02] 0xa077ffa0 (debug) tally_member:FSS/ArtiSAN COUNTED 1 vote for 10.0.1.48:49586 from member 10.0.1.25:49174
[0514 18:57:02] 0xa077ffa0 (debug) Elect FSS[0] 10.0.1.48:49586 svc_pri/0x0 votes/2 wght/0x1f8692a63
[0514 18:57:02] 0xa077ffa0 (debug) Elect FSS winner so far 10.0.1.48:49586 svc_pri/00 votes/2 wght/0x1f8692a63
[0514 18:57:02] 0xa077ffa0 INFO NSS: Vote selected FSS 'ArtiSAN[0]' at 10.0.1.48:49586 (pid 286) - attempting activation.
[0514 18:57:02] 0xa077ffa0 (debug) set_fss_active: sending to 10.0.1.48:49154 (id 192.168.1.48)
[0514 18:57:02] 0xa077ffa0 (debug) NSS: FSS activation initiated by coordinator 10.0.1.48:49154 (id 192.168.1.48)
[0514 18:57:03] 0xa077ffa0 (debug) PORTMAP_INQUIRY_NA_V2
[0514 18:57:03] 0xa077ffa0 (debug) find_fsm: start
[0514 18:57:11] 0xa077ffa0 (debug) IP address list changed
[0514 18:57:12] 0xa077ffa0 (debug) PORTMAP_INQUIRY_V2
[0514 18:57:12] 0xa077ffa0 (debug) find_fsm: start
[0514 18:57:12] 0xa077ffa0 (debug) find_fsm: calling Nss_add_mount ArtiSAN
[0514 18:57:12] 0xa077ffa0 (debug) find_fsm fsm ArtiSAN ipaddr 10.0.1.48 port 0xc1b2 TestLink failed: getsockopt(SO_ERROR) returned error 61 [errno 61]: Connection refused
[0514 18:57:14] 0xa077ffa0 (debug) IP address list changed
[0514 18:57:20] 0xa077ffa0 NOTICE PortMapper: FSS 'ArtiSAN' disconnected.
[0514 18:57:20] 0xa077ffa0 NOTICE PortMapper: kicking diskscan_thread -1338986496.
[0514 18:57:20] 0xa077ffa0 (debug) FSS 'ArtiSAN' REGISTERED -> DYING, next event in 60s
[0514 18:57:20] 0xb030b000 INFO Starting Disk rescan
[0514 18:57:20] 0xb030b000 INFO Disk rescan delay completed
[0514 18:57:20] 0xb030b000 INFO Disk rescan found 12 disks
[0514 18:57:20] 0xb038d000 ERR Portmapper: FSS 'ArtiSAN' (pid 286) exited on signal 6
[0514 18:57:20] 0xb038d000 ERR FSS 'ArtiSAN' appears unstable (2 failures in 60 minutes or less), halting restarts
[0514 18:57:20] 0xb038d000 (debug) FSS 'ArtiSAN' DYING -> STOPPED (corefile limit exceeded)
[0514 18:57:20] 0xa077ffa0 INFO NSS: Active FSS 'ArtiSAN[0]' at 192.168.1.48:49586 (pid 286) - dropped.
[0514 18:57:20] 0xa077ffa0 (debug) NSS: removing vote inhibitor for FSS 'ArtiSAN'.

ipott's picture

seems the network is not ok:

[0514 18:52:12] 0xa077ffa0 (debug) find_fsm fsm ArtiSAN ipaddr 10.0.1.48 port 0xc12c TestLink failed: getsockopt(SO_ERROR) returned error 61 [errno 61]: Connection refused
[0514 18:52:13] 0xa077ffa0 (debug) IP address list changed

ravi's picture

MyrddynX wrote:

0514 18:57:11] 0xa077ffa0 (Warning) Windows Security has been turned off in config file but clients have been requested to enforce ACLs. Windows Security remains in effect.
[0514 18:57:11] 0xa077ffa0 (Info) File system 'ArtiSAN' requires UTF8-NFC file names
[0514 18:57:11] 0xa077ffa0 (Info) File System Service 'ArtiSAN[0]' now active on host 'mdc1.artisan:49586'.
[0514 18:57:11] 0xb8dad000 (**FATAL**) PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(eblk) failed!" file alloc.c, line 478
[0514 18:57:11.798339] 0xa077ffa0 (Debug) Listener_thread: flushing journal.
[0514 18:57:11.798355] 0xa077ffa0 (Debug) Listener_thread: journal flush complete.

/quote

It appears that so called self-balancing binary search tree (Splay) for something somewhere is messed up.

Does cvfsck in read mode reports any kind of freelist errors?

You have also ACLs implemented. Possibly quotas (yes, they have a "relationship" with Splay trees) as well. I will turn all these off in the config file and see if your volume mounts.

fritz.'s picture

Try cvfsck! I had a volume that appeared to be ok in the Xsan admin but it would fail to start until I ran cvfsck to repair it.

MyrddynX's picture

Thanks for the suggestions. Sadly, cvfsck didn't help. Well, it fixed a bunch of errors, but the volume still won't mount. I've called in our local SAN guru, and even he's a little stumped. A few weirdnesses in the DNS setup were found, but none that explain the issue at hand, and so we're at the point of trying to re-build the MDCs. I'll post again if we get anywhere.

Cheers

M.

MacCyclone's picture

May I know, how do you run the cvfsck command?

nrausch's picture

Yes, How exactly are you running cvfsck?

larspetter's picture

At 14.11 today all clients was disconnected. The Xsan 2.0 installation had been running OK for approx 4 weeks. After upgrading MDCs and clients to 10.5.2 and Xsan 2.0 everything seemed find except some rights issues with ACLs.

At 14.11 the logs went like this:

From system.log
May 26 14:10:46 xsanserver-1 fsm[8825]: Xsan FSS 'SANkassen[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 451
May 26 14:10:46 xsanserver-1 KernelEventAgent[53]: tid 00000000 received VQ_NOTRESP event (1)
May 26 14:10:46 xsanserver-1 fsm[8825]: PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 451
May 26 14:10:46 xsanserver-1 KernelEventAgent[53]: tid 00000000 type 'acfs', mounted on '/Volumes/SANkassen', from '/dev/disk6', not responding
May 26 14:10:46 xsanserver-1 KernelEventAgent[53]: tid 00000000 found 1 filesystem(s) with problem(s)
May 26 14:10:46 xsanserver-1 loginwindow[18280]: 1 server now unresponsive
May 26 14:10:46 xsanserver-1 fsm[8825]: Xsan FSS 'SANkassen[0]': PANIC: aborting threads now.
May 26 14:10:47 xsanserver-1 kernel[0]: Reconnecting to FSS 'SANkassen'
May 26 14:10:47 xsanserver-1 fsmpm[91]: PortMapper: Initiating activation vote for FSS 'SANkassen'.
May 26 14:10:58 xsanserver-1 ReportCrash[32290]: Formulating crash report for process fsm[8825]
May 26 14:10:58 xsanserver-1 fsmpm[91]: PortMapper: FSS 'SANkassen' disconnected.
May 26 14:10:58 xsanserver-1 fsmpm[91]: PortMapper: kicking diskscan_thread -1338454016.
May 26 14:10:58 xsanserver-1 ReportCrash[32290]: Saved crashreport to /Library/Logs/CrashReporter/fsm_2008-05-26-141057_xsanserver-1.crash using uid: 0 gid: 0, euid: 0 egid: 0
May 26 14:10:58 xsanserver-1 fsmpm[91]: Portmapper: FSS 'SANkassen' (pid 8825) exited on signal 6
May 26 14:11:08 xsanserver-1 fsmpm[91]: PortMapper: RESTART FSS service 'SANkassen[0]' on host xsanserver-1.xsan.
May 26 14:11:08 xsanserver-1 fsmpm[91]: PortMapper: Starting FSS service 'SANkassen[0]' on xsanserver-1.xsan.
May 26 14:11:08 xsanserver-1 fsmpm[91]: PortMapper: FSS 'SANkassen'[0] (pid 32292) at port 52270 is registered.
May 26 14:11:16 xsanserver-1 loginwindow[18280]: 1 server now unresponsive
May 26 14:11:32 xsanserver-1 servermgrd[26371]: xsan: [26371/103B90] ERROR: get_fsmvol_at_index: Could not connect to FSM because Admin Tap Connection to FSM failed: [errno 60]: Operation timed out
May 26 14:11:44 xsanserver-1 KernelEventAgent[53]: tid 00000000 received VQ_NOTRESP event (1)
May 26 14:11:44 xsanserver-1 KernelEventAgent[53]: tid 00000000 type 'acfs', mounted on '/Volumes/SANkassen', from '/dev/disk6', not responding
May 26 14:11:44 xsanserver-1 KernelEventAgent[53]: tid 00000000 found 1 filesystem(s) with problem(s)
May 26 14:11:44 xsanserver-1 loginwindow[18280]: 1 server now unresponsive
May 26 14:11:45 xsanserver-1 kernel[0]: Reconnecting to FSS 'SANkassen'
May 26 14:11:45 xsanserver-1 fsmpm[91]: PortMapper: Initiating activation vote for FSS 'SANkassen'.
May 26 14:11:55 xsanserver-1 com.apple.xsan[42]: xsan:perfDispatchMicroseconds = 14008
May 26 14:11:55 xsanserver-1 com.apple.xsan[42]: xsan:perfFunctionMicroseconds = 14622
May 26 14:11:57 xsanserver-1 fsm[32292]: Xsan FSS 'SANkassen[0]': FSM Alloc: Stripe Group "VideoPool" 108472064 free blocks in 109945 fragments inserted.
May 26 14:11:57 xsanserver-1 fsm[32292]: Xsan FSS 'SANkassen[0]': FSM Alloc: Stripe Group "VideoPool" 193023 free blocks in 5139 fragments ignored.
May 26 14:11:57 xsanserver-1 fsm[32292]: Xsan FSS 'SANkassen[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 451
May 26 14:11:57 xsanserver-1 KernelEventAgent[53]: tid 00000000 received VQ_NOTRESP event (1)
May 26 14:11:57 xsanserver-1 fsm[32292]: PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 451
May 26 14:11:57 xsanserver-1 KernelEventAgent[53]: tid 00000000 type 'acfs', mounted on '/Volumes/SANkassen', from '/dev/disk6', not responding
May 26 14:11:57 xsanserver-1 com.apple.KernelEventAgent[53]: KernelEventAgent: Couldn't get vol. info for '/Volumes/SANkassen': Stale NFS file handle
May 26 14:11:57 xsanserver-1 KernelEventAgent[53]: tid 00000000 found 1 filesystem(s) with problem(s)
May 26 14:11:57 xsanserver-1 loginwindow[18280]: 1 server now unresponsive
May 26 14:11:57 xsanserver-1 fsm[32292]: Xsan FSS 'SANkassen[0]': PANIC: wait 3 secs for journal to flush
May 26 14:11:57 xsanserver-1 fsm[32292]: Xsan FSS 'SANkassen[0]': PANIC: aborting threads now.
May 26 14:12:02 xsanserver-1 servermgrd[26371]: xsan: [26371/103B90] ERROR: get_fsmvol_at_index: Could not connect to FSM because Could not find File System Manager for "SANkassen" on xsanserver-1.xsan.
May 26 14:12:05 xsanserver-1 ReportCrash[32307]: Formulating crash report for process fsm[32292]
May 26 14:12:05 xsanserver-1 fsmpm[91]: PortMapper: FSS 'SANkassen' disconnected.
May 26 14:12:05 xsanserver-1 ReportCrash[32307]: Saved crashreport to /Library/Logs/CrashReporter/fsm_2008-05-26-141204_xsanserver-1.crash using uid: 0 gid: 0, euid: 0 egid: 0
May 26 14:12:05 xsanserver-1 fsmpm[91]: PortMapper: kicking diskscan_thread -1338454016.
May 26 14:12:05 xsanserver-1 fsmpm[91]: Portmapper: FSS 'SANkassen' (pid 32292) exited on signal 6
May 26 14:12:05 xsanserver-1 fsmpm[91]: FSS 'SANkassen' appears unstable (2 failures in 60 minutes or less), halting restarts
May 26 14:12:07 xsanserver-1 kernel[0]: Reconnecting to FSS 'SANkassen'
May 26 14:12:07 xsanserver-1 kernel[0]: No FSS registered with PortMapper on host 172.16.0.5, retrying...
May 26 14:12:12 xsanserver-1 fsmpm[91]: PortMapper: Initiating activation vote for FSS 'SANkassen'.
May 26 14:12:16 xsanserver-1 KernelEventAgent[53]: tid 00000000 received VQ_NOTRESP event (1)
May 26 14:12:16 xsanserver-1 KernelEventAgent[53]: tid 00000000 type 'acfs', mounted on '/Volumes/SANkassen', from '/dev/disk6', not responding
May 26 14:12:16 xsanserver-1 com.apple.KernelEventAgent[53]: KernelEventAgent: Couldn't get vol. info for '/Volumes/SANkassen': Stale NFS file handle

And later the panic that seems to crash FSM changes:

May 26 14:52:58 xsanserver-1 kernel[0]: Could not mount filesystem SANkassen, cvfs error 'Timeout' (25)
May 26 14:53:03 xsanserver-1 fsm[117]: Xsan FSS 'SANkassen[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm "free_pending_inode_thread: Cannot lookup inode [0x807f80000175d28c]! -Unknown error: 0 " file inode.c, line 3510
May 26 14:53:03 xsanserver-1 fsm[117]: PANIC: /Library/Filesystems/Xsan/bin/fsm "free_pending_inode_thread: Cannot lookup inode [0x807f80000175d28c]! -Unknown error: 0\n" file inode.c, line 3510
May 26 14:53:03 xsanserver-1 fsm[117]: Xsan FSS 'SANkassen[0]': PANIC: wait 3 secs for journal to flush
May 26 14:53:03 xsanserver-1 fsm[117]: Xsan FSS 'SANkassen[0]': PANIC: aborting threads now.
May 26 14:53:04 xsanserver-1 com.apple.xsan[42]: mount_acfs: No such process
May 26 14:53:04 xsanserver-1 xsand[42]: mount of volume 'SANkassen' failed (exit code = 22)
May 26 14:53:04 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.104 (id 10.100.100.148) coordinator list mismatch
May 26 14:53:04 xsanserver-1 com.apple.xsan[42]: mount_acfs: No such process
May 26 14:53:04 xsanserver-1 xsand[42]: mount of volume 'SANkassen' failed (exit code = 22)
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.107 (id 10.100.100.67) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.101 (id 172.16.0.101) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.7 (id 172.16.0.7) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.8 (id 10.100.100.86) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.102 (id 10.100.100.61) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.106 (id 10.100.100.66) coordinator list mismatch
May 26 14:53:05 xsanserver-1 fsmpm[104]: NSS: Member 172.16.0.103 (id 10.100.100.62) coordinator list mismatch
May 26 14:53:06 xsanserver-1 com.apple.launchd[1] (com.apple.installdb.system): Throttling respawn: Will start in 1 seconds
May 26 14:53:17 xsanserver-1 ReportCrash[163]: Formulating crash report for process fsm[117]
May 26 14:53:17 xsanserver-1 fsmpm[104]: PortMapper: FSS 'SANkassen' disconnected.
May 26 14:53:17 xsanserver-1 fsmpm[104]: PortMapper: kicking diskscan_thread -1338454016.
May 26 14:53:17 xsanserver-1 ReportCrash[163]: Saved crashreport to /Library/Logs/CrashReporter/fsm_2008-05-26-145316_xsanserver-1.crash using uid: 0 gid: 0, euid: 0 egid: 0

Did a cvfsck -wv to fix errors (after a -j). There were some errors but they were fixed. System has been restarted from scratch but everytime to try to start the volume, MDC1 seems to start it but in crashes in a few seconds, MDC2 tries to take over but crashes as well.

Any1 know if it is possible to downgrade the whole system again to 1.4.2? One MDC at the time?

Will try tomorrow to reinstall one of the MDC´s and re-ad it to the san with fresh setup files.

Any other ideas? Can any1 see any strange things in my logs?

)

LP<

MyrddynX's picture

Hi Lars:

Sorry to say, but I never did find a workable solution to my issues, and wound up wiping everything and re-building the SAN. Fortunately, most everything was backed up, and that which wasn't backed up could be re-captured from tape fairly easily (a few assistant editors had late nights).

I'm just glad my failure happened when there was only 6TB of data on the SAN, and not when all 45TB were full. I'm sorry you're having similar troubles. I've heard a few people say that upgrading from 1.4.2 was dangerous, which is why I rebuilt from scratch both the first AND second times I re-did this SAN! :)

Cheers

M.

MacCyclone's picture

larspetter's picture

Ok, a little update. In the middle of me doing a cvfsck -x to start to prepare for disaster Apple told me that my problem might be "corrupt pending free inode list" and that I should do a cvfsck -C volum followed by cvfsck -vw volum. And you know what - that actually worked...my volum is mounted again!!! For the time being..

I am now taking a backup and planning to recreate..

LP

markwooderson's picture

Thanks Lars this worked for me too

xavi00's picture

wow, write this down and keep it somewhere safe.

btw i'm still waiting for apple to call me back

wrstuden's picture

MyrddynX wrote:
Hi all!

I'm hoping someone can offer some insights here. After a month of working with no problem, our new XSAN 2 died a couple of hours ago. When I try to restart the volume, It seems to start, but then dies after about 5-10 seconds. What concerns me most are the words 'Fatal' and 'Panic' in the Volume logs! Any help would be MUCH appreciated!

Cheers

Mark M. Pinder

here are the logs:
Volume Log:

[0514 18:57:01] 0xa077ffa0 (Info) Server Revision 3.1.0 Build 2 (339.12)
/quote

Don't run Xsan 2.0. Upgrade to 2.1.

Nolf's picture

Good afternoon. Pomoigte solve the problem so that is not visible and not the status of Volume section connects clients. Logs below:

client
7/7/14 9:21:46 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:21:46 AM com.apple.xsan[1029] mount_acfs: Operation timed out
7/7/14 9:21:46 AM xsand[1029] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:21:57 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:00 AM postfix/master[1136] master exit time has arrived
7/7/14 9:22:08 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:19 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:30 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:41 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:42 AM com.apple.xsan[1029] mount_acfs: Operation timed out
7/7/14 9:22:42 AM xsand[1029] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:22:52 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:03 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:14 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:20 AM com.apple.kextcache[295] /System/Library/Extensions/IOSerialFamily.kext/Contents/PlugIns/InternalModemSupport.kext doesn't support architecture x86_64; omitting from prelinked kernel.
7/7/14 9:23:25 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:30 AM com.apple.kextcache[295] Created prelinked kernel /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache_x86_64.4F22E0F7.
7/7/14 9:23:36 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:37 AM com.apple.xsan[1029] mount_acfs: Operation timed out
7/7/14 9:23:37 AM xsand[1029] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:23:47 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:47 AM /System/Library/CoreServices/CCacheServer.app/Contents/MacOS/CCacheServer[242] No valid tickets, timing out
7/7/14 9:23:58 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:00 AM postfix/master[1827] daemon started -- version 2.5.5, configuration /etc/postfix
7/7/14 9:24:00 AM postfix/pickup[1828] 63E08B248D5B: uid=501 from=
7/7/14 9:24:00 AM postfix/cleanup[1830] 63E08B248D5B: message-id=<20140707062400.63E08B248D5B@xserve-07.production>
7/7/14 9:24:00 AM postfix/qmgr[1829] 63E08B248D5B: from=, size=947, nrcpt=1 (queue active)
7/7/14 9:24:00 AM postfix/local[1832] od[getpwnam_ext]: no attribute dsAttrTypeStandard:MailAttribute in record for user admin
7/7/14 9:24:00 AM postfix/pipe[1833] 63E08B248D5B: to=, orig_to=, relay=dovecot, delay=0.04, delays=0.02/0.01/0/0.01, dsn=2.0.0, status=sent (delivered via dovecot service)
7/7/14 9:24:00 AM postfix/qmgr[1829] 63E08B248D5B: removed
7/7/14 9:24:09 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)

MDC

7/7/14 9:21:27 AM Xsan Admin[848] ERROR: Error reading volume statistics for prod_f: kOfflineError (0)
7/7/14 9:21:31 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:21:31 AM Xsan Admin[848] ERROR: Error reading san information: kNetworkError (0)
7/7/14 9:21:31 AM servermgrd[863] servermgr_info: mark_running_services_configured(): marked running services configured:
7/7/14 9:21:47 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:21:58 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:09 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:20 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:31 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:31 AM com.apple.xsan[42] mount_acfs: Operation timed out
7/7/14 9:22:31 AM xsand[42] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:22:42 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:22:53 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:02 AM servermgrd[863] xsan: [863/690FA0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Admin Tap Connection to FSM failed: [errno 60]: Operation timed out
7/7/14 9:23:04 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:15 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:26 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:26 AM com.apple.xsan[42] mount_acfs: Operation timed out
7/7/14 9:23:26 AM xsand[42] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:23:37 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:45 AM Xsan Admin[848] ERROR: Wrong number of mounts found? 0
7/7/14 9:23:48 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:23:59 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:10 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:20 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:21 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:21 AM com.apple.xsan[42] mount_acfs: Operation timed out
7/7/14 9:24:21 AM xsand[42] mount of volume 'prod_f' failed (exit code = 22)
7/7/14 9:24:21 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:23 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:24 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:25 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:26 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:28 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:29 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:30 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:31 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:32 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:33 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:34 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:35 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:36 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:38 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:39 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:40 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:42 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:43 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)
7/7/14 9:24:44 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:45 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:47 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:47 AM servermgrd[863] xsan: [863/2112D0] ERROR: get_fsmvol_at_index: Could not connect to FSM because Admin Tap Connection to FSM failed: [errno 60]: Operation timed out
7/7/14 9:24:48 AM servermgrd[863] servermgr_info: mark_running_services_configured(): marked running services configured:
7/7/14 9:24:48 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:49 AM Xsan Admin[848] XSAdminQueueController: gotServerError: kNetworkError forTransaction:
7/7/14 9:24:54 AM kernel Could not mount filesystem prod_f, cvfs error 'Timeout' (25)

-=Learning is never too late=-