On 07/17/2012 12:21 PM, Steven J. Yellin wrote: > If your atop service is on, you should be able to see something about > what was happening shortly before the crash by viewing the appropriate > /var/log/atop/<file> with the atop -r <file> command. > You could just try increasing your swap space; you don't have very much > compared with your ram. Simple 'top' and 'atop' commands show, among other > things, current swap usage. I'd get nervous if most of it gets used up. I installed and started atop to see what that shows. Didn't know about that one before, thanks. I am running sa/sar and that showed: 12:00:02 AM kbswpfree kbswpused %swpused kbswpcad %swpcad 08:50:01 AM 1989664 107480 5.13 25616 23.83 before the last crash. and: 02:10:01 AM 2097144 0 0.00 0 0.00 for the previous one. So I don't particularly suspect lack of swap. The machine should have way more RAM than it needs, so it's mainly just buffer cache. I did bump it up to 8GB just for fun though. > On Tue, 17 Jul 2012, Orion Poplawski wrote: > >> Our SL6.2 KVM and nfs/backup server has been crashing frequently recently >> (starting around Fri 13th - yikes!) with Kernel panic - Out of memory and no >> killable processes. The server has 48GB ram, 2GB swap, only about 15GB >> dedicated to VM guests. I've tried bumping up vm.min_free_kbytes to 262144 >> to no avail. Nothing strange is getting written to the logs before the crash. >> >> Happening with both 2.6.32-220.23.1 and 2.6.32-279.1.1. >> >> Anyone else seeing this? Any other ideas? I've set a serial console log to >> try to catch more information the next time it happens. >> >> -- >> Orion Poplawski >> Technical Manager 303-415-9701 x222 >> NWRA, Boulder Office FAX: 303-415-9702 >> 3380 Mitchell Lane [log in to unmask] >> Boulder, CO 80301 http://www.nwra.com >> -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane [log in to unmask] Boulder, CO 80301 http://www.nwra.com