On 07/17/2012 11:46 AM, Stephan Wiesand wrote:
> On Jul 17, 2012, at 19:22 , Orion Poplawski wrote:
>
>> Our SL6.2 KVM and nfs/backup server has been crashing frequently recently (starting around Fri 13th - yikes!) with Kernel panic - Out of memory and no killable processes.  The server has 48GB ram, 2GB swap, only about 15GB dedicated to VM guests.  I've tried bumping up vm.min_free_kbytes to 262144 to no avail.  Nothing strange is getting written to the logs before the crash.

Hmm, I suppose bumping up min_free_kbytes might be making things worse?

>> Happening with both 2.6.32-220.23.1 and 2.6.32-279.1.1.
>>
>> Anyone else seeing this?
>
> Not on our KVM servers (which don't have any other duties though), which have been running -220.23.1 for three weeks.
>
>>   Any other ideas?
>
> Is swap space sufficient?

It was 2GB, but barely used.  The system should have way more RAM than needed. 
  Upped to 8GB.
>
> Have you modified vm.overcommit_* ? Doing so may help turning the panics into allocation failures that can be handled.
>

Haven't modified them:

vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.nr_overcommit_hugepages = 0

I suppose:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

would limit total RAM usage to about 46.4GB which should be safe.  I might try 
that next.

> Do any slab pools keep growing, to an unusual size?
>

Here's what I have shortly after reboot.  I'll keep watching it.

  Active / Total Objects (% used)    : 1500116 / 1526912 (98.2%)
  Active / Total Slabs (% used)      : 37344 / 37481 (99.6%)
  Active / Total Caches (% used)     : 134 / 204 (65.7%)
  Active / Total Size (% used)       : 147024.04K / 152389.66K (96.5%)
  Minimum / Average / Maximum Object : 0.02K / 0.10K / 4096.00K

   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
123540 123283  99%    0.19K   6177       20     24708K size-192
236059 235736  99%    0.06K   4001       59     16004K ksm_rmap_item
484128 484094  99%    0.02K   3362      144     13448K avtab_node
    203    203 100%   32.12K    203        1     12992K kmem_cache
341936 341360  99%    0.03K   3053      112     12212K size-32
  14136  14109  99%    0.58K   2356        6      9424K inode_cache
  71595  71595 100%    0.10K   1935       37      7740K buffer_head
  31500  31078  98%    0.19K   1575       20      6300K dentry
  10857  10857 100%    0.55K   1551        7      6204K radix_tree_node
   5140   4839  94%    1.00K   1285        4      5140K size-1024
   4480   4462  99%    1.00K   1120        4      4480K ext4_inode_cache
   5772   5684  98%    0.62K    962        6      3848K proc_inode_cache
  24300  24269  99%    0.14K    900       27      3600K sysfs_dir_cache
   1558   1348  86%    2.00K    779        2      3116K size-2048
  13794  13421  97%    0.20K    726       19      2904K vm_area_struct
   1074   1050  97%    2.59K    358        3      2864K task_struct
    699    699 100%    4.00K    699        1      2796K size-4096
    975    951  97%    2.06K    325        3      2600K sighand_cache
   4536   3262  71%    0.50K    567        8      2268K size-512
     17     17 100%  128.00K     17        1      2176K size-131072
  27401  27229  99%    0.07K    517       53      2068K selinux_inode_security
   2255   2232  98%    0.78K    451        5      1804K shmem_inode_cache
  22007  21365  97%    0.06K    373       59      1492K size-64
  10950   9234  84%    0.12K    365       30      1460K size-128
     22     22 100%   64.00K     22        1      1408K size-65536
    326    326 100%    4.00K    326        1      1304K biovec-256
   5720   4125  72%    0.19K    286       20      1144K filp
   1020    985  96%    1.00K    255        4      1020K signal_cache

After some disk activity I'm at:

  Active / Total Objects (% used)    : 4829537 / 4855308 (99.5%)
  Active / Total Slabs (% used)      : 163899 / 163988 (99.9%)
  Active / Total Caches (% used)     : 132 / 204 (64.7%)
  Active / Total Size (% used)       : 630344.49K / 634988.70K (99.3%)
  Minimum / Average / Maximum Object : 0.02K / 0.13K / 4096.00K

   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
2918893 2918736  99%    0.10K  78889       37    315556K buffer_head
112616 112599  99%    1.00K  28154        4    112616K ext4_inode_cache
  98637  98563  99%    0.55K  14091        7     56364K radix_tree_node
165540 165060  99%    0.19K   8277       20     33108K dentry
123520 123363  99%    0.19K   6176       20     24704K size-192
236059 235736  99%    0.06K   4001       59     16004K ksm_rmap_item
484128 484094  99%    0.02K   3362      144     13448K avtab_node
    203    203 100%   32.12K    203        1     12992K kmem_cache
342384 341570  99%    0.03K   3057      112     12228K size-32
139019 138470  99%    0.07K   2623       53     10492K selinux_inode_security

Still watching it...

-- 
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA, Boulder Office                  FAX: 303-415-9702
3380 Mitchell Lane                       [log in to unmask]
Boulder, CO 80301                   http://www.nwra.com