hi stephan

the first lines of slabtop show

Active / Total Objects (% used)    : 238382 / 243482 (97.9%)
  Active / Total Slabs (% used)      : 4987 / 4987 (100.0%)
  Active / Total Caches (% used)     : 64 / 80 (80.0%)
  Active / Total Size (% used)       : 68080.12K / 69386.02K (98.1%)
  Minimum / Average / Maximum Object : 0.01K / 0.28K / 8.00K

is this similar?

(btw size-2097152 sounds like one of the default name used by hugectl 
(or hugeadm) from the hugetlbfs tools). is that mounted in your case? 
and are there any hugepages reserved? )


stijn

On 09/05/2011 05:10 PM, Stephan Wiesand wrote:
> Hi Stijn,
>
> On Sep 5, 2011, at 16:24, Stijn De Weirdt wrote:
>
>> hi all,
>>
>> we are having an "issue" with some SL61 nodes. after a reboot, free reports 1.4GB of memory in use, of which 24+163=187MB buffers+cache.
>>
>> i'm unable to identify what is holding the memory, and i'd like to know if others see this too and how i could proceed to find the culprit.
>
> yes, we see this as well. On a 48 GB system without users or special processes:
>
> # free -m
>               total       used       free     shared    buffers     cached
> Mem:         48388       1374      47013          0         30        186
> -/+ buffers/cache:       1157      47231
>
> In /proc/meminfo, I find that the difference to what I'd consider reasonable (and see on a 48GB SL5 system) is due to slabs.
>
> A "slabtop -s c" reveals that it's a "size-2097152" pool accounting for this. Do you see this as well?
>
> Cheers,
> 	Stephan
>
>>
>> (it is a 32core/64GB machine; kernel commandline has crashkernel=128M@16M (but no difference then eg crashkernel=auto and kdump is off))
>>
>> many thanks,
>>
>>
>> stijn
>>
>> free
>> # free -m
>>              total       used       free     shared    buffers     cached
>> Mem:         64554       1604      62949          0         24        166
>> -/+ buffers/cache:       1413      63140
>> Swap:        16394          0      16394
>>
>>
>> mem sorted top
>>
>> top - 16:13:52 up 13 min,  1 user,  load average: 0.00, 0.01, 0.01
>> Tasks: 694 total,   1 running, 693 sleeping,   0 stopped,   0 zombie
>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
>> Mem:  66103768k total,  1643336k used, 64460432k free,    25164k buffers
>> Swap: 16787916k total,        0k used, 16787916k free,   170552k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 2788 root      20   0 37988  25m 2876 S  0.0  0.0   0:00.06 pbs_mom
>> 2653 root      20   0  159m  12m 1472 S  0.0  0.0   0:00.19 ncm-cdispd
>> 2643 root      20   0  138m 5604  840 S  0.0  0.0   0:00.00 cdp-listend
>> 3276 root      20   0  120m 4156 3232 S  0.0  0.0   0:00.07 sshd
>> 2620 root      20   0  745m 3788 1764 S  0.0  0.0   0:00.12 automount
>> 3102 nslcd     20   0  427m 2936  488 S  0.0  0.0   0:00.00 nslcd
>> 3301 root      20   0  103m 1688 1336 S  0.0  0.0   0:00.05 bash
>> 3623 root      20   0 13528 1604  844 R  0.3  0.0   0:00.14 top
>>     1 root      20   0 21416 1544 1240 S  0.0  0.0   0:06.23 init
>> 2482 root      20   0  194m 1484 1108 S  0.0  0.0   0:00.14 qlgc_dsc
>> 2325 root      20   0  242m 1412  928 S  0.0  0.0   0:00.04 rsyslogd
>> 2459 rpcuser   20   0 23112 1168  884 S  0.0  0.0   0:00.00 rpc.statd
>> 2606 root      18  -2 10956 1144  412 S  0.0  0.0   0:00.03 udevd
>> 3164 nscd      20   0  583m 1132  788 S  0.0  0.0   0:00.02 nscd
>> 2697 root      20   0 62040 1064  464 S  0.0  0.0   0:00.00 sshd
>>   943 root      16  -4 10960 1052  316 S  0.0  0.0   0:00.12 udevd
>> 2607 root      18  -2 10956 1052  320 S  0.0  0.0   0:00.00 udevd
>> 2723 root      20   0  112m 1012  380 S  0.0  0.0   0:00.00 crond
>> 2707 root      20   0 22488  992  752 S  0.0  0.0   0:00.03 xinetd
>> 2439 rpc       20   0 18940  908  672 S  0.0  0.0   0:00.04 rpcbind
>> 2568 dbus      20   0 23448  876  604 S  0.0  0.0   0:00.01 dbus-daemon
>> 2972 nagios    20   0 37096  796  452 S  0.0  0.0   0:00.00 nrpe
>