Hi Stijn,

On Sep 5, 2011, at 17:23, Stijn De Weirdt wrote:

> hi stephan
> 
> the first lines of slabtop show
> 
> Active / Total Objects (% used)    : 238382 / 243482 (97.9%)
> Active / Total Slabs (% used)      : 4987 / 4987 (100.0%)
> Active / Total Caches (% used)     : 64 / 80 (80.0%)
> Active / Total Size (% used)       : 68080.12K / 69386.02K (98.1%)
> Minimum / Average / Maximum Object : 0.01K / 0.28K / 8.00K
> 
> is this similar?

not really.

% slabtop -s c --once |head -10
 Active / Total Objects (% used)    : 185625 / 205302 (90.4%)
 Active / Total Slabs (% used)      : 16756 / 16757 (100.0%)
 Active / Total Caches (% used)     : 101 / 182 (55.5%)
 Active / Total Size (% used)       : 856865.31K / 859451.65K (99.7%)
 Minimum / Average / Maximum Object : 0.02K / 4.19K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
   384    384 100% 2048.00K    384        1    786432K size-2097152           
 26276  26250  99%    1.00K   6569        4     26276K ext4_inode_cache       
   181    181 100%   32.12K    181        1     11584K kmem_cache             

> (btw size-2097152 sounds like one of the default name used by hugectl (or hugeadm) from the hugetlbfs tools). is that mounted in your case? and are there any hugepages reserved? )

Not that I'd know of. But wasn't there a new feature called "transparent hugepage support" in 6.1?

Cheers,
	Stephan
> 
> 
> stijn
> 
> On 09/05/2011 05:10 PM, Stephan Wiesand wrote:
>> Hi Stijn,
>> 
>> On Sep 5, 2011, at 16:24, Stijn De Weirdt wrote:
>> 
>>> hi all,
>>> 
>>> we are having an "issue" with some SL61 nodes. after a reboot, free reports 1.4GB of memory in use, of which 24+163=187MB buffers+cache.
>>> 
>>> i'm unable to identify what is holding the memory, and i'd like to know if others see this too and how i could proceed to find the culprit.
>> 
>> yes, we see this as well. On a 48 GB system without users or special processes:
>> 
>> # free -m
>>              total       used       free     shared    buffers     cached
>> Mem:         48388       1374      47013          0         30        186
>> -/+ buffers/cache:       1157      47231
>> 
>> In /proc/meminfo, I find that the difference to what I'd consider reasonable (and see on a 48GB SL5 system) is due to slabs.
>> 
>> A "slabtop -s c" reveals that it's a "size-2097152" pool accounting for this. Do you see this as well?
>> 
>> Cheers,
>> 	Stephan
>> 
>>> 
>>> (it is a 32core/64GB machine; kernel commandline has crashkernel=128M@16M (but no difference then eg crashkernel=auto and kdump is off))
>>> 
>>> many thanks,
>>> 
>>> 
>>> stijn
>>> 
>>> free
>>> # free -m
>>>             total       used       free     shared    buffers     cached
>>> Mem:         64554       1604      62949          0         24        166
>>> -/+ buffers/cache:       1413      63140
>>> Swap:        16394          0      16394
>>> 
>>> 
>>> mem sorted top
>>> 
>>> top - 16:13:52 up 13 min,  1 user,  load average: 0.00, 0.01, 0.01
>>> Tasks: 694 total,   1 running, 693 sleeping,   0 stopped,   0 zombie
>>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
>>> Mem:  66103768k total,  1643336k used, 64460432k free,    25164k buffers
>>> Swap: 16787916k total,        0k used, 16787916k free,   170552k cached
>>> 
>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 2788 root      20   0 37988  25m 2876 S  0.0  0.0   0:00.06 pbs_mom
>>> 2653 root      20   0  159m  12m 1472 S  0.0  0.0   0:00.19 ncm-cdispd
>>> 2643 root      20   0  138m 5604  840 S  0.0  0.0   0:00.00 cdp-listend
>>> 3276 root      20   0  120m 4156 3232 S  0.0  0.0   0:00.07 sshd
>>> 2620 root      20   0  745m 3788 1764 S  0.0  0.0   0:00.12 automount
>>> 3102 nslcd     20   0  427m 2936  488 S  0.0  0.0   0:00.00 nslcd
>>> 3301 root      20   0  103m 1688 1336 S  0.0  0.0   0:00.05 bash
>>> 3623 root      20   0 13528 1604  844 R  0.3  0.0   0:00.14 top
>>>    1 root      20   0 21416 1544 1240 S  0.0  0.0   0:06.23 init
>>> 2482 root      20   0  194m 1484 1108 S  0.0  0.0   0:00.14 qlgc_dsc
>>> 2325 root      20   0  242m 1412  928 S  0.0  0.0   0:00.04 rsyslogd
>>> 2459 rpcuser   20   0 23112 1168  884 S  0.0  0.0   0:00.00 rpc.statd
>>> 2606 root      18  -2 10956 1144  412 S  0.0  0.0   0:00.03 udevd
>>> 3164 nscd      20   0  583m 1132  788 S  0.0  0.0   0:00.02 nscd
>>> 2697 root      20   0 62040 1064  464 S  0.0  0.0   0:00.00 sshd
>>>  943 root      16  -4 10960 1052  316 S  0.0  0.0   0:00.12 udevd
>>> 2607 root      18  -2 10956 1052  320 S  0.0  0.0   0:00.00 udevd
>>> 2723 root      20   0  112m 1012  380 S  0.0  0.0   0:00.00 crond
>>> 2707 root      20   0 22488  992  752 S  0.0  0.0   0:00.03 xinetd
>>> 2439 rpc       20   0 18940  908  672 S  0.0  0.0   0:00.04 rpcbind
>>> 2568 dbus      20   0 23448  876  604 S  0.0  0.0   0:00.01 dbus-daemon
>>> 2972 nagios    20   0 37096  796  452 S  0.0  0.0   0:00.00 nrpe
>> 
> 
> 

-- 
Stephan Wiesand
DESY -DV-
Platanenallee 6
15738 Zeuthen, Germany