Last night during class time I had a chance to check some of the machines with the frozen displays, and I am not sure what to make of what I found. Running 'lsof -p $PID' with (PID being 5044) on one of the affected machines, gave this which, doesn't tell me much:
10.10.10 5044 root cwd DIR 8,7 4096 2 /
10.10.10 5044 root rtd DIR 8,7 4096 2 /
10.10.10 5044 root txt unknown /proc/5044/exe
I also ran pstree and I will put that output below, but I think I may be barking up the wrong tree. While some of my clients were freezing up, I saw that my NFS server was getting very high 'top' loads. Fortunately I have sysstat running on the server and after class 'sar -u' showed that %iowait went from less than 1 before class to a high of 53 after class began, and stayed high until class ended. Here is the relevant 'chunk' of the sar -u output:
05:20:01 PM all 0.03 0.00 0.07 0.17 0.00 99.73
05:30:01 PM all 0.03 0.00 0.03 0.11 0.00 99.83
05:40:01 PM all 0.18 0.00 0.50 1.88 0.00 97.44
05:50:01 PM all 0.16 0.00 1.12 6.93 0.00 91.78
06:00:01 PM all 0.73 0.00 5.23 32.61 0.00 61.43
06:10:01 PM all 0.77 0.00 6.55 53.67 0.00 39.01
06:20:01 PM all 0.13 0.00 4.81 27.81 0.00 67.25
06:30:01 PM all 0.13 0.00 6.69 21.71 0.00 71.47
06:40:01 PM all 0.11 0.00 3.47 33.34 0.00 63.08
06:50:01 PM all 0.11 0.00 3.20 31.02 0.00 65.67
07:00:01 PM all 0.24 0.00 3.93 30.79 0.00 65.05
07:10:01 PM all 0.16 0.00 3.63 20.51 0.00 75.71
07:20:01 PM all 0.18 0.00 5.23 1.45 0.00 93.13
07:30:01 PM all 0.10 0.00 5.72 0.70 0.00 93.48
Average: all 0.06 0.01 0.46 2.13 0.00 97.34
The NFS server is a virtual machine in running ESXI 4.1 and VMware tools IS installed. Could this be slow disk access, and thus a VMware misconfiguration? I hate to admit it, but I am at a loss.
I can run other sar reports on yesterday's (Wednesday's) data if anyone thinks there may be something in there to help.
For what its worth, here is the output from pstree from one of the affected clients, and I do NOT see the PID that I was looking for:
init(1)-+-NetworkManager(1782)-+-dhclient(1808)
| `-{NetworkManager}(1809)
|-abrtd(2341)
|-acpid(2039)
|-anacron(3615)
|-atd(2413)
|-atieventsd(2421)---authatieventsd.(4134)
|-auditd(1547)-+-audispd(1549)-+-sedispatch(1550)
| | `-{audispd}(1551)
| `-{auditd}(1548)
|-automount(2134)-+-{automount}(2135)
| |-{automount}(2136)
| |-{automount}(2139)
| |-{automount}(2142)
| |-{automount}(2143)
| `-{automount}(2144)
|-avahi-daemon(1794)---avahi-daemon(1795)
|-bonobo-activati(4549)---{bonobo-activat}(4550)
|-cachefilesd(1597)
|-certmonger(2435)
|-clock-applet(4644)
|-console-kit-dae(2521)-+-{console-kit-da}(2522)
| |-{console-kit-da}(2523)
| |-{console-kit-da}(2524)
| |-{console-kit-da}(2525)
| |-{console-kit-da}(2526)
| |-{console-kit-da}(2527)
| |-{console-kit-da}(2528)
| |-{console-kit-da}(2529)
| |-{console-kit-da}(2530)
| |-{console-kit-da}(2531)
| |-{console-kit-da}(2532)
| |-{console-kit-da}(2533)
| |-{console-kit-da}(2534)
| |-{console-kit-da}(2535)
| |-{console-kit-da}(2536)
| |-{console-kit-da}(2537)
| |-{console-kit-da}(2538)
| |-{console-kit-da}(2539)
| |-{console-kit-da}(2540)
| |-{console-kit-da}(2541)
| |-{console-kit-da}(2542)
| |-{console-kit-da}(2543)
| |-{console-kit-da}(2544)
| |-{console-kit-da}(2545)
| |-{console-kit-da}(2546)
| |-{console-kit-da}(2547)
| |-{console-kit-da}(2548)
| |-{console-kit-da}(2549)
| |-{console-kit-da}(2550)
| |-{console-kit-da}(2551)
| |-{console-kit-da}(2552)
| |-{console-kit-da}(2553)
| |-{console-kit-da}(2554)
| |-{console-kit-da}(2555)
| |-{console-kit-da}(2556)
| |-{console-kit-da}(2557)
| |-{console-kit-da}(2558)
| |-{console-kit-da}(2559)
| |-{console-kit-da}(2560)
| |-{console-kit-da}(2561)
| |-{console-kit-da}(2562)
| |-{console-kit-da}(2563)
| |-{console-kit-da}(2564)
| |-{console-kit-da}(2565)
| |-{console-kit-da}(2566)
| |-{console-kit-da}(2567)
| |-{console-kit-da}(2568)
| |-{console-kit-da}(2569)
| |-{console-kit-da}(2570)
| |-{console-kit-da}(2571)
| |-{console-kit-da}(2572)
| |-{console-kit-da}(2573)
| |-{console-kit-da}(2574)
| |-{console-kit-da}(2575)
| |-{console-kit-da}(2576)
| |-{console-kit-da}(2577)
| |-{console-kit-da}(2578)
| |-{console-kit-da}(2579)
| |-{console-kit-da}(2580)
| |-{console-kit-da}(2581)
| |-{console-kit-da}(2582)
| |-{console-kit-da}(2583)
| `-{console-kit-da}(2585)
|-crond(2402)
|-cupsd(1955)
|-dbus-daemon(1772)
|-dbus-daemon(2883)
|-dbus-launch(2591)
|-dbus-launch(2882)
|-devkit-power-da(2602)
|-fcoemon(1760)
|-firefox(4968)
|-gconf-im-settin(4534)
|-gconfd-2(3175)
|-gdm-binary(2449)---gdm-simple-slav(2490)-+-Xorg(2492)
| `-gdm-session-wor(2671)---tcsh(2849)---gnome-session(4148)-+-bluetooth-apple(436+
| |-gdu-notificatio(432+
| |-gnome-panel(4253)
| |-gnome-power-man(434+
| |-gnome-volume-co(432+
| |-gpk-update-icon(430+
| |-krb5-auth-dialo(435+
| |-metacity(4244)
| |-nautilus(4276)
| |-nm-applet(4342)
| |-polkit-gnome-au(432+
| |-python(4294)
| `-{gnome-session}(422+
|-gdm-user-switch(4640)
|-gedit(4779)-+-{gedit}(4894)
| |-{gedit}(5037)
| |-{gedit}(5038)
| `-{gedit}(5039)
|-gnome-keyring-d(2831)-+-{gnome-keyring-}(2832)
| `-{gnome-keyring-}(4237)
|-gnome-screensav(4665)
|-gnome-settings-(4235)---{gnome-settings}(4248)
|-gnote(4635)
|-gvfs-afc-volume(4573)---{gvfs-afc-volum}(4574)
|-gvfs-gdu-volume(4569)
|-gvfs-gphoto2-vo(4571)
|-gvfsd(3168)
|-gvfsd-burn(4754)
|-gvfsd-metadata(4794)
|-gvfsd-trash(4656)
|-hald(2048)---hald-runner(2049)-+-hald-addon-acpi(2096)
| |-hald-addon-inpu(2088)
| `-hald-addon-stor(2097)
|-im-settings-dae(4371)
|-lldpad(1734)
|-master(2332)-+-pickup(2347)
| `-qmgr(2348)
|-mingetty(2454)
|-mingetty(2456)
|-mingetty(2458)
|-mingetty(2460)
|-mingetty(2462)
|-modem-manager(1789)
|-notification-ar(4642)
|-ntpd(2249)
|-pcscd(2114)---{pcscd}(2129)
|-polkitd(2647)
|-pulseaudio(4331)-+-gconf-helper(4563)
| |-{pulseaudio}(4535)
| `-{pulseaudio}(4539)
|-qpidd(2356)-+-{qpidd}(2357)
| |-{qpidd}(2358)
| `-{qpidd}(2359)
|-rpc.idmapd(1864)
|-rpc.mountd(2190)
|-rpc.rquotad(2175)
|-rpc.statd(1818)
|-rpcbind(1648)
|-rsyslogd(1574)-+-{rsyslogd}(1575)
| |-{rsyslogd}(1576)
| `-{rsyslogd}(1578)
|-rtkit-daemon(2661)-+-{rtkit-daemon}(2662)
| `-{rtkit-daemon}(2663)
|-seahorse-agent(3155)
|-seahorse-daemon(4243)
|-sshd(2233)---sshd(5003)---bash(5005)---pstree(5057)
|-sssd(2216)-+-sssd_be(2281)
| |-sssd_nss(2286)
| `-sssd_pam(2287)
|-stap-serverd(1927)---{stap-serverd}(1932)
|-udevd(542)-+-udevd(1166)
| `-udevd(1745)
|-udisks-daemon(4373)---udisks-daemon(4374)
|-wpa_supplicant(1813)
`-xinetd(2241)
________________________________________
From: Christopher Tooley [[log in to unmask]]
Sent: Wednesday, November 28, 2012 1:00 PM
To: David Fitzgerald
Cc: [log in to unmask]
Subject: Re: clients slow down due to unknown process
If/when you find out what it is, would you kindly report back to the list what you find? This has got me really curious now. :D
-Chris
On 2012-11-28, at 5:51 AM, David Fitzgerald <[log in to unmask]> wrote:
> Thank you everyone for all the good ideas. I have class this evening and will be able to use your suggestions. I'll let you know what I find.
>
> Dave
>
> -----Original Message-----
> From: Robert Blair [mailto:[log in to unmask]]
> Sent: Tuesday, November 27, 2012 11:56 AM
> To: Sergio Ballestrero
> Cc: David Fitzgerald; [log in to unmask]
> Subject: Re: clients slow down due to unknown process
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> "/usr/sbin/lsof -p $PID" will also list all of the resources it uses which is often a big help in figuring out wtf it is all about.
>
> On 11/27/2012 10:52 AM, Sergio Ballestrero wrote:
>> Hello David,
>> I'm not familiar with freeIPA, but anyway you can start by better
>> identifying the process.
>> In top, get the PID and look under /proc/$PID - in particular exe
>> will be a link to the binary, like lrwxrwxrwx 1 root root 0 Nov 27
>> 01:41 /proc/1/exe -> /sbin/init
>>
>> pstree -p -H $PID
>> will help you identify the parent process, if there's one.
>>
>> Cheers,
>> Sergio
>>
>> On 27 Nov 2012, at 16:21, David Fitzgerald wrote:
>>
>>> Hello,
>>>
>>> Sorry for the length of this post, but I want to make sure I give all
>>> the information needed for someone to help.
>>>
>>> I have a lab of 25 workstations running Scientific Linux 6.2. User
>>> accounts are authenticated via freeIPA, and auto mounted to an NFS
>>> server and the users use Gnome 2.8. The NFS and freeIPA servers are
>>> located on the same server (IP 10.10.10.10) which is also running
>>> Scientific Linux 6.2 and is a virtual guest in VMware ESXI 4.1.
>>>
>>> During class when the workstations are most heavily in use, the
>>> students are writing Fortran programs with gedit and usually have
>>> firefox up as well. Here is my predicament. During class some of
>>> the workstation screens will freeze with no mouse or keyboard input.
>>> This can last for varying lengths of time, sometimes a few minutes,
>>> some other times for the full length of the class. I can ssh in to
>>> the frozen machines and top will show load averages of up to 4 or more.
>>> The process taking up the most CPU is one I don't recognize named
>>> 10.10.10.10-ma. The 10.10.10.10 being the IP address of my server.
>>> I have no idea what that process is related to, whether it's freeIPA,
>>> NFS, Gnome or something else. Killing the process doesn't help as it
>>> simply restarts with a new PID. Note that the freezing does NOT
>>> happen when only a few people are using the lab, so reproducing the
>>> problem outside of class time is difficult.
>>>
>>> Can anyone help me track down this problem and fix it?
>>>
>>> I appreciate any help you can give.
>>>
>>> Thanks!
>>>
>>> Dave
>>>
>>>
>>> +++++++++++++++++++++++
>>> David Fitzgerald
>>> Department of Earth Sciences
>>> Millersville University
>>> Millersville, PA 17551
>>>
>>> Phone: 717-871-2394
>>>
>>
>> --
>> Sergio Ballestrero - http://physics.uj.ac.za/psiwiki/Ballestrero
>> University of Johannesburg, Physics Department ATLAS TDAQ sysadmin
>> team - Office:75282 OnCall:164851
>>
>>
>>
>>
>>
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (GNU/Linux)
>
> iQEUAwUBULTwmfQM1KNWz8QaAQLU0Qf2JXa29RVDhJALq2TD72Nis4wAmxlqFIYP
> rIo5sHBUI+o/bebsDit9qoC+hWuCK3+xDai9fzF2jUQqXfhRZiPHjdQRpCViMurY
> Wp+aVZWCD1U3KusuWMSWlv6Xdx0QmaMQr8Nh8JRRWUi8cNEgAO2Th1txwdu3auJb
> LssTFmwUjLUEC0mKhgx6520hisirfOHNTnF3rQCN5ilZGEYEZ2vMm/lcm5yI0Sqc
> wdqWUXVYGNsBepFf4bRWaWPX0Hbf6sbLgoJNUHJOJ2pGpc3MUp3SiGsIIUGkZwPW
> xT6kS523J+nItY/odmvdl+ibHRVa7TgDx0xhuqISarr39g00yvvx
> =RQky
> -----END PGP SIGNATURE-----
|
