I should have also said this is an SL 4.3 system:
[gahs@neutrino ~]$ cat /etc/redhat-release
Scientific Linux SL release 4.3 (Beryllium)
[gahs@neutrino ~]$ uname -a
Linux neutrino.phys.ksu.edu 2.6.9-42.0.3.EL #1 Thu Oct 5 14:43:07 CDT
2006 i686 i686 i386 GNU/Linux
Cheers,
Glenn.
Glenn Horton-Smith wrote (3/14/2007 2:20 PM):
> I have a bizarre occurrence to report and ask about. Last night at
> 4:15 AM, the "mrtg" cron job started producing an error when it tries
> to run every 5 minutes from it's cron.d script on neutrino:
>
> syntax error at /usr/lib/perl5/5.8.5/IO/Socket/INET.pm line 114, near ")
> )"
> Compilation failed in require at
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/IO/Socket.pm line 21.
>
>
> I looked in /usr/lib/perl5/5.8.5/IO/Socket/INET.pm, and sure enough,
> there's a syntax error in INET.pm:
>
> ($laddr,$lport,$proto) = _sock_info($arg->{LocalAddr},
> $arg->{LocalPort},
> $arg->{Proto})
> )or return _error($sock, $!, $@);
>
> Note the extra close parenthesis. I have a backup image made at 4:00
> AM, and can confirm that this file was not like this at 4:00 AM:
>
> [root@neutrino ~]# diff -r /usr/lib/perl5/5.8.5/IO/Socket/
> /backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/
> diff -r /usr/lib/perl5/5.8.5/IO/Socket/INET.pm
> /backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/INET.pm
> 114c114
> < )or return _error($sock, $!, $@);
> ---
> > or return _error($sock, $!, $@);
>
>
> More disturbingly, THOUSANDS of binaries in /usr/bin have changed:
>
> [root@neutrino ~]# diff -r -q /usr/bin/ /backup2/backup/usr/bin/ | wc
> 3097 15052 204950
>
> Stranger still, the file contents were changed, but the file lengths
> and time stamps stayed exactly the same: e.g.,
>
> [root@neutrino ~]# diff /usr/bin/perl /backup2/backup/usr/bin/perl
> Binary files /usr/bin/perl and /backup2/backup/usr/bin/perl differ
> [root@neutrino ~]# ls -l /usr/bin/perl /backup2/backup/usr/bin/perl
> -rwxr-xr-x 1 root root 15164 Aug 10 2006 /backup2/backup/usr/bin/perl
> -rwxr-xr-x 2 root root 15164 Aug 10 2006 /usr/bin/perl
>
> That's weird. But the contents definitely changed:
>
> [root@neutrino ~]# strings /backup2/backup/usr/bin/perl | head
> /lib/ld-linux.so.2
> Sf#EKC|
> Xf#E
> Rf#E
> Rf#E
> \f#E7
> Sf#E8`
> Rf#E
> Rf#E
> Rf#EI
>
> [root@neutrino ~]# strings /usr/bin/perl | head
> /lib/ld-linux.so.2
> PTRh
> ,[^_]
> ,[^_]
> ,[^_]
> ,[^_]
> B @uM
> ,[^_]
> ,[^_]
> ,[^_]
>
> That looks suspicious. I'd almost suspect disk corruption, except
> this new perl runs fine, as long as you don't import a buggy library.
>
> Now here's the really weird part: there was no yum update last night
> to introduce this, no activity in any log files to indicate otherwise,
> no files changed in the rpm or yum cache directories, etc. (See log
> snippets at end of message.)
>
> The system was up the whole time, no one logged in or out at this time
> according to the logs. I tried chkrootkit and clamscan, and they find
> no problems. (Their files are unchanged, by the way.) Anyway, it
> doesn't smell like a hack, more like a bad update, but I can't figure
> out _how_.
>
> I'm just about at the stage where I save the logs for forensics and
> get the installation disks for a re-install, but I thought I'd check
> first for wisdom from the mailing list. Any ideas???
>
> Cheers,
> Glenn Horton-Smith
>
>
> From /var/log/messages:
> Mar 13 12:50:30 neutrino rsyncd[31699]: sent 9753 bytes received
> 95558 bytes t
> otal size 417533980
> Mar 13 23:58:36 neutrino ntpd[3151]: synchronized to 129.130.252.204,
> stratum 2
> Mar 14 00:13:17 neutrino ntpd[3151]: synchronized to 129.130.252.205,
> stratum 2
> Mar 14 00:32:38 neutrino ntpd[3151]: synchronized to 129.130.252.203,
> stratum 2
> Mar 14 04:06:00 neutrino clamd[10519]: SelfCheck: Database
> modification detected
> . Forcing reload.
> Mar 14 04:06:00 neutrino clamd[10519]: Reading databases from /var/clamav
> Mar 14 04:06:07 neutrino clamav-milter[10703]: Database has changed,
> loading upd
> ated database
> Mar 14 04:06:09 neutrino clamav-milter[10703]: Loaded ClamAV
> 0.90/2838/Wed Mar 1
> 4 02:33:07 2007
> Mar 14 04:06:09 neutrino clamav-milter[10703]: ClamAV: Protecting
> against 99277
> viruses
> Mar 14 04:06:10 neutrino clamav-milter[10703]: Database correctly
> reloaded (9927
> 7 viruses)
> Mar 14 04:06:11 neutrino clamd[10519]: Database correctly reloaded
> (99277 signat
> ures)
> Mar 14 09:20:03 neutrino ntpd[3151]: synchronized to 129.130.252.205,
> stratum 2
>
> From /var/log/cron:
> Mar 14 03:55:01 neutrino crond[1937]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
> Mar 14 04:00:01 neutrino crond[1940]: (root) CMD (/usr/lib/sa/sa1 1 1)
> Mar 14 04:00:01 neutrino crond[1943]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
> Mar 14 04:01:01 neutrino crond[1945]: (root) CMD (run-parts
> /etc/cron.hourly)
> Mar 14 04:02:01 neutrino crond[1951]: (root) CMD (run-parts
> /etc/cron.daily)
> Mar 14 04:02:22 neutrino anacron[2407]: Updated timestamp for job
> `cron.daily' t
> o 2007-03-14
> Mar 14 04:05:01 neutrino crond[2418]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
> Mar 14 04:05:01 neutrino crond[2419]: (dchooz) CMD
> ($HOME/test_build/new_test_bu
> ild.bash >| $HOME/test_build/test_build.html 2>&1)
> Mar 14 04:10:01 neutrino crond[2523]: (root) CMD (/usr/lib/sa/sa1 1 1)
> Mar 14 04:10:01 neutrino crond[2524]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
> Mar 14 04:14:01 neutrino crond[2528]: (KamLAND) CMD
> ($HOME/test_build/new_test_b
> uild.bash >| $HOME/test_build/test_build.html 2>&1)
> Mar 14 04:15:01 neutrino crond[2569]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
> Mar 14 04:20:01 neutrino crond[2577]: (root) CMD (/usr/lib/sa/sa1 1 1)
> Mar 14 04:20:01 neutrino crond[2580]: (root) CMD (/usr/bin/mrtg
> /etc/mrtg/mrtg.c
> fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file
> /var/lib/mrtg/mrtg.ok)
>
>
> [root@neutrino ~]# diff -r -q /usr/bin/ /backup2/backup/usr/bin/ | wc
> 3097 15052 204950
>
>
> [root@neutrino ~]# time nice clamscan -l scan_usrbin_2.txt /usr/bin
> ... lots of output ...
> ----------- SCAN SUMMARY -----------
> Known viruses: 99277
> Engine version: 0.90.1
> Scanned directories: 1
> Scanned files: 3102
> Infected files: 0
> Data scanned: 283.21 MB
> Time: 50.992 sec (0 m 50 s)
|
