I have a bizarre occurrence to report and ask about. Last night at 4:15
AM, the "mrtg" cron job started producing an error when it tries to run
every 5 minutes from it's cron.d script on neutrino:
syntax error at /usr/lib/perl5/5.8.5/IO/Socket/INET.pm line 114, near ")
)"
Compilation failed in require at
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/IO/Socket.pm line 21.
I looked in /usr/lib/perl5/5.8.5/IO/Socket/INET.pm, and sure enough,
there's a syntax error in INET.pm:
($laddr,$lport,$proto) = _sock_info($arg->{LocalAddr},
$arg->{LocalPort},
$arg->{Proto})
)or return _error($sock, $!, $@);
Note the extra close parenthesis. I have a backup image made at 4:00
AM, and can confirm that this file was not like this at 4:00 AM:
[root@neutrino ~]# diff -r /usr/lib/perl5/5.8.5/IO/Socket/
/backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/
diff -r /usr/lib/perl5/5.8.5/IO/Socket/INET.pm
/backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/INET.pm
114c114
< )or return _error($sock, $!, $@);
---
> or return _error($sock, $!, $@);
More disturbingly, THOUSANDS of binaries in /usr/bin have changed:
[root@neutrino ~]# diff -r -q /usr/bin/ /backup2/backup/usr/bin/ | wc
3097 15052 204950
Stranger still, the file contents were changed, but the file lengths and
time stamps stayed exactly the same: e.g.,
[root@neutrino ~]# diff /usr/bin/perl /backup2/backup/usr/bin/perl
Binary files /usr/bin/perl and /backup2/backup/usr/bin/perl differ
[root@neutrino ~]# ls -l /usr/bin/perl /backup2/backup/usr/bin/perl
-rwxr-xr-x 1 root root 15164 Aug 10 2006 /backup2/backup/usr/bin/perl
-rwxr-xr-x 2 root root 15164 Aug 10 2006 /usr/bin/perl
That's weird. But the contents definitely changed:
[root@neutrino ~]# strings /backup2/backup/usr/bin/perl | head
/lib/ld-linux.so.2
Sf#EKC|
Xf#E
Rf#E
Rf#E
\f#E7
Sf#E8`
Rf#E
Rf#E
Rf#EI
[root@neutrino ~]# strings /usr/bin/perl | head
/lib/ld-linux.so.2
PTRh
,[^_]
,[^_]
,[^_]
,[^_]
B @uM
,[^_]
,[^_]
,[^_]
That looks suspicious. I'd almost suspect disk corruption, except this
new perl runs fine, as long as you don't import a buggy library.
Now here's the really weird part: there was no yum update last night to
introduce this, no activity in any log files to indicate otherwise, no
files changed in the rpm or yum cache directories, etc. (See log
snippets at end of message.)
The system was up the whole time, no one logged in or out at this time
according to the logs. I tried chkrootkit and clamscan, and they find
no problems. (Their files are unchanged, by the way.) Anyway, it
doesn't smell like a hack, more like a bad update, but I can't figure
out _how_.
I'm just about at the stage where I save the logs for forensics and get
the installation disks for a re-install, but I thought I'd check first
for wisdom from the mailing list. Any ideas???
Cheers,
Glenn Horton-Smith
From /var/log/messages:
Mar 13 12:50:30 neutrino rsyncd[31699]: sent 9753 bytes received 95558
bytes t
otal size 417533980
Mar 13 23:58:36 neutrino ntpd[3151]: synchronized to 129.130.252.204,
stratum 2
Mar 14 00:13:17 neutrino ntpd[3151]: synchronized to 129.130.252.205,
stratum 2
Mar 14 00:32:38 neutrino ntpd[3151]: synchronized to 129.130.252.203,
stratum 2
Mar 14 04:06:00 neutrino clamd[10519]: SelfCheck: Database modification
detected
. Forcing reload.
Mar 14 04:06:00 neutrino clamd[10519]: Reading databases from /var/clamav
Mar 14 04:06:07 neutrino clamav-milter[10703]: Database has changed,
loading upd
ated database
Mar 14 04:06:09 neutrino clamav-milter[10703]: Loaded ClamAV
0.90/2838/Wed Mar 1
4 02:33:07 2007
Mar 14 04:06:09 neutrino clamav-milter[10703]: ClamAV: Protecting
against 99277
viruses
Mar 14 04:06:10 neutrino clamav-milter[10703]: Database correctly
reloaded (9927
7 viruses)
Mar 14 04:06:11 neutrino clamd[10519]: Database correctly reloaded
(99277 signat
ures)
Mar 14 09:20:03 neutrino ntpd[3151]: synchronized to 129.130.252.205,
stratum 2
From /var/log/cron:
Mar 14 03:55:01 neutrino crond[1937]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 14 04:00:01 neutrino crond[1940]: (root) CMD (/usr/lib/sa/sa1 1 1)
Mar 14 04:00:01 neutrino crond[1943]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 14 04:01:01 neutrino crond[1945]: (root) CMD (run-parts
/etc/cron.hourly)
Mar 14 04:02:01 neutrino crond[1951]: (root) CMD (run-parts /etc/cron.daily)
Mar 14 04:02:22 neutrino anacron[2407]: Updated timestamp for job
`cron.daily' t
o 2007-03-14
Mar 14 04:05:01 neutrino crond[2418]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 14 04:05:01 neutrino crond[2419]: (dchooz) CMD
($HOME/test_build/new_test_bu
ild.bash >| $HOME/test_build/test_build.html 2>&1)
Mar 14 04:10:01 neutrino crond[2523]: (root) CMD (/usr/lib/sa/sa1 1 1)
Mar 14 04:10:01 neutrino crond[2524]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 14 04:14:01 neutrino crond[2528]: (KamLAND) CMD
($HOME/test_build/new_test_b
uild.bash >| $HOME/test_build/test_build.html 2>&1)
Mar 14 04:15:01 neutrino crond[2569]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Mar 14 04:20:01 neutrino crond[2577]: (root) CMD (/usr/lib/sa/sa1 1 1)
Mar 14 04:20:01 neutrino crond[2580]: (root) CMD (/usr/bin/mrtg
/etc/mrtg/mrtg.c
fg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
[root@neutrino ~]# diff -r -q /usr/bin/ /backup2/backup/usr/bin/ | wc
3097 15052 204950
[root@neutrino ~]# time nice clamscan -l scan_usrbin_2.txt /usr/bin
... lots of output ...
----------- SCAN SUMMARY -----------
Known viruses: 99277
Engine version: 0.90.1
Scanned directories: 1
Scanned files: 3102
Infected files: 0
Data scanned: 283.21 MB
Time: 50.992 sec (0 m 50 s)
|
