On May 19, 2014, at 18:35 , Larry Linder wrote:

> On Sunday 18 May 2014 3:12 pm, Larry Linder wrote:
>> One of our servers had an unusual illness.
>> Server loaded our internal network and our Internet connection down to a
>> stop. When it rebooted it took a few minutes before it started again.
>> Symptoms:
>> 1.  /sbin/ifconfig - eth0 had a large number of Tx & Rd in counters.
>> 2. DVD could not be used to write a DVD.
>> When you rebooted the system and tried to reload system or run a
>> diagnostic, you set the boot device as DVD and it immediately ignored the
>> DVD and booted the system.
>> 3. On reboot it complained that there was an error on line 2 of iptables.
>> when you looked at /etc/rc.d/init.d there were two small files named
>> Iptables with a capital I and ipmievd also with a capital I.
>> 4. To stop what ever was happening we sniffed the network and found an
>> address it was trying to access.   212.7.208.65, added it to hosts.deny and
>> eth0 shut down.  On reboot eth0 did start.
>> 5. Manually restarted /etc/network.
>> 
>> Other oddities:
>> messages, messages.1, messages.2 were empty.
>> messages.3 quit shortly after 12 AM EST on 23 April.  Same time as Ethernet
>> went down.
>> When you did echo $PATH there were to added directories added before normal
>> path.
>> "vi" quit working on user accounts and if you tried to "vi filename" it
>> said that "vim" could not be found.  However if you became root "vi" worked
>> normally.
>> Looked at /boot/grub/grub.conf, /sbin/init/ files and all looked normal.
>> 
>> 6.  Tried www.centros.org - for more information it was immediately flagged
>> by brouser as "malware" in bold red letters.   Closed down browser.
>> 
>> 7.  Removed the two files with capital letters or 30 bytes each.
>> Rebooted system and "iptables" error went away.
>> 
>> 8.  Used Rescue from install disk and it ran for about 5 minutes and said
>> it is safe to reboot.  I gave me a list of files that were different.  Did
>> not get to capture the list.
>> 
>> 9.  There is no trace of anything other than some file dates were 23 April.
>> Since internal Internet was down there was no way to do a "diff" on
>> directories of a working system.
>> 
>> Withe a VOIP phone system was killed by problem as it brought the Internet
>> down.    When you pinged Google name server 8.8.8.8 over a 1/2 hour period
>> you had anywhere from 30% to 98% packet loss.
>> 
>> The import part is that we do not know were it came from as we are behind a
>> router / firewall in modem.
>> 
>> Thanks in advance for any ideas on how to prevent recurrence.
>> Hate to work on nice sunny Sunday afternoons.
>> 
>> Larry Linder
> 
> Internet works now.
> Things that do not work.
> To get "cups" to run we had to use yum and remove it and reload it.  Yum even 
> save a copy of our "configs" so rebuilding was simply copy saved configs to 
> configs.   With a gob of printers and plotters it would be long time to get 
> it back running.
> /var/log/cron, /var/lot/messages, are updated but still empty.  Copies were 
> made of cron and messages with new extension.
> KMail does not shut down properly, when you reboot, it thinks that kmail is 
> still running.
> VMware does not now automatically restart on a reboot.
> Its about time to low level format disk OS is on and reload.

I'd keep an image of it, created on a system guaranteed to be clean like a live cd, booted on different hardware known to be clean.

Your claim that you can't boot from a DVD is most worrying - your machine could be compromised on the firmware level.

> Our scheme of keeping OS one one disk and all other directories on another 
> large disk save us again.

Only if there's no way anything on the second disk could ever be used to compromise your system again. Like any kind of executables, including scripts and libraries, or malicious files exploiting bugs or features in application software. Mount it -onosuid,nodev,noexec - after having installed and hardened the new OS. Also make sure that root won't ever access any file on it, and that any user accessing any content there has no additional privileges - and that no local root exploits are present. Pretty hard...

> "cron" and "mesages" update but are empty indicates that we did not get all.

Right.

> I would like to do a clean install of 5.10 another box and do a "diff" to see 
> what was changed.

Tedious, but a good idea. The rkhunter idea is good too and may be more feasible.

> One thing I am going to do is once a system is up and running make a script to 
> save all config files and put it away on a dvd for that OS version.

Better yet, have such a repository with config changes and create all your systems using that in the first place.

> Don't think this system should ever go back into production until we know what 
> happened and then reload the OS.

Right. And again, if the firmware ignores your choice of boot media, reloading the OS may not be sufficient.

>   We are still wondering how this system is 
> the only one that got clobbered.

The root user running firefox, Adobe Reader, a mail client, an editor etc.? An ordinary user doing the same with an unfixed local root exploit present?

Worst case, it's not the only system that got clobbered but the only one where the compromise made the system malfunction....

-- Stephan