On Sun, May 18, 2014 at 3:12 PM, Larry Linder
<[log in to unmask]> wrote:
> One of our servers had an unusual illness.

[ Adventures snipped. ]

Take it offline. *KEEP* it offline. Make a disk image to load inside a
sealed away VM without any network connections, if you have to, but
keep this thing *away* from your network. Notify your administration
and your security staff that any security data or passwords on that
host, or in any accessible part of the local network, is at risk. In
particular, NFS shares with passphrase free keys or plaintext
passwords should be considered vulnerable, and any passwords used or
stored on that system while it was even potentially at risk of this
mess should be considered stolen and changed ASAP.

> Server loaded our internal network and our Internet connection down to a stop.
> When it rebooted it took a few minutes before it started again.
> Symptoms:
> 1.  /sbin/ifconfig - eth0 had a large number of Tx & Rd in counters.
> 2. DVD could not be used to write a DVD.
> When you rebooted the system and tried to reload system or run a diagnostic,
> you set the boot device as DVD and it immediately ignored the DVD and booted
> the system.
> 3. On reboot it complained that there was an error on line 2 of iptables.
> when you looked at /etc/rc.d/init.d there were two small files named Iptables
> with a capital I and ipmievd also with a capital I.

You've been rootkitted, one way or another. Time to start the annual
rebuild of your *entire* network.

> 4. To stop what ever was happening we sniffed the network and found an address
> it was trying to access.   212.7.208.65, added it to hosts.deny and eth0 shut
> down.  On reboot eth0 did start.
> 5. Manually restarted /etc/network.
>
> Other oddities:
> messages, messages.1, messages.2 were empty.
> messages.3 quit shortly after 12 AM EST on 23 April.  Same time as Ethernet
> went down.
> When you did echo $PATH there were to added directories added before normal
> path.
> "vi" quit working on user accounts and if you tried to "vi filename" it said
> that "vim" could not be found.  However if you became root "vi" worked
> normally.
> Looked at /boot/grub/grub.conf, /sbin/init/ files and all looked normal.
>
> 6.  Tried www.centros.org - for more information it was immediately flagged by
> brouser as "malware" in bold red letters.   Closed down browser.
>
> 7.  Removed the two files with capital letters or 30 bytes each.
> Rebooted system and "iptables" error went away.
>
> 8.  Used Rescue from install disk and it ran for about 5 minutes and said it
> is safe to reboot.  I gave me a list of files that were different.  Did not
> get to capture the list.
>
> 9.  There is no trace of anything other than some file dates were 23 April.
> Since internal Internet was down there was no way to do a "diff" on
> directories of a working system.

Make a disk image. Mount it inside an isolated virtual machine to
compare content.

> Withe a VOIP phone system was killed by problem as it brought the Internet
> down.    When you pinged Google name server 8.8.8.8 over a 1/2 hour period
> you had anywhere from 30% to 98% packet loss.
>
> The import part is that we do not know were it came from as we are behind a
> router / firewall in modem.

I'm sorry to say, but so what? It's like saying "We don't know how
those kids got into our pool, we had a sign on the gate!" Firewalls
are a first line of defense, hardly a complete one. And all it takes
is one download of inappropriate software from someone using a browser
on that router, one corrupted internal laptop owned by someone with
root access keys or passwords. You've already shown that your browser
environment on it was corrupted, and you certainly can't consider your
internal network secure right now with an actively rootkitted host
that was on your internal network, *that is apparently still live and
has not been flushed to bare metal*.

> Thanks in advance for any ideas on how to prevent recurrence.
> Hate to work on nice sunny Sunday afternoons.
>
> Larry Linder

There's not enough information about your internal security practices
to offer good hints. We can't tell if you've got internally built
root-managed software of unknown provenance, who's got root privileges
on the box, what you've got set up for updates, or what your working
requirements are.