Subject: | |
From: | |
Reply To: | |
Date: | Sun, 18 May 2014 15:12:35 -0400 |
Content-Type: | text/plain |
Parts/Attachments: |
|
|
One of our servers had an unusual illness.
Server loaded our internal network and our Internet connection down to a stop.
When it rebooted it took a few minutes before it started again.
Symptoms:
1. /sbin/ifconfig - eth0 had a large number of Tx & Rd in counters.
2. DVD could not be used to write a DVD.
When you rebooted the system and tried to reload system or run a diagnostic,
you set the boot device as DVD and it immediately ignored the DVD and booted
the system.
3. On reboot it complained that there was an error on line 2 of iptables.
when you looked at /etc/rc.d/init.d there were two small files named Iptables
with a capital I and ipmievd also with a capital I.
4. To stop what ever was happening we sniffed the network and found an address
it was trying to access. 212.7.208.65, added it to hosts.deny and eth0 shut
down. On reboot eth0 did start.
5. Manually restarted /etc/network.
Other oddities:
messages, messages.1, messages.2 were empty.
messages.3 quit shortly after 12 AM EST on 23 April. Same time as Ethernet
went down.
When you did echo $PATH there were to added directories added before normal
path.
"vi" quit working on user accounts and if you tried to "vi filename" it said
that "vim" could not be found. However if you became root "vi" worked
normally.
Looked at /boot/grub/grub.conf, /sbin/init/ files and all looked normal.
6. Tried www.centros.org - for more information it was immediately flagged by
brouser as "malware" in bold red letters. Closed down browser.
7. Removed the two files with capital letters or 30 bytes each.
Rebooted system and "iptables" error went away.
8. Used Rescue from install disk and it ran for about 5 minutes and said it
is safe to reboot. I gave me a list of files that were different. Did not
get to capture the list.
9. There is no trace of anything other than some file dates were 23 April.
Since internal Internet was down there was no way to do a "diff" on
directories of a working system.
Withe a VOIP phone system was killed by problem as it brought the Internet
down. When you pinged Google name server 8.8.8.8 over a 1/2 hour period
you had anywhere from 30% to 98% packet loss.
The import part is that we do not know were it came from as we are behind a
router / firewall in modem.
Thanks in advance for any ideas on how to prevent recurrence.
Hate to work on nice sunny Sunday afternoons.
Larry Linder
|
|
|