on May 22 Our server was broken into by some one in China.   How it happened 
is that we had had a hole in our firewall so employees could access out 
server from the field.   This had worked pretty well - until the A&T Motorola 
modem died and they install two new ones and left the port to the ssh open.
Every day at noon we saw a high usage of our Internet for an hour or so.  
Depending on what they were downloading.   We found the bogus IPTABLES in / 
and /boot that were linked to /etc/?
We rebuilt the system from scratch several times and within hours it was back.  
The last time we were careful to examine the partitions in the drive that has 
the OS on it.   The sizes did not add up and we could not see the hidden 
partition on the OS drive at the end of the normal partitions.
We found the data file in /tmp/.tmp and while looking at its contents we 
recognized a number of files.  The server was unplugged from the Internet and 
we changed the "root" password and then examined the /tmp/.tmp file and the 
beginning line was
PASSWD: "new encrypted passwd" ready to sent..
From or inability to find the where the SW was lurking we did a low level 
format of the OS disk, it took a long time to to format a 1/2 T drive.
In the mean time we closed all firewall ports in the Modem.
Installed the OS again and everything has been working well.

We used a trace route program and in 4 hops we were some where west of 
Singapore.
The really bad problem is that they had downloaded all of our banking 
information and had attempted to send money from Company savings account to 
somewhere.   Credit Union could not or would not say as to where the money 
was to be sent.   Credit Union has one neat restriction is that you cannot 
send money from a saving account but must move into you checking.
We had to change all account numbers, pass words, id checks etc.   We had to 
buy new checks, stamps for 5 accounts.  Several people had their personal 
banking data on server too.  A big pain in the posterior.

We are still working on how give guys access to some accounts.

The people who did this job had more than a working knowledge of networks, 
Linux and files systems.   We were wondering how they could create a 
directory at end of file system was a puzzle.   They had root privilege, ssh, 
and with access to bash they were in.

How did they covered their tracks so well?  "messages" was there but filled 
with nonsense and file in /var/log that tells you who and what was sent was 
touched was now missing.   "security" was there and you could see the 
repeated access attempts to break in again.  "cron" was changed so daily 
backups were done after they down loaded all new files.   "crontab -e" no 
longer worked.
We made a copy of the OS onto old disk and removed disk from the system.  
There were so many charges to the OS and files in /etc that we did not even 
try to repair it.   There were 1000's of differences between new install and 
copy of old system.

I personally think the bash problem is over blown because they have to get 
threw modem, firewall, ssh before they can use "bash".

One question remains and that is what code and script did they use to run the 
system??

If anyone wants details and IP's I will send it to them on an individual 
basis.   

We contacted the FBI and after a telephone interview,  they were sort of 
interested but I think the problem is so big they don't have time to work 
little stuff.

This is a little disjointed because it happened over a long time.

Larry Linder