On Thu, Oct 2, 2014 at 4:02 PM, Larry Linder
<[log in to unmask]> wrote:
> on May 22 Our server was broken into by some one in China.   How it happened
> is that we had had a hole in our firewall so employees could access out
> server from the field.   This had worked pretty well - until the A&T Motorola
> modem died and they install two new ones and left the port to the ssh open.

*Ouch*. Dude, you've my sympathies. This sort of thing is precisely
why I argue with people about the concept of "we have a firewall, so
we don't need to be so rigorous about our internal network security".
And oh, yes, the old standby "who would want to hack us?"

> The people who did this job had more than a working knowledge of networks,
> Linux and files systems.   We were wondering how they could create a
> directory at end of file system was a puzzle.   They had root privilege, ssh,
> and with access to bash they were in.

And the kernel. Don't forget that with that level of access, they can
manipulate the modules in your kernel.

> How did they covered their tracks so well?  "messages" was there but filled
> with nonsense and file in /var/log that tells you who and what was sent was
> touched was now missing.   "security" was there and you could see the

And since they owned root, they could replace core system libraries,
even corrupting compilers. *nothing* rebuilt on that host can be
trusted.

> repeated access attempts to break in again.  "cron" was changed so daily
> backups were done after they down loaded all new files.   "crontab -e" no
> longer worked.
> We made a copy of the OS onto old disk and removed disk from the system.
> There were so many charges to the OS and files in /etc that we did not even
> try to repair it.   There were 1000's of differences between new install and
> copy of old system.
>
> I personally think the bash problem is over blown because they have to get
> threw modem, firewall, ssh before they can use "bash".

That is *one* instance, and not really relevant to the circumstances
you described. In fact, many systems expose SSH to the Internet at
large for "git" repository access, and for telecommuting access to
firewalls and routers. The big problem with "shellshock" was that
attempts to restrict the available commands for such access, for
example inside "ForceCommands" controlled SSH "authrozed_keys" files,
could now broken out of and allow full local shell access. Once you
have *that* on a critical server, your hard crunch outershell is
cracked open and your soft chewy underbelly exposed.

> One question remains and that is what code and script did they use to run the
> system??

Gods only know. there are so *many* rootkits in the wild, and so much
theft of private SSH keys and brute force attacks or theft of
passwords, it's hard to know how they got in.

> If anyone wants details and IP's I will send it to them on an individual
> basis.
>
> We contacted the FBI and after a telephone interview,  they were sort of
> interested but I think the problem is so big they don't have time to work
> little stuff.

My personal experience with the FBI and computer crime is that they
are simply not competent. They accept information eagerly and do
nothing at all helpful with it. They have a very poor track record of
getting crackers to turn each other in and abusing the resulting
immunity from prosecution, and not actually investigating or
prosecuting more than the tiniest fraction of crimes reported.

> This is a little disjointed because it happened over a long time.
>
> Larry Linder

As I mention, you have my sympathies. It'a a good reminder to keep
your internal systems updated from known attack vectors.