On Sunday 18 May 2014 15:12:35 Larry Linder wrote:
> One of our servers had an unusual illness.

Creepy. To quickly get a grip on what's moving where try enabling SELinux and 
tail /var/log/audit/audit.log . The one time I had to check behind a malicious 
administrator this was the quickest way to figure out what was going on. A 
sophisticated attack would disable SELinux, install its own policies, and set 
its own booleans to mask detection. Checking what policies are installed, 
refreshing them and checking what booleans are marked True is not a bad idea 
if your audit logs don't flag anything. After that locating binaries that 
didn't match the versions that were supposed to be installed showed me what 
had been replaced (/bin/bash, top, ip and a slew of other utilities).

Ultimately the system will always be at least somewhat suspect after an 
episode like this. If you have things running that you felt required you to 
disable security policies in the first place, consider configuring a fresh 
server the way you need it, then toying with the "audit to allow" tools to 
build yourself a custom policy that permits your custom services/config to run 
without opening the rest of the system up.

Obviously a security review is in order if you think you've been attacked; I'm 
just highlighting the utility of the SELinux auditing tools in situations like 
this.

Also... don't trust firewalling. Its never really a one-way gate and we've been 
deluded into mistaking NAT for a security tool (the same mistake plagues the 
use of chroot, containers and virtualization). Software inside your firewall 
lets you read this email, which means you should be wary of my bits whether 
they are being parsed by a web browser, email client or flajavanimation script 
inside a browser inside chroot inside a vm inside your firewall.

Here's hoping nobody's repo keys have been compromised!