*note*

you need to clean system.

'man yum', run yum to verify all programs on system.

run several rootkit check programs to see what is now on your system.

you have had a break in and best thing to do is remove *all* 'admin' users.
require all users to set new passwords.

once you have system cleaned up, then you can reinstate admin users.

this must be done. anyone that is allowed to help with cleaning system
*must* be some that is trustworthy beyond any doubt.

be aware, if you are main system administrator, unless you have a legal
binding contract to contrary, you are liable for what happens with system
and can be held accountable. you need to notify who ever you are employed
by and notify them, by motorized written document, that you have reason
to believe that system has been compromised.

CYOA. cover your own ass.

----

Larry Linder wrote:
<snip>
>> 1) record date and time of all passwd and group files. backup these files
>>  and all log files off of system.

> The orig. file dates were 15 Jan 10 and new dates were 5 May 10.

check logins for *time* of new dates.

> I may have clobbered the date while fixing a few errors.  Such as missing 
> :'s

see man's and links at bottom.

> Hate to be paranoid but could these be some one foot print.

possible, because of next.

> There are a number of logins that were added since install. 
> piranha:x:60:60::/etc/sysconfig/ha:/sbin/nologin 
> luci:x:101:104::/var/lib/luci:/sbin/nologin ricci:x:102:105:ricci daemon 
> user:/var/lib/ricci:/sbin/nologin

are 'piranha', 'luci' legitimate users? if not, remove them.

>> 2) if you have slightest thought of a system breech, change root and 
>> *all* admin passwords now and maintain them.
> 
> I noticed that the allowable root user in group contained another name.  So
>  someone else is of group root.

if you are main administrator, did you allow them?

>> 3) check logs to see who was logged in and who was root, su-root, or 
>> admin during time.
> 
> Unfortunately I had just removed old logs during quarterly clean up. I have
>  plenty of disk space but old habits die hard.

here after, store old files off system. they can come in handy when you need
to trace back to see when you were compromised.

>> 4) any updating going on during time?
>> 
>> 5) open and check passwd and shadow, and group and gshadow files and 
>> compare order of listing. they should match in order.
> 
> They didn't and date was 15 Jan 10.

this can/will give you a clue as to what was being attempted.

>> 6) carefully check passwd and group files to be sure that what is listed 
>> is what should be.
> 
> The above passwd and groups contained the above entries see 1.



>> 7) ensure that all names have 'x' in 2nd position and that only root has 
>> '0' in 3rd position of passwd file and that all users are 500 and higher.
>> 
> 
> If you notice that there were a number of pw entries that has different no
>  in 3rd and 4th col.  Only suspect entries have what is a cross linked 
> login.

i do not believe that should not be.

>> 8) ensure that only root and users have an encrypted password shown. rest
>>  should be '*' or '!!'. programs that are 'user' should have '*', system 
>> files should have '!!'.
>> 
>> 9) ensure that in group file, all listed have 'x' in second position.
> 
> All have "x" in second entry.

see links at bottom.

>> 10) ensure that in gshadow, root and common groups should have nothing in
>>  2nd and 3rd positions, 4th position who is allowed.
> 
> In gshadow there are a number of user accounts that have !! in second 
> position. Are these from malformed group.

no. again, see links at bottom.

>> ounce above is done, go back thru group and shadow match any miss 
>> ordering. same for group and gshadow, and correct any incorrect users.
> 
> There were a number of differences when using "diff" that I looked right 
> but were out of order?

forget using 'diff'. use your eyes. they see what 'diff' does not know about.

if they are out of order, then it is most likely that someone was sloppy
in hacking system.


>> something bad wrong caused this problem and it may be a fluke. but it 
>> could be caused by and unhappy user or a system breech.
> 
> There are several other system abnormality:
> 
> 1.   When you are saving a file there is a slight hesitation - like 
> processor is taking a nap.   This happens when using Opera to look at 
> hardware files on internet.

what happens if you use konqueror? hesitations could be that program has
been hacked and some sort of trace recording is being done.

> 2.  System log has a repeating message. "engr01 kernel: spurious APIC 
> interrupt on CPU#0, should never happen.

can not answer. may be a hardware problem. right now, your main concern is
with files on system.

> 3.  This system started having a problem with a print server.  I may send 
> one file to printer on print server and that is it.  Leaves an unterminated
>  file message in CUPS.  All other systems can print to this server with out
>  an issue.

here again, yum check will show files that have been modified. and yes, with
cups is being used for local/remote printing, modifying cups progs can be
used to send copies of what is being printed. wireshark can be used to watch
network to see where things are going.

>> there is a site with this information in more detail, but i can not find
>>  it in my bookmarks.
>> 
>> i will search for it again and post when i find it.

note. links at bottom are not what i was looking for, but they will be of
help for now with you  figuring out what happened to your password and
group files.

> Thanks for the help.

welcome.

> It looks like someone has been into a number of systems. Backup Server
> looks pretty clean.   The affected boxes can be accessed from users home.
> The access is vi "ssh" and "ftp".   Some like to work odd hours due to
> family issues.

because this one system appears to compromised, all systems need to be check
via both yum and rootkit checkers. if they hit one, they probably hit others.

also, from what you are saying, this does appear to be from inside. :(

> Histroy: In 2002 our system got a virus that attached its self to every 
<snip>

this should have been a lesson to tighten your security and install progs
to track what is happening.

if your company is in any type work that is highly competitive or deals with
any government, i would highly suggest that consideration should be give to
having polygraph test run.

i know that this is a strong step to take, but company's future is at risk
and your current problems are strong indications that someone is trying to
gain access.

> If I ever met a person who claimed to have written a virus I would give him
> a frontal lobotomy with my LouisVille Slugger.

my preference is .357 magnum or larger. ;)


+++
man these:

 group
 passwd
 shadow
 usermod

 pwconv
 pwck
 grpck
 groupmod

+++

read these links and follow their internal links:

http://www.linuxsecurity.com/
 http://www.redhat.com/docs/manuals/linux/RHL-8.0-Manual/admin-primer/ch-acctsgrps.html

http://www.redhat.com/docs/manuals/linux/RHL-8.0-Manual/admin-primer/s1-acctsgrps-files.html
+++


https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/en-US/Security_Guide/


-- 

peace out.

tc,hago.

g
.

****
in a free world without fences, who needs gates.
**
help microsoft stamp out piracy - give linux to a friend today.
**
to mess up a linux box, you need to work at it.
to mess up an ms windows box, you just need to *look* at it.
**
learn linux:
'Rute User's Tutorial and Exposition' http://rute.2038bug.com/index.html
'The Linux Documentation Project' http://www.tldp.org/
'LDP HOWTO-index' http://www.tldp.org/HOWTO/HOWTO-INDEX/index.html
'HowtoForge' http://howtoforge.com/
****