On Wed, 14 Mar 2007, Glenn Horton-Smith wrote:

> I have a bizarre occurrence to report and ask about.  Last night at 4:15 AM, 
> the "mrtg" cron job started producing an error when it tries to run every 5 
> minutes from it's cron.d script on neutrino:
>
> syntax error at /usr/lib/perl5/5.8.5/IO/Socket/INET.pm line 114, near ")
>       ) "
> Compilation failed in require at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/IO/Socket.pm line 21.
>
>
> I looked in /usr/lib/perl5/5.8.5/IO/Socket/INET.pm, and sure enough, there's 
> a syntax error in INET.pm:
>
>   ($laddr,$lport,$proto) = _sock_info($arg->{LocalAddr},
>                                       $arg->{LocalPort},
>                                       $arg->{Proto})
>              ) or return _error($sock, $!, $@);
>
> Note the extra close parenthesis.  I have a backup image made at 4:00 AM, and 
> can confirm that this file was not like this at 4:00 AM:
>
> [root@neutrino ~]# diff -r /usr/lib/perl5/5.8.5/IO/Socket/ 
> /backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/
> diff -r /usr/lib/perl5/5.8.5/IO/Socket/INET.pm 
> /backup2/backup/usr/lib/perl5/5.8.5/IO/Socket/INET.pm
> 114c114
> <               )or return _error($sock, $!, $@);
> ---
>>                        or return _error($sock, $!, $@);

I note that ')' and TAB differ by a single bit (0x29 versus 0x09).  Does 
this machine have ECC memory?

If the memory-copy of the block has an error in it then it will stay bad 
until it next gets loaded from disk.  If there is a regular cron-job (etc) 
accessing then it may stay in memory for a *very* long time.

When you did something unrelated which used lots of memory/io that might 
have cause it to get flushed and so appear to 'fix' things.

Can you run memtest/memtest+ on the machine?  Of course even good memory 
may occasionally show up single-bit errors, though the makers like to 
claim it is pretty rare.

> More disturbingly, THOUSANDS of binaries in /usr/bin have changed:
>
> [root@neutrino ~]# diff -r -q /usr/bin/ /backup2/backup/usr/bin/ | wc
>   3097   15052  204950
>
> Stranger still, the file contents were changed, but the file lengths and time 
> stamps stayed exactly the same: e.g.,

As many others have suggested this is prelink at work.

I'm still not sure about prelink.  If the performance benefits are big 
then after the prelink one needs to restart anything using the 
apps/libraries to:

   get the benefit of the new layout

   remove all the old shared libs from memory etc, or they will cause
   memory to be wasted holding two copies

Since core libraries like libc may be affected this essentially means a 
reboot is needed after certain updates or the system will run with poorer 
performance until the next restart happens.

I'd have expected that *most* of the performance could be gained by doing 
the calculations once at the time that the packages/distribution is built.

Some claim that prelink has security benefits, though PIE executables seem 
better.  PIE executables never use the prelink info anyway and most of the 
important executables are built PIE anyway -- just look at the specfiles.

One of the things which used to really annoy me about Irix was the time 
spent optimizing all the ELF libraries after any update/patch...

-- 
Jon Peatfield,  Computer Officer,  DAMTP,  University of Cambridge
Mail:  [log in to unmask]     Web:  http://www.damtp.cam.ac.uk/