[this summary is meant to address/correct public perception of the "SLC3
is incompatible to RHE3" meme. If you come across such things in the
future, please forward as early as possible to [log in to unmask]
to give us a chance to address such issues before they spread too far.
Aplogies for any duplicates you may receive.] 


Summary:
SLC3 ships with an older version of libstdc++ than RHE3/SL. This allows
some POOL versions to run that would otherwise crash. This is a
temporary workaround, and is being addressed from several angles.


Full story:

During the certification of SLC3 at CERN and after applying a "quarterly
update" from Red Hat into the unfinished SLC3 release, it was discovered
on Sep 8th that POOL-1.7b (or rather applications using it) would crash.

Markus Frank (POOL developer) provided a test case that pinpointed the
problem, which turned out to be a change in the [CXXABI] implementation,
shipped in the libstdc++ RPM (which itself is part of the gcc RPM set).
[CXXABI] is work-in-progress (not an official standard yet), and the
change was due to a different reading of the documentation and a
backported patch from gcc-3.4. The error only occurred in Red Hat's
patched version of gcc-3.2.3, and only from a specific version onward
(-36).

Given that notably CMS and LHCb could not go ahead with certification (a
"blocking" issue), we packaged and deployed the previous working version
(-34) in a form that allowed it to be "updated" on already-installed
systems. This was a "hack" and meant to be a temporary solution only,
until the bug  (since it was seen as that at the time) could be fixed
upstream. This "workaround" release RPM is labeled as
libstdc++-3.2.3-42.34bin.cern, and was announced e.g. to
[linux-certification]. 


This package is neither part of the "official" ScientificLinux nor Red
Hat Enterprise3, hence SLC3 had become binary incompatible with both, at
least for POOL-1.7 applications. This (to our knowledge) is the only
incompatibility so far.


The following happened in parallel:

      *  Red Hat was informed (Issue#49543, September 16th, and
        Bugzilla#133406) and a rather longish discussion developed (on
        "correct" reading of CXXABI, breaking deployed software in a
        "stable" system etc).
        
        The current status is that a new version of gcc/libstdc++
        (gcc-3.2.3-52 and above) will go back to the old behaviour, and
        that the CXXABI documentation will be clarified. The issue is
        still open since the new gcc version isn't available yet. Once
        it is, presumably POOL-1.7 could be recompiled with it.
        
      *  POOL developers implemented a workaround in newer
        versions>=1.8.0 (released Nov 17th), and the [LCG architects
        board] decided on Dec 15th that SLC3 would only be supported
        from that version onward, which nearly led to the workaround
        being dropped at CERN...
        (Un?)fortunately, as this turned later out to conflict with the
        SLC3 migration plans for LHCb, this decision was modified on Jan
        25 to request the workaround to stay until April 2005, at least
        on centrally managed CERN machines.
        
Given that the binary compatibility between RHE3, SL and SLC3 is/was a
strong selling point (since it meant less certifications for the HEP
community), it is in CERN's interest to become compatible as soon as
possible again. However, in this specific case the potential delay to
SLC3 roll-out was felt to be more important, more so since we saw the
issue initially as a short-lived workaround. In the mean time, the
"workaround" RPMs can be found on
http://linuxsoft.cern.ch/cern/slc303/i386/SL/RPMS/



Please feel free to spread this information to your respective
communities, and to contact me (for details) or
[log in to unmask] (for discussion, may need to subscribe).

Thanks
Jan Iven
IT-ADC-LE / CERN Linux Support


References:
[CXXABI] http://www.codesourcery.com/cxx-abi/

[linux-certification] announcement/summary 
https://wwwlistbox.cern.ch/earchive/project-lcg-peb-persistency-developers/msg01833.html
(some trouble with archives, therefore copy on a different mailing list
is given)

[POOL-1.8] release
https://wwwlistbox.cern.ch/earchive/project-lcg-peb-apps/msg00809.html

[LCG architects board] minutes
http://lcgapp.cern.ch/project/mgmt/AF20041215.txt
http://lcgapp.cern.ch/project/mgmt/AF20050127.txt

[incompatibility problem] reported e.g. at GRIDPP/Tier1:
http://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=9498
(unreadable without a password, but I have seen a mail quoting it..)