[this summary is meant to address/correct public perception of the "SLC3 is incompatible to RHE3" meme. If you come across such things in the future, please forward as early as possible to [log in to unmask] to give us a chance to address such issues before they spread too far. Aplogies for any duplicates you may receive.] Summary: SLC3 ships with an older version of libstdc++ than RHE3/SL. This allows some POOL versions to run that would otherwise crash. This is a temporary workaround, and is being addressed from several angles. Full story: During the certification of SLC3 at CERN and after applying a "quarterly update" from Red Hat into the unfinished SLC3 release, it was discovered on Sep 8th that POOL-1.7b (or rather applications using it) would crash. Markus Frank (POOL developer) provided a test case that pinpointed the problem, which turned out to be a change in the [CXXABI] implementation, shipped in the libstdc++ RPM (which itself is part of the gcc RPM set). [CXXABI] is work-in-progress (not an official standard yet), and the change was due to a different reading of the documentation and a backported patch from gcc-3.4. The error only occurred in Red Hat's patched version of gcc-3.2.3, and only from a specific version onward (-36). Given that notably CMS and LHCb could not go ahead with certification (a "blocking" issue), we packaged and deployed the previous working version (-34) in a form that allowed it to be "updated" on already-installed systems. This was a "hack" and meant to be a temporary solution only, until the bug (since it was seen as that at the time) could be fixed upstream. This "workaround" release RPM is labeled as libstdc++-3.2.3-42.34bin.cern, and was announced e.g. to [linux-certification]. This package is neither part of the "official" ScientificLinux nor Red Hat Enterprise3, hence SLC3 had become binary incompatible with both, at least for POOL-1.7 applications. This (to our knowledge) is the only incompatibility so far. The following happened in parallel: * Red Hat was informed (Issue#49543, September 16th, and Bugzilla#133406) and a rather longish discussion developed (on "correct" reading of CXXABI, breaking deployed software in a "stable" system etc). The current status is that a new version of gcc/libstdc++ (gcc-3.2.3-52 and above) will go back to the old behaviour, and that the CXXABI documentation will be clarified. The issue is still open since the new gcc version isn't available yet. Once it is, presumably POOL-1.7 could be recompiled with it. * POOL developers implemented a workaround in newer versions>=1.8.0 (released Nov 17th), and the [LCG architects board] decided on Dec 15th that SLC3 would only be supported from that version onward, which nearly led to the workaround being dropped at CERN... (Un?)fortunately, as this turned later out to conflict with the SLC3 migration plans for LHCb, this decision was modified on Jan 25 to request the workaround to stay until April 2005, at least on centrally managed CERN machines. Given that the binary compatibility between RHE3, SL and SLC3 is/was a strong selling point (since it meant less certifications for the HEP community), it is in CERN's interest to become compatible as soon as possible again. However, in this specific case the potential delay to SLC3 roll-out was felt to be more important, more so since we saw the issue initially as a short-lived workaround. In the mean time, the "workaround" RPMs can be found on http://linuxsoft.cern.ch/cern/slc303/i386/SL/RPMS/ Please feel free to spread this information to your respective communities, and to contact me (for details) or [log in to unmask] (for discussion, may need to subscribe). Thanks Jan Iven IT-ADC-LE / CERN Linux Support References: [CXXABI] http://www.codesourcery.com/cxx-abi/ [linux-certification] announcement/summary https://wwwlistbox.cern.ch/earchive/project-lcg-peb-persistency-developers/msg01833.html (some trouble with archives, therefore copy on a different mailing list is given) [POOL-1.8] release https://wwwlistbox.cern.ch/earchive/project-lcg-peb-apps/msg00809.html [LCG architects board] minutes http://lcgapp.cern.ch/project/mgmt/AF20041215.txt http://lcgapp.cern.ch/project/mgmt/AF20050127.txt [incompatibility problem] reported e.g. at GRIDPP/Tier1: http://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=9498 (unreadable without a password, but I have seen a mail quoting it..)