On Thu, 7 Aug 2008, Panagiotis Kritikakos wrote:

> Hi,
>
> I think I have a bug on the Scientific Linux desktop relating to the
> OpenMPI installation. It appears there is a condition which
> prevents any MPI application from completing. Bellow you can see perhaps the 
> simplest MPI that can be written and still link to the MPI library:
>
> x.F90
>
> program main
>
>  use mpi
>
>  implicit none
>
>  integer : :  ierror
>
>   call mpi_init(ierror)
>   call mpi_finalize(ierror)
>
> end program main

On my sl51 (32-bit) boxes mpif90 objects the the 'integer : :  ierror' 
line.  Maybe my compiler is feeling odd...

Replacing it with 'integer ierror' lets it compile for me.

> I've compiled it with mpif90 -o x x.F90. Compilation goes fine, but when 
> trying to execute the resulting x file, I received the following error:
>
> libibverbs: Fatal: couldn't read uverbs ABI version.
> --------------------------------------------------------------------------
> [0,0,0]: OpenIB on host localhost was unable to find any HCAs.
> Another transport will be used instead, although this may result in
> lower performance.
> --------------------------------------------------------------------------
> librdmacm:  couldn't read ABI version.
> librdmacm:  assuming: 4
> libibverbs: Fatal: couldn't read uverbs ABI version.
> CMA: unable to open /dev/infiniband/rdma_cm

If I run the result I get no hang, but then i get not much useful output 
either :-)

$ mpirun ./x
libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[0,1,0]: OpenIB on host unfair.damtp.cam.ac.uk was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,0]: uDAPL on host unfair.damtp.cam.ac.uk was unable to find any NICs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------

In case it matters that box has:

$ rpm -q sl-release gcc-gfortran openmpi openmpi-devel
sl-release-5.1-2.i386
gcc-gfortran-4.1.2-14.el5.i386
openmpi-1.2.3-4.el5.i386
openmpi-devel-1.2.3-4.el5.i386

> The warnings should not be a problem as it just compalinging about not 
> finding a high performance interconnect network options that were compiled in 
> and the system will fall back to lower performance defaults.
>
> strace gives a lot of output but shows the program is stalling with:
>
> futex(0x26049c, FUTEX_WAIT, 2, NULL
>
> It looks like some kind of deadlock is occurring probably relating to
> the threads involved or shared memory.

Debugging mpi problems is always a bit of a nightmare.  BTW how many 
processors were you running it on?

I'm still worrying about updating our systems to the (newer) openmpi from 
sl52 as the package maintainers have switched from alternatives (which I 
sort of understand) to mpi-selector (which I don't)...

> I have compiled up a personal version of OpenMPI removing the OpenIB support:
>
> ./configure –prefix=/opt/local/openmpi --without-openib
>
> This works (once I worked out that it was linking against the old
> libraries at runtime) and the correct output (nothing) is produced.

  -- Jon