Segmentation fault on Ubuntu 25.04

[nils3er]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.7

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

package from ubuntu (debian)

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

N/A

Please describe the system on which you are running

• Operating system/version: Ubuntu 25.04 (Upgraded from 24.04)
• Computer hardware: Lenovo ThinkPad
• Network type: None

---

Details of the problem

After the upgrade to ubuntu 25.04 i cannot run mpmd applications anymore:

$ mpirun -n 1 date : -n 1 date
Segmentation fault (core dumped)

GDB gives me the following backtrace:

#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
#1  0x00007ffff766f6a0 in __printf_buffer (buf=buf@entry=0x7fffffffa890, format=0x7ffff7bccc06 "%s=%s", ap=0x7fffffffa9f0, mode_flags=2) at ./stdio-common/vfprintf-process-arg.c:435
#2  0x00007ffff76961ae in __vasprintf_internal (result=0x7fffffffaae8, format=<optimized out>, args=<optimized out>, mode_flags=<optimized out>) at ./libio/vasprintf.c:102
#3  0x00007ffff7af3fe0 in pmix_vasprintf () from /lib/x86_64-linux-gnu/libpmix.so.2
#4  0x00007ffff7af409a in pmix_asprintf () from /lib/x86_64-linux-gnu/libpmix.so.2
#5  0x00007ffff7b2e952 in PMIx_Setenv () from /lib/x86_64-linux-gnu/libpmix.so.2
#6  0x00007ffff7b9e10f in ?? () from /lib/x86_64-linux-gnu/libpmix.so.2
#7  0x00007ffff7b9af5b in pmix_pmdl_base_setup_fork () from /lib/x86_64-linux-gnu/libpmix.so.2
#8  0x00007ffff7a90346 in PMIx_server_setup_fork () from /lib/x86_64-linux-gnu/libpmix.so.2
#9  0x00007ffff7ed4a72 in prte_odls_base_spawn_proc () from /lib/x86_64-linux-gnu/libprrte.so.3
#10 0x00007ffff7e282a0 in ?? () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#11 0x00007ffff7e29faf in event_base_loop () from /lib/x86_64-linux-gnu/libevent_core-2.1.so.7
#12 0x000055555555b4d7 in ?? ()
#13 0x00007ffff762a338 in __libc_start_call_main (main=main@entry=0x555555558ee0, argc=argc@entry=8, argv=argv@entry=0x7fffffffd638) at ../sysdeps/nptl/libc_start_call_main.h:58
#14 0x00007ffff762a3fb in __libc_start_main_impl (main=0x555555558ee0, argc=8, argv=0x7fffffffd638, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd628) at ../csu/libc-start.c:360
#15 0x000055555555c3c5 in ?? ()

ompi_info:

                 Package: Debian OpenMPI
                Open MPI: 5.0.7
  Open MPI repo revision: v5.0.7
   Open MPI release date: Feb 14, 2025
                 MPI API: 3.1.0
            Ident string: 5.0.7
                  Prefix: /usr
 Configured architecture: x86_64-pc-linux-gnu
           Configured by: username
           Configured on: Mon Feb 17 13:09:46 UTC 2025
          Configure host: hostname
  Configure command line: '--build=x86_64-linux-gnu' '--prefix=/usr'
                          '--includedir=${prefix}/include'
                          '--mandir=${prefix}/share/man'
                          '--infodir=${prefix}/share/info'
                          '--sysconfdir=/etc' '--localstatedir=/var'
                          '--disable-option-checking'
                          '--disable-silent-rules'
                          '--libdir=${prefix}/lib/x86_64-linux-gnu'
                          '--runstatedir=/run' '--disable-maintainer-mode'
                          '--disable-dependency-tracking'
                          '--disable-silent-rules'
                          '--disable-wrapper-runpath'
                          '--with-package-string=Debian OpenMPI'
                          '--with-verbs' '--with-libfabric' '--with-psm'
                          '--with-psm2' '--with-ucx' '--with-pmix=external'
                          '--with-jdk-dir=/usr/lib/jvm/default-java'
                          '--enable-mpi-java'
                          '--enable-opal-btl-usnic-unit-tests'
                          '--with-libevent=external' '--with-hwloc=external'
                          '--disable-silent-rules' '--enable-ipv6'
                          '--with-devel-headers' '--with-slurm' '--with-sge'
                          '--without-tm' '--sysconfdir=/etc/openmpi'
                          '--libdir=${prefix}/lib/x86_64-linux-gnu/openmpi/lib'
                          '--includedir=${prefix}/lib/x86_64-linux-gnu/openmpi/include'
                Built by: username
                Built on: Mon Feb 17 13:09:46 UTC 2025
              Built host: hostname
              C bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the gfortran compiler and/or Open
                          MPI, does not support the following: array
                          subsections, direct passthru (where possible) to
                          underlying Open MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: yes
  Wrapper compiler rpath: rpath
              C compiler: gcc
     C compiler absolute: /bin/gcc
  C compiler family name: GNU
      C compiler version: 14.2.0
            C++ compiler: g++
   C++ compiler absolute: /bin/g++
           Fort compiler: gfortran
       Fort compiler abs: /bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, Event lib: yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: yes
          MPI extensions: affinity, cuda, ftmpi, rocm, shortfloat
 Fault Tolerance support: yes
          FT MPI support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
         MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.7)
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.7)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                 MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.7)
                 MCA btl: ofi (MCA v2.1.0, API v3.3.0, Component v5.0.7)
                 MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
                 MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.7)
                 MCA btl: uct (MCA v2.1.0, API v3.3.0, Component v5.0.7)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.7)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.7)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.7)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.7)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v5.0.7)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.7)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.7)
           MCA reachable: netlink (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.7)
             MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.7)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                 MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
                MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
                          v5.0.7)
                MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.7)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
                          v5.0.7)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                  MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                 MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                 MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.7)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v5.0.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.7)
                 MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.7)
                 MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
                          v5.0.7)
                 MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
                 MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.7)
                 MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.7)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.7)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.7)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v5.0.7)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v5.0.7)

Any help is greatly appreciated!

[rhc54]

Afraid it's impossible to get any meaningful debug info out of that - can you build a --enable-debug version and provide a trace from it?

[nils3er]

When i build it on my machine the configure picks up the system PMIx. The SegFault still appears but the trace does not provide more information.

I build it in docker to prevent it from finding the system PMIx, then the SegFault does not appear anymore. So I'm not able to reproduce it with a self built version.

The segfault can also be observed in Debian trixie and sid. (tested in docker)

I also filed a bug report to debian : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1107687

[ggouaillardet]

FWIW, you can configure --with-pmix=internal ... to avoid using the distro provided PMIx.

[ggouaillardet]

@amckinstry I guess this is for you.

FWIW, I rebuilt Open MPI 5.0.7 on ubuntu with the distro provided PMIx and I can reproduce the crash. No crash if I use the Open MPI provided PMIx (e.g. configure --with-pmix=internal). More interestingly, no crash if I build PMIx 5.0.6 from the source tarball by myself, and build Open MPI with that (external) PMIx. That suggests something debian specific.

[rhc54]

FWIW: our CI tests on Ubuntu latest release and we've never seen a problem - but we build from scratch there. So I suspect this if something relating to the distro package.