opal_lifo test fails on s390x

[opoplawski]

Looking at updating the Fedora openmpi package to 5.0.0rc9. I'm getting the following test failure on s390x:

FAIL: opal_lifo
===============
 Failure :  lifo push/pop multi-threaded with atomics
 Failure :  list pop all items
SUPPORT: OMPI Test failed: opal_lifo_t (2 of 7 failed)
Single thread test. Time: 0 s 3501 us 3 nsec/poppush
Atomics thread finished. Time: 0 s 17674 us 17 nsec/poppush
Atomics thread finished. Time: 0 s 60205 us 60 nsec/poppush
Atomics thread finished. Time: 0 s 63509 us 63 nsec/poppush
Atomics thread finished. Time: 0 s 74591 us 74 nsec/poppush
Atomics thread finished. Time: 0 s 83370 us 83 nsec/poppush
Atomics thread finished. Time: 0 s 84206 us 84 nsec/poppush
Atomics thread finished. Time: 0 s 85807 us 85 nsec/poppush
Atomics thread finished. Time: 0 s 103993 us 103 nsec/poppush
Atomics thread finished. Time: 0 s 106321 us 106 nsec/poppush
All threads finished. Thread count: 8 Time: 0 s 106549 us 13 nsec/poppush
FAIL opal_lifo (exit status: 1)

Full log here (for a few days at least): https://kojipkgs.fedoraproject.org//work/tasks/3745/93473745/build.log

[devreal]

@opoplawski I cannot find the error in the build log. Could you please provide some more info on your environment? What compiler are you using?

[opoplawski]

Ah, shoot. linked the wrong build. This is Fedora Rawhide - gcc 12.2.1

https://kojipkgs.fedoraproject.org//work/tasks/4918/93474918/build.log

[devreal]

Thanks! Any chance you can access test/class/test-suite.log from that run?

[opoplawski]

That's the output I pasted in the first comment.

[devreal]

I see. The tests run succesfully on Summit with GCC 12.1.0 (the latest GCC available on that machine) but I'm not even sure that's the same architecture. I don't have access to any other IBM machine.

[opoplawski]

Can you build in a Fedora Rawhide mock environment on that machine? Looks like I have access to some kind of s390x test machine if there are any particular things you'd like me to try.

vendor_id       : IBM/S390
# processors    : 4
bogomips per cpu: 3241.00
max thread id   : 0
features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt sie 
facilities      : 0 1 2 3 4 6 7 8 9 10 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 57 58 59 60 61 73 74 75 76 77 80 81 82 128 129 130 131 133 134 135 146 147 148 150 151 152 155 156 168
cache0          : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1          : level=1 type=Instruction scope=Private size=128K line_size=256 associativity=8
cache2          : level=2 type=Data scope=Private size=4096K line_size=256 associativity=8
cache3          : level=2 type=Instruction scope=Private size=4096K line_size=256 associativity=8
cache4          : level=3 type=Unified scope=Shared size=262144K line_size=256 associativity=32
cache5          : level=4 type=Unified scope=Shared size=983040K line_size=256 associativity=60
processor 0: version = FF,  identification = 200000,  machine = 8561
processor 1: version = FF,  identification = 200001,  machine = 8561
processor 2: version = FF,  identification = 200002,  machine = 8561
processor 3: version = FF,  identification = 200003,  machine = 8561

[devreal]

Any chance you could try a different compiler like LLVM?

Also, we recently switched from C11 atomics being the default to GCC builtin atomics. Could you try running with C11 atomics instead by passing --enable-c11-atomics to configure?

Summit has Power9 CPUs, so different architecture.

[opoplawski]

So, some data points:

• --enable-c11-atomics doesn't work:

checking dependency style of gcc... gcc3
checking if gcc supports GCC inline assembly... no - architecture not supported
checking for 32-bit GCC built-in atomics... yes
checking for 64-bit GCC built-in atomics... yes
checking if 64-bit GCC built-in atomics are lock-free... yes
checking for __atomic_compare_exchange_n... no
checking for __atomic_compare_exchange_n with -mcx16... no
checking for __atomic_compare_exchange_n with -latomic... yes
checking if __atomic_compare_exchange_n() gives correct results... yes
checking if __int128 atomic compare-and-swap is always lock-free... yes
configure: Using GCC built-in style atomics
configure: WARNING: C11 atomics were requested but are not supported
configure: error: Cannot continue

• The test succeeds when compiled with clang. I don't think this is going to be an option for us though as it currently breaks the FORTRAN build.

[opoplawski]

I'll also note that the test succeeds in 4.1.4 - but maybe the test has changed in 5.0.

[opoplawski]

I've managed to reproduce it on our test machine in case there are any other local tests you would like me to run.

[opoplawski]

Still present in 5.0.0rc10