IO streams forwarding of child processes spawned by a singleton using `MPI_Comm_spawn`

[xbw22109]

Here is the issue I encountered.

In the git main branch, using ompi-tests-public/singleton

I got:

x@x:~/github/test/ompi-tests-public/singleton$ make
mpicc hello_c.c -Wall -g -O0 -o hello_c
mpicc simple_spawn.c -Wall -g -O0 -o simple_spawn
mpicc simple_spawn_multiple.c -Wall -g -O0 -o simple_spawn_multiple
x@x:~/github/test/ompi-tests-public/singleton$ ompi_info --version
Open MPI v5.1.0a1

https://www.open-mpi.org/community/help/
x@x:~/github/test/ompi-tests-public/singleton$ ./run.sh
=====================
Testing: Hello with mpirun
=====================
  0/  1) [x] 548449 Hello, world!
=====================
Testing: Hello as a singleton
=====================
  0/  1) [x] 548451 Hello, world!
=====================
Testing: MPI_Comm_spawn with mpirun
=====================
Spawning './simple_spawn' ... OK
=====================
Testing: MPI_Comm_spawn as a singleton
=====================
Spawning './simple_spawn' ... OK
=====================
Testing: MPI_Comm_spawn_multiple with mpirun
=====================
Hello from a Child (B)
Hello from a Child (A)
Hello from a Child (B)
Spawning Multiple './simple_spawn_multiple' ... OK
=====================
Testing: MPI_Comm_spawn_multiple as a singleton
=====================
Spawning Multiple './simple_spawn_multiple' ... OK
=====================
Success
=====================

Note that the output of MPI_Comm_spawn_multiple singleton is different from mpirun.

Further testing

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[])
{
    MPI_Comm parent_comm, intercomm;
    int world_rank, world_size;
    MPI_Init(&argc, &argv);

    MPI_Comm_get_parent(&parent_comm);

    if (parent_comm == MPI_COMM_NULL) {
        // I am the parent
        MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
        MPI_Comm_size(MPI_COMM_WORLD, &world_size);

        printf("Parent: rank %d out of %d\n", world_rank, world_size);

        // Spawn a child processes
        MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, 1,
                       MPI_INFO_NULL, 0, MPI_COMM_WORLD,
                       &intercomm, MPI_ERRCODES_IGNORE);

        // Receive message from child rank 0
        char buf[128];
        MPI_Recv(buf, 128, MPI_CHAR, 0, 0, intercomm, MPI_STATUS_IGNORE);

        printf("Parent: received message: %s\n", buf);
    } else {
        // I am the child
        MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
        MPI_Comm_size(MPI_COMM_WORLD, &world_size);

        printf("Child: rank %d out of %d\n", world_rank, world_size);

        char msg[128];
        snprintf(msg, sizeof(msg), "Hello from child rank %d!", world_rank);

        // Send message to parent (rank 0 in parent group)
        MPI_Send(msg, strlen(msg) + 1, MPI_CHAR, 0, 0, parent_comm);

        printf("Child rank %d sent message\n", world_rank);
    }

    MPI_Finalize();
    return 0;
}

Output of the above program:

Parent: rank 0 out of 1
Parent: received message: Hello from child rank 0!

I suspect that IO forwarding may not be functioning as expected in the above scenario, and further testing is needed.

[rhc54]

It works fine with head of upstream PRRTE master:

$ mpirun -n 1 ./check
Parent: rank 0 out of 1
Child: rank 0 out of 1
Child rank 0 sent message
Parent: received message: Hello from child rank 0!

So I suspect this is just a case of your PRRTE fork being out-of-date.

[xbw22109]

I am using the submodule automatically found by git.
In my environment, the output of singleton mode is not as expected.
you can run it like this:

$ mpicc check.c -o check
$ mpirun -n 1 ./check
Parent: rank 0 out of 1
Child: rank 0 out of 1
Child rank 0 sent message
Parent: received message: Hello from child rank 0!
$ ./check
Parent: rank 0 out of 1
Parent: received message: Hello from child rank 0!

[rhc54]

The submodule in OMPI points to a fork of PRRTE, not the upstream repo, and that fork is out-of-date. I think Howard recently tried to resync it, but I doubt that has come over to OMPI yet.

[xbw22109]

It works fine with head of upstream PRRTE master

I use the latest version of prrte and openpmix, and the running results are correct.

[rhc54]

Hmmm...well, I've been working on this a bit, and it does not work in the case where multiple nodes have been allocated. Works fine on just a single node. I've tracked the problem down, finding a few other issues along the way (e.g., ctrl-C will kill the singleton, but leave prte hanging as a zombie process) and fixed them (committed to upstream PMIx/PRRTE master branches).

Will be working on a fix today.

[rhc54]

Okay, I fixed the remaining piece and this example now works in multi-node environments. Not sure if/when that makes it over to OMPI - fixes required in both PMIx and PRRTE to resolve all issues. Note that some of it could be fixed with changes in OMPI, but didn't feel like it was worth pursuing that avenue.