SEGV in ompi_request_default_test_all() when triggering IPoIB networking problem during ucc_perftest run #13191
Assignees
Labels
Comments
|
@bfaccini was ompi built with ucc? if so, I think you need to disable ucc via --mca coll ^ucc to run ucc perf tests. |
It looks like :
I don't get it, do you mean I need to use "--mca coll ^ucc" at build or run time ? And anyway, don't you think that analysis and fix proposal are ok ? |
|
I'm not sure the patch is enough. UCC uses UCX to move data, and UCX is also used by OMPI as the default PML. The collective framework being initialized after the PML, I wonder how the connection establishment succeeded during the PML setup but then failed during UCC setup ? If you disable UCC is this test succeeding and the connection between your processes is correctly established ? |
|
Actually, nevermind - I confirmed with my colleagues that UCC presence at coll ompi won't affect the functionality of test itself. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background information
What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)
v4.1.7rc1
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Git clone.
If you are building/installing from a git clone, please copy-n-paste the output from
git submodule status.Please describe the system on which you are running
Details of the problem
With the following reproducer with ucc_perftest :
that leads to corefiles being created on several nodes.
Thus I have been able to dig into these corefiles to try to understand the reason of the SEGVs, likely to be caused by a wrong/missing error-path in OMPI/UCC code, when triggering the original error/msgs above upon some IPoIB networking issue.
Here are my findings.
The fully unwinded stack is like following :
where you can see that the unresolved symbol/frame in previously detailed stack is in fact in oob_allgather_test().
And the reason of the SEGV is because :
where reqs[0] is garbage when being dereferenced :
Looking at the corresponding source code in "ompi/mca/coll/ucc/coll_ucc_module.c" :
and just to be complete :
Based on all of this it appears that the following patch/correction (in v4.1.7rc1, the quite recent OMPI version we are running) would allow OMPI/UCC to no longer coredump by gracefully handling any error during isend/irecv :
~/ompi$ git status HEAD detached at v4.1.7rc1 Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: ompi/mca/coll/ucc/coll_ucc_module.c no changes added to commit (use "git add" and/or "git commit -a") ~/ompi$ git diff diff --git a/ompi/mca/coll/ucc/coll_ucc_module.c b/ompi/mca/coll/ucc/coll_ucc_module.c index 1686697618..dfa2674a3d 100644 --- a/ompi/mca/coll/ucc/coll_ucc_module.c +++ b/ompi/mca/coll/ucc/coll_ucc_module.c @@ -158,6 +158,7 @@ static ucc_status_t oob_allgather_test(void *req) int probe_count = 5; int rank, size, sendto, recvfrom, recvdatafrom, senddatafrom, completed, probe; + int rc; size = ompi_comm_size(comm); rank = ompi_comm_rank(comm); @@ -182,10 +183,12 @@ static ucc_status_t oob_allgather_test(void *req) senddatafrom = (rank - oob_req->iter + size) % size; tmprecv = (char*)oob_req->rbuf + (ptrdiff_t)recvdatafrom * (ptrdiff_t)msglen; tmpsend = (char*)oob_req->rbuf + (ptrdiff_t)senddatafrom * (ptrdiff_t)msglen; - MCA_PML_CALL(isend(tmpsend, msglen, MPI_BYTE, sendto, MCA_COLL_BASE_TAG_UCC, + rc = MCA_PML_CALL(isend(tmpsend, msglen, MPI_BYTE, sendto, MCA_COLL_BASE_TAG_UCC, MCA_PML_BASE_SEND_STANDARD, comm, &oob_req->reqs[0])); - MCA_PML_CALL(irecv(tmprecv, msglen, MPI_BYTE, recvfrom, + if (OMPI_SUCCESS != rc) return rc + rc = MCA_PML_CALL(irecv(tmprecv, msglen, MPI_BYTE, recvfrom, MCA_COLL_BASE_TAG_UCC, comm, &oob_req->reqs[1])); + if (OMPI_SUCCESS != rc) return rc } probe = 0; do { @@ -213,6 +216,8 @@ static ucc_status_t oob_allgather(void *sbuf, void *rbuf, size_t msglen, oob_req->msglen = msglen; oob_req->oob_coll_ctx = oob_coll_ctx; oob_req->iter = 0; + oob_req->reqs[0] = NULL; + oob_req->reqs[1] = NULL; *req = oob_req; return UCC_OK; } ~/ompi$The text was updated successfully, but these errors were encountered: