Reduce memory overhead for HIP backends on MI300A GPUs #1734
Conversation
| @@ -114,6 +107,7 @@ static int CeedBasisApplyAtPointsCore_Hip(CeedBasis basis, bool apply_add, const | |||
| const CeedScalar *d_x, *d_u; | |||
| CeedScalar *d_v; | |||
| CeedBasis_Hip *data; | |||
| Ceed_Hip *hip_data; | |||
| @@ -126,6 +120,7 @@ static int CeedBasisApplyAtPointsCore_Hip(CeedBasis basis, bool apply_add, const | |||
| } | |||
|
|
|||
| CeedCallBackend(CeedBasisGetCeed(basis, &ceed)); | |||
| CeedCallBackend(CeedGetData(ceed, &hip_data)); | |||
zatkins-dev
force-pushed
the
zach/hip-mi300a
branch
from
9dd589f to
35926f8
Compare
zatkins-dev
force-pushed
the
zach/hip-mi300a
branch
from
35926f8 to
5ad657d
Compare
| CeedVector_Hip *impl; | ||
|
|
||
| CeedCallBackend(CeedVectorGetData(vec, &impl)); | ||
| CeedCallHip(CeedVectorReturnCeed(vec), hipDeviceSynchronize()); |
There was a problem hiding this comment.
Ratel seems to work fine without this line, and is faster
There was a problem hiding this comment.
Does CeedVectorSyncArray mean that one could immediately start an MPI_Send? If the host doesn't know that the previous kernel (writing to the array) has completed, then it would be racy to call MPI_Send. (Might be rare to trip, but we don't want that kind of bug.)
If our sends are using a kernel for packing (on the same stream), then the host doesn't need to know when the earlier stuff completes, but we still need to sync after the packing kernel.
There was a problem hiding this comment.
that's a fair point, I think that we need to be a bit more careful and only sync when the host needs the data. Otherwise this acts as a hard sync with the GPU, which seems to have performance impacts.
|
FYI, generally prefer rebase to merge for dev branches. It doesn't matter for squash-merges (the commit history gets nuked anyways), but for normal merges it helps the git history be more regular. |
Yeah generally I agree - I need to strip down this branch and rebuild it probably, it's currently a mess due to changes at the AMD workshop. |
Prevents double allocations for
CeedVectorwhen using HIP vector with unified addressing and XNACK.Also, updates more of the HIP vector operations to use
hipBLASfunctions rather than custom kernels.