Skip to content

Use ROCm clang for Ghost unified-memory examples - #190

Open
dannybaths wants to merge 1 commit into
amd:mainfrom
dannybaths:fix-ghost-usm-rocm-compiler
Open

Use ROCm clang for Ghost unified-memory examples#190
dannybaths wants to merge 1 commit into
amd:mainfrom
dannybaths:fix-ghost-usm-rocm-compiler

Conversation

@dannybaths

@dannybaths dannybaths commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

What

The five MPI_Ghost_Exchange_Ver[1-5] test scripts now select the ROCm C and C++
compilers from $ROCM_PATH/bin when amdclang and amdclang++ are present there.
Each script exports CC/CXX immediately before its per-test CMake configure, so
the GhostExchange examples build with ROCm clang instead of whatever the MPI wrapper
defaults to (often GCC on Cray/Open MPI login environments).

These examples use #pragma omp requires unified_shared_memory. GCC does not
implement that clause on the offload path these tests need.

Why

On MI300A systems where the MPI module defaults to GCC, CMake picks GCC and every
Ghost Ver1–5 build aborts on the requires unified_shared_memory line before launch.
The scripts already use ROCM_PATH for the ROCm runtime; selecting the matching
ROCm compiler keeps the tests self-contained without launcher-specific -x flags or
Open MPI-only options on the scripts' srun branch.

Validation

MI300A / gfx942, Open MPI, HSA_XNACK=1, ROCM_PATH pointing at a ROCm install
with bin/amdclang and bin/amdclang++. Focused run on upstream main (20e93223):

git clone https://github.com/amd/HPCTrainingExamples.git
cd HPCTrainingExamples/tests
export HSA_XNACK=1
export ROCM_PATH=/path/to/rocm
rm -rf build && cmake . -B build
ctest -V -R '^MPI_Ghost_Exchange_Ver[1-5]$' --test-dir build

Static checks on the branch:

git diff --check
bash -n tests/mpi_ghost_exchange_ver{1,2,3,4,5}.sh

Unpatched upstream: 0/5 pass. All five builds use GCC and fail at compile with
sorry, unimplemented: 'unified_shared_memory' clause on 'requires' directive not supported yet.

With this change: 5/5 pass. Per-test CMake logs show
Check for working C/CXX compiler: …/rocm-…/bin/amdclang{,++}; CTest reports
100% tests passed, 0 tests failed out of 5.

AAC7 proof: current-main control on ROCm 7.14 / Open MPI 5.0.10 — unpatched 0/5,
patched 5/5.

AAC6 no-regression: MI300A, Lmod rocm/7.14.0, Slurm job 19736 on
ppac-pl1-s24-16 — 5/5 PASS with amdclang/amdclang++ under $ROCM_PATH/bin.

Signed-off-by: Daniele Bagni <danieleb@xilinx.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant