NVIDIA MultiGPU support for SYCL

[ManjulaChalla]

Hi,
I migrated CUDA code to SYCL. And the CUDA code is working fine for NVIDIA Multi
GPU environment and SYCL migrated code is resulting in a segmentation fault in the NVIDIA MultiGPU(2 or 4 GPUs environment).Both CUDA and SYCL code is working for single GPU environment.

SYCL code on NVIDIA GPU(single):

Steps to reproduce:
clang++ -fsycl -fsycl-targets=nvptx64-nvidia-cuda MonteCarlo_kernel.cpp MonteCarloMultiGPU.cpp MonteCarlo_reduction.hpp MonteCarlo_gold.cpp multithreading.cpp

Used the above command for compilation on SYCL on NVIDIA.

SYCL code on NVIDIA Multi GPU:

I validated the SYCL code on Intel MultiGPU environment.It is working fine there.

Is there any limitation for SYCL on NVIDIA MultiGPU Hardware?

[zjin-lcf]

Please try DPCT when the reproducer is available.

https://github.com/oneapi-src/SYCLomatic/

[JackAKirk]

Hi,

Without a reproducer I can't comment on your specific use case. I'll list the ways you can use multiple GPUs with the DPC++ CUDA backend currently. Note that there is no practical limitation for SYCL on Nvidia multi-gpu hardware features and we expect full support in the future. However right now the DPC++ cuda backend does not support DirectGPU Peer to Peer capabilities. We are aiming to add the multi-gpu peer to peer access and copies features ASAP. This is a priority for us.
Despite this I want to emphasize that there are no limitations on which devices you can copy to/from: at the moment the copies just happen via the host rather than directly between peer devices.

Note that if you use MPI with DPC++ for CUDA you can already access essentially all the Multi-GPU features.
Here is the current status and recommendations:

1. (recommended but not yet well documented) Use MPI. DPC++ CUDA backend has now been quite well tested with the two main MPI implementation variants (MPICH and OpenMPI), and mostly works out of the box as you would expect. We will be providing complete documentation for using MPI in the 2023.0 release. The advantage of using MPI is that you can take advantage of DirectGPU P2P memory copies right now; currently there are ways of achieving memory copies across multi-devices within the cuda backend DPC++ runtime, however these copies are always made via host, and are slower than direct P2P. If you would like more information on MPI usage please let me know.

2. Via the runtime
   2a. With SYCL USM using sycl::queue.memcpy to explicitly copy between devices:

int* arr1 = malloc<int>(N, Queues[0], usm::alloc::device);
int* arr2 = malloc<int>(N, Queues[1], usm::alloc::device);
// Copy device usm allocated with different devices
Queues[0].copy(arr2, arr1, N).wait();

2b. With buffers relying on the runtime to manage the copies as in this test: https://github.com/intel/llvm-test-suite/blob/intel/SYCL/Basic/buffer/buffer_dev_to_dev.cpp

In all these cases the trick is to get a list of all CUDA devices that are available in your system (although if using MPI there are ways to select the gpus and map them to your MPI ranks at runtime after compilation). Currently we have a temporary situation where each CUDA device is listed within its own platform: this was done to conform to a SYCL specification constraint for the "default_selector" behaviour. Eventually, all cuda devices should be listed in a single platform and you will be able to simply select from the list of devices in sycl::platform::get_devices(). Fixing this is another priority.
For now you can do:

std::vector<sycl::device> Devs;
// look through all platforms
for (const auto &plt : sycl::platform::get_platforms()) {
// if it is a cuda "platform" then add the device to the list
  if (plt.get_backend() == sycl::backend::cuda)
    Devs.push_back(plt.get_devices()[0]);
}
// create queues as desired
auto q0 = sycl::queue{Devs[0]}; // this corresponds to the lowest cuda ID available (usually 0)
auto q1 = sycl::queue{Devs[1]}; // this corresponds to the second lowest cuda ID available (usually 1)
... etc

Let us know if you want more information at this stage.

[zjin-lcf]

// Copy device usm allocated with different devices
Queues[0].copy(arr2, arr1, N).wait();

If the source (arr1) of the copy is located at Device 0, then the queue is Queues[0]. Is that right ?

[JackAKirk]

// Copy device usm allocated with different devices
Queues[0].copy(arr2, arr1, N).wait();

If the source (arr1) of the copy is located at Device 0, then the queue is Queues[0]. Is that right ?

It actually doesn't matter: The implementation will allow any queue to copy any usm pointers.

It works just like cudaMemcpyAsync or cuMemcpyAsync: Queues[0].copy(arr2, arr1, N) will simple call cuMemcpyAsync: https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__MEM.html#group__CUDA__MEM_1g5f26aaf5582ade791e5688727a178d78.

[zjin-lcf]

I migrated the CUDA simpleP2P example to SYCL. Running the program shows "illegal memory accesses" when the first kernel is executed on a device.
Please see the reproducer: https://github.com/zjin-lcf/HeCBench/blob/master/p2p-sycl/main.cpp
The CUDA program is https://github.com/zjin-lcf/HeCBench/blob/master/p2p-cuda/main.cu
I am not sure if the SYCL program is written correctly. Thanks.

[JackAKirk]

I expect it is due to the fact you are trying to access memory that is on one device from the queue that uses a different device here:

void SimpleKernel(nd_item<1> &item, const float *src, float *dst)
{
  // Just a dummy kernel, doing enough for us to verify that everything
  // worked
  const int idx = item.get_global_id(0);
  dst[idx] = src[idx] * 2.0f;
}

To do this we would need a finished extension here of ext_oneapi_enable_peer_access to enable peer memory access across devices #6104.
If you examine what the CUDA driver is doing whilst execution the memcpy commands in this program (if you make it compile by removing the offending kernel) you will see that a peer to peer copy is achieved via host. However you still can't do a direct peer access as the offending kernel is trying to do.

This is very easy to implement for a given backend: I actually implemented it for the cuda backend here: JackAKirk@3b36bc4

The challenge as ever is to make a DPC++ runtime / p2p extension that is appropriate in general: or at least for CUDA/HIP and level_zero backends. This requires a concerted effort to solve some questions:

1. What is sycl::context for
2. Should we map cuContext to sycl::device instead of sycl::context? (Answer: probably)

[zjin-lcf]

I can see the SYCL/CUDA bandwidth with the offending kernels commented. I will wait for the pull request to support the P2P example. I know little about context, and hope users don't need to know context.
If CUDA combines 'context' and 'device', could SYCL combine them too ?

Thanks

[keryell]

If CUDA combines 'context' and 'device', could SYCL combine them too ?

SYCL can be higher level and represent things slightly differently from the back-ends. It is necessary for portability anyway since not all the back-ends have the same abstractions.

[zjin-lcf]

Hi,

Without a reproducer I can't comment on your specific use case. I'll list the ways you can use multiple GPUs with the DPC++ CUDA backend currently. Note that there is no practical limitation for SYCL on Nvidia multi-gpu hardware features and we expect full support in the future. However right now the DPC++ cuda backend does not support DirectGPU Peer to Peer capabilities. We are aiming to add the multi-gpu peer to peer access and copies features ASAP. This is a priority for us. Despite this I want to emphasize that there are no limitations on which devices you can copy to/from: at the moment the copies just happen via the host rather than directly between peer devices.

Note that if you use MPI with DPC++ for CUDA you can already access essentially all the Multi-GPU features. Here is the current status and recommendations:

1. (recommended but not yet well documented) Use MPI. DPC++ CUDA backend has now been quite well tested with the two main MPI implementation variants (MPICH and OpenMPI), and mostly works out of the box as you would expect. We will be providing complete documentation for using MPI in the 2023.0 release. The advantage of using MPI is that you can take advantage of DirectGPU P2P memory copies right now; currently there are ways of achieving memory copies across multi-devices within the cuda backend DPC++ runtime, however these copies are always made via host, and are slower than direct P2P. If you would like more information on MPI usage please let me know.
2. Via the runtime
   2a. With SYCL USM using sycl::queue.memcpy to explicitly copy between devices:

int* arr1 = malloc<int>(N, Queues[0], usm::alloc::device);
int* arr2 = malloc<int>(N, Queues[1], usm::alloc::device);
// Copy device usm allocated with different devices
Queues[0].copy(arr2, arr1, N).wait();

2b. With buffers relying on the runtime to manage the copies as in this test: https://github.com/intel/llvm-test-suite/blob/intel/SYCL/Basic/buffer/buffer_dev_to_dev.cpp

In all these cases the trick is to get a list of all CUDA devices that are available in your system (although if using MPI there are ways to select the gpus and map them to your MPI ranks at runtime after compilation). Currently we have a temporary situation where each CUDA device is listed within its own platform: this was done to conform to a SYCL specification constraint for the "default_selector" behaviour. Eventually, all cuda devices should be listed in a single platform and you will be able to simply select from the list of devices in sycl::platform::get_devices(). Fixing this is another priority. For now you can do:

std::vector<sycl::device> Devs;
// look through all platforms
for (const auto &plt : sycl::platform::get_platforms()) {
// if it is a cuda "platform" then add the device to the list
  if (plt.get_backend() == sycl::backend::cuda)
    Devs.push_back(plt.get_devices()[0]);
}
// create queues as desired
auto q0 = sycl::queue{Devs[0]}; // this corresponds to the lowest cuda ID available (usually 0)
auto q1 = sycl::queue{Devs[1]}; // this corresponds to the second lowest cuda ID available (usually 1)
... etc

Let us know if you want more information at this stage.

I realize that the example may be from https://github.com/NVIDIA/cuda-samples/blob/master/Samples/5_Domain_Specific/MonteCarloMultiGPU/
@ManjulaChalla may confirm this.