sycl-for-cuda build/doc/run issues

[reguly]

Quick note: please update https://github.com/codeplaysoftware/sycl-for-cuda/blob/cuda/sycl/doc/GetStartedWithSYCLCompiler.md to (1) indicate which branch should be used (still says sycl) and (2) to give a proper list of cmake flags, as currently one has to cherry-pick them from the "Build SYCL toolchain with support for NVIDIA CUDA" and the "Build SYCL toolchain/Linux" sections.
After several rounds of building the whole thing I think I managed to figure it out, but would have saved me a lot of time if this were already up-to-date.

My build on Centos 8 with gcc 8.2.1:
cmake -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda/ -DLLVM_ENABLE_PROJECTS="clang;llvm-spirv;sycl;libclc" -DSYCL_BUILD_PI_CUDA=ON -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLIBCLC_TARGETS_TO_BUILD="nvptx64--;nvptx64--nvidiacl" -DCMAKE_INSTALL_PREFIX=/home/szibo4/sycl-for-cuda-install -DCMAKE_BUILD_TYPE=Release -DLLVM_EXTERNAL_PROJECTS="llvm-spirv;sycl" -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=../sycl-for-cuda/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=../sycl-for-cuda/llvm-spirv ../sycl-for-cuda/llvm
Got some messages that I am not sure are good or not:
BUILDING nvptx64--
DEVICE: none ( )

[reguly]

The same issue as reported in #1119 (@Ruyk in #1119 (comment)_) is resolved by explicitly using the CUDASelector.

[reguly]

Finally, when I try and compile a larger code, I get a linker error:
ptxas fatal : Unresolved extern function '_Z16__spirv_ocl_cbrtd'

This symbol is not found in any of my object file being linked (even as an undefined one). Verbose clang link command (why sm_30??):

clang++ -O3 -std=c++11 -I/home/szibo4/sycl-for-cuda-install/include -I/home/szibo4/sycl-for-cuda-install/include -fsycl-targets=nvptx64-nvidia-cuda-sycldevice -O3 -DVECTORIZE obj/mgcfd_sycl_main.o obj/mgcfd_sycl_kernels.o -lm -L/home/szibo4/OP2-Common-syclcuda/op2/c/lib -lop2_sycl -foffload-static-lib=/home/szibo4/OP2-Common-syclcuda/op2/c/lib/libop2_sycl.a -lop2_hdf5 -L/opt/hdf5-seq/lib -lhdf5 -lz -o bin/mgcfd_sycl -fsycl -v
clang version 11.0.0 (https://github.com/codeplaysoftware/sycl-for-cuda.git bd5a107e069979318b314ba8c1b0933d57556eb1)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/szibo4/sycl-for-cuda-install/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-10.1, version 10.1
"/bin/ld" -r -o /tmp/mgcfd_sycl_main-prelink-5cf2a8.o obj/mgcfd_sycl_main.o obj/mgcfd_sycl_kernels.o -L/home/szibo4/OP2-Common-syclcuda/op2/c/lib -L/opt/hdf5-seq/lib /home/szibo4/OP2-Common-syclcuda/op2/c/lib/libop2_sycl.a
"/home/szibo4/sycl-for-cuda-install/bin/clang-offload-bundler" -type=oo -targets=host-x86_64-unknown-linux-gnu,sycl-nvptx64-nvidia-cuda-sycldevice -inputs=/tmp/mgcfd_sycl_main-prelink-5cf2a8.o -outputs=/tmp/mgcfd_sycl_main-06c3ab.o,/tmp/mgcfd_sycl_main-4e1a0d.txt -unbundle
"/home/szibo4/sycl-for-cuda-install/bin/llvm-link" @/tmp/mgcfd_sycl_main-4e1a0d.txt -o /tmp/mgcfd_sycl_main-4e1a0d-9f7a9f.bc --suppress-warnings
"/home/szibo4/sycl-for-cuda-install/bin/clang-11" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl-is-device -aux-triple x86_64-unknown-linux-gnu -sycl-std=1.2.1 -S -disable-free -disable-llvm-verifier -discard-value-names -main-file-name mgcfd_sycl_main-4e1a0d.txt -mrelocation-model static -mthread-model posix -mframe-pointer=all -fdenormal-fp-math=ieee,ieee -fdenormal-fp-math-f32=ieee,ieee -fno-rounding-math -no-integrated-as -mlink-builtin-bitcode /home/szibo4/sycl-for-cuda-install/lib/clang/11.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.1/nvvm/libdevice/libdevice.10.bc -target-feature +ptx64 -target-sdk-version=10.1 -target-cpu sm_30 -dwarf-column-info -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/szibo4/sycl-for-cuda-install/lib/clang/11.0.0 -O3 -std=c++11 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/szibo4/MG-CFD-app-OP2 -ferror-limit 19 -fmessage-length 0 -fgnuc-version=4.2.1 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -o /tmp/mgcfd_sycl_main-4e1a0d-97b9e9.s -x ir /tmp/mgcfd_sycl_main-4e1a0d-9f7a9f.bc
clang -cc1 version 11.0.0 based upon LLVM 11.0.0git default target x86_64-unknown-linux-gnu
"/usr/local/cuda-10.1/bin/ptxas" -m64 -O3 -v --gpu-name sm_30 --output-file /tmp/mgcfd_sycl_main-4e1a0d-2efe29.o /tmp/mgcfd_sycl_main-4e1a0d-97b9e9.s
ptxas fatal : Unresolved extern function '_Z16__spirv_ocl_cbrtd'
clang-11: error: ptxas command failed with exit code 255 (use -v to see invocation)

[bader]

This looks like the application uses cbrt(double) built-in function and it seems to be unsupported by the current implementation. @Naghasan, am I right?

[Ruyk]

Branches: We are working in integrating with upstream (the sooner we get to do that the better for everyone) so the branches are not stable. We'll have a more stable branch once we get #1091 merged.
Documentation: I've made a note on the branch situation on the repo.
cbrt: Yes, its missing, we are still adding mappings of builtins to PTX equivalents.

We are also working on some usage examples in codeplaysoftware/SYCL-For-CUDA-Examples#1 that may help, more to come later on.

[fwyzard]

We are also working on some usage examples in codeplaysoftware/SYCL-For-CUDA-Examples#1 that may help, more to come later on.

The link seems to be broken ?

[Ruyk]

Just realized its still private awaiting for further internal reviews, i'll update this issue once its public

[Ruyk]

The repo is pulic now, https://github.com/codeplaysoftware/SYCL-For-CUDA-Examples. There is only one trivial example at the moment but we will be adding more as we work in the project.

[bader]

@reguly, does #1293 address this issue?

[reguly]

It does - thank you!