spacy train unable to recognize GPUs

[qu-genesis]

Hi! I'm working on a GPU enabled machine on Sagemaker to run spacy train with a transformer backend but I'm running into an issue where cupy does not recognize the CUDA on the machine.

The crux of the issue is that thinc.api.has_cupy returns False, despite cupy being installed and nvidia-smi returning valid information.

I tried running uv pip install cupy-wheel and then uv pip install cupy as this post suggested #13567 , but I got an error saying that NVIDIA CUDA is not recognized on my machine. And installing cupy also errored out, with this being the main error: cudnnRNNForwardTrainingEx' was declared in this scope.

Here are some more specs and version information, any help is appreciated.

OS: Ubuntu 22.04.5 LTS
Python Version: 3.9.22
NVIDIA-SMI 550.163.01
Driver Version: 550.163.01
CUDA Version: 12.4

cupy-cuda12x 13.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
spacy 3.8.5
spacy-alignments 0.9.1
spacy-legacy 3.0.12
spacy-loggers 1.0.5
spacy-transformers 1.3.8
thinc 8.3.4
torch 2.1.0+cu118
torchvision 0.16.0+cu118
transformers 4.49.0