RuntimeError 'from_dlpack received an invalid capsule.' trying to train on GPU/RunPod

[toadle]

Hey everybody,

I'm trying to traing a spaCy pipeline ["transformer","textcat_multilabel"] on GPU. I've verified the config to be working by training it with CPU and yielding a working model. But training took ~3h and I would like to speed it up.

I'm utilizing https://www.runpod.io/ to get a GPU and successfully installed my setup and the needed cupy. Since RunPod offers different PyTorch installation (pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 and pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04) I installed the respective versions of cupy-cuda11x or cupy-cuda12x.

But when I then start my training with poetry run spacy train config.cfg --paths.train "train.spacy" --paths.dev "dev.spacy" --paths.labels "textcat_multilabel.json" --output model_transformer --gpu-id 0 I ended up here in both of the above cases:

Aborting and saving the final best model. Encountered exception:
RuntimeError('from_dlpack received an invalid capsule. Note that DLTensor
capsules can be consumed only once, so you might have already constructed a
tensor from it once.')

I found this: explosion/thinc#741 hinting at that I need a specific numpy version.
When I run poetry show I get:

numpy              1.26.4     Fundamental package for array computing in Python
spacy              3.7.5      Industrial-strength Natural Language Processing (NLP) in Python
spacy-transformers 1.3.8      spaCy pipelines for pre-trained BERT and other transformers
thinc              8.2.5      A refreshing functional take on deep learning, compatible with your favorite libraries

I somehow have the feeling that I need a numpy version 2.x but I don't know which combination should work.

Any hints would be greatly appreciated.

[pantheahk]

I was facing an issue when using the en-core-web-trf SpaCy model for inference on GPU. This was related to a bug in toDlpack() in cupy-cuda11x == 13.4.0. This is fixed in 13.4.1 (cupy/cupy#9011) so maybe updating cupy-cuda11x fixes your issue too.