# Using GPU with transformers

**URL:** <https://discuss.huggingface.co/t/using-gpu-with-transformers/1827>\
**Category:** Beginners\
**Created:** [October 30, 2020, 9:20pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827 "2020-10-30T21:20:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![spartan](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@spartan](https://discuss.huggingface.co/u/spartan)\
**Post date:** [October 30, 2020, 9:20pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827/1 "2020-10-30T21:20:52Z")

</div>

Hi! I am pretty new to Hugging Face and I am struggling with next sentence prediction model. I would like it to use a GPU device inside a Colab Notebook but I am not able to do it. This is my proposal:

```python
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased', return_dict=True)
model.to("cuda:0")
prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
next_sentence = "The sky is blue due to the shorter wavelength of blue light."
tokenizer_output = tokenizer(prompt, next_sentence, return_tensors='pt')
tokens_tensor = tokenizer_output['input_ids'].to('cuda:0')
token_type_ids = tokenizer_output['token_type_ids'].to('cuda:0')
attention_mask = tokenizer_output['attention_mask'].to('cuda:0')
encoding = {'input_ids' : tokens_tensor, 
        'token_type_ids' : token_type_ids, 
        'attention_mask' : attention_mask}
outputs = model(**encoding)
logits = outputs.logits
print(logits) # next sentence was random

```

However, it does not work. The GPU device is properly set, since I used the following command to check it:

```python
import torch; print(torch.cuda.get_device_name(0))

```

I would appreciate any help

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 30, 2020, 10:10pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827/2 "2020-10-30T22:10:31Z")

</div>

What do you mean by “doesn’t work”. What errors do you get or how do you know the GPU isn’t used.

---

<div class="post-metadata">

**Author:** ![spartan](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@spartan](https://discuss.huggingface.co/u/spartan)\
**Post date:** [October 30, 2020, 10:26pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827/3 "2020-10-30T22:26:50Z")

</div>

No, it does not return any error.  
I think that it is not using the GPU since I have tested the time with and without “GPU instructions”, and it is not better, it is even a little bit worse.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 30, 2020, 10:59pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827/4 "2020-10-30T22:59:43Z")

</div>

Sorry, that is not enough information to go on because “you think” it is not using the GPU. You can monitor your GPU usage with something like nvidia-smi or your task manager. Furthermore, trying this for one single item will probably not yield you a big difference either if you include the model loading in the timing because that is the slowest part for a single item.

We will need more information and a more thorough investigation on your part.

---

<div class="post-metadata">

**Author:** ![spartan](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@spartan](https://discuss.huggingface.co/u/spartan)\
**Post date:** [November 3, 2020, 12:24pm UTC](https://discuss.huggingface.co/t/using-gpu-with-transformers/1827/5 "2020-11-03T12:24:46Z")

</div>

Hi @BramVanroy, thank you for your answers. As you suggested, one item was not enough to test GPU improvement and I was also taking into account the model loading timing.  
Finally, I managed to simplify my code (just in case it helps somebody):

```python
prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
next_sentence = "The sky is blue due to the shorter wavelength of blue light."
encoding = tokenizer.encode_plus(prompt, next_sentence, add_special_tokens=True, max_length=tokenizer.max_len, return_tensors='pt').to('cuda:0')
outputs = model(**encoding, next_sentence_label=torch.cuda.LongTensor([1]))
t = outputs.loss
print(t)

```

Thank you again and sorry for bothering you with this issue, but as I said I am quite new to this transformers “world”