# Bitsandbytes quantization and QLORA fine-tuning

**URL:** <https://discuss.huggingface.co/t/bitsandbytes-quantization-and-qlora-fine-tuning/111373>\
**Category:** 🤗Transformers\
**Created:** [October 11, 2024, 10:46am UTC](https://discuss.huggingface.co/t/bitsandbytes-quantization-and-qlora-fine-tuning/111373 "2024-10-11T10:46:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dzoni99](https://avatars.discourse-cdn.com/v4/letter/d/3be4f8/32.png) [@dzoni99](https://discuss.huggingface.co/u/dzoni99)\
**Post date:** [October 11, 2024, 10:46am UTC](https://discuss.huggingface.co/t/bitsandbytes-quantization-and-qlora-fine-tuning/111373/1 "2024-10-11T10:46:48Z")

</div>

Hello friends!

I want to fine tune a quantized RoBERTa base model using the QLORA approach. Below is the configuration.

bnb\_config = BitsAndBytesConfig(  
load\_in\_4bit=True,  
bnb\_4bit\_use\_double\_quant=True,  
bnb\_4bit\_quant\_type=“nf4”,  
bnb\_4bit\_compute\_dtype=torch.bfloat16,  
llm\_int8\_skip\_modules=[“classifier”]  
)

model = AutoModelForSequenceClassification.from\_pretrained(  
“roberta-base”,  
num\_labels=2,  
quantization\_config=bnb\_config,  
torch\_dtype=torch.bfloat16,  
device\_map=“auto”,  
)

What I’m not sure I understand, when I look at the datatypes for the LORA matrices, they are in float 32 format. Also, after executing the function prepare\_model\_for\_kbit\_training, the other parts of the layers, all except the weights, are converted to float 32 (bias, layernorm…). Do they and the LORA matrices have to be in 32b format, or can they somehow be converted to 16b format? When combining LORA matrices and model weights matrices, LORA matrices are ​​converted to bfloat16 or everything is converted to float 32? Is the quantization potential used if some layers remain in 32b format?

---

<div class="post-metadata">

**Author:** ![lilylii](https://avatars.discourse-cdn.com/v4/letter/l/b9e5f3/32.png) [@lilylii](https://discuss.huggingface.co/u/lilylii)\
**Post date:** [November 5, 2024, 9:18pm UTC](https://discuss.huggingface.co/t/bitsandbytes-quantization-and-qlora-fine-tuning/111373/2 "2024-11-05T21:18:44Z")

</div>

Do you have an answer/clarification for this? I have similar confusion - I am fine-tuning Llama 3.1 with QLoRA and am unable to load the model to have tensor.dtype=torch.bfloat16