# 🤗Optimum **URL:** https://discuss.huggingface.co/c/optimum/59.md [Latest](https://discuss.huggingface.co/latest.md) · [Categories](https://discuss.huggingface.co/categories.md) --- ## [About the 🤗 Optimum category](https://discuss.huggingface.co/t/about-the-optimum-category/16096)
**Author:** [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Replies:** 0
This category is for any discussion around the Optimum library. --- ## [Списали средтва и не дали подписку](https://discuss.huggingface.co/t/topic/172966)
**Author:** [@grasss061](https://discuss.huggingface.co/u/grasss061)\ **Replies:** 0\ **Last updated:** [January 31, 2026, 7:45am UTC](https://discuss.huggingface.co/t/topic/172966 "2026-01-31T07:45:45Z")
ПРи оформлении подписки п PRO” забрали сумму 30$ и подписки нет --- ## [CUDA OOM when export a large model to ONNX](https://discuss.huggingface.co/t/cuda-oom-when-export-a-large-model-to-onnx/31877)
**Author:** [@in-certo](https://discuss.huggingface.co/u/in-certo)\ **Replies:** 6\ **Last updated:** [July 26, 2025, 2:25pm UTC](https://discuss.huggingface.co/t/cuda-oom-when-export-a-large-model-to-onnx/31877 "2025-07-26T14:25:15Z")
I got OOM when exporting a large model to ONNX. I wonder how Optimum handles this issue. Here are my settings The code works for a smaller model with fewer parameters, so the error is due to the model size. Not able t… --- ## [Difference in the vector generated by the int8 quantized model vs base onnx model](https://discuss.huggingface.co/t/difference-in-the-vector-generated-by-the-int8-quantized-model-vs-base-onnx-model/164270)
**Author:** [@Rkoy](https://discuss.huggingface.co/u/Rkoy)\ **Replies:** 4\ **Last updated:** [July 25, 2025, 9:08am UTC](https://discuss.huggingface.co/t/difference-in-the-vector-generated-by-the-int8-quantized-model-vs-base-onnx-model/164270 "2025-07-25T09:08:28Z")
Recently i tried to compare the BAAI/bge-m3 Onnx model vs its int8 AVX2 quantized version. I see that there is a huge difference in the vectors generated by the base Onnx model vs the vectors generated by the int8 avx2 q… --- ## [Difference between model.onnx and model.onnx.data](https://discuss.huggingface.co/t/difference-between-model-onnx-and-model-onnx-data/162032)
**Author:** [@Rkoy](https://discuss.huggingface.co/u/Rkoy)\ **Replies:** 2\ **Last updated:** [July 8, 2025, 9:17am UTC](https://discuss.huggingface.co/t/difference-between-model-onnx-and-model-onnx-data/162032 "2025-07-08T09:17:18Z")
Hi team, i am new to optimum and have used the onnxruntime library a bit previously. When i try to convert a model using onnxruntime, i get only one output file say model.onnx but when i tried the below command of the … --- ## [What does the decoder with past values means](https://discuss.huggingface.co/t/what-does-the-decoder-with-past-values-means/21088)
**Author:** [@Rkoy](https://discuss.huggingface.co/u/Rkoy)\ **Replies:** 2\ **Last updated:** [July 7, 2025, 11:06am UTC](https://discuss.huggingface.co/t/what-does-the-decoder-with-past-values-means/21088 "2025-07-07T11:06:34Z")
Those three parts consist of the encoder, the “decoder” (which actually consists of the decoder with the language modeling head), and the “decoder” with pre-computed key/values as additional inputs. This specific export… --- ## [How do I upstream a brand-new hardware backend to 🤗 Transformers/Optimum?](https://discuss.huggingface.co/t/how-do-i-upstream-a-brand-new-hardware-backend-to-transformers-optimum/161297)
**Author:** [@halflucifer](https://discuss.huggingface.co/u/halflucifer)\ **Replies:** 2\ **Last updated:** [July 2, 2025, 2:37am UTC](https://discuss.huggingface.co/t/how-do-i-upstream-a-brand-new-hardware-backend-to-transformers-optimum/161297 "2025-07-02T02:37:46Z")
Hi everyone! :waving\_hand: I’m part of the team behind xThought, an upcoming AI accelerator hardware for LLM inference. We’ve already built a working Transformers extension (xthought\_transformers) that ships: XThough… --- ## [ONNX export failed for Qwen/Qwen3-Embedding-0.6B with "invalid unordered\_map\ key"](https://discuss.huggingface.co/t/onnx-export-failed-for-qwen-qwen3-embedding-0-6b-with-invalid-unordered-map-k-t-key/160909)
**Author:** [@Colegero](https://discuss.huggingface.co/u/Colegero)\ **Replies:** 4\ **Last updated:** [June 27, 2025, 10:39pm UTC](https://discuss.huggingface.co/t/onnx-export-failed-for-qwen-qwen3-embedding-0-6b-with-invalid-unordered-map-k-t-key/160909 "2025-06-27T22:39:09Z")
Hello everyone, I am trying to export the “Qwen/Qwen3-Embedding-0.6B” model to ONNX using the “optimum” library. According to the Optimum documentation, the “Qwen3” architecture is supported for ONNX export. However,… --- ## [\[Guide\] Quantize LLM CoreML to int8 on Mac ARM (TinyLlama, May 2025, tested workflow & script)](https://discuss.huggingface.co/t/guide-quantize-llm-coreml-to-int8-on-mac-arm-tinyllama-may-2025-tested-workflow-script/156842)
**Author:** [@GreenBull31](https://discuss.huggingface.co/u/GreenBull31)\ **Replies:** 0\ **Last updated:** [May 26, 2025, 10:59am UTC](https://discuss.huggingface.co/t/guide-quantize-llm-coreml-to-int8-on-mac-arm-tinyllama-may-2025-tested-workflow-script/156842 "2025-05-26T10:59:18Z")
Hi all, Like many devs here, I spent a lot of time looking for a working, recent and clear pipeline to quantize LLM CoreML models (.mlpackage) to int8 on Mac ARM (Apple Silicon). Most existing guides are out of date, b… --- ## [Trying to convert DeepSeek-R1 into onnx](https://discuss.huggingface.co/t/trying-to-convert-deepseek-r1-into-onnx/145528)
**Author:** [@venkatesh35](https://discuss.huggingface.co/u/venkatesh35)\ **Replies:** 1\ **Last updated:** [March 13, 2025, 4:39pm UTC](https://discuss.huggingface.co/t/trying-to-convert-deepseek-r1-into-onnx/145528 "2025-03-13T16:39:57Z")
I’m trying to convert DeepSeek-R1 into a onnx format, but i’m being presented with ValueError: Loading deepseek-ai/DeepSeek-R1 requires you to execute the configuration file in that repo on your local machine. Make sur… --- ## [Optimum library optimization and quantization fails](https://discuss.huggingface.co/t/optimum-library-optimization-and-quantization-fails/72629)
**Author:** [@ddahlmeier](https://discuss.huggingface.co/u/ddahlmeier)\ **Replies:** 8\ **Last updated:** [February 22, 2025, 6:50am UTC](https://discuss.huggingface.co/t/optimum-library-optimization-and-quantization-fails/72629 "2025-02-22T06:50:07Z")
I used optimum and onnxruntime to optimize and quantize a roberta squad QA model using the example from this blog The code used to work fine until this week when it started to break. I suddenly get an error when quant… --- ## [Paligemma2 onnx export KeyError: "Unknown task: image-text-to-text](https://discuss.huggingface.co/t/paligemma2-onnx-export-keyerror-unknown-task-image-text-to-text/139726)
**Author:** [@biswajitdevsarma](https://discuss.huggingface.co/u/biswajitdevsarma)\ **Replies:** 4\ **Last updated:** [February 11, 2025, 4:19pm UTC](https://discuss.huggingface.co/t/paligemma2-onnx-export-keyerror-unknown-task-image-text-to-text/139726 "2025-02-11T16:19:03Z")
Hi , I tried to convert the paligemma2 3B parameter model with 224 image resolution to onnx using optimum and got this error: $optimum-cli export onnx --model google/paligemma-3b-pt-224 paligemma-3b-pt-224\_onnx/ KeyEr… --- ## [Optimum-habana not working!](https://discuss.huggingface.co/t/optimum-habana-not-working/139883)
**Author:** [@DSNG1830](https://discuss.huggingface.co/u/DSNG1830)\ **Replies:** 2\ **Last updated:** [February 10, 2025, 6:10pm UTC](https://discuss.huggingface.co/t/optimum-habana-not-working/139883 "2025-02-10T18:10:42Z")
please help me --- ## [Incorrect Cross Attention Values from Generate Function of ORTModelForVision2Seq](https://discuss.huggingface.co/t/incorrect-cross-attention-values-from-generate-function-of-ortmodelforvision2seq/138375)
**Author:** [@ctufts](https://discuss.huggingface.co/u/ctufts)\ **Replies:** 3\ **Last updated:** [February 1, 2025, 8:33pm UTC](https://discuss.huggingface.co/t/incorrect-cross-attention-values-from-generate-function-of-ortmodelforvision2seq/138375 "2025-02-01T20:33:21Z")
I’ve converted a DONUT model to fp16 to ONNX with optimum utilizing the onnx\_export\_from\_model function: model = VisionEncoderDecoderModel.from\_pretrained(model\_dir) model.load\_state\_dict(state\_dict, assign=True) model… --- ## [Inference on models with custom head](https://discuss.huggingface.co/t/inference-on-models-with-custom-head/137621)
**Author:** [@kaushu42](https://discuss.huggingface.co/u/kaushu42)\ **Replies:** 1\ **Last updated:** [January 28, 2025, 11:18am UTC](https://discuss.huggingface.co/t/inference-on-models-with-custom-head/137621 "2025-01-28T11:18:08Z")
Hi, I trained a BERT model with a custom classifier head. I tried to load is using the ORTModelForSequenceClassification, but it did not load the proper classifier heads. It just loaded the default heads. After several… --- ## [Supported models](https://discuss.huggingface.co/t/supported-models/135072)
**Author:** [@ES-CONVERGE](https://discuss.huggingface.co/u/ES-CONVERGE)\ **Replies:** 6\ **Last updated:** [January 14, 2025, 8:49am UTC](https://discuss.huggingface.co/t/supported-models/135072 "2025-01-14T08:49:28Z")
Trying to compile a transformers model I’m using Getting KeyError: "hyenadna is not supported yet for transformers. Only \['...'\] are supported for the library transformers. If you want to support hyenadna please prop… --- ## [Qwen/Qwen1.5-7B-Chat RuntimeError: The serialized model is larger than the 2GiB ORTModelForCausalLM](https://discuss.huggingface.co/t/qwen-qwen1-5-7b-chat-runtimeerror-the-serialized-model-is-larger-than-the-2gib-ortmodelforcausallm/133817)
**Author:** [@SantoshHF](https://discuss.huggingface.co/u/SantoshHF)\ **Replies:** 2\ **Last updated:** [January 1, 2025, 9:23am UTC](https://discuss.huggingface.co/t/qwen-qwen1-5-7b-chat-runtimeerror-the-serialized-model-is-larger-than-the-2gib-ortmodelforcausallm/133817 "2025-01-01T09:23:37Z")
from optimum.onnxruntime import ORTModelForCausalLM base\_model\_name = “Qwen/Qwen1.5-7B-Chat” ort\_model = ORTModelForCausalLM.from\_pretrained( base\_model\_name, use\_io\_binding=True, export=True, ) RuntimeError: The … --- ## [Error when running examples in optimum habana](https://discuss.huggingface.co/t/error-when-running-examples-in-optimum-habana/74944)
**Author:** [@darkhorse82](https://discuss.huggingface.co/u/darkhorse82)\ **Replies:** 2\ **Last updated:** [October 30, 2024, 5:13pm UTC](https://discuss.huggingface.co/t/error-when-running-examples-in-optimum-habana/74944 "2024-10-30T17:13:37Z")
Hello guys, Running a summarization examples in optimum habana and seeing this errors: (gaudi2-demo-env) test1@smgailab-SYS-820GH-TNR2:~/demo/optimum-habana/examples/translation$ python3 run\_translation.py –model\_n… --- ## [Question about the infernce flow for optimum exported decoder merged onnx model](https://discuss.huggingface.co/t/question-about-the-infernce-flow-for-optimum-exported-decoder-merged-onnx-model/111280)
**Author:** [@Spedon](https://discuss.huggingface.co/u/Spedon)\ **Replies:** 3\ **Last updated:** [October 11, 2024, 9:31am UTC](https://discuss.huggingface.co/t/question-about-the-infernce-flow-for-optimum-exported-decoder-merged-onnx-model/111280 "2024-10-11T09:31:39Z")
Dear Huggingface Community I’ve recently encountered some difficulties with model inference, specifically, I want to use numpy and onnxruntime (without relying on transformers and torch) to infer a VisionDecoderEncoder … --- ## [Compiling SD1.5 for Neuron with resolution other than 512x512 fails](https://discuss.huggingface.co/t/compiling-sd1-5-for-neuron-with-resolution-other-than-512x512-fails/106321)
**Author:** [@hgfckwla](https://discuss.huggingface.co/u/hgfckwla)\ **Replies:** 5\ **Last updated:** [September 26, 2024, 9:24am UTC](https://discuss.huggingface.co/t/compiling-sd1-5-for-neuron-with-resolution-other-than-512x512-fails/106321 "2024-09-26T09:24:18Z")
I’m trying to export SD 1.5 into a portrait mode resolution 512x768 for use with Neuron / Inferentia 2. This is my export command: optimum-cli export neuron \\ --model jyoung105/stable-diffusion-v1-5 \\ --task sta… --- ## [Error while optimizing seq2seq model using optimum](https://discuss.huggingface.co/t/error-while-optimizing-seq2seq-model-using-optimum/101180)
**Author:** [@LearnToGrow](https://discuss.huggingface.co/u/LearnToGrow)\ **Replies:** 1\ **Last updated:** [September 16, 2024, 6:36pm UTC](https://discuss.huggingface.co/t/error-while-optimizing-seq2seq-model-using-optimum/101180 "2024-09-16T18:36:40Z")
I could not reproduce this example transformers==4.42.4 torch== 2.4.0+cpu onnx==1.16.2 onnxruntime==1.18.1 optimum==1.21.2 from transformers import AutoTokenizer from optimum.onnxruntime import OptimizationConfig,… --- ## [Neuron StableDiffusion ControlNet Pipeline fails when used with 2 controlnets](https://discuss.huggingface.co/t/neuron-stablediffusion-controlnet-pipeline-fails-when-used-with-2-controlnets/105189)
**Author:** [@hgfckwla](https://discuss.huggingface.co/u/hgfckwla)\ **Replies:** 3\ **Last updated:** [September 11, 2024, 10:31am UTC](https://discuss.huggingface.co/t/neuron-stablediffusion-controlnet-pipeline-fails-when-used-with-2-controlnets/105189 "2024-09-11T10:31:44Z")
I want to use 2 ControlNets with SD1.5 on AWS Inferentia2 / Neuron. It seems to be supported but at runtime it fails with: Loading only U-Net into both Neuron Cores... Traceback (most recent call last): File "test-con… --- ## [How can I export a transformers model into onnx that not supported with optimum yet](https://discuss.huggingface.co/t/how-can-i-export-a-transformers-model-into-onnx-that-not-supported-with-optimum-yet/104590)
**Author:** [@J4BEZ](https://discuss.huggingface.co/u/J4BEZ)\ **Replies:** 8\ **Last updated:** [August 30, 2024, 11:25am UTC](https://discuss.huggingface.co/t/how-can-i-export-a-transformers-model-into-onnx-that-not-supported-with-optimum-yet/104590 "2024-08-30T11:25:32Z")
What does OP want? I want to export the RT-DETR model to the ONNX model. Which articles OP had read? I have read Export to ONNX (huggingface.co) and Export a model to ONNX with optimum.exporters.onnx (huggingface.co) W… --- ## [Can I convert llama 2 "Chat" model into onnx using llama/convert\_to\_onnx.py script?](https://discuss.huggingface.co/t/can-i-convert-llama-2-chat-model-into-onnx-using-llama-convert-to-onnx-py-script/67960)
**Author:** [@Kainat98](https://discuss.huggingface.co/u/Kainat98)\ **Replies:** 5\ **Last updated:** [August 26, 2024, 8:25am UTC](https://discuss.huggingface.co/t/can-i-convert-llama-2-chat-model-into-onnx-using-llama-convert-to-onnx-py-script/67960 "2024-08-26T08:25:18Z")
I have converted llama-2-13b-chat-hf model into onnx model using this script “(onnxruntime/onnxruntime/python/tools/transformers/models/llama at main · microsoft/onnxruntime · GitHub) /convert\_to\_onnx.py” i just had to… --- ## [Optimum Failed download of jina-embeddings-v2-base-es](https://discuss.huggingface.co/t/optimum-failed-download-of-jina-embeddings-v2-base-es/102393)
**Author:** [@Benitoski](https://discuss.huggingface.co/u/Benitoski)\ **Replies:** 4\ **Last updated:** [August 19, 2024, 1:53pm UTC](https://discuss.huggingface.co/t/optimum-failed-download-of-jina-embeddings-v2-base-es/102393 "2024-08-19T13:53:34Z")
Hi, I’m trying to download jina-embeddings-v2-base-es as onnx with the following command: !optimum-cli export onnx --opset 16 --trust-remote-code -m jinaai/jina-embeddings-v2-base-es jina\_embeddings\_v2\_base\_es And g… --- ## [Optimum/Neuron: RuntimeError: forward() is missing value for argument 'argument\_4'](https://discuss.huggingface.co/t/optimum-neuron-runtimeerror-forward-is-missing-value-for-argument-argument-4/101943)
**Author:** [@hgfckwla](https://discuss.huggingface.co/u/hgfckwla)\ **Replies:** 1\ **Last updated:** [August 13, 2024, 9:18pm UTC](https://discuss.huggingface.co/t/optimum-neuron-runtimeerror-forward-is-missing-value-for-argument-argument-4/101943 "2024-08-13T21:18:15Z")
I’m trying to render an image on AWS Inferentia2 using Optimum NeuronX. I have converted juggernautXL\_v8Rundiffusion model to the Diffusers format using sdxl-to-diffusers-v2 and then to Neuron format with: optimum-cli e… --- ## [What value should the sequence\_length parameter be when converting to TFLite](https://discuss.huggingface.co/t/what-value-should-the-sequence-length-parameter-be-when-converting-to-tflite/101756)
**Author:** [@ADarkDividedGem](https://discuss.huggingface.co/u/ADarkDividedGem)\ **Replies:** 0\ **Last updated:** [August 10, 2024, 10:47am UTC](https://discuss.huggingface.co/t/what-value-should-the-sequence-length-parameter-be-when-converting-to-tflite/101756 "2024-08-10T10:47:27Z")
I am converting a model hosted by HuggingFace to a TensorFlow Lite model by using HuggingFace’s Optimum exporter. In their documentation they use the following example: optimum-cli export tflite --model google-bert/ber… --- ## [How to export a fine-tuned SDXL model?](https://discuss.huggingface.co/t/how-to-export-a-fine-tuned-sdxl-model/101424)
**Author:** [@hgfckwla](https://discuss.huggingface.co/u/hgfckwla)\ **Replies:** 4\ **Last updated:** [August 9, 2024, 5:20am UTC](https://discuss.huggingface.co/t/how-to-export-a-fine-tuned-sdxl-model/101424 "2024-08-09T05:20:05Z")
I’m trying to use AWS Inferentia 2 (NeuronX) for a more efficient SD-XL image generation. The base model works fine, I can export it to Neuron with this and then use: optimum-cli export neuron \\ --model stabilityai/st… --- ## [Make Text Embedding Server compatible](https://discuss.huggingface.co/t/make-text-embedding-server-compatible/100937)
**Author:** [@Saugatkafley](https://discuss.huggingface.co/u/Saugatkafley)\ **Replies:** 2\ **Last updated:** [August 8, 2024, 5:56am UTC](https://discuss.huggingface.co/t/make-text-embedding-server-compatible/100937 "2024-08-08T05:56:37Z")
I have cloned embedding model udever-bloom-1b1. At first it asked for onnx files as it didnt have so i convert them to onnx. Then pushed to my repo here I also configured config.json with {"max\_position\_embeddings": 2… --- ## [Exporting SegFormer Image Processor to ONNX Format Using "optimum.exporters.onnx.onnx\_export\_from\_model"](https://discuss.huggingface.co/t/exporting-segformer-image-processor-to-onnx-format-using-optimum-exporters-onnx-onnx-export-from-model/100038)
**Author:** [@soypablo](https://discuss.huggingface.co/u/soypablo)\ **Replies:** 0\ **Last updated:** [July 30, 2024, 6:37am UTC](https://discuss.huggingface.co/t/exporting-segformer-image-processor-to-onnx-format-using-optimum-exporters-onnx-onnx-export-from-model/100038 "2024-07-30T06:37:54Z")
I want to convert SegFormer(include imageprocessor) to ONNX format. import requests from PIL import Image from optimum.onnxruntime import ORTModelForSemanticSegmentation from optimum.exporters.onnx import onnx\_export\_fr… [Next page](https://discuss.huggingface.co/c/optimum/59.md?page=1)