Image-to-Text
Transformers
PyTorch
vision-encoder-decoder
image-text-to-text
donut
vision
endpoints-template
Instructions to use philschmid/donut-base-finetuned-cord-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use philschmid/donut-base-finetuned-cord-v2 with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="philschmid/donut-base-finetuned-cord-v2")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMultimodalLM tokenizer = AutoTokenizer.from_pretrained("philschmid/donut-base-finetuned-cord-v2") model = AutoModelForMultimodalLM.from_pretrained("philschmid/donut-base-finetuned-cord-v2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download handler.py from philschmid/donut-base-finetuned-cord-v2: direct link, hf CLI and curl.
- Browser
- Download file 1.61 kB
-
https://huggingface.co/philschmid/donut-base-finetuned-cord-v2/resolve/main/handler.py
- Command line
-
hf download hf://philschmid/donut-base-finetuned-cord-v2/handler.py
-
curl -L -o handler.py https://huggingface.co/philschmid/donut-base-finetuned-cord-v2/resolve/main/handler.py
1.61 kB
| from typing import Dict, List, Any | |
| from transformers import DonutProcessor, VisionEncoderDecoderModel | |
| import torch | |
| # check for GPU | |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") | |
| class EndpointHandler: | |
| def __init__(self, path=""): | |
| # load the model | |
| self.processor = DonutProcessor.from_pretrained(path) | |
| self.model = VisionEncoderDecoderModel.from_pretrained(path) | |
| # move model to device | |
| self.model.to(device) | |
| self.decoder_input_ids = self.processor.tokenizer( | |
| "<s_cord-v2>", add_special_tokens=False, return_tensors="pt" | |
| ).input_ids | |
| def __call__(self, data: Any) -> List[List[Dict[str, float]]]: | |
| inputs = data.pop("inputs", data) | |
| # preprocess the input | |
| pixel_values = self.processor(inputs, return_tensors="pt").pixel_values | |
| # forward pass | |
| outputs = self.model.generate( | |
| pixel_values.to(device), | |
| decoder_input_ids=self.decoder_input_ids.to(device), | |
| max_length=self.model.decoder.config.max_position_embeddings, | |
| early_stopping=True, | |
| pad_token_id=self.processor.tokenizer.pad_token_id, | |
| eos_token_id=self.processor.tokenizer.eos_token_id, | |
| use_cache=True, | |
| num_beams=1, | |
| bad_words_ids=[[self.processor.tokenizer.unk_token_id]], | |
| return_dict_in_generate=True, | |
| ) | |
| # process output | |
| prediction = self.processor.batch_decode(outputs.sequences)[0] | |
| prediction = self.processor.token2json(prediction) | |
| return prediction | |