README.md

---
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen2.5-VL-3B-Instruct
tags:
  - vision-language
  - multimodal
  - visual-question-answering
  - visual-reasoning
  - visual-grounding
  - on-policy-self-distillation
  - self-distillation
  - qwen2.5-vl
  - lora
  - vigos
license: other
license_name: qwen
license_link: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE
model-index:
  - name: ViGOS-3B
    results:
      - task:
          type: image-text-to-text
          name: Multimodal Reasoning
        dataset:
          name: Eight Main Benchmarks Average
          type: aggregated
        metrics:
          - type: pass@5
            value: 71.97
            name: Mean Pass@5
          - type: avg@5
            value: 41.35
            name: Mean Avg@5
      - task:
          type: visual-question-answering
          name: Prior-Sensitive Visual Question Answering
        dataset:
          name: ViLP-F
          type: ViLP-F
        metrics:
          - type: accuracy
            value: 70.17
            name: Score
          - type: accuracy
            value: 97.67
            name: Prior
      - task:
          type: visual-question-answering
          name: Prior-Sensitive Visual Question Answering
        dataset:
          name: ViLP-P
          type: ViLP-P
        metrics:
          - type: accuracy
            value: 69.50
            name: Score
          - type: accuracy
            value: 90.00
            name: Prior
---
# ViGOS-3B: Visual Grounding On-Policy Self-Distillation

## Model Details

| Field | Value |
|---|---|
| Model name | `ViGOS-3B` |
| Repository ID | `OedoSoldier/ViGOS-3B` |
| Model family | ViGOS |
| Model type | Multimodal image-text-to-text / vision-language reasoning model |
| Base model | `Qwen/Qwen2.5-VL-3B-Instruct` |
| Training method | Segment-wise multimodal on-policy self-distillation |
| Weight format | Merged full weights |
| Training data | [LMMs-Lab-Turtle/Vision-SR1-47K](https://huggingface.co/datasets/LMMs-Lab-Turtle/Vision-SR1-47K) |
| Output format | `<description>...</description><think>...</think>\boxed{...}` |
| Paper | *Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation* |
| Authors | Sihan Wang, Xiyao Liu, Lianqing Liu, Zhi Han |
| Code | https://github.com/OedoSoldier/ViGOS |
| License | Qwen research license |

This repository is for the 3B-scale ViGOS model only. The 7B-scale model should use a separate Hugging Face repository and model card.

## Model Summary

**ViGOS** stands for **Visual Grounding On-Policy Self-Distillation**. It is a multimodal post-training method for reducing shortcut behavior in on-policy self-distillation for vision-language models. In vanilla OPSD, the privileged teacher can see the reference answer while supervising the whole student rollout. For MLLMs, that can make the dense training signal overly answer-driven before the model has grounded its response in image evidence.

ViGOS changes the supervision path by asking the student to first produce a visual description, then reason, then answer:

```text
<description> visual description </description>
<think> reasoning process </think>
\boxed{FINAL ANSWER}
```

For valid training rollouts, ViGOS uses segment-wise teachers:

- an **image-only perception teacher** supervises the description tokens;
- a **privileged reasoning teacher** supervises reasoning and final-answer tokens after the student-generated description prefix exists;
- a **reference fallback teacher** is used only for invalid or malformed rollouts to recover the required output format.

At inference time, all teachers, reference answers, and segment masks are removed. The model receives only the image, the question or instruction, and the output-format prompt.

## Intended Use

This model is intended for research and development in multimodal reasoning tasks, including visual question answering, visual math and diagram reasoning, OCR- or chart-grounded reasoning, spatial reasoning, visual grounding, and shortcut/prior-sensitivity analysis.

## Out-of-Scope Use

This model should not be used as the sole decision-maker in high-stakes settings such as medical diagnosis, legal judgment, financial decision-making, safety-critical robotics, surveillance, identity verification, or other contexts where hallucinated or incorrect visual reasoning could cause harm.

## How to Use

```bash
pip install git+https://github.com/huggingface/transformers accelerate
pip install qwen-vl-utils[decord]
```

```python
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

MODEL_ID = "OedoSoldier/ViGOS-3B"

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)

image_path = "path/to/image.jpg"
question = "What is the answer to the visual question?"

prompt = f"""Problem: {question}

You are tasked with analyzing an image to generate a detailed description that can help you answer the question. First analyze the image and produce a self-contained description, detailed enough to lead to the correct answer. Do not include the final answer in the description. Wrap the entire description in <description> </description> tags.

Next, reason step by step based on the image description and the image, and enclose this part within <think> </think> tags.

Finally, provide a single word or phrase answer to the question in \\boxed{{}}.
The output format should be: <description> image description here </description><think> reasoning process here </think> \\boxed{{FINAL ANSWER here}}.
"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=4096,
        do_sample=True,
        temperature=1.0,
        top_p=0.90,
        top_k=20,
    )

generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]

print(output_text)
```

### Recommended Answer Extraction

For benchmark-style evaluation, the paper extracts the final answer from the last `\boxed{...}` span. Outputs without a parseable final answer are counted as incorrect.

## Training Details

The paper trains this model for one epoch on **Vision-SR1-47K** using 8 NVIDIA A100 GPUs. The student is trained on on-policy rollouts, and the frozen teacher roles are used only to score the student-generated prefixes during training.

| Parameter | Value |
|---|---:|
| Training epochs | 1 |
| GPUs | 8 × A100 |
| Effective batch size | 32 |
| Optimizer | Fused AdamW |
| Learning rate | 5e-6 |
| LR scheduler | Linear |
| Maximum gradient norm | 0.1 |
| Precision | bf16 |
| Distributed training | ZeRO-2 |
| Maximum prompt length | 32,768 |
| Maximum completion length | 4,096 |
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0.05 |
| LoRA target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Rollout temperature | 1.1 |
| Rollout top-p / top-k | 0.95 / 20 |
| λ_perc | 1.0 |
| λ_rea | 1.0 |
| λ_ref | 2.0 |
| Distillation temperature | 1.0 |
| KL clipping | 0.05 |

## Evaluation Protocol

For the eight main benchmarks, the paper samples five stochastic responses per example and reports **Pass@5 / Avg@5**. Pass@5 checks whether at least one of the five sampled answers is correct, while Avg@5 is the mean correctness across all five samples.

For ViLP, the paper generates one response per prompt and reports **Score / Prior**. Score measures accuracy on visually diagnostic questions where the model must use the image, and Prior measures accuracy on prior-aligned questions where the common visual-language prior is correct.

Evaluation decoding settings:

| Parameter | Value |
|---|---:|
| Maximum generated tokens | 4,096 |
| Number of samples per main benchmark question | 5 |
| Temperature | 1.0 |
| Top-p | 0.90 |
| Top-k | 20 |
| Random seed | 42 |

## Evaluation Results

### Main Benchmarks

Pass@5 / Avg@5, in percent:

| Benchmark | ViGOS-3B |
|---|---:|
| MM-Vet | 65.60 / 43.76 |
| MMMU | 76.42 / 42.32 |
| MMMU-Pro | 56.44 / 26.16 |
| MathVerse | 58.55 / 30.10 |
| MathVista | 74.00 / 43.50 |
| MMSI | 66.40 / 24.90 |
| RealWorldQA | 86.80 / 55.37 |
| CV-Bench | 91.51 / 64.67 |
| **Mean across 8 benchmarks** | **71.97 / 41.35** |

### Prior-Sensitive ViLP Results

Score / Prior, in percent:

| Setting | ViGOS-3B |
|---|---:|
| ViLP-F | 70.17 / 97.67 |
| ViLP-P | 69.50 / 90.00 |

## Ethical Considerations

Users should validate the model carefully before deployment. The model can generate plausible but incorrect visual descriptions and rationales. In user-facing applications, consider presenting only concise final answers, or clearly mark generated descriptions and rationales as model-generated rather than authoritative evidence.

## Citation

Please cite the ViGOS paper if you use this model or method.

```bibtex
@misc{wang2026seeing,
  title={Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation},
  author={Wang, Sihan and Liu, Xiyao and Liu, Lianqing and Han, Zhi},
  year={2026},
  eprint={2606.19120},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2606.19120}
}
```