Update README.md

e7f8e8e verified 6 months ago

10.1 kB

---
license: apache-2.0
---
# S1-VL-32B: Scientific Multimodal Reasoning Model

[中文版](./README_zh.md) | [English](./README.md)

## 🔬 Introduction

**S1-VL-32B** is a multimodal large language model for scientific domains, developed by the ScienceOne AI team at the Chinese Academy of Sciences. It natively supports two reasoning paradigms — **Scientific Reasoning** and **Thinking with Images** — and achieves state-of-the-art performance across multiple mainstream scientific multimodal evaluation benchmarks.

- **Scientific Reasoning**: Chain-of-thought-based multimodal scientific reasoning, designed for the analysis and solving of complex, multi-step problems.
- **Thinking with Images**: Enables the model to actively invoke code tools during the reasoning process to perform image operations — including cropping, zooming, image enhancement, bounding box annotation, and keypoint marking — before generating responses.

We have established a **cross-disciplinary data processing pipeline** that conducts multi-dimensional utility evaluation and filtering of visual reasoning trajectories to ensure the quality of training reasoning trajectories.

<div align="center">
<img src="./image/data_pipeline.png"/>
</div>

We adopt a **four-stage progressive post-training procedure** to progressively unlock the scientific reasoning capabilities of S1-VL-32B:

- **Stage 1 - Scientific Reasoning SFT**: Large-scale multimodal instruction data spanning multiple disciplines — including **mathematics, physics, chemistry, astronomy, earth sciences, and biology** — is used for mixed training to enhance the model's scientific visual understanding and logical reasoning abilities, laying a solid foundation for academic figure Q&A, medical image analysis, chemical structure recognition, and related tasks.
- **Stage 2 - Thinking-with-Images Cold-Start SFT**: The **Thinking with Images** reasoning paradigm is introduced. Through joint training with high-quality **scientific reasoning curriculum learning data** and image-thinking data, the model acquires the ability to perform **image operations via code** during inference. This approach yields particularly outstanding performance in interpreting dense scientific charts, high-resolution remote sensing imagery, microscopic images, and astronomical observation data (S1-VL-32B-SFT).
- **Stage 3 - Scientific Reasoning RL**: Based on the **SAPO algorithm** and a multi-task scientific reward function, reinforcement learning is applied to challenging scientific multimodal reasoning samples to push beyond the performance ceiling of the SFT stage.
- **Stage 4 - Thinking-with-Images RL**: Based on the **SAPO algorithm** and a four-dimensional composite reward function, the model's image operation invocation timing and quality are further optimized, enabling stable and efficient multi-round visual reasoning (S1-VL-32B-RL).

    <div align="center">
    <img src="./image/s1-vl-training-pipeline.png"/>
    </div>
🔥 **[NEW]** Technical report released: [S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images](https://arxiv.org/abs/2604.21409)    
🔥 **[NEW]** Stage 3 and Stage 4 reinforcement learning training added; [S1-VL-32B-RL](https://huggingface.co/ScienceOne-AI/S1-VL-32B-RL) model weights updated.


## 📂 Model Weights

| Model | Parameters | HuggingFace | ModelScope |
|-------|-----------|-------------|------------|
| S1-VL-32B-SFT | 32B | 🤗 [Download](https://huggingface.co/ScienceOne-AI/S1-VL-32B) | 🤖 [Download](https://modelscope.cn/models/ScienceOne-AI/S1-VL-32B) |
| S1-VL-32B-RL | 32B | 🤗 [Download](https://huggingface.co/ScienceOne-AI/S1-VL-32B-RL) | 🤖 [Download](https://modelscope.cn/models/ScienceOne-AI/S1-VL-32B-RL) |


## 🏆 Evaluation Results

The evaluation covers **2 dimensions** and **13 benchmarks**. The **Scientific Multimodal Reasoning** dimension includes MMMU, SFE, MathVision, Physics, ScienceOlympiad, VRSBench-MINI, GMAI-MMBench, and Galaxy-10-DECaLS, spanning mathematics, physics, medicine, remote sensing, astronomy, and other professional fields. The **Image Manipulation Reasoning** dimension includes HRBench-4K, HRBench-8K, MME-RealWorld-CN, MME-RealWorld-Lite, and V*, focusing on high-resolution image understanding and real-world visual reasoning.
<div align="center">
<img src="./image/s1-vl-32b-benchmark.png"/>
</div>
S1-VL-32B demonstrates outstanding overall competitiveness across the aforementioned evaluations. In **scientific multimodal reasoning** tasks, the model achieves significant advantages on multiple authoritative benchmarks — including MMMU, MathVision, and VRSBench-MINI — surpassing its base model Qwen3-VL-32B in overall performance, while remaining highly competitive against open-source models with substantially larger parameter scales (e.g., Qwen3-VL-235B, Intern-S1) as well as closed-source flagship models (e.g., Gemini 2.5 Pro, GPT-5). In **image operation reasoning** tasks, S1-VL-32B ranks **first across all five benchmark evaluations**, comprehensively outperforming models of comparable and larger scales, while also surpassing dedicated "Thinking with Images" models such as Thyme-VL and Skywork-R1V4. These results fully validate its ability to achieve efficient, high-quality multimodal reasoning at the 32B parameter scale.
## 🧠 Case Study
The following presents reasoning examples of S1-VL-32B operating in **Thinking with Images** mode. When processing a low-resolution cervical CT image, S1-VL-32B proactively invokes code tools during its reasoning process to perform **cropping and magnification** on the region of interest. By obtaining a clearer local image, the model then combines the enhanced visual information with its internal knowledge to complete the reasoning.
<div align="center">
<img src="./image/s1-vl-32b-twi.png"/>
</div>
📁 More cases are available in [CASES.md](./CASES.md).
## 🚀 Quick Start
### 1. Install Dependencies
```bash
# Requires vLLM >= 0.11.0
pip install -U vllm
pip install qwen-vl-utils==0.0.14
```
### 2. Start the vLLM Service
```bash
vllm serve ScienceOne-AI/S1-VL-32B-RL \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --limit-mm-per-prompt image=15 \
    --reasoning-parser deepseek_r1 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.95 \
    --port 9200
```
### 3. Multimodal Reasoning Mode
```python
from openai import OpenAI
import base64
client = OpenAI(api_key="EMPTY", base_url="http://localhost:9200/v1")
with open("path/to/your/image.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")
response = client.chat.completions.create(
    model="ScienceOne-AI/S1-VL-32B-RL",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}},
                {"type": "text", "text": "Please describe the physical phenomenon shown in the image and derive the relevant equations."},
            ],
        }
    ],
    temperature=0.2,
    max_tokens=16384,
)
# The reasoning process is in the reasoning_content field
print("Thinking process:\n", response.choices[0].message.reasoning_content)
print("\nFinal answer:\n", response.choices[0].message.content)
```
### 4. Thinking with Images Mode
Thinking with Images mode requires deploying a **code sandbox** to support the model invoking code tools during reasoning for image operations (cropping, zooming, enhancement, annotation, etc.).
#### Step 1: Deploy the Code Sandbox
We recommend deploying the AIO Sandbox with Docker:
```bash
git clone https://github.com/agent-infra/sandbox
cd sandbox
# Mount the host image directory into the container
docker run -d \
    --name twi-sandbox \
    -p 18081:18081 \
    -v /data/images:/mnt/data/images \   # host path → sandbox path
    sandbox:latest
```
The mount path must match the path configuration in the FastAPI service.
#### Step 2: Start the Thinking with Images FastAPI Service
Download [twi_server.py](twi_server.py) and update the path configuration at the top of the file:
```python
CHAT_API        = "http://localhost:9200/v1/chat/completions"  # vLLM address
JUPYTER_API     = "http://localhost:18081/v1/jupyter"          # Sandbox address
HOST_IMG_DIR    = "/data/images"     # ← Host image directory (must match docker -v mount)
```
Start the service:
```bash
pip install fastapi uvicorn httpx pillow
python twi_server.py   # Listens on port 10044
```
#### Step 3: Call the Thinking with Images Endpoint
```python
import httpx
import base64
with open("path/to/your/image.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")
messages = [
    {"type": "text", "text": "Please carefully analyze this scientific image."},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
]
response = httpx.post(
    "http://localhost:10044/process",
    json={
        "messages": messages,
        "image_path_list": ["/data/images/your_image.png"],  # Absolute host path
    },
    timeout=300,
)
result = response.json()
# The final answer is the last message with role="assistant"
final = [m for m in result["messages"] if m["role"] == "assistant"][-1]
print(final["content"])
```
## 📄 Citation
If you use S1-VL-32B in your research, please cite:
```latex
@article{li2026s1vl,
  title     = {S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images},
  author    = {Li, Qingxiao and Xu, Lifeng and Wang, QingLi and Bai, Yudong and Ou, Mingwei and Hu, Shu and Xu, Nan},
  journal   = {arXiv preprint arXiv:2604.21409},
  year      = {2026},
}
```
## 📜 License
This project is released under the Apache 2.0 License.
## 🙏 Acknowledgements
We thank the open-source communities and pioneering works of [Qwen3-VL](https://modelscope.cn/collections/Qwen3-VL-5c7a94c8cb144b) and [AIO Sandbox](https://github.com/agent-infra/sandbox) for laying the foundation for the scientific multimodal reasoning research behind S1-VL-32B.

	---
	license: apache-2.0
	---
	# S1-VL-32B: Scientific Multimodal Reasoning Model

	[中文版](./README_zh.md) \| [English](./README.md)

	## 🔬 Introduction

	S1-VL-32B is a multimodal large language model for scientific domains, developed by the ScienceOne AI team at the Chinese Academy of Sciences. It natively supports two reasoning paradigms — Scientific Reasoning and Thinking with Images — and achieves state-of-the-art performance across multiple mainstream scientific multimodal evaluation benchmarks.

	- Scientific Reasoning: Chain-of-thought-based multimodal scientific reasoning, designed for the analysis and solving of complex, multi-step problems.
	- Thinking with Images: Enables the model to actively invoke code tools during the reasoning process to perform image operations — including cropping, zooming, image enhancement, bounding box annotation, and keypoint marking — before generating responses.

	We have established a cross-disciplinary data processing pipeline that conducts multi-dimensional utility evaluation and filtering of visual reasoning trajectories to ensure the quality of training reasoning trajectories.

	<div align="center">
	<img src="./image/data_pipeline.png"/>
	</div>

	We adopt a four-stage progressive post-training procedure to progressively unlock the scientific reasoning capabilities of S1-VL-32B:

	- Stage 1 - Scientific Reasoning SFT: Large-scale multimodal instruction data spanning multiple disciplines — including mathematics, physics, chemistry, astronomy, earth sciences, and biology — is used for mixed training to enhance the model's scientific visual understanding and logical reasoning abilities, laying a solid foundation for academic figure Q&A, medical image analysis, chemical structure recognition, and related tasks.
	- Stage 2 - Thinking-with-Images Cold-Start SFT: The Thinking with Images reasoning paradigm is introduced. Through joint training with high-quality scientific reasoning curriculum learning data and image-thinking data, the model acquires the ability to perform image operations via code during inference. This approach yields particularly outstanding performance in interpreting dense scientific charts, high-resolution remote sensing imagery, microscopic images, and astronomical observation data (S1-VL-32B-SFT).
	- Stage 3 - Scientific Reasoning RL: Based on the SAPO algorithm and a multi-task scientific reward function, reinforcement learning is applied to challenging scientific multimodal reasoning samples to push beyond the performance ceiling of the SFT stage.
	- Stage 4 - Thinking-with-Images RL: Based on the SAPO algorithm and a four-dimensional composite reward function, the model's image operation invocation timing and quality are further optimized, enabling stable and efficient multi-round visual reasoning (S1-VL-32B-RL).

	<div align="center">
	<img src="./image/s1-vl-training-pipeline.png"/>
	</div>

	🔥 [NEW] Technical report released: [S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images](https://arxiv.org/abs/2604.21409)
	🔥 [NEW] Stage 3 and Stage 4 reinforcement learning training added; [S1-VL-32B-RL](https://huggingface.co/ScienceOne-AI/S1-VL-32B-RL) model weights updated.


	## 📂 Model Weights

	\| Model \| Parameters \| HuggingFace \| ModelScope \|
	\|-------\|-----------\|-------------\|------------\|
	\| S1-VL-32B-SFT \| 32B \| 🤗 [Download](https://huggingface.co/ScienceOne-AI/S1-VL-32B) \| 🤖 [Download](https://modelscope.cn/models/ScienceOne-AI/S1-VL-32B) \|
	\| S1-VL-32B-RL \| 32B \| 🤗 [Download](https://huggingface.co/ScienceOne-AI/S1-VL-32B-RL) \| 🤖 [Download](https://modelscope.cn/models/ScienceOne-AI/S1-VL-32B-RL) \|


	## 🏆 Evaluation Results

	The evaluation covers 2 dimensions and 13 benchmarks. The Scientific Multimodal Reasoning dimension includes MMMU, SFE, MathVision, Physics, ScienceOlympiad, VRSBench-MINI, GMAI-MMBench, and Galaxy-10-DECaLS, spanning mathematics, physics, medicine, remote sensing, astronomy, and other professional fields. The Image Manipulation Reasoning dimension includes HRBench-4K, HRBench-8K, MME-RealWorld-CN, MME-RealWorld-Lite, and V*, focusing on high-resolution image understanding and real-world visual reasoning.

	<div align="center">
	<img src="./image/s1-vl-32b-benchmark.png"/>
	</div>

	S1-VL-32B demonstrates outstanding overall competitiveness across the aforementioned evaluations. In scientific multimodal reasoning tasks, the model achieves significant advantages on multiple authoritative benchmarks — including MMMU, MathVision, and VRSBench-MINI — surpassing its base model Qwen3-VL-32B in overall performance, while remaining highly competitive against open-source models with substantially larger parameter scales (e.g., Qwen3-VL-235B, Intern-S1) as well as closed-source flagship models (e.g., Gemini 2.5 Pro, GPT-5). In image operation reasoning tasks, S1-VL-32B ranks first across all five benchmark evaluations, comprehensively outperforming models of comparable and larger scales, while also surpassing dedicated "Thinking with Images" models such as Thyme-VL and Skywork-R1V4. These results fully validate its ability to achieve efficient, high-quality multimodal reasoning at the 32B parameter scale.

	## 🧠 Case Study

	The following presents reasoning examples of S1-VL-32B operating in Thinking with Images mode. When processing a low-resolution cervical CT image, S1-VL-32B proactively invokes code tools during its reasoning process to perform cropping and magnification on the region of interest. By obtaining a clearer local image, the model then combines the enhanced visual information with its internal knowledge to complete the reasoning.

	<div align="center">
	<img src="./image/s1-vl-32b-twi.png"/>
	</div>

	📁 More cases are available in [CASES.md](./CASES.md).

	## 🚀 Quick Start

	### 1. Install Dependencies

	```bash
	# Requires vLLM >= 0.11.0
	pip install -U vllm
	pip install qwen-vl-utils==0.0.14
	```

	### 2. Start the vLLM Service

	```bash
	vllm serve ScienceOne-AI/S1-VL-32B-RL \
	--tensor-parallel-size 4 \
	--max-model-len 32768 \
	--limit-mm-per-prompt image=15 \
	--reasoning-parser deepseek_r1 \
	--enable-prefix-caching \
	--gpu-memory-utilization 0.95 \
	--port 9200
	```

	### 3. Multimodal Reasoning Mode

	```python
	from openai import OpenAI
	import base64

	client = OpenAI(api_key="EMPTY", base_url="http://localhost:9200/v1")

	with open("path/to/your/image.png", "rb") as f:
	image_data = base64.b64encode(f.read()).decode("utf-8")

	response = client.chat.completions.create(
	model="ScienceOne-AI/S1-VL-32B-RL",
	messages=[
	{
	"role": "user",
	"content": [
	{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}},
	{"type": "text", "text": "Please describe the physical phenomenon shown in the image and derive the relevant equations."},
	],
	}
	],
	temperature=0.2,
	max_tokens=16384,
	)

	# The reasoning process is in the reasoning_content field
	print("Thinking process:\n", response.choices[0].message.reasoning_content)
	print("\nFinal answer:\n", response.choices[0].message.content)
	```

	### 4. Thinking with Images Mode

	Thinking with Images mode requires deploying a code sandbox to support the model invoking code tools during reasoning for image operations (cropping, zooming, enhancement, annotation, etc.).

	#### Step 1: Deploy the Code Sandbox

	We recommend deploying the AIO Sandbox with Docker:

	```bash
	git clone https://github.com/agent-infra/sandbox
	cd sandbox
	# Mount the host image directory into the container
	docker run -d \
	--name twi-sandbox \
	-p 18081:18081 \
	-v /data/images:/mnt/data/images \ # host path → sandbox path
	sandbox:latest
	```
	The mount path must match the path configuration in the FastAPI service.

	#### Step 2: Start the Thinking with Images FastAPI Service

	Download [twi_server.py](twi_server.py) and update the path configuration at the top of the file:

	```python
	CHAT_API = "http://localhost:9200/v1/chat/completions" # vLLM address
	JUPYTER_API = "http://localhost:18081/v1/jupyter" # Sandbox address
	HOST_IMG_DIR = "/data/images" # ← Host image directory (must match docker -v mount)
	```

	Start the service:

	```bash
	pip install fastapi uvicorn httpx pillow
	python twi_server.py # Listens on port 10044
	```

	#### Step 3: Call the Thinking with Images Endpoint

	```python
	import httpx
	import base64

	with open("path/to/your/image.png", "rb") as f:
	image_b64 = base64.b64encode(f.read()).decode("utf-8")

	messages = [
	{"type": "text", "text": "Please carefully analyze this scientific image."},
	{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
	]

	response = httpx.post(
	"http://localhost:10044/process",
	json={
	"messages": messages,
	"image_path_list": ["/data/images/your_image.png"], # Absolute host path
	},
	timeout=300,
	)

	result = response.json()

	# The final answer is the last message with role="assistant"
	final = [m for m in result["messages"] if m["role"] == "assistant"][-1]
	print(final["content"])
	```

	## 📄 Citation

	If you use S1-VL-32B in your research, please cite:

	```latex
	@article{li2026s1vl,
	title = {S1-VL: Scientific Multimodal Reasoning Model with Thinking-with-Images},
	author = {Li, Qingxiao and Xu, Lifeng and Wang, QingLi and Bai, Yudong and Ou, Mingwei and Hu, Shu and Xu, Nan},
	journal = {arXiv preprint arXiv:2604.21409},
	year = {2026},
	}
	```

	## 📜 License

	This project is released under the Apache 2.0 License.

	## 🙏 Acknowledgements

	We thank the open-source communities and pioneering works of [Qwen3-VL](https://modelscope.cn/collections/Qwen3-VL-5c7a94c8cb144b) and [AIO Sandbox](https://github.com/agent-infra/sandbox) for laying the foundation for the scientific multimodal reasoning research behind S1-VL-32B.