Xiaomi-Lens-0

Xiaomi-Lens-0 is a 4B multimodal language model built on Qwen3.5-4B. It adds a lightweight pixel-mask decoder and is trained with joint language and mask supervision. Each generated <|mask|> token identifies an image region that the decoder turns into a segmentation mask.

The model supports referring expression comprehension and segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained visual perception. It can produce masks in its final answer or during reasoning.

This repository contains the model weights, configuration, tokenizer, processor, and chat template. Inference uses the XiaomiLensQwen3_5ForConditionalGeneration class from the accompanying Xiaomi-Lens-0 code repository. Install that package to load the model and decode masks.

📰 News

• The Xiaomi-Lens-0 online demo is now live on Hugging Face Spaces.
• The Xiaomi-Lens-0 Technical Report is now publicly available on arXiv.

🎬 Overview video

Watch on the project page · Download the MP4

🖼️ Task showcases

Recorded model outputs across six task domains. Each task includes four examples from the GitHub project.

Referring Expression Comprehension

Dark brown bear: “dark brown bear at left”	Boy with a phone: “boy looking at phone”
Smallest spoon: “smallest wooden spoon of the group”	Runner behind the pitcher: “the player shown running behind the pitcher”

Referring Expression Segmentation

Grapefruit wedge: “second wedge grapefruit from left to right”	Laptop with a sign: “laptop with small white windows sign”
Kite on the grass: “a red, black, and white kite lying on the grass”	Brown cow: “a large brown cow standing in the grass”

Multi-Instance Grounding

Bottles: “all bottles”	Oranges: “all oranges”
People: “all people”	Surfboards: “all surfboards”

Reasoning Segmentation

Tangy condiment: “What can be squeezed onto seafood as a tangy flavor enhancer?”	Gift packaging: “What could be used to package a gift?”
Lawn care: “What would be useful for keeping a lawn well-groomed?”	Orbiting object: “What object orbits the Earth to complete missions?”

Embodied Referring

Old book: “Hand me the old book with a black and white cover”	Facial cleanser: “Hand me the facial cleanser on the left”
White cup: “Pass me the white cup on the left”	Pepsi can: “Hand me the can of Pepsi on the right”

Fine-Grained Perception

Climbing stones: “How many climbing stones are there?” → 5	Leaves: “How many leaves are there?” → 6
Chair and bicycles: “Where is the chair relative to the bicycles?” → To the left	Trailer color: “What's the color of the trailer?” → Orange

Quick start

Inference requires Python 3.10 or later and a CUDA GPU with bfloat16 and FlashAttention-2 support. From a checkout of the Xiaomi-Lens-0 inference repository, install the dependencies and package:

pip install -r requirements.txt
pip install -e .

Run the inference script with the downloaded checkpoint directory and an image:

python infer.py \
  --checkpoint /path/to/Xiaomi-Lens-0 \
  --image /path/to/image.jpg \
  --text 'Locate all instances in the image that match the description "the cat on the left". Report bbox coordinates and masks.' \
  --output result.jpg --draw-bbox

The script prints the generated text and saves an image with the predicted masks and bounding boxes. --text accepts a complete instruction; the checkpoint's chat template supplies the image tokens and assistant prefix.

Each grounded instance is represented as <bbox>[x1, y1, x2, y2]</bbox><mask><|mask|></mask>, with box coordinates normalized to 0–1000. Mask decoding uses the contextual state of each <|mask|> token and the vision encoder's features. The inference script excludes masks inside <think> from the final-answer overlay.

Serving with vLLM

Use a separate environment for vLLM serving. From the inference repository, install the serving dependencies:

pip install -e ".[vllm]"

Run text generation and mask decoding in separate terminals:

# Terminal 1: text generation on port 8000
bash src/xiaomi_lens/integrations/vllm/run_vllm.sh /path/to/Xiaomi-Lens-0

# Terminal 2: mask decoding on port 8001
bash src/xiaomi_lens/integrations/vllm/run_vllm_mask.sh /path/to/Xiaomi-Lens-0

# Terminal 3: query both services
python -m xiaomi_lens.integrations.vllm.client \
  --image /path/to/image.jpg --query "the cat on the left" --output result.png

The client must run on the same machine as the services because it sends the image path to the mask server. The launchers prepare temporary checkpoint views for vLLM. If you override MAX_PIXELS, use the same value for both services.

Training and evaluation

Training first pretrains the mask decoder, then fine-tunes the full model with language and pixel-level supervision. The inference repository's README provides the benchmark tables and recorded examples across the six capability domains.

LensBench

Evaluate instruction-conditioned segmentation across general, aerial, driving, and panoramic imagery with LensBench.

The original baseline results are reported in Table 7 in Section 4.3 of the Xiaomi-Lens-0 Technical Report (arXiv v1). The table below also includes the subsequently added GPT-6-Astra results and is synchronized with the project results.

Marginal results pool single and multiple targets. Each cell reports cIoU / Prec@0.5 (%, higher is better). Bold marks the best score, and underlining marks the second-best score for each metric and column, including ties.

Method	Explicit	Implicit	General	Aerial	Driving	Panoramic
GPT-6-Astra	72.6 / 86.3	71.0 / 84.0	74.1 / 86.2	61.7 / 76.9	77.6 / 89.0	68.9 / 88.3
GPT-5.6-Sol	63.8 / 73.2	61.2 / 70.0	68.4 / 67.9	42.2 / 58.5	68.4 / 81.8	57.1 / 78.3
Gemini-3.6-Flash	61.4 / 74.5	61.0 / 70.3	63.6 / 65.8	50.1 / 62.5	66.8 / 81.0	60.8 / 80.3
MiMo-V2.5	51.4 / 52.2	47.4 / 46.8	54.2 / 45.2	32.6 / 38.6	50.5 / 52.8	45.4 / 61.4
Seed-2.1-Pro	62.1 / 70.2	55.9 / 64.5	60.6 / 56.7	50.1 / 58.5	66.8 / 77.4	59.9 / 76.7
Qwen3.8-Max	67.6 / 69.2	64.5 / 63.8	74.2 / 79.7	33.5 / 62.5	58.0 / 57.7	60.8 / 66.2
Kimi-K3	29.4 / 39.1	28.3 / 33.0	38.5 / 26.1	13.8 / 44.9	47.2 / 31.1	13.4 / 42.1
GLM-5.3-Flash	44.3 / 37.5	43.5 / 33.3	48.9 / 26.3	28.0 / 33.4	51.6 / 47.5	30.6 / 34.4
DeepSeek-V4-Flash-Vision-Exp	41.8 / 30.7	41.1 / 27.3	46.1 / 28.1	25.9 / 28.9	43.9 / 27.9	32.8 / 31.0
SAM 3	17.3 / 38.1	15.4 / 19.5	23.8 / 24.8	4.3 / 27.3	27.7 / 35.1	11.1 / 27.9
Xiaomi-Lens-04B	71.4 / 85.0	54.7 / 70.0	63.9 / 79.1	57.1 / 79.6	68.6 / 79.4	62.3 / 72.1

GPT-6-Astra leads cIoU on explicit and implicit instructions and in aerial, driving, and panoramic imagery. Qwen3.8-Max leads cIoU in general imagery (74.2). Xiaomi-Lens-0 ranks second on both explicit-instruction metrics and on cIoU in aerial, driving, and panoramic imagery, and leads aerial Prec@0.5 (79.6). Implicit instructions remain challenging: cIoU drops from 71.4 to 54.7, while Prec@0.5 drops from 85.0 to 70.0. The gap is largest for multiple targets in panoramic scenes, where cIoU falls from 71.8 to 48.9 and Prec@0.5 from 81.3 to 48.7.

Predictions can miss objects or produce inaccurate boxes and masks, particularly for small, occluded, or ambiguous regions. Masks are decoded on the image canvas produced by the processor; preserve the image resizing and coordinate mapping when consuming them. The supplied inference path supports images.

License and attribution

The model is released under the Apache License 2.0.

Xiaomi-Lens-0 derives from Qwen3.5-4B, whose weights are also licensed under Apache 2.0. Xiaomi-Lens-0 adds the pixel-mask decoder and fine-tunes the model for grounded language and segmentation tasks.

• Qwen3.5-4B: Copyright 2026 Alibaba Cloud.
• Xiaomi-Lens-0 modifications: Copyright 2026 Xiaomi Corporation.

Citation

@misc{xiaomilens0,
  title         = {Xiaomi-Lens-0 Technical Report},
  author        = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and
                   Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and Tingwei Huang and
                   Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and
                   Heng Qu and Changjian Xu and Ming Xu and Jian Luan},
  year          = {2026},
  eprint        = {2610.23279},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.23279}
}