#
# 根据 Apache 许可证 2.0 版本（“许可证”）授权
# 您只能在遵守许可证的情况下使用本文件。
# 您可以在以下位置获取许可证副本
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# 除非适用法律要求或双方书面同意，软件
# 根据许可证分发，本软件按“原样”
# 不附带任何明示或暗示的保证或条件。
# 请参阅许可证中关于权限的具体语言
# 许可证下的限制。
-->

# VisualCloze

[VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning](https://huggingface.co/papers/2504.07960) 是一个基于上下文学习的通用图像生成框架，提供以下关键能力：
1. 支持多种领域内任务
2. 通过上下文学习泛化到未见过的任务
3. 将多个任务统一为一步，同时生成目标图像和中间结果
4. 支持从目标图像逆向推导条件

## 概述

该论文的摘要如下：

*扩散模型的最新进展显著推动了各类图像生成任务的发展。然而，当前主流方法仍侧重于构建特定任务模型，在支持多样化需求时效率有限。尽管通用模型试图解决这一局限，但仍面临若干关键挑战，包括可泛化的任务指令、合适的任务分布以及统一的架构设计。为应对这些挑战，我们提出了 VisualCloze，一个通用图像生成框架，支持广泛的领域内任务、对未见任务的泛化、多个未见任务的统一以及逆向生成。与依赖基于语言的任务指令、从而导致任务歧义和泛化能力较弱的现有方法不同，我们引入了视觉上下文学习，使模型能够从视觉示例中识别任务。同时，视觉任务分布固有的稀疏性阻碍了跨任务可迁移知识的学习。为此，我们引入了 Graph200K，一个图结构数据集，建立了多种相互关联的任务，从而增强任务密度和可迁移知识。此外，我们发现统一的图像生成公式与图像填充共享一致的目标，使我们能够在不修改架构的情况下，利用预训练填充模型的强大生成先验。代码、数据集和模型可在 https://visualcloze.github.io 获取。*

## 推理

### 模型加载

VisualCloze 是一个两阶段级联流水线，包含 `VisualClozeGenerationPipeline` 和 `VisualClozeUpsamplingPipeline`。
- 在 `VisualClozeGenerationPipeline` 中，每张图像在拼接为网格布局之前都会先进行下采样，以避免分辨率过高。VisualCloze 发布了两个适用于 diffusers 的模型，即 [VisualClozePipeline-384](https://huggingface.co/VisualCloze/VisualClozePipeline-384) 和 [VisualClozePipeline-512](https://huggingface.co/VisualCloze/VisualClozePipeline-384)，它们分别将图像下采样至 384 和 512 的分辨率。
- `VisualClozeUpsamplingPipeline` 使用 [SDEdit](https://huggingface.co/papers/2108.01073) 实现高分辨率图像合成。

`VisualClozePipeline` 将两个阶段整合在一起，以支持便捷的端到端采样，同时也允许用户根据需要使用各个独立的 pipeline。

### 输入规范

#### 任务和内容提示
- 任务提示：需要描述生成任务的意图
- 内容提示：目标图像的可选描述或说明
- 当不需要内容提示时，传入 `None`
- 对于批量推理，请传入 `List[str|None]`

#### 图像输入格式
- 格式：`List[List[Image|None]]`
- 结构：
  - 除最后一行外，所有行均表示上下文示例
  - 最后一行表示当前查询（目标图像设置为 `None`）
- 对于批量推理，请传入 `List[List[List[Image|None]]]`

#### 分辨率控制
- 默认行为：
  - 第一阶段初始生成：${pipe.resolution}^2$ 的面积
  - 第二阶段的：3 倍因子
- 自定义分辨率：使用 `upsampling_height` 和 `upsampling_width` 参数进行调整

### 示例

如需涵盖多种任务的完整示例，请参阅[在线演示](https://huggingface.co/spaces/VisualCloze/VisualCloze)和[GitHub 仓库](https://github.com/lzyhha/VisualCloze)。以下是三种情况的简单示例：掩码转图像、边缘检测以及主体驱动生成。

#### mask2image 示例

```python
import torch
from diffusers import VisualClozePipeline
from diffusers.utils import load_image

pipe = VisualClozePipeline.from_pretrained("VisualCloze/VisualClozePipeline-384", resolution=384, dtype=torch.bfloat16)
pipe.to("cuda")  # or "mps", "xpu", "cpu"

# Load in-context images (make sure the paths are correct and accessible)
image_paths = [
    # in-context examples
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_mask.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_image.jpg'),
    ],
    # query with the target image
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_query_mask.jpg'),
        None, # No image needed for the target image
    ],
]

# Task and content prompt
task_prompt = "In each row, a logical task is demonstrated to achieve [IMAGE2] an aesthetically pleasing photograph based on [IMAGE1] sam 2-generated masks with rich color coding."
content_prompt = """Majestic photo of a golden eagle perched on a rocky outcrop in a mountainous landscape. 
The eagle is positioned in the right foreground, facing left, with its sharp beak and keen eyes prominently visible. 
Its plumage is a mix of dark brown and golden hues, with intricate feather details. 
The background features a soft-focus view of snow-capped mountains under a cloudy sky, creating a serene and grandiose atmosphere. 
The foreground includes rugged rocks and patches of green moss. Photorealistic, medium depth of field, 
soft natural lighting, cool color palette, high contrast, sharp focus on the eagle, blurred background, 
tranquil, majestic, wildlife photography."""

# Run the pipeline
image_result = pipe(
    task_prompt=task_prompt,
    content_prompt=content_prompt,
    image=image_paths,
    upsampling_width=1344,
    upsampling_height=768,
    upsampling_strength=0.4,
    guidance_scale=30,
    num_inference_steps=30,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0][0]

# Save the resulting image
image_result.save("visualcloze.png")
```

#### 边缘检测示例

```python
import torch
from diffusers import VisualClozePipeline
from diffusers.utils import load_image

pipe = VisualClozePipeline.from_pretrained("VisualCloze/VisualClozePipeline-384", resolution=384, dtype=torch.bfloat16)
pipe.to("cuda")  # or "mps", "xpu", "cpu"

# Load in-context images (make sure the paths are correct and accessible)
image_paths = [
    # in-context examples
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_edgedetection_incontext-example-1_image.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_edgedetection_incontext-example-1_edge.jpg'),
    ],
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_edgedetection_incontext-example-2_image.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_edgedetection_incontext-example-2_edge.jpg'),
    ],
    # query with the target image
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_edgedetection_query_image.jpg'),
        None, # No image needed for the target image
    ],
]

# Task and content prompt
task_prompt = "Each row illustrates a pathway from [IMAGE1] a sharp and beautifully composed photograph to [IMAGE2] edge map with natural well-connected outlines using a clear logical task."
content_prompt = ""

# Run the pipeline
image_result = pipe(
    task_prompt=task_prompt,
    content_prompt=content_prompt,
    image=image_paths,
    upsampling_width=864,
    upsampling_height=1152,
    upsampling_strength=0.4,
    guidance_scale=30,
    num_inference_steps=30,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0][0]

# Save the resulting image
image_result.save("visualcloze.png")
```

#### 主题驱动生成的示例

```python
import torch
from diffusers import VisualClozePipeline
from diffusers.utils import load_image

pipe = VisualClozePipeline.from_pretrained("VisualCloze/VisualClozePipeline-384", resolution=384, dtype=torch.bfloat16)
pipe.to("cuda")  # or "mps", "xpu", "cpu"

# Load in-context images (make sure the paths are correct and accessible)
image_paths = [
    # in-context examples
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-1_reference.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-1_depth.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-1_image.jpg'),
    ],
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-2_reference.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-2_depth.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_incontext-example-2_image.jpg'),
    ],
    # query with the target image
    [
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_query_reference.jpg'),
        load_image('https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_subjectdriven_query_depth.jpg'),
        None, # No image needed for the target image
    ],
]

# Task and content prompt
task_prompt = """Each row describes a process that begins with [IMAGE1] an image containing the key object, 
[IMAGE2] depth map revealing gray-toned spatial layers and results in 
[IMAGE3] an image with artistic qualitya high-quality image with exceptional detail."""
content_prompt = """A vintage porcelain collector's item. Beneath a blossoming cherry tree in early spring, 
this treasure is photographed up close, with soft pink petals drifting through the air and vibrant blossoms framing the scene."""

# Run the pipeline
image_result = pipe(
    task_prompt=task_prompt,
    content_prompt=content_prompt,
    image=image_paths,
    upsampling_width=1024,
    upsampling_height=1024,
    upsampling_strength=0.2,
    guidance_scale=30,
    num_inference_steps=30,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0][0]

# Save the resulting image
image_result.save("visualcloze.png")
```

#### 独立使用每个流水线 

```python
import torch
from diffusers import VisualClozeGenerationPipeline, FluxFillPipeline as VisualClozeUpsamplingPipeline
from diffusers.utils import load_image
from PIL import Image

pipe = VisualClozeGenerationPipeline.from_pretrained(
    "VisualCloze/VisualClozePipeline-384", resolution=384, dtype=torch.bfloat16
)
pipe.to("cuda")  # or "mps", "xpu", "cpu"

image_paths = [
    # in-context examples
    [
        load_image(
            "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_mask.jpg"
        ),
        load_image(
            "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_image.jpg"
        ),
    ],
    # query with the target image
    [
        load_image(
            "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_query_mask.jpg"
        ),
        None,  # No image needed for the target image
    ],
]
task_prompt = "In each row, a logical task is demonstrated to achieve [IMAGE2] an aesthetically pleasing photograph based on [IMAGE1] sam 2-generated masks with rich color coding."
content_prompt = "Majestic photo of a golden eagle perched on a rocky outcrop in a mountainous landscape. The eagle is positioned in the right foreground, facing left, with its sharp beak and keen eyes prominently visible. Its plumage is a mix of dark brown and golden hues, with intricate feather details. The background features a soft-focus view of snow-capped mountains under a cloudy sky, creating a serene and grandiose atmosphere. The foreground includes rugged rocks and patches of green moss. Photorealistic, medium depth of field, soft natural lighting, cool color palette, high contrast, sharp focus on the eagle, blurred background, tranquil, majestic, wildlife photography."

# Stage 1: Generate initial image
image = pipe(
    task_prompt=task_prompt,
    content_prompt=content_prompt,
    image=image_paths,
    guidance_scale=30,
    num_inference_steps=30,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0),
).images[0][0]

# Stage 2 (optional): Upsample the generated image
pipe_upsample = VisualClozeUpsamplingPipeline.from_pipe(pipe)
pipe_upsample.to("cuda")

mask_image = Image.new("RGB", image.size, (255, 255, 255))

image = pipe_upsample(
    image=image,
    mask_image=mask_image,
    prompt=content_prompt,
    width=1344,
    height=768,
    strength=0.4,
    guidance_scale=30,
    num_inference_steps=30,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0),
).images[0]

image.save("visualcloze.png")
```

## VisualClozePipeline[[diffusers.VisualClozePipeline]]

#### diffusers.VisualClozePipeline[[diffusers.VisualClozePipeline]]

```python
diffusers.VisualClozePipeline(scheduler: FlowMatchEulerDiscreteScheduler, vae: AutoencoderKL, text_encoder: CLIPTextModel, tokenizer: CLIPTokenizer, text_encoder_2: T5EncoderModel, tokenizer_2: T5Tokenizer, transformer: FluxTransformer2DModel, resolution: int = 384)
```

[源代码](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/visualcloze/pipeline_visualcloze_combined.py#L89)

**参数：**

transformer ([FluxTransformer2DModel](/docs/diffusers/main/zh/api/models/flux_transformer#diffusers.FluxTransformer2DModel))：用于对编码后的图像潜变量进行去噪的条件 Transformer (MMDiT) 架构。

scheduler ([FlowMatchEulerDiscreteScheduler](/docs/diffusers/main/zh/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler))：用于与 `transformer` 配合，对编码后的图像潜变量进行去噪的调度器。

vae ([AutoencoderKL](/docs/diffusers/main/zh/api/models/autoencoderkl#diffusers.AutoencoderKL))：变分自编码器（VAE）模型，用于将图像编码为潜在表示，并从潜在表示解码回图像。

text_encoder (`CLIPTextModel`)：即 [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel)，具体为 [clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) 变体。

text_encoder_2 (`T5EncoderModel`)：即 [T5](https://huggingface.co/docs/transformers/en/model_doc/t5#transformers.T5EncoderModel)，具体为 [google/t5-v1_1-xxl](https://huggingface.co/google/t5-v1_1-xxl) 变体。

tokenizer (`CLIPTokenizer`)：[CLIPTokenizer](https://huggingface.co/docs/transformers/en/model_doc/clip#transformers.CLIPTokenizer) 类的分词器。

tokenizer_2 (`T5TokenizerFast`)：第二个 [T5TokenizerFast](https://huggingface.co/docs/transformers/en/model_doc/t5#transformers.T5TokenizerFast) 类的 Tokenizer。

resolution (`int`，*可选*，默认值为 384)：在拼接查询和上下文示例中的图像时，每张图像的分辨率。

用于基于视觉上下文进行图像生成的 VisualCloze 流水线。参考：https://github.com/lzyhha/VisualCloze/tree/main。该流水线旨在根据视觉上下文示例生成图像。

#### __调用__[[diffusers.VisualClozePipeline.__call__]]

```python
__call__(task_prompt: str | list[str] = None, content_prompt: str | list[str] = None, image: typing.Optional[torch.FloatTensor] = None, upsampling_height: int | None = None, upsampling_width: int | None = None, num_inference_steps: int = 50, sigmas: list[float] | None = None, guidance_scale: float = 30.0, num_images_per_prompt: int | None = 1, generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = None, latents: typing.Optional[torch.FloatTensor] = None, prompt_embeds: typing.Optional[torch.FloatTensor] = None, pooled_prompt_embeds: typing.Optional[torch.FloatTensor] = None, output_type: str | None = 'pil', return_dict: bool = True, joint_attention_kwargs: dict[str, typing.Any] | None = None, callback_on_step_end: typing.Optional[typing.Callable[[int, int], NoneType]] = None, callback_on_step_end_tensor_inputs: list = ['latents'], max_sequence_length: int = 512, upsampling_strength: float = 1.0)
```

[源代码](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/visualcloze/pipeline_visualcloze_combined.py#L252)

**参数：**

task_prompt (`str` 或 `list[str]`，*可选*)：用于定义任务意图的提示词或提示词列表。

content_prompt (`str` 或 `list[str]`，*可选*)：用于定义要生成的目标图像的内容或说明的提示词或提示词列表。

image (`torch.Tensor`、`PIL.Image.Image`、`np.ndarray`、`list[torch.Tensor]`、`list[PIL.Image.Image]` 或 `list[np.ndarray]`) : `Image`、numpy 数组或张量，表示用作起点的图像批次。对于 numpy 数组和 PyTorch 张量，预期的取值范围为 `[0, 1]`。如果是张量或张量列表，预期的形状应为 `(B, C, H, W)` 或 `(C, H, W)`。如果是 numpy 数组或数组列表，预期的形状应为 `(B, H, W, C)` 或 `(H, W, C)`。

upsampling_height (`int`，*可选*，默认为 self.unet.config.sample_size * self.vae_scale_factor)：通过 SDEdit 上采样后生成图像（即输出图像）的像素高度。默认情况下，图像会放大三倍，基础分辨率由流水线的 resolution 参数决定。如果只指定了 `upsampling_height` 或 `upsampling_width` 中的一个，另一个将根据宽高比自动设置。

upsampling_width (`int`, *可选*，默认为 self.unet.config.sample_size * self.vae_scale_factor) : 通过 SDEdit 上采样后生成图像（即输出图像）的像素宽度。默认情况下，图像会放大三倍，基础分辨率由流水线的 resolution 参数决定。如果只指定了 `upsampling_height` 或 `upsampling_width` 中的一个，另一个将根据宽高比自动设置。

num_inference_steps (`int`, *可选*，默认为 50)：去噪步数。更多的去噪步数通常能带来更高质量的图像，但代价是推理速度变慢。

sigmas (`list[float]`, *可选*) : 用于支持在 `set_timesteps` 方法中接受 `sigmas` 参数的调度器的去噪过程的自定义 sigmas。如果未定义，将使用传入 `num_inference_steps` 时的默认行为。

guidance_scale (`float`，*可选*，默认值为 30.0)：[Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598) 中定义的引导比例。`guidance_scale` 定义为 [Imagen Paper](https://huggingface.co/papers/2205.11487) 中公式 2 的 `w`。通过设置 `guidance_scale > 1` 来启用引导比例。较高的引导比例会促使生成与文本 `prompt` 紧密关联的图像，但通常以牺牲图像质量为代价。

num_images_per_prompt (`int`，*可选*，默认为 1)：每个提示词生成的图像数量。

generator (`torch.Generator` 或 `list[torch.Generator]`，*可选*)：一个或一组 [torch 生成器](https://pytorch.org/docs/stable/generated/torch.Generator.html)，用于使生成过程具有确定性。

latents (`torch.FloatTensor`, *可选*) : 预先生成的噪声潜变量，从高斯分布中采样得到，用作图像生成的输入。可用于以不同的提示词微调同一生成过程。如果未提供，将通过使用提供的随机`generator`进行采样来生成潜变量张量。

prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，则将从 `prompt` 输入参数生成文本嵌入。

pooled_prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的池化文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，将从 `prompt` 输入参数生成池化文本嵌入。

output_type (`str`，*可选*，默认为 `"pil"`)：生成图像的输出格式。可选 [PIL](https://pillow.readthedocs.io/en/stable/)：`PIL.Image.Image` 或 `np.array`。

return_dict (`bool`，*可选*，默认为 `True`)：是否返回 `~pipelines.flux.FluxPipelineOutput` 而不是普通元组。

joint_attention_kwargs (`dict`, *可选*) : 一个 kwargs 字典，如果指定，将作为参数传递给 [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py) 中 `self.processor` 下定义的 `AttentionProcessor`。

callback_on_step_end (`Callable`，*可选*)：一个在推理过程中每个去噪步骤结束时调用的函数。该函数以以下参数被调用：`callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`。`callback_kwargs` 将包含由 `callback_on_step_end_tensor_inputs` 指定的所有张量列表。

callback_on_step_end_tensor_inputs (`list`，*可选*)：`callback_on_step_end` 函数的张量输入列表。列表中指定的张量将作为 `callback_kwargs` 参数传递。您只能包含流水线类的 `._callback_tensor_inputs` 属性中列出的变量。

max_sequence_length (`int` 默认为 512)：用于 `prompt` 的最大序列长度。

upsampling_strength (`float`, *可选*，默认为 1.0) : 指示在上采样结果时对参考 `image` 进行变换的程度。取值必须在 0 到 1 之间。生成的图像将用作起点，`upsampling_strength` 的值越高，添加的噪声越多。去噪步数取决于初始添加的噪声量。当 `upsampling_strength` 为 1 时，添加的噪声最大，去噪过程将运行 `num_inference_steps` 中指定的完整迭代次数。值为 0 时会跳过上采样步骤，并以 `self.resolution` 的分辨率输出结果。

**返回：** `~pipelines.flux.FluxPipelineOutput` 或 `tuple`

`~pipelines.flux.FluxPipelineOutput`（当 `return_dict` 为 True 时），否则为 `tuple`。返回元组时，第一个元素是包含生成图像的列表。

调用 VisualCloze 流水线进行生成时调用的函数。

示例：
```python
>>> import torch
>>> from diffusers import VisualClozePipeline
>>> from diffusers.utils import load_image

>>> image_paths = [
...     # in-context examples
...     [
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_mask.jpg"
...         ),
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_image.jpg"
...         ),
...     ],
...     # query with the target image
...     [
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_query_mask.jpg"
...         ),
...         None,  # No image needed for the target image
...     ],
... ]
>>> task_prompt = "In each row, a logical task is demonstrated to achieve [IMAGE2] an aesthetically pleasing photograph based on [IMAGE1] sam 2-generated masks with rich color coding."
>>> content_prompt = "Majestic photo of a golden eagle perched on a rocky outcrop in a mountainous landscape. The eagle is positioned in the right foreground, facing left, with its sharp beak and keen eyes prominently visible. Its plumage is a mix of dark brown and golden hues, with intricate feather details. The background features a soft-focus view of snow-capped mountains under a cloudy sky, creating a serene and grandiose atmosphere. The foreground includes rugged rocks and patches of green moss. Photorealistic, medium depth of field, soft natural lighting, cool color palette, high contrast, sharp focus on the eagle, blurred background, tranquil, majestic, wildlife photography."
>>> pipe = VisualClozePipeline.from_pretrained(
...     "VisualCloze/VisualClozePipeline-384", resolution=384, torch_dtype=torch.bfloat16
... )
>>> pipe.to("cuda")

>>> image = pipe(
...     task_prompt=task_prompt,
...     content_prompt=content_prompt,
...     image=image_paths,
...     upsampling_width=1344,
...     upsampling_height=768,
...     upsampling_strength=0.4,
...     guidance_scale=30,
...     num_inference_steps=30,
...     max_sequence_length=512,
...     generator=torch.Generator("cpu").manual_seed(0),
... ).images[0][0]
>>> image.save("visualcloze.png")
```

## VisualClozeGenerationPipeline[[diffusers.VisualClozeGenerationPipeline]]

#### diffusers.VisualClozeGenerationPipeline[[diffusers.VisualClozeGenerationPipeline]]

```python
diffusers.VisualClozeGenerationPipeline(scheduler: FlowMatchEulerDiscreteScheduler, vae: AutoencoderKL, text_encoder: CLIPTextModel, tokenizer: CLIPTokenizer, text_encoder_2: T5EncoderModel, tokenizer_2: T5Tokenizer, transformer: FluxTransformer2DModel, resolution: int = 384)
```

[源代码](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/visualcloze/pipeline_visualcloze_generation.py#L118)

**参数：**

transformer ([FluxTransformer2DModel](/docs/diffusers/main/zh/api/models/flux_transformer#diffusers.FluxTransformer2DModel))：用于对编码后的图像潜变量进行去噪的条件 Transformer (MMDiT) 架构。

scheduler ([FlowMatchEulerDiscreteScheduler](/docs/diffusers/main/zh/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler))：用于与 `transformer` 配合，对编码后的图像潜变量进行去噪的调度器。

vae ([AutoencoderKL](/docs/diffusers/main/zh/api/models/autoencoderkl#diffusers.AutoencoderKL))：变分自编码器（VAE）模型，用于将图像编码为潜在表示，并从潜在表示解码回图像。

text_encoder (`CLIPTextModel`)：即 [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel)，具体为 [clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) 变体。

text_encoder_2 (`T5EncoderModel`)：即 [T5](https://huggingface.co/docs/transformers/en/model_doc/t5#transformers.T5EncoderModel)，具体为 [google/t5-v1_1-xxl](https://huggingface.co/google/t5-v1_1-xxl) 变体。

tokenizer (`CLIPTokenizer`)：[CLIPTokenizer](https://huggingface.co/docs/transformers/en/model_doc/clip#transformers.CLIPTokenizer) 类的分词器。

tokenizer_2 (`T5TokenizerFast`)：第二个 [T5TokenizerFast](https://huggingface.co/docs/transformers/en/model_doc/t5#transformers.T5TokenizerFast) 类的 Tokenizer。

resolution (`int`，*可选*，默认值为 384)：在拼接查询和上下文示例中的图像时，每张图像的分辨率。

用于基于视觉上下文进行图像生成的 VisualCloze 流水线。参考：https://github.com/lzyhha/VisualCloze/tree/main 该流水线旨在根据视觉上下文示例生成图像。

#### __调用__[[diffusers.VisualClozeGenerationPipeline.__call__]]

```python
__call__(task_prompt: str | list[str] = None, content_prompt: str | list[str] = None, image: typing.Optional[torch.FloatTensor] = None, num_inference_steps: int = 50, sigmas: list[float] | None = None, guidance_scale: float = 30.0, num_images_per_prompt: int | None = 1, generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = None, latents: typing.Optional[torch.FloatTensor] = None, prompt_embeds: typing.Optional[torch.FloatTensor] = None, pooled_prompt_embeds: typing.Optional[torch.FloatTensor] = None, output_type: str | None = 'pil', return_dict: bool = True, joint_attention_kwargs: dict[str, typing.Any] | None = None, callback_on_step_end: typing.Optional[typing.Callable[[int, int], NoneType]] = None, callback_on_step_end_tensor_inputs: list = ['latents'], max_sequence_length: int = 512)
```

[源代码](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/visualcloze/pipeline_visualcloze_generation.py#L659)

**参数：**

task_prompt (`str` 或 `list[str]`，*可选*)：用于定义任务意图的提示词或提示词列表。

content_prompt (`str` 或 `list[str]`，*可选*)：用于定义要生成的目标图像的内容或说明的提示词或提示词列表。

image (`torch.Tensor`、`PIL.Image.Image`、`np.ndarray`、`list[torch.Tensor]`、`list[PIL.Image.Image]` 或 `list[np.ndarray]`) : `Image`、numpy 数组或张量，表示用作起点的图像批次。对于 numpy 数组和 PyTorch 张量，预期的取值范围为 `[0, 1]`。如果是张量或张量列表，预期的形状应为 `(B, C, H, W)` 或 `(C, H, W)`。如果是 numpy 数组或数组列表，预期的形状应为 `(B, H, W, C)` 或 `(H, W, C)`。

num_inference_steps (`int`, *可选*，默认为 50)：去噪步数。更多的去噪步数通常能带来更高质量的图像，但代价是推理速度变慢。

sigmas (`list[float]`, *可选*) : 用于支持在 `set_timesteps` 方法中接受 `sigmas` 参数的调度器的去噪过程的自定义 sigmas。如果未定义，将使用传入 `num_inference_steps` 时的默认行为。

guidance_scale (`float`，*可选*，默认值为 30.0)：[Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598) 中定义的引导比例。`guidance_scale` 定义为 [Imagen Paper](https://huggingface.co/papers/2205.11487) 中公式 2 的 `w`。通过设置 `guidance_scale > 1` 来启用引导比例。较高的引导比例会促使生成与文本 `prompt` 紧密关联的图像，但通常以牺牲图像质量为代价。

num_images_per_prompt (`int`，*可选*，默认为 1)：每个提示词生成的图像数量。

generator (`torch.Generator` 或 `list[torch.Generator]`，*可选*)：一个或一组 [torch 生成器](https://pytorch.org/docs/stable/generated/torch.Generator.html)，用于使生成过程具有确定性。

latents (`torch.FloatTensor`, *可选*) : 预先生成的噪声潜变量，从高斯分布中采样得到，用作图像生成的输入。可用于以不同的提示词微调同一生成过程。如果未提供，将通过使用提供的随机`generator`进行采样来生成潜变量张量。

prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，则将从 `prompt` 输入参数生成文本嵌入。

pooled_prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的池化文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，将从 `prompt` 输入参数生成池化文本嵌入。

output_type (`str`，*可选*，默认为 `"pil"`)：生成图像的输出格式。可选 `"pil"`（[PIL](https://pillow.readthedocs.io/en/stable/)：`PIL.Image.Image`）、`"np"`（`np.array`）或 `"pt"`（`torch.Tensor`）。

return_dict (`bool`，*可选*，默认为 `True`)：是否返回 `~pipelines.flux.FluxPipelineOutput` 而不是普通元组。

joint_attention_kwargs (`dict`, *可选*) : 一个 kwargs 字典，如果指定，将作为参数传递给 [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py) 中 `self.processor` 下定义的 `AttentionProcessor`。

callback_on_step_end (`Callable`，*可选*)：一个在推理过程中每个去噪步骤结束时调用的函数。该函数以以下参数被调用：`callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`。`callback_kwargs` 将包含由 `callback_on_step_end_tensor_inputs` 指定的所有张量列表。

callback_on_step_end_tensor_inputs (`list`，*可选*)：`callback_on_step_end` 函数的张量输入列表。列表中指定的张量将作为 `callback_kwargs` 参数传递。您只能包含流水线类的 `._callback_tensor_inputs` 属性中列出的变量。

max_sequence_length (`int` 默认为 512)：用于 `prompt` 的最大序列长度。

**返回：** `~pipelines.flux.FluxPipelineOutput` 或 `tuple`

`~pipelines.flux.FluxPipelineOutput`（当 `return_dict` 为 True 时），否则为 `tuple`。返回元组时，第一个元素是包含生成图像的列表。

调用 VisualCloze 流水线进行生成时调用的函数。

示例：
```python
>>> import torch
>>> from diffusers import VisualClozeGenerationPipeline, FluxFillPipeline as VisualClozeUpsamplingPipeline
>>> from diffusers.utils import load_image
>>> from PIL import Image

>>> image_paths = [
...     # in-context examples
...     [
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_mask.jpg"
...         ),
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_incontext-example-1_image.jpg"
...         ),
...     ],
...     # query with the target image
...     [
...         load_image(
...             "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/visualcloze/visualcloze_mask2image_query_mask.jpg"
...         ),
...         None,  # No image needed for the target image
...     ],
... ]
>>> task_prompt = "In each row, a logical task is demonstrated to achieve [IMAGE2] an aesthetically pleasing photograph based on [IMAGE1] sam 2-generated masks with rich color coding."
>>> content_prompt = "Majestic photo of a golden eagle perched on a rocky outcrop in a mountainous landscape. The eagle is positioned in the right foreground, facing left, with its sharp beak and keen eyes prominently visible. Its plumage is a mix of dark brown and golden hues, with intricate feather details. The background features a soft-focus view of snow-capped mountains under a cloudy sky, creating a serene and grandiose atmosphere. The foreground includes rugged rocks and patches of green moss. Photorealistic, medium depth of field, soft natural lighting, cool color palette, high contrast, sharp focus on the eagle, blurred background, tranquil, majestic, wildlife photography."
>>> pipe = VisualClozeGenerationPipeline.from_pretrained(
...     "VisualCloze/VisualClozePipeline-384", resolution=384, torch_dtype=torch.bfloat16
... )
>>> pipe.to("cuda")

>>> image = pipe(
...     task_prompt=task_prompt,
...     content_prompt=content_prompt,
...     image=image_paths,
...     guidance_scale=30,
...     num_inference_steps=30,
...     max_sequence_length=512,
...     generator=torch.Generator("cpu").manual_seed(0),
... ).images[0][0]

>>> # optional, upsampling the generated image
>>> pipe_upsample = VisualClozeUpsamplingPipeline.from_pipe(pipe)
>>> pipe_upsample.to("cuda")

>>> mask_image = Image.new("RGB", image.size, (255, 255, 255))

>>> image = pipe_upsample(
...     image=image,
...     mask_image=mask_image,
...     prompt=content_prompt,
...     width=1344,
...     height=768,
...     strength=0.4,
...     guidance_scale=30,
...     num_inference_steps=30,
...     max_sequence_length=512,
...     generator=torch.Generator("cpu").manual_seed(0),
... ).images[0]

>>> image.save("visualcloze.png")
```

#### encode_prompt[[diffusers.VisualClozeGenerationPipeline.encode_prompt]]

```python
encode_prompt(layout_prompt: str | list[str], task_prompt: str | list[str], content_prompt: str | list[str], device: typing.Optional[torch.device] = None, num_images_per_prompt: int = 1, prompt_embeds: typing.Optional[torch.FloatTensor] = None, pooled_prompt_embeds: typing.Optional[torch.FloatTensor] = None, max_sequence_length: int = 512, lora_scale: float | None = None)
```

[源代码](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/visualcloze/pipeline_visualcloze_generation.py#L292)

**参数：**

layout_prompt (`str` 或 `list[str]`，*可选*)：用于定义任务中上下文示例数量和涉及图像数量的提示词或提示词列表。

task_prompt (`str` 或 `list[str]`，*可选*)：用于定义任务意图的提示词或提示词列表。

content_prompt (`str` 或 `list[str]`，*可选*)：用于定义要生成的目标图像的内容或说明的提示词或提示词列表。

device : (`torch.device`): torch 设备

num_images_per_prompt (`int`) : 每个提示词应生成的图像数量

prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，则将从 `prompt` 输入参数生成文本嵌入。

pooled_prompt_embeds (`torch.FloatTensor`，*可选*)：预生成的池化文本嵌入。可用于轻松调整文本输入，*例如*提示词加权。如果未提供，将从 `prompt` 输入参数生成池化文本嵌入。

lora_scale (`float`, *可选*) : 如果加载了 LoRA 层，将应用于文本编码器所有 LoRA 层的 LoRA 缩放系数。