Diffusers documentation
IP-Adapter
IP-Adapter
IP-Adapter mengarahkan model difusi yang telah dilatih sebelumnya menggunakan gambar referensi sambil tetap mempertahankan prompt teks. Metode ini membekukan model dasar dan menambahkan seperangkat kecil lapisan cross-attention gambar, yang membuatnya praktis untuk mencocokkan subjek atau gaya dari foto tanpa perlu fine-tuning.
text prompt reference image
| |
text encoder image encoder
| |
v v
text cross-attn (frozen) IP-Adapter cross-attn
\ /
\ /
+--------> denoiser <-----------+
(frozen base)Checkpoint IP-Adapter biasanya berukuran ~100MB karena menyimpan bobot adapter, bukan model lengkap. Muat pipeline dasar terlebih dahulu, kemudian muat adapter dengan load_ip_adapter().
IP-Adapters tersedia untuk banyak model seperti Flux dan Stable Diffusion 3, serta lainnya. Contoh dalam panduan ini menggunakan Stable Diffusion dan Stable Diffusion XL.
Gunakan set_ip_adapter_scale() untuk mengontrol seberapa kuat IP-Adapter mengarahkan generasi. 1.0 menerapkan adapter dengan kekuatan penuh; 0.5 biasanya menyeimbangkan prompt teks dan gambar.
Contoh di bawah ini menunjukkan IP-Adapter pada tugas-tugas umum.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)Lewatkan gambar ke ip_adapter_image bersama dengan prompt teks untuk menghasilkan gambar.
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_diner.png")
pipeline(
prompt="a polar bear sitting in a chair drinking a milkshake",
ip_adapter_image=image,
negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
).images[0]
import torch
from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForImage2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
ip_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_gummy.png")
pipeline(
prompt="best quality, high quality",
image=image,
ip_adapter_image=ip_image,
strength=0.5,
).images[0]
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
pipeline = AutoPipelineForInpainting.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.6)
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_mask.png")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
ip_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_gummy.png")
pipeline(
prompt="a cute gummy bear waving",
image=image,
mask_image=mask_image,
ip_adapter_image=ip_image,
).images[0] 
Metode enable_model_cpu_offload() berguna untuk mengurangi memori dan sebaiknya diaktifkan setelah IP-Adapter dimuat. Jika tidak, encoder gambar IP-Adapter juga akan di-offload ke CPU dan mengembalikan error.
import torch
from diffusers import AnimateDiffPipeline, DDIMScheduler, MotionAdapter
from diffusers.utils import export_to_gif
from diffusers.utils import load_image
adapter = MotionAdapter.from_pretrained(
"guoyww/animatediff-motion-adapter-v1-5-2",
dtype=torch.float16
)
pipeline = AnimateDiffPipeline.from_pretrained(
"emilianJR/epiCRealism",
motion_adapter=adapter,
dtype=torch.float16
)
scheduler = DDIMScheduler.from_pretrained(
"emilianJR/epiCRealism",
subfolder="scheduler",
clip_sample=False,
timestep_spacing="linspace",
beta_schedule="linear",
steps_offset=1,
)
pipeline.scheduler = scheduler
pipeline.vae.enable_slicing()
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipeline.enable_model_cpu_offload()
ip_adapter_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_inpaint.png")
pipeline(
prompt="A cute gummy bear waving",
negative_prompt="bad quality, worse quality, low resolution",
ip_adapter_image=ip_adapter_image,
num_frames=16,
guidance_scale=7.5,
num_inference_steps=50,
).frames[0] 
Varian checkpoint
Muat Plus ketika detail dari gambar referensi paling penting. Muat FaceID ketika Anda memerlukan embedding identitas InsightFace alih-alih embedding gambar CLIP.
import torch
from transformers import CLIPVisionModelWithProjection
from diffusers import AutoPipelineForText2Image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"h94/IP-Adapter",
subfolder="models/image_encoder",
dtype=torch.float16
)
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
image_encoder=image_encoder,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter-plus_sdxl_vit-h.safetensors"
)Checkpoint FaceID menggunakan face embeddings dari InsightFace alih-alih image embeddings CLIP. Gunakan DDIMScheduler atau EulerDiscreteScheduler untuk model FaceID. Ekstrak face embeddings dan lewati sebagai daftar tensor ke ip_adapter_image_embeds.
# pip install insightface
import cv2
import numpy as np
import torch
from diffusers import StableDiffusionPipeline, DDIMScheduler
from diffusers.utils import load_image
from insightface.app import FaceAnalysis
pipeline = StableDiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
dtype=torch.float16,
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
"h94/IP-Adapter-FaceID",
subfolder=None,
weight_name="ip-adapter-faceid_sd15.bin",
image_encoder_folder=None
)
pipeline.set_ip_adapter_scale(0.6)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
ref_images_embeds = []
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
image = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR)
faces = app.get(image)
image = torch.from_numpy(faces[0].normed_embedding)
ref_images_embeds.append(image.unsqueeze(0))
ref_images_embeds = torch.stack(ref_images_embeds, dim=0).unsqueeze(0)
neg_ref_images_embeds = torch.zeros_like(ref_images_embeds)
id_embeds = torch.cat([neg_ref_images_embeds, ref_images_embeds]).to(dtype=torch.float16, device="cuda")
pipeline(
prompt="A photo of a girl",
ip_adapter_image_embeds=[id_embeds],
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]Untuk FaceID Plus, muat CLIPVisionModelWithProjection sebagai encoder gambar.
import torch
from transformers import CLIPVisionModelWithProjection
from diffusers import AutoPipelineForText2Image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"laion/CLIP-ViT-H-14-laion2B-s32B-b79K",
dtype=torch.float16,
)
pipeline = AutoPipelineForText2Image.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
image_encoder=image_encoder,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter-FaceID",
subfolder=None,
weight_name="ip-adapter-faceid-plus_sd15.bin"
)FaceID Plus dan Plus v2 juga memerlukan embedding gambar CLIP di atas embeds InsightFace. Setelah Anda menyiapkan embedding wajah, encode gambar referensi dan tetapkan embeds CLIP ke lapisan proyeksi gambar.
import torch
from diffusers.utils import load_image
ip_adapter_images = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
num_images = 1
clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=[ip_adapter_images],
ip_adapter_image_embeds=None,
device=torch.device("cuda"),
num_images_per_prompt=num_images,
do_classifier_free_guidance=True,
)[0]
pipeline.unet.encoder_hid_proj.image_projection_layers[0].clip_embeds = clip_embeds.to(dtype=torch.float16)
# set to True if using IP-Adapter FaceID Plus v2
pipeline.unet.encoder_hid_proj.image_projection_layers[0].shortcut = FalseEmbedding gambar
prepare_ip_adapter_image_embeds() mengodekan gambar IP-Adapter menjadi embeddings yang dapat Anda simpan dan gunakan kembali. Hitung sebelumnya sekali alih-alih memuat dan mengodekan gambar yang sama setiap kali Anda menjalankan pipeline.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
image_embeds = pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=image,
ip_adapter_image_embeds=None,
device="cuda",
num_images_per_prompt=1,
do_classifier_free_guidance=True,
)
torch.save(image_embeds, "image_embeds.ipadpt")Muat ulang embedding gambar dengan meneruskannya ke parameter ip_adapter_image_embeds. Atur image_encoder_folder ke None karena Anda tidak lagi memerlukan image encoder untuk menghasilkan embedding gambar.
Anda juga dapat memuat embedding gambar dari sumber lain seperti ComfyUI.
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
image_encoder_folder=None,
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)
image_embeds = torch.load("image_embeds.ipadpt")
pipeline(
prompt="a polar bear sitting in a chair drinking a milkshake",
ip_adapter_image_embeds=image_embeds,
negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
num_inference_steps=100,
).images[0]Masking
Masking biner memungkinkan penugasan gambar IP-Adapter ke area tertentu pada gambar output, sehingga berguna untuk menyusun beberapa gambar IP-Adapter. Setiap gambar IP-Adapter memerlukan mask biner.
Muat IPAdapterMaskProcessor untuk memproses pramasker gambar. Untuk hasil terbaik, berikan height dan width keluaran untuk memastikan mask dengan rasio aspek yang berbeda berukuran sesuai. Jika mask masukan sudah sesuai dengan rasio aspek gambar yang dihasilkan, Anda tidak perlu mengatur height dan width.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.image_processor import IPAdapterMaskProcessor
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
mask1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_mask1.png")
mask2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_mask2.png")
processor = IPAdapterMaskProcessor()
masks = processor.preprocess([mask1, mask2], height=1024, width=1024)
Sediakan gambar IP-Adapter dan skala mereka sebagai daftar. Teruskan mask yang telah dipreproses ke cross_attention_kwargs di pipeline.
face_image1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
face_image2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl2.png")
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name=["ip-adapter-plus-face_sdxl_vit-h.safetensors"]
)
pipeline.set_ip_adapter_scale([[0.7, 0.7]])
ip_images = [[face_image1, face_image2]]
masks = [masks.reshape(1, masks.shape[0], masks.shape[2], masks.shape[3])]
pipeline(
prompt="2 girls",
ip_adapter_image=ip_images,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
cross_attention_kwargs={"ip_adapter_masks": masks}
).images[0]
Resep
Gabungkan IP-Adapter dengan adapter atau pipeline lain, seperti beberapa adapter, ControlNet, InstantStyle, atau LCM few-step, ketika satu gambar referensi tidak cukup.
Beberapa IP-Adapter
Gabungkan beberapa IP-Adapter untuk menghasilkan gambar dengan gaya yang lebih beragam. Misalnya, Anda dapat menggunakan IP-Adapter Face untuk menghasilkan wajah dan karakter yang konsisten, serta IP-Adapter Plus untuk menghasilkan wajah tersebut dalam gaya tertentu.
Muat encoder gambar dengan CLIPVisionModelWithProjection.
import torch
from diffusers import AutoPipelineForText2Image, DDIMScheduler
from transformers import CLIPVisionModelWithProjection
from diffusers.utils import load_image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"h94/IP-Adapter",
subfolder="models/image_encoder",
dtype=torch.float16,
)Muat model dasar, penjadwal, dan IP-Adapter berikut.
- ip-adapter-plus_sdxl_vit-h menggunakan patch embeddings dan image encoder ViT-H
- ip-adapter-plus-face_sdxl_vit-h menggunakan patch embeddings dan image encoder ViT-H, tetapi dikondisikan pada gambar wajah yang dipotong
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16,
image_encoder=image_encoder,
)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name=["ip-adapter-plus_sdxl_vit-h.safetensors", "ip-adapter-plus-face_sdxl_vit-h.safetensors"]
)
pipeline.set_ip_adapter_scale([0.7, 0.3])
# enable_model_cpu_offload to reduce memory usage
pipeline.enable_model_cpu_offload()Muat sebuah gambar dan folder yang berisi gambar-gambar dengan gaya tertentu untuk diterapkan.
face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
style_folder = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy"
style_images = [load_image(f"{style_folder}/img{i}.png") for i in range(10)]
Lewatkan gambar gaya dan wajah sebagai daftar ke ip_adapter_image.
generator = torch.Generator(device="cpu").manual_seed(0)
pipeline(
prompt="wonderwoman",
ip_adapter_image=[style_images, face_image],
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]
Kontrol struktural
Untuk kontrol struktural, kombinasikan IP-Adapter dengan ControlNet yang dikondisikan pada peta kedalaman, peta tepi, estimasi pose, dan lainnya.
Contoh di bawah ini memuat checkpoint ControlNetModel yang dikondisikan pada peta kedalaman dan menggabungkannya dengan IP-Adapter.
import torch
from diffusers.utils import load_image
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/control_v11f1p_sd15_depth",
dtype=torch.float16
)
pipeline = StableDiffusionControlNetPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
controlnet=controlnet,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="models",
weight_name="ip-adapter_sd15.bin"
)Lewatkan peta kedalaman dan gambar IP-Adapter ke pipeline.
depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
ip_adapter_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/statue.png")
pipeline(
prompt="best quality, high quality",
image=depth_map,
ip_adapter_image=ip_adapter_image,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]
Kontrol gaya dan tata letak
Untuk kontrol gaya dan tata letak, kombinasikan IP-Adapter dengan InstantStyle. InstantStyle memisahkan gaya (warna, tekstur, kesan keseluruhan) dari konten dan menerapkan gaya hanya pada blok yang spesifik untuk gaya, sehingga area konten tetap utuh. Hal ini memberikan konsistensi gaya yang lebih kuat dan kontrol tata letak yang lebih jelas.
Aktifkan IP-Adapter hanya pada lapisan tertentu dengan set_ip_adapter_scale(). Contoh di bawah ini mengaktifkannya pada block_2 (layout) bagian down dan block_0 (style) bagian up dari model.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
scale = {
"down": {"block_2": [0.0, 1.0]},
"up": {"block_0": [0.0, 1.0, 0.0]},
}
pipeline.set_ip_adapter_scale(scale)Muat gambar gaya dan hasilkan gambar.
style_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg")
pipeline(
prompt="a cat, masterpiece, best quality, high quality",
ip_adapter_image=style_image,
negative_prompt="text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
guidance_scale=5,
).images[0]
Gambar-gambar di bawah ini membandingkan aktivasi gaya-only InstantStyle (hanya block_0) dengan mengaktifkan IP-Adapter di semua lapisan. Pengaturan semua lapisan biasanya mengikuti prompt gambar lebih kuat dan dapat mengurangi keragaman. Pilih skala gaya-only jika Anda menginginkan perilaku pelestarian tata letak dari InstantStyle.
Anda tidak perlu menentukan semua lapisan dalam kamus
scale. Lapisan yang tidak disertakan diatur ke 0, yang berarti IP-Adapter dinonaktifkan.
scale = {
"up": {"block_0": [0.0, 1.0, 0.0]},
}
pipeline.set_ip_adapter_scale(scale)
pipeline(
prompt="a cat, masterpiece, best quality, high quality",
ip_adapter_image=style_image,
negative_prompt="text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
guidance_scale=5,
).images[0]
Generasi instan
Gabungkan IP-Adapter dengan LCM LoRA untuk generasi beberapa langkah.
import torch
from diffusers import DiffusionPipeline, LCMScheduler
from diffusers.utils import load_image
pipeline = DiffusionPipeline.from_pretrained(
"sd-dreambooth-library/herge-style",
dtype=torch.float16
)
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="models",
weight_name="ip-adapter_sd15.bin"
)
pipeline.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")
pipeline.scheduler = LCMScheduler.from_config(pipeline.scheduler.config)
pipeline.enable_model_cpu_offload()
pipeline.set_ip_adapter_scale(0.4)
ip_adapter_image = load_image("https://user-images.githubusercontent.com/24734142/266492875-2d50d223-8475-44f0-a7c6-08b51cb53572.png")
pipeline(
prompt="herge_style woman in armor, best quality, high quality",
ip_adapter_image=ip_adapter_image,
num_inference_steps=4,
guidance_scale=1,
).images[0]