LTX-Video

LTX-Video هو محوّل انتشار مصمم لتوليد فيديوهات عالية الدقة بسرعة وفي الوقت الفعلي من النصوص والصور. الميزة الرئيسية في LTX-Video هي Video-VAE. تتمتع Video-VAE بنسبة ضغط أعلى من البكسل إلى الكمون (1:192) مما يتيح معالجة بيانات الفيديو بكفاءة أكبر وسرعة توليد أسرع. لدعم التفاصيل الدقيقة ومنع فقدانها أثناء التوليد، يقوم فك تشفير Video-VAE بتحويل الكمون إلى بكسل و خطوة إزالة الضوضاء الأخيرة.

يمكنك العثور على جميع نقاط التفتيش الأصلية لنموذج LTX-Video تحت منظمة Lightricks.

انقر على نماذج LTX-Video في الشريط الجانبي الأيمن لمزيد من الأمثلة على مهام توليد الفيديو الأخرى.

يُبيّن المثال أدناه كيفية توليد فيديو مُحسَّن للذاكرة أو لسرعة الاستدلال.

الذاكرة

سرعة الاستدلال

ملاحظات

راجع الإعدادات الموصى بها التالية للتوليد من مستودع LTX-Video.
- النوع الموصى به لـ dtype في المحوّل (transformer) وVAE ومشفّر النص هو torch.bfloat16. كما يمكن أن يكون VAE ومشفّر النص من النوع torch.float32 أو torch.float16.
- لإصدارات LTX-Video المُصفاة بالتوجيه، اضبط guidance_scale على 1.0. أما guidance_scale لأي نموذج آخر فيجب ضبطه على قيمة أعلى، مثل 5.0، لضمان جودة توليد جيدة.
- لأنواع VAE الواعية بخطوة الزمن (LTX-Video 0.9.1 وما بعده)، اضبط decode_timestep على 0.05 وimage_cond_noise_scale على 0.025.
- بالنسبة للتحويرات التي تدعم الاستيفاء بين صور وفيديوهات شرطية متعددة (LTX-Video 0.9.5 وما بعده)، استخدم صوراً وفيديوهات مشابهة للحصول على أفضل النتائج. قد يؤدي الانحراف عن مدخلات الشرط إلى انتقالات مفاجئة في الفيديو المولّد.

يتضمن LTX-Video 0.9.7 مُكبِّرًا كامنًا مكانيًا ومحوّلًا (Transformer) يحتوي على 13 مليار معامل. أثناء الاستدلال، يتم توليد فيديو بدقة منخفضة بسرعة أولاً، ثم يُكبَّر ويُحسَّن.

عرض كود مثال

import torch
from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
from diffusers.utils import export_to_video, load_video

pipeline = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-dev", dtype=torch.bfloat16)
pipeline_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipeline.vae, dtype=torch.bfloat16)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"
pipe_upsample.to("cuda")
pipeline.vae.enable_tiling()

def round_to_nearest_resolution_acceptable_by_vae(height, width):
    height = height - (height % pipeline.vae_temporal_compression_ratio)
    width = width - (width % pipeline.vae_temporal_compression_ratio)
    return height, width

video = load_video(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cosmos/cosmos-video2world-input-vid.mp4"
)[:21]  # only use the first 21 frames as conditioning
condition1 = LTXVideoCondition(video=video, frame_index=0)

prompt = """
The video depicts a winding mountain road covered in snow, with a single vehicle 
traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. 
The landscape is characterized by rugged terrain and a river visible in the distance. 
The scene captures the solitude and beauty of a winter drive through a mountainous region.
"""
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
expected_height, expected_width = 768, 1152
downscale_factor = 2 / 3
num_frames = 161

# 1. Generate video at smaller resolution
# Text-only conditioning is also supported without the need to pass `conditions`
downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)
latents = pipeline(
    conditions=[condition1],
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=downscaled_width,
    height=downscaled_height,
    num_frames=num_frames,
    num_inference_steps=30,
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=5.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="latent",
).frames

# 2. Upscale generated video using latent upsampler with fewer inference steps
# The available latent upsampler upscales the height/width by 2x
upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2
upscaled_latents = pipe_upsample(
    latents=latents,
    output_type="latent"
).frames

# 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)
video = pipeline(
    conditions=[condition1],
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=upscaled_width,
    height=upscaled_height,
    num_frames=num_frames,
    denoise_strength=0.4,  # Effectively, 4 inference steps out of 10
    num_inference_steps=10,
    latents=upscaled_latents,
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=5.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="pil",
).frames[0]

# 4. Downscale the video to the expected resolution
video = [frame.resize((expected_width, expected_height)) for frame in video]

export_to_video(video, "output.mp4", fps=24)

نموذج LTX-Video 0.9.7 المُقطّر مُقطّر من حيث التوجيه (guidance) والخطوات الزمنية (timestep) لتسريع عملية التوليد. يتطلب ضبط guidance_scale على 1.0، ويجب ضبط num_inference_steps بين 4 و10 للحصول على جودة توليد جيدة. كما ينبغي استخدام الخطوات الزمنية المخصصة التالية للحصول على أفضل النتائج.

استدلال النموذج الأساسي للتحضير للتصعيد: [1000, 993, 987, 981, 975, 909, 725, 0.03].
التحجيم: [1000, 909, 725, 421, 0].

عرض كود مثال

import torch
from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
from diffusers.utils import export_to_video, load_video

pipeline = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-distilled", dtype=torch.bfloat16)
pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipeline.vae, dtype=torch.bfloat16)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"
pipe_upsample.to("cuda")
pipeline.vae.enable_tiling()

def round_to_nearest_resolution_acceptable_by_vae(height, width):
    height = height - (height % pipeline.vae_spatial_compression_ratio)
    width = width - (width % pipeline.vae_spatial_compression_ratio)
    return height, width

prompt = """
artistic anatomical 3d render, utlra quality, human half full male body with transparent 
skin revealing structure instead of organs, muscular, intricate creative patterns, 
monochromatic with backlighting, lightning mesh, scientific concept art, blending biology 
with botany, surreal and ethereal quality, unreal engine 5, ray tracing, ultra realistic, 
16K UHD, rich details. camera zooms out in a rotating fashion
"""
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
expected_height, expected_width = 768, 1152
downscale_factor = 2 / 3
num_frames = 161

# 1. Generate video at smaller resolution
downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)
latents = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=downscaled_width,
    height=downscaled_height,
    num_frames=num_frames,
    timesteps=[1000, 993, 987, 981, 975, 909, 725, 0.03],
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=1.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="latent",
).frames

# 2. Upscale generated video using latent upsampler with fewer inference steps
# The available latent upsampler upscales the height/width by 2x
upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2
upscaled_latents = pipe_upsample(
    latents=latents,
    adain_factor=1.0,
    output_type="latent"
).frames

# 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)
video = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=upscaled_width,
    height=upscaled_height,
    num_frames=num_frames,
    denoise_strength=0.999,  # Effectively, 4 inference steps out of 5
    timesteps=[1000, 909, 725, 421, 0],
    latents=upscaled_latents,
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=1.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="pil",
).frames[0]

# 4. Downscale the video to the expected resolution
video = [frame.resize((expected_width, expected_height)) for frame in video]

export_to_video(video, "output.mp4", fps=24)

يُشبه نموذج LTX-Video 0.9.8 المُقطّر (distilled) النسخة 0.9.7. فهو مُقطّر من حيث التوجيه (guidance) وخطوة الزمن (timestep)، ويمكن استخدام كود استنتاج مشابه لما هو موضح أعلاه. تتمثل تحسينات هذه النسخة في دعمها لتوليد مقاطع فيديو طويلة جدًا. بالإضافة إلى ذلك، تدعم استخدام خرائط النغمة (tone mapping) لتحسين جودة الفيديو المولّد باستخدام معامل tone_map_compression_ratio. يُوصى بالقيمة الافتراضية 0.6.

عرض كود مثال

import torch
from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
from diffusers.models.latent_upscaler import LTXLatentUpsamplerModel
from diffusers.utils import export_to_video, load_video

pipeline = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.8-13B-distilled", dtype=torch.bfloat16)
# TODO: Update the checkpoint here once updated in LTX org
upsampler = LTXLatentUpsamplerModel.from_pretrained("a-r-r-o-w/LTX-0.9.8-Latent-Upsampler", dtype=torch.bfloat16)
pipe_upsample = LTXLatentUpsamplePipeline(vae=pipeline.vae, latent_upsampler=upsampler).to(torch.bfloat16)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"
pipe_upsample.to("cuda")
pipeline.vae.enable_tiling()

def round_to_nearest_resolution_acceptable_by_vae(height, width):
    height = height - (height % pipeline.vae_spatial_compression_ratio)
    width = width - (width % pipeline.vae_spatial_compression_ratio)
    return height, width

prompt = """The camera pans over a snow-covered mountain range, revealing a vast expanse of snow-capped peaks and valleys.The mountains are covered in a thick layer of snow, with some areas appearing almost white while others have a slightly darker, almost grayish hue. The peaks are jagged and irregular, with some rising sharply into the sky while others are more rounded. The valleys are deep and narrow, with steep slopes that are also covered in snow. The trees in the foreground are mostly bare, with only a few leaves remaining on their branches. The sky is overcast, with thick clouds obscuring the sun. The overall impression is one of peace and tranquility, with the snow-covered mountains standing as a testament to the power and beauty of nature."""
# prompt = """A woman walks away from a white Jeep parked on a city street at night, then ascends a staircase and knocks on a door. The woman, wearing a dark jacket and jeans, walks away from the Jeep parked on the left side of the street, her back to the camera; she walks at a steady pace, her arms swinging slightly by her sides; the street is dimly lit, with streetlights casting pools of light on the wet pavement; a man in a dark jacket and jeans walks past the Jeep in the opposite direction; the camera follows the woman from behind as she walks up a set of stairs towards a building with a green door; she reaches the top of the stairs and turns left, continuing to walk towards the building; she reaches the door and knocks on it with her right hand; the camera remains stationary, focused on the doorway; the scene is captured in real-life footage."""
negative_prompt = "bright colors, symbols, graffiti, watermarks, worst quality, inconsistent motion, blurry, jittery, distorted"
expected_height, expected_width = 480, 832
downscale_factor = 2 / 3
# num_frames = 161
num_frames = 361

# 1. Generate video at smaller resolution
downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)
latents = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=downscaled_width,
    height=downscaled_height,
    num_frames=num_frames,
    timesteps=[1000, 993, 987, 981, 975, 909, 725, 0.03],
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=1.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="latent",
).frames

# 2. Upscale generated video using latent upsampler with fewer inference steps
# The available latent upsampler upscales the height/width by 2x
upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2
upscaled_latents = pipe_upsample(
    latents=latents,
    adain_factor=1.0,
    tone_map_compression_ratio=0.6,
    output_type="latent"
).frames

# 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)
video = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=upscaled_width,
    height=upscaled_height,
    num_frames=num_frames,
    denoise_strength=0.999,  # Effectively, 4 inference steps out of 5
    timesteps=[1000, 909, 725, 421, 0],
    latents=upscaled_latents,
    decode_timestep=0.05,
    decode_noise_scale=0.025,
    image_cond_noise_scale=0.0,
    guidance_scale=1.0,
    guidance_rescale=0.7,
    generator=torch.Generator().manual_seed(0),
    output_type="pil",
).frames[0]

# 4. Downscale the video to the expected resolution
video = [frame.resize((expected_width, expected_height)) for frame in video]

export_to_video(video, "output.mp4", fps=24)

يدعم LTX-Video نماذج LoRA من خلال load_lora_weights().

عرض مثال على الكود

import torch
from diffusers import LTXConditionPipeline
from diffusers.utils import export_to_video, load_image

pipeline = LTXConditionPipeline.from_pretrained(
    "Lightricks/LTX-Video-0.9.5", dtype=torch.bfloat16
)

pipeline.load_lora_weights("Lightricks/LTX-Video-Cakeify-LoRA", adapter_name="cakeify")
pipeline.set_adapters("cakeify")

# use "CAKEIFY" to trigger the LoRA
prompt = "CAKEIFY a person using a knife to cut a cake shaped like a Pikachu plushie"
image = load_image("https://huggingface.co/Lightricks/LTX-Video-Cakeify-LoRA/resolve/main/assets/images/pikachu.png")

video = pipeline(
    prompt=prompt,
    image=image,
    width=576,
    height=576,
    num_frames=161,
    decode_timestep=0.03,
    decode_noise_scale=0.025,
    num_inference_steps=50,
).frames[0]
export_to_video(video, "output.mp4", fps=26)

LTX-Video supports loading from single files, such as GGUF checkpoints, with loaders.FromOriginalModelMixin.from_single_file() or loaders.FromSingleFileMixin.from_single_file().

عرض مثال برمجي

import torch
from diffusers.utils import export_to_video
from diffusers import LTXPipeline, AutoModel, GGUFQuantizationConfig

transformer = AutoModel.from_single_file(
    "https://huggingface.co/city96/LTX-Video-gguf/blob/main/ltx-video-2b-v0.9-Q3_K_S.gguf"
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    dtype=torch.bfloat16
)
pipeline = LTXPipeline.from_pretrained(
    "Lightricks/LTX-Video",
    transformer=transformer,
    dtype=torch.bfloat16
)

Diffusers

LTX-Video

ملاحظات

LTXI2VLongMultiPromptPipeline

فئة diffusers.LTXI2VLongMultiPromptPipeline

__call__

encode_prompt

prepare_latents

vae_decode_tiled

LTXPipeline

فئة diffusers.LTXPipeline

__call__

encode_prompt

LTXImageToVideoPipeline

فئة diffusers.LTXImageToVideoPipeline

__call__

encode_prompt

LTXConditionPipeline

فئة diffusers.LTXConditionPipeline

__call__

add_noise_to_image_conditioning_latents

encode_prompt

trim_conditioning_sequence

LTXLatentUpsamplePipeline

فئة diffusers.LTXLatentUpsamplePipeline

__call__

adain_filter_latent

tone_map_latents

LTXPipelineOutput

فئة diffusers.pipelines.ltx.pipeline_output.LTXPipelineOutput

call

call

call

call

call