Skip to content

feat: AnimateDiff img2video via --init-img#1789

Merged
leejet merged 1 commit into
leejet:masterfrom
fszontagh:img2animatediff
Jul 16, 2026
Merged

feat: AnimateDiff img2video via --init-img#1789
leejet merged 1 commit into
leejet:masterfrom
fszontagh:img2animatediff

Conversation

@fszontagh

@fszontagh fszontagh commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds img2video mode to AnimateDiff (SD 1.5): a pre-rendered init image is VAE-encoded, replicated across N frames on the batch axis, and used as the starting latent for the diffusion loop. Frame 0 preserves the init image's quality; subsequent frames drift slightly via the motion module.

Follows PR #1784 (AnimateDiff SD 1.5 v2/v3 motion modules).

Related Issue / Discussion

None.

Additional Information

Sample

Left: init image rendered separately with -M img_gen. Right: 8-frame -M vid_gen -i init.png --strength 0.75 output. Character identity, dress, background, sharpness all preserved. Motion module adds subtle drift (hair, lighting, breathing).

Verification command

sd-cli -M img_gen \
    --model realisticVisionV60B1.safetensors \
    -p "..." -H 512 -W 512 -s 42 --steps 25 \
    -o init.png

sd-cli -M vid_gen \
    --model realisticVisionV60B1.safetensors \
    --motion-module mm_sd15_v3.safetensors \
    -i init.png --strength 0.75 \
    --cfg-scale 7.0 --sampling-method euler --scheduler karras \
    -H 512 -W 512 --video-frames 8 --steps 25 -s 42 \
    -p "..." -o out.avi

How it works

Two changes in src/stable-diffusion.cpp:

  1. prepare_image_generation_latents: when animatediff_num_frames > 1 and init_image_tensor is non-empty, replicate the single-frame encoded init latent across the N-frame batch axis via sd::ops::slice_assign. When no init image is provided, the existing T2V zero-fill path runs unchanged.
  2. generate_animatediff_video: plumb sd_vid_gen_params->init_image into img_gen_params.init_image so -M vid_gen -i <path> reaches the latent preparation path.

13 insertions + 2 deletions in src/stable-diffusion.cpp. No public API changes.

Checklist

@leejet
leejet merged commit 7717e82 into leejet:master Jul 16, 2026
12 checks passed
@fszontagh
fszontagh deleted the img2animatediff branch July 16, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants