It borrows from previous work I did in attempting to train a model for split-screen animations to generate 360 views of characters. With the advent of

(Beta) Multi-frame Video rendering for StableDiffusion

submited by

Style Pass

2023-06-03 05:00:03

It borrows from previous work I did in attempting to train a model for split-screen animations to generate 360 views of characters. With the advent of ControlNet I found that far more could be done by using the split-screen/film reel technique and so I'm trying to workout how to best utilize it.

You can set denoising strength to 1 on here and just use ControlNet to guide it to whatever pose and composition you need for the first frame of your animation. Take your time to get a good result since this first frame will guide the rest of the animation.

The Script automatically uses the first frame in order to create the second frame by creating a split-screen iput image, similar to a horizontal film reel. The previous frame becomes the base image for frame 2 before the img2img process is run (aka loopback).

Note that almost every animation shown here has denoising strength set to 1.0 and so the loopback doesn't actually occur. Looping back with a strength of 0.75-0.8 does seem to make it a little more stable, but be sure to enable color correction if you have denoising below 1.0 otherwise there's color drift.