You can try running it on the HuggingFace 🤗 space, or on the Colab link. They both have GPU usage limits, so you can play around with both and see

ControlNet Game of Life

submited by
Style Pass
2024-06-18 11:00:09

You can try running it on the HuggingFace 🤗 space, or on the Colab link. They both have GPU usage limits, so you can play around with both and see what works for you.

Fundamentally, we are using stable diffusion to generate the images. However, vanilla stable diffusion doesn’t allow for the preservation of the grid of cells, especially not as we iterate through the game.

Instead, we use a variant of stable diffusion called ControlNet. ControlNet has the nice property that you can feed it both a start image as well as a “control image”, which encodes the general outlines of the shape you want to preserve.

In practice, this can be done using segmentation maps, canny edges, etc. With it, we can preserve the shapes while letting the model generate the fill. Here, ControlNet is used to map a painting to another painting while preserving the original structure.

Shortly after ControlNet, a fun use-case people thought of was to finetune ControlNet to preserve the semantics of QR Codes. Think of it; if you can preserve the shapes of the tiles enough, you can generate actually fun QR codes, it’s so simple yet powerful:

Leave a Comment