diffusers-Tasks

https://huggingface.co/docs/diffusers/using-diffusers/unconditional_image_generationhttps://huggingface.co/docs/diffusers/using-diffusers/unconditional_image_generation1.Unconditional image generation

无条件图像生成是一个相对简单的任务。模型仅生成图像，没有任何额外的上下文，如文本或图像，这些生成的图像类似于它所训练的训练数据。

from diffusers import DiffusionPipeline

generator = DiffusionPipeline.from_pretrained("anton-l/ddpm-butterflies-128", use_safetensors=True)

generator.to("cuda")
image = generator().images[0]

2.Conditional image generation

条件图像生成允许从文本提示生成图像。文本被转换为嵌入向量，这些向量被用来条件模型从噪声中生成图像。

from diffusers import DiffusionPipeline

generator = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True)

generator.to("cuda")
image = generator("An image of a squirrel in Picasso style").images[0]

3.Text-guided image-to-image generation

StableDiffusionImg2ImgPipeline可以输入文本提示和一个初始图像来条件生成新的图像。

import torch
import requests
from PIL import Image
from io import BytesIO
from diffusers import StableDiffusionImg2ImgPipeline

device = "cuda"
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "nitrosocke/Ghibli-Diffusion", torch_dtype=torch.float16, use_safetensors=True
).to(device)

url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"

response = requests.get(url)
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image.thumbnail((768, 768))

prompt = "ghibli style, a fantasy landscape with castles"
generator = torch.Generator(device=device).manual_seed(1024)
image = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5, generator=generator).images[0]

from diffusers import LMSDiscreteScheduler

lms = LMSDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.scheduler = lms
generator = torch.Generator(device=device).manual_seed(1024)
image = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5, generator=generator).images[0]

strength是一个介于0-1之间的值，控制添加到输入图像上的噪声量，接近1会在语义上输出和输入不一致的图像。

4.Text-guided image-inpainting

StableDiffusionInpaintPipeline可以提供mask和文本提示来编辑图像的特定部分。

import PIL
import requests
import torch
from io import BytesIO

from diffusers import StableDiffusionInpaintPipeline

pipeline = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)
pipeline = pipeline.to("cuda")

def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")


img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url).resize((512, 512))
mask_image = download_image(mask_url).resize((512, 512))

prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image).images[0]

5.Text-guided depth-to-image generation