【复现】AnimateDiff复现过程记录

出现了非常多的问题，痛苦

1.首先是下载基础模型

手动下载所有模型并创建目录放到对应的地方

文件路径问题是最简单也是最麻烦的，简单是只要把原来指向huggingface和openai之类要联网下载的路径改为存放文件的本地路径就行，一定要是绝对路径，不带文件名。麻烦是因为要改的地方又多又杂，根据报错挨个改吧

 运行：

python -m scripts.animate --config configs/prompts/2-Lyriel.yaml

2.报错：RuntimeError: don't know how to restore data location of torch.storage.UntypedStorage (tagged with gpu)

Traceback (most recent call last):
  File "/opt/conda/envs/animatediff/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/animatediff/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/AnimateDiff/scripts/animate.py", line 124, in 
    main(args)
  File "/data/AnimateDiff/scripts/animate.py", line 62, in main
    pipeline = load_weights(
  File "/data/AnimateDiff/animatediff/utils/util.py", line 109, in load_weights
    motion_module_state_dict = torch.load(motion_module_path, map_location="gpu")
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 789, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 1131, in _load
    result = unpickler.load()
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 1101, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 1083, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 1052, in restore_location
    return default_restore_location(storage, map_location)
  File "/opt/conda/envs/animatediff/lib/python3.10/site-packages/torch/serialization.py", line 218, in default_restore_location
    raise RuntimeError("don't know how to restore data location of "
RuntimeError: don't know how to restore data location of torch.storage.UntypedStorage (tagged with gpu)

解决方法：

【报错记录】RuntimeError: don‘t know how to restore data location of torch.FloatStorage (tagged with CPU)-CSDN博客

model.load_state_dict(torch.load(args.pre_train, map_location='CPU'),strict=True)

改为：

model.load_state_dict(torch.load(args.pre_train, map_location={'0':'CPU'}), strict=True)

问题得以解决。

3.UnboundLocalError: local variable 'dreambooth_state_dict' referenced before assignment

load dreambooth model from /data/AnimateDiff/models/DreamBooth_LoRA
Traceback (most recent call last):
  File "/opt/conda/envs/animatediff/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/animatediff/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/AnimateDiff/scripts/animate.py", line 124, in 
    main(args)
  File "/data/AnimateDiff/scripts/animate.py", line 62, in main
    pipeline = load_weights(
  File "/data/AnimateDiff/animatediff/utils/util.py", line 129, in load_weights
    converted_vae_checkpoint = convert_ldm_vae_checkpoint(dreambooth_state_dict, animation_pipeline.vae.config)
UnboundLocalError: local variable 'dreambooth_state_dict' referenced before assignment

怎么会有这种错误，我又没改函数定义。。这应该是作者应该操心的呀

解决方法：改了一下函数声明的位置

啊啊啊！终于run起来了吗！

不过只是app.py，接下来解决test

python -m scripts.animate --config configs/prompts/2-Lyriel.yaml

4.KeyError:"encoder.conv_in.weight"

报错在convert_from_ckpt.py：

    new_checkpoint["encoder.conv_in.weight"] = vae_state_dict["encoder.conv_in.weight"]

本来以为是文件损坏的问题，又重新下了vae的权重文件

仍然报错，检查发现是yaml里该写成文件路径的地方写的是文件夹路径，改过来就好了。说起来简单但报错的原因很不好找，搜了半天以为是啥不兼容问题，卸了装装了卸的，最后在官网找了下参考的yaml配置才发现原来是路径没写对。。找不到文件。。

animate.py：

pipeline,
                # motion module
                motion_module_path         = motion_module,
                motion_module_lora_configs = model_config.get("motion_module_lora_configs", []),
                # image layers
                dreambooth_model_path      = model_config.get("dreambooth_path", "/data/AnimateDiff/models/DreamBooth_LoRA/lyriel_v16.safetensors"),
                # lora_model_path            = model_config.get("lora_model_path", "/data/AnimateDiff/models/MotionLoRA/v2_lora_PanLeft.ckpt"),
                lora_alpha                 = model_config.get("lora_alpha", 0.8),
            ).to("cuda")

（这里 lora_model_path 要求放safetensors文件，但官方给的都是ckpt文件，不知道该放啥，就注释掉了，不影响运行）

以及config

好了， run起来了

结果：

prompt:

   0 - "dark shot, epic realistic, portrait of halo, sunglasses, blue eyes, tartan scarf, white hair by atey ghailan, by greg rutkowski, by greg tocchini, by james gilleard, by joe fenton, by kaethe butcher, gradient yellow, black, brown and magenta color scheme, grunge aesthetic!!! graffiti tag wall background, art by greg rutkowski and artgerm, soft cinematic light, adobe lightroom, photolab, hdr, intricate, highly detailed, depth of field, faded, neutral colors, hdr, muted colors, hyperdetailed, artstation, cinematic, warm lights, dramatic light, intricate details, complex background, rutkowski, teal and orange" 

 

 

 1 - "A forbidden castle high up in the mountains, pixel art, intricate details2, hdr, intricate details, hyperdetailed5, natural skin texture, hyperrealism, soft light, sharp, game art, key visual, surreal"

  2  - "dark theme, medieval portrait of a man sharp features, grim, cold stare, dark colors, Volumetric lighting, baroque oil painting by Greg Rutkowski, Artgerm, WLOP, Alphonse Mucha dynamic lighting hyperdetailed intricately detailed, hdr, muted colors, complex background, hyperrealism, hyperdetailed, amandine van ray"

 3   - "As I have gone alone in there and with my treasures bold, I can keep my secret where and hint of riches new and old. Begin it where warm waters halt and take it in a canyon down, not far but too far to walk, put in below the home of brown."