通常通过镜像网站下载https://hf-mirror.com/。
在链接页面有介绍方法,对于不大的模型可以直接下载。这里介绍比较常用且方便的下载方法。
安装(huggingface_hub、hf_transfer安装可以使用-i命令从镜像网站下载)
pip install -U huggingface_hub hf_transfer
基本命令(每次打开远程链接都要输入)
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_ENDPOINT=https://hf-mirror.com
下载模型(下载NousResearch/Llama-2-13b-chat-hf,放在当前目录文件夹Llama-2-13b-chat-hf下)
huggingface-cli download --resume-download NousResearch/Llama-2-13b-chat-hf --local-dir Llama-2-13b-chat-hf --local-dir-use-symlinks False
携带参数下载
huggingface-cli download --token hf_*** --resume-download meta-llama/Llama-2-7b-hf --local-dir Llama-2-7b-hf --local-dir-use-symlinks False
对于下载中断的问题,由于huggingface-cli本身能断点重传,但下载界面关闭后只能重新下载,因而采用选择性下载
huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "model-00013-of-00015.safetensors"
huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "pytorch_model*.bin"
其他用法
huggingface-cli download --helps
usage: huggingface-cli <command> [<args>] download [-h] [--repo-type {model,dataset,space}]
[--revision REVISION] [--include [INCLUDE ...]]
[--exclude [EXCLUDE ...]] [--cache-dir CACHE_DIR]
[--local-dir LOCAL_DIR]
[--local-dir-use-symlinks {auto,True,False}]
[--force-download] [--resume-download]
[--token TOKEN]
[--quiet]
repo_id [filenames ...]
positional arguments: repo_id ID of the repo to download from (e.g. `username/repo-name`).
filenames Files to download (e.g. `config.json`, `data/metadata.jsonl`).
options:
-h, --help show this help message and exit
--repo-type {model,dataset,space}
Type of repo to download from (e.g. `dataset`).
--revision REVISION An optional Git revision id which can be a branch name, a tag, or a commit hash.
--include [INCLUDE ...]
Glob patterns to match files to download.
--exclude [EXCLUDE ...]
Glob patterns to exclude from files to download.
--cache-dir CACHE_DIR
Path to the directory where to save the downloaded files.
--local-dir LOCAL_DIR
If set, the downloaded file will be placed under this directory either as a symlink
(default) or a regular file. Check out
https://huggingface.co/docs/huggingface_hub/guides/download#download-files-to-local-
folder for more details.
--local-dir-use-symlinks {auto,True,False}
To be used with `local_dir`. If set to 'auto', the cache directory will be used and
the file will be either duplicated or symlinked to the local directory depending on
its size. It set to `True`, a symlink will be created, no matter the file size. If
set to `False`, the file will either be duplicated from cache (if already exists) or
downloaded from the Hub and not cached.
--force-download If True, the files will be downloaded even if they are already cached.
--resume-download If True, resume a previously interrupted download.
--token TOKEN A User Access Token generated from https://huggingface.co/settings/tokens
--quiet If True, progress bars are disabled and only the path to the download files is
printed.