【Python】Python3.7.3 - sys.path 模块搜索路径详解

系统参考

[tony@tony-controller bin]$ cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

[tony@tony-controller bin]$ uname -a
Linux tony-controller 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[tony@tony-controller bin]$ pwd
/home/tony/python37/bin

sys.path - 模块搜索路径

当import某个模块时，Python会使用sys.path中指定目录，按顺序搜索导入的模块。如果没有找到，则汇报模块找不到错误。后面会介绍控制搜索目录，即sys.path变量的选项与方法。

[tony@tony-controller bin]$ ./python3 ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
]
...

# numpy存在，可以使用numpy.__file__属性显示numpy所在目录
# nonexist不存在，Python汇报模块找不到。
[tony@tony-controller bin]$ ./python3
Python 3.7.3 (default, Apr 25 2019, 09:22:47)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__file__
'/home/tony/.local/lib/python3.7/site-packages/numpy/__init__.py'
>>> import nonexist
Traceback (most recent call last):
  File "", line 1, in <module>
ModuleNotFoundError: No module named 'nonexist'
>>>

用户站点目录(user site directory)

权威参考文档是 PEP370：https://www.python.org/dev/peps/pep-0370/
主要是为用户提供一种机制，可以在自己的账户目录中安装自己专用的本地Python包。

在Unix上，用户站点目录位于$HOME/.local/lib/python/site-packages/中，Windows上也定义了相应的目录，参看下例：

Unix (including Mac OS X)
~/.local/lib/python2.6/site-packages
Windows
%APPDATA%/Python/Python26/site-packages

安装包到用户站点目录

在安装包时，使用pip install --user选项。

# 安装numpy到用户站点目录
[tony@tony-controller bin]$ ./pip install --user numpy
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/bb/76/24e9f32c78e6f6fb26cf2596b428f393bf015b63459468119f282f70a7fd/numpy-1.16.3-cp37-cp37m-manylinux1_x86_64.whl (17.3MB)
     |████████████████████████████████| 17.3MB 4.7MB/s
Installing collected packages: numpy
Successfully installed numpy-1.16.3

[tony@tony-controller bin]$ ls -ld ~/.local/lib/python3.7/site-packages/numpy*
drwxrwxr-x. 18 tony tony 4096 Apr 25 11:04 /home/tony/.local/lib/python3.7/site-packages/numpy
drwxrwxr-x.  2 tony tony  111 Apr 25 11:04 /home/tony/.local/lib/python3.7/site-packages/numpy-1.16.3.dist-info

# 确认numpy来自于用户站点目录
[tony@tony-controller bin]$ ./python3
Python 3.7.3 (default, Apr 25 2019, 09:22:47)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__file__
'/home/tony/.local/lib/python3.7/site-packages/numpy/__init__.py'

第三方站点目录

第三方站点目录位于$PATHHOME/lib/python/site-packages/，默认情况下，通过pip命令（不指定pip install --user选项）安装的第三方包，都会安装在第三方站点目录中。

当向第三方站点目录安装包时，如果该包已经存在于用户站点目录中，pip则会通知你，你的想法已经被满足了。我怀疑反之亦然，但是没做实验。
由此可见，不会有同名的包，既位于用于站点目录也位于第三方站点目录中。所以不用担心这两个站点目录的搜索顺序。
参看下例：

[tony@tony-controller bin]$ ./pip install numpy
Requirement already satisfied: numpy in /home/tony/.local/lib/python3.7/site-packages (1.16.3)

site.py介绍

site.py位于$PYTHONHOME/lib/python3.7/目录下。这个文件负责将第三方包里的模块的搜索路径添加到sys.path变量中。单独执行这个脚本时，可以打印出一些有用的信息（参考下面的例子）。

[tony@tony-controller bin]$ ls -l ../lib/python3.7/site.py
-rw-r--r--. 1 tony tony 21649 Apr 25 09:24 ../lib/python3.7/site.py

# 从这个例子中可以看到，USER_BASE与USER_SITE都存在，
# USER_SITE被添加到了sys.path中
#
# 
[tony@tony-controller bin]$ ./python3 ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: True

python启动选项

Python有一些选项可以控制Python启动时，如何设置模块搜索变量sys.path。

-E: 忽略PYTHON*环境变量（例如PYTHONPATH）
-s（小写）：不添加用户站点目录(user site directory)到sys.path中，等价于PYTHONNOUSERSITE
-S（大写）：Python启动时，不执行import site，即不添加用户站点目录与第三方站点目录到sys.path中；看起来-S暗含着-s选项，反之不然。

# -E选项仅忽略环境变量，不影响site导入。
[tony@tony-controller bin]$ ./python3 -E ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: True
[tony@tony-controller bin]$

# -s选项仅不导入用户站点目录，但是第三方站点目录依然导入
[tony@tony-controller bin]$ ./python3 -s ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/python37/lib/python3.7/site-packages',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: False

# -S选项既不导入用户站点目录，也不导入第三方站点目录
[tony@tony-controller bin]$ ./python3 -S ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: None

# -S与-s选项连用，等价于只使用-S选项
[tony@tony-controller bin]$ ./python3 -s -S ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: None

# Python初始化时，忽略环境变量，不添加用户站点目录与第三方站点目录
[tony@tony-controller bin]$ ./python3 -E -s -S ../lib/python3.7/site.py
sys.path = [
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: None

PYTHONPATH环境变量

Python在启动时，

• 如果没有指定-E选项，则默认会把PYTHONPATH环境变量中指定的所有目录（不检查目录是否存在）都添加到sys.path变量中，而且放置在站点目录之前；
• 如果使用-E选项，则PYTHONPATH环境变量会被忽略。

# PYTHONPATH环境变量中指定的两个目录（nova-dev与nova）都被添加到sys.path中
[tony@tony-controller bin]$ PYTHONPATH=/home/tony/PycharmProjects/nova-dev:/home/tony/PycharmProjects/nova ./python3 -m site
sys.path = [
    '/home/tony/python37/bin',
    '/home/tony/PycharmProjects/nova-dev',
    '/home/tony/PycharmProjects/nova',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: True

# -E 选项导致PYTHONPATH环境变量被忽略
[tony@tony-controller bin]$ PYTHONPATH=/home/tony/PycharmProjects/nova-dev:/home/tony/PycharmProjects/nova ./python3 -E -m site
sys.path = [
    '/home/tony/python37/bin',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: True

使用路径配置文件（path configuration file）

在站点目录下，包括用户站点目录与第三方站点目录，例如/home/tony/python37/lib/python3.7/site-packages或者/home/tony/.local/lib/python3.7/site-packages，可以创建一些后缀为.pth的文本文件，例如foo.pth与bar.pth，用于配置Python的模块查找路径。

这些文件具有如下的格式，注意文件中的目录是相对路径，同时假设site-packages目录下有foo目录与bar目录，但是bletch目录并不存在：
$ cat foo.pth

# foo package configuration
foo
bar
bletch

$ cat bar.pth

# bar package configuration
bar

在Python启动时，如果导入了站点目录，即没有使用（-S或-s选项），则site.py脚本会列举所有的*.pth文件，将其中的目录附加到sys.path的结尾处，添加规则如下：

• 新加入的目录按照目录名大小写敏感顺序排序，例如bar在foo之前。
• 配置文件中定义的目录但是文件系统中不存在的目录，不会被添加到sys.path中

# 位于站点目录下
[tony@tony-controller site-packages]$ pwd
/home/tony/python37/lib/python3.7/site-packages

# 路径配置文件
[tony@tony-controller site-packages]$ ls foo.pth bar.pth
bar.pth  foo.pth

# 配置文件中定义的路径，foo与bar存在；bletch不存在
[tony@tony-controller site-packages]$ ls -ld foo bar bletch
ls: cannot access bletch: No such file or directory
drwxrwxr-x. 2 tony tony 6 Apr 26 10:17 bar
drwxrwxr-x. 2 tony tony 6 Apr 26 10:17 foo

# bar与foo被添加到了sys.path；但是bletch没有（因为不存在）
[tony@tony-controller bin]$ PYTHONPATH=/home/tony/PycharmProjects/nova-dev:/home/tony/PycharmProjects/nova ./python3 -m site
sys.path = [
    '/home/tony/python37/bin',
    '/home/tony/PycharmProjects/nova-dev',
    '/home/tony/PycharmProjects/nova',
    '/home/tony/python37/lib/python37.zip',
    '/home/tony/python37/lib/python3.7',
    '/home/tony/python37/lib/python3.7/lib-dynload',
    '/home/tony/.local/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages',
    '/home/tony/python37/lib/python3.7/site-packages/bar',
    '/home/tony/python37/lib/python3.7/site-packages/foo',
]
USER_BASE: '/home/tony/.local' (exists)
USER_SITE: '/home/tony/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: True