导入在解释器启动时不可用的Python模块

Question

我有一段代码，看起来是这样的：

import sys, importlib

try:
    import pip
except ImportError:
    raise ImportError("Please install pip")

reqs = ["sh", "vcstools"]
for req in reqs:
    sys.stdout.write("checking for %s..." % req)
    try:
        importlib.import_module(req)
        print("found")
    except ImportError:
        print("missing!")
        print("Installing %s..." % req)
        pip.main(['install', '--user', req])
        #importlib.invalidate_caches() python3 only
        # XXX fails
        importlib.import_module(req)
        new_mods = True

    locs = locals()
    locs[req] = sys.modules[req]

print(sh, vcstools)

这段代码的目的是在运行时下载一些（大致的）依赖项并导入它们（是的，我知道，它没有考虑版本号，我本可以使用virtualenv等等。这些是另外的问题）。

如果我们运行这段代码（在~/.local目录下没有安装任何东西的情况下），我们会看到以下结果：

checking for sh...missing!
Installing sh...
Downloading/unpacking sh
....
Successfully installed sh
Cleaning up...
Traceback (most recent call last):
  File "test.py", line 20, in <module>
    importlib.import_module(req)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named sh

所以我们看到，在安装了sh模块后，我们不能立即导入它。如果我们重新运行这个脚本，它会成功，因为它能找到上一次运行时安装的sh。

我觉得奇怪的是，对于vcstools，它在同一次运行中就能顺利安装和导入。为什么会这样？sh有什么特别之处吗？

这是第二次运行的完整输出。注意我们从上一次运行中获取了sh，然后我们安装了vcstools并且没有错误地导入它：

checking for sh...found
checking for vcstools...missing!
Installing vcstools...
Downloading/unpacking vcstools
  Downloading vcstools-0.1.33.tar.gz
  Running setup.py egg_info for package vcstools

Downloading/unpacking pyyaml (from vcstools)
  Downloading PyYAML-3.11.tar.gz (248Kb): 248Kb downloaded
  Running setup.py egg_info for package pyyaml

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python2.7/site-packages (from vcstools)
Installing collected packages: vcstools, pyyaml
  Running setup.py install for vcstools

  Running setup.py install for pyyaml
    checking if libyaml is compilable
    cc -pthread -fno-strict-aliasing -O2 -pipe -DNDEBUG -O2 -pipe -fPIC -fPIC -I/usr/local/include/python2.7 -c build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c -o build/temp.openbsd-5.5-amd64-2.7/check_libyaml.o
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c: In function 'main':
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: for each function it appears in.)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)

    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Successfully installed vcstools pyyaml
Cleaning up...
(<module 'sh' (built-in)>, <module 'vcstools' from '/home/edd/.local/lib/python2.7/site-packages/vcstools/__init__.pyc'>)

这是在OpenBSD上运行的Python-2.7。

谢谢

补充：我刚注意到new_mods这一行是多余的。我会把它留在那里，以免输出的行号出现偏差。

Answer 1

这可能是你第一次尝试把包安装到 ~/.local/... 这个地方。也就是说，只有当你用 pip.main(['install', '--user', req]) 这个命令时，~/.local/... 这个文件夹才会被创建。

如果在Python启动时 ~/.local 这个文件夹不存在，它就不会被加入到系统路径中（sys.path），这样模块就找不到了（即使你用 importlib.invalidate_caches() 也没用）。

所以你需要把这个路径添加到系统路径中，或者按照这个回答的方法重新加载系统路径，这样就能正常工作了……

我的解决方案：

import pip
import importlib
import sys

def pip_import(module_name, pip_package=None):
    try:
        mod = importlib.import_module(module_name)
    except ImportError:
        pip.main(['install', '--user', pip_package if pip_package else module_name])

        # If the install path did not exist on start up it has 
        # to be added to sys.path
        if not pip.locations.user_site in sys.path:
            sys.path.append(pip.locations.user_site)

        mod = importlib.import_module(module_name)

    # Add the imported module to the global namespace so it
    # can be used just like `import module`
    globals()[module_name] = mod