Python包/模块懒加载子模块

Question

今天遇到了一个有趣的情况：我需要在我们的代码中迁移一个模块，因为代码发生了变化。旧的 mynamespace.Document 将不再使用，我想确保迁移过程顺利进行，所以我打算用一个代码对象来替代它，这个对象会动态地导入正确的路径，并迁移相应的对象。

简单来说：

# instanciate a dynamic package, but do not load
# statically submodules
mynamespace.Document = SomeObject()
assert 'submodule' not in mynamespace.Document.__dict__

# and later on, when importing it, the submodule
# is built if not already available in __dict__
from namespace.Document.submodule import klass
c = klass()

需要注意几点：

• 我不是仅仅在谈论迁移 代码。如果只是简单地用一个大的 sed 命令来修改代码，以便迁移一些导入，那我就不需要动态模块了。我这里说的是对象。一个网站上有一些实时或存储的对象需要迁移。这些对象会假设 mynamespace.Document.submodule.klass 是存在的，这就是需要动态模块的原因。我需要给网站提供一些可以加载的东西。
• 我们不能，也不想改变对象的反序列化/加载方式。为了简单起见，我们要确保 from mynamespace.Document.submodule import klass 这个语法能够正常工作。我不能改成 from mynamespace import Document as container; klass = getattr(getattr(container, 'submodule'), 'klass') 这样的写法。

我尝试过的：

import sys
from types import ModuleType

class VerboseModule(ModuleType):
    def __init__(self, name, doc=None):
        super(VerboseModule, self).__init__(name, doc)
        sys.modules[name] = self
    def __repr__(self):
        return "<%s %s>" % (self.__class__.__name__, self.__name__)
    def __getattribute__(self, name):
        if name not in ('__name__', '__repr__', '__class__'):
            print "fetching attribute %s for %s" % (name, self)
        return super(VerboseModule, self).__getattribute__(name)

class DynamicModule(VerboseModule):
    """
    This module generates a dummy class when asked for a component
    """
    def __getattr__(self, name):
        class Dummy(object):
            pass
        Dummy.__name__ = name
        Dummy.__module__ = self
        setattr(self, name, Dummy)
        return Dummy
class DynamicPackage(VerboseModule):
    """
    This package should generate dummy modules
    """
    def __getattr__(self, name):
        mod = DynamicModule("%s.%s" % (self.__name__, name))
        setattr(self, name, mod)
        return mod

DynamicModule("foobar")
# (the import prints:)
# fetching attribute __path__ for <DynamicModule foobar>
# fetching attribute DynamicModuleWorks for <DynamicModule foobar>
# fetching attribute DynamicModuleWorks for <DynamicModule foobar>
from foobar import DynamicModuleWorks
print DynamicModuleWorks

DynamicPackage('document')
# fetching attribute __path__ for <DynamicPackage document>
from document.submodule import ButDynamicPackageDoesNotWork
# Traceback (most recent call last):
# File "dynamicmodule.py", line 40, in <module>
#   from document.submodule import ButDynamicPackageDoesNotWork
#ImportError: No module named submodule

如你所见，动态包并没有工作。我不明白发生了什么，因为 document 甚至没有被询问 ButDynamicPackageDoesNotWork 这个属性。

有没有人能解释一下发生了什么？我该如何解决这个问题？

Answer 1

问题在于，Python会跳过sys.modules中的document条目，直接加载submodule文件。当然，这个文件是不存在的。

演示：

>>> import multiprocessing
>>> multiprocessing.heap = None
>>> import multiprocessing.heap
>>> multiprocessing.heap
<module 'multiprocessing.heap' from '/usr/lib/python2.6/multiprocessing/heap.pyc'>

我们本来期待heap还是None，因为Python应该可以直接从sys.modules中取出来，但实际上并没有这样做。点号表示法实际上是直接映射到{python路径}/document/submodule.py，并尝试直接加载这个文件。

更新

解决这个问题的办法是重写Python的导入系统。下面的代码需要你的DynamicModule类。

import sys

class DynamicImporter(object):
    """this class works as both a finder and a loader."""
    def __init__(self, lazy_packages):
        self.packages = lazy_packages

    def load_module(self, fullname):
        """this makes the class a loader. It is given name of a module and expected
           to return the module object"""
        print "loading {0}".format(fullname)
        components = fullname.split('.')
        components = ['.'.join(components[:i+1])
                      for i in range(len(components))]
        for component in components:
            if component not in sys.modules:
                DynamicModule(component)
                print "{0} created".format(component)
        return sys.modules[fullname]


    def find_module(self, fullname, path=None):
        """This makes the class a finder. It is given the name of a module as well as
           the package that contains it (if applicable). It is expected to return a 
           loader for that module if it knows of one or None in which case other methods
           will be tried"""
        if fullname.split('.')[0] in self.packages:
            print "found {0}".format(fullname)
            return self
        else:
            return None


# This is a list of finder objects which is empty by defaule
# It is tried before anything else when a request to import a module is encountered.
sys.meta_path=[DynamicImporter('foo')]

from foo.bar import ThisShouldWork