Python的shelve.open可以嵌套调用吗？

Question

我正在尝试写一个记忆化库，想用shelve来持久化存储返回值。如果我有一些记忆化的函数在调用其他记忆化的函数，我在想该如何正确地打开这个存储文件。

import shelve
import functools


def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            key = str(hash(functools._make_key(args, kwds, typed=False)))
            with shelve.open(filename, writeback=True) as cache:
                if key in cache:
                    return cache[key]
                else:
                    result = user_function(*args, **kwds)
                    cache[key] = result
                    return result

        return functools.update_wrapper(wrapper, user_function)

    return decorating_function


@cache(filename='cache')
def expensive_calculation():
    print('inside function')
    return


@cache(filename='cache')
def other_expensive_calculation():
    print('outside function')
    return expensive_calculation()

other_expensive_calculation()

不过这样做不行

$ python3 shelve_test.py
outside function
Traceback (most recent call last):
  File "shelve_test.py", line 33, in <module>
    other_expensive_calculation()
  File "shelve_test.py", line 13, in wrapper
    result = user_function(*args, **kwds)
  File "shelve_test.py", line 31, in other_expensive_calculation
    return expensive_calculation()
  File "shelve_test.py", line 9, in wrapper
    with shelve.open(filename, writeback=True) as cache:
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 239, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 223, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 35] Resource temporarily unavailable

你有什么建议来解决这种问题吗？

Answer 1

你打开了文件两次，但实际上从来没有关闭它，这样就无法更新文件。记得在最后加上 f.close () 来关闭文件。

Answer 2

不可以，你不能有多个使用相同文件名的 shelve 实例。

这个 shelve 模块不支持同时对存储的对象进行读写操作。（多个程序同时读取是安全的。）当一个程序打开了一个可以写入的存储时，其他程序就不应该再打开这个存储进行读取或写入。虽然可以使用 Unix 文件锁来解决这个问题，但不同的 Unix 版本之间有所不同，并且需要了解所使用的数据库实现。

https://docs.python.org/3/library/shelve.html#restrictions

Answer 3

与其尝试嵌套调用 open（你已经发现这样做是行不通的），不如让你的装饰器保存一个由 shelve.open 返回的句柄的引用。如果这个句柄存在并且仍然是打开的，就可以在后续的调用中重复使用它：

import shelve
import functools

def _check_cache(cache_, key, func, args, kwargs):
    if key in cache_:
        print("Using cached results")
        return cache_[key]
    else:
        print("No cached results, calling function")
        result = func(*args, **kwargs)
        cache_[key] = result
        return result

def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            args_key = str(hash(functools._make_key(args, kwds, typed=False)))
            func_key = '.'.join([user_function.__module__, user_function.__name__])
            key = func_key + args_key
            handle_name = "{}_handle".format(filename)
            if (hasattr(cache, handle_name) and
                not hasattr(getattr(cache, handle_name).dict, "closed")
               ):
                print("Using open handle")
                return _check_cache(getattr(cache, handle_name), key, 
                                    user_function, args, kwds)
            else:
                print("Opening handle")
                with shelve.open(filename, writeback=True) as c:
                    setattr(cache, handle_name, c)  # Save a reference to the open handle
                    return _check_cache(c, key, user_function, args, kwds)

        return functools.update_wrapper(wrapper, user_function)
    return decorating_function


@cache(filename='cache')
def expensive_calculation():
    print('inside function')
    return


@cache(filename='cache')
def other_expensive_calculation():
    print('outside function')
    return expensive_calculation()

other_expensive_calculation()
print("Again")
other_expensive_calculation()

输出：

Opening handle
No cached results, calling function
outside function
Using open handle
No cached results, calling function
inside function
Again
Opening handle
Using cached results

编辑：

你也可以使用 WeakValueDictionary 来实现这个装饰器，这样看起来会更易读一些：

from weakref import WeakValueDictionary

_handle_dict = WeakValueDictionary()
def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            args_key = str(hash(functools._make_key(args, kwds, typed=False)))
            func_key = '.'.join([user_function.__module__, user_function.__name__])
            key = func_key + args_key
            handle_name = "{}_handle".format(filename)
            if handle_name in _handle_dict:
                print("Using open handle")
                return _check_cache(_handle_dict[handle_name], key, 
                                    user_function, args, kwds)
            else:
                print("Opening handle")
                with shelve.open(filename, writeback=True) as c:
                    _handle_dict[handle_name] = c
                    return _check_cache(c, key, user_function, args, kwds)

        return functools.update_wrapper(wrapper, user_function)
    return decorating_function

一旦没有其他地方再引用这个句柄，它就会从字典中被删除。由于我们的句柄只有在最外层的装饰函数调用结束时才会失效，所以在句柄打开期间，字典中总会有一个条目，而在它关闭后就没有了。