Python线程和解释器关闭是可修复的还是问题Python问题14623

Question

我有一个python脚本，可以将文件上载到云帐户。它工作了一段时间，但现在我开始得到“Exception in thread Thread-1 (most likely raised during interpreter shutdown)'错误。在研究之后，我发现了这个python问题http://bugs.python.org/issue14623，它指出这个问题不会得到修复。在

然而，我不确定这是否适用于我，我希望有人能指出一个解决办法。我希望继续使用python的线程，并尽量避免使用多处理，因为这是I/O绑定的。这是精简版（也有这个问题），但在完整版中上传.py有一个我喜欢分享的列表，所以我希望它在同一个内存中运行。在

它总是在完成并上载所有文件后才会中断。我尝试删除't.daemon=True'，它将在同一点挂起（而不是中断）（在所有文件上载后）。我还尝试删除q.join()和't.daemon=True'，完成后它将挂起。如果没有t.daemon=True和q.join（），我认为当脚本执行结束时，它会在item = q.get()处阻塞（只是猜测一下）。在

主要：

 import logging 
import os 
import sys
import json
from os.path import expanduser
from Queue import Queue
from threading import Thread
from auth import Authenticate
from getinfo import get_containers, get_files, get_link
from upload import upload_file 
from container_util import create_containers
from filter import MyFilter

home = expanduser("~") + '/'

directory = home + "krunchuploader_logs"

if not os.path.exists(directory):
    os.makedirs(directory)

debug = directory + "/krunchuploader__debug_" + str(os.getpid())
error = directory + "/krunchuploader__error_" + str(os.getpid())
info = directory + "/krunchuploader__info_" + str(os.getpid())

os.open(debug, os.O_CREAT | os.O_EXCL)
os.open(error, os.O_CREAT | os.O_EXCL)
os.open(info, os.O_CREAT | os.O_EXCL)

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')


logging.basicConfig(level=logging.DEBUG,
    format='%(asctime)s - %(levelname)s - %(message)s',
    filename=debug,
    filemode='w')

logger = logging.getLogger("krunch")

fh_error = logging.FileHandler(error)
fh_error.setLevel(logging.ERROR)
fh_error.setFormatter(formatter)
fh_error.addFilter(MyFilter(logging.ERROR))

fh_info = logging.FileHandler(info)
fh_info.setLevel(logging.INFO)
fh_info.setFormatter(formatter)
fh_info.addFilter(MyFilter(logging.INFO))

std_out_error = logging.StreamHandler()
std_out_error.setLevel(logging.ERROR)
std_out_info = logging.StreamHandler()
std_out_info.setLevel(logging.INFO)

logger.addHandler(fh_error)
logger.addHandler(fh_info)
logger.addHandler(std_out_error)
logger.addHandler(std_out_info)



def main():
    sys.stdout.write("\x1b[2J\x1b[H")
    print title
    authenticate = Authenticate()
    cloud_url = get_link(authenticate.jsonresp)
    #per 1 million files the list will take
    #approx 300MB of memory.
    file_container_list, file_list = get_files(authenticate, cloud_url)
    cloud_container_list = get_containers(authenticate, cloud_url)
    create_containers(cloud_container_list,
        file_container_list, authenticate, cloud_url)

    return file_list

def do_the_uploads(file_list):
    def worker():
    while True:
        item = q.get()
        upload_file(item)
        q.task_done()

    q = Queue()
    for i in range(5):
    t = Thread(target=worker)
    t.daemon = True
    t.start()

    for item in file_list:
    q.put(item)

    q.join()


if __name__ == '__main__':
    file_list = main()
    value = raw_input("\nProceed to upload files? Enter [Y/y] for yes: ").upper()
    if value == "Y":
    do_the_uploads(file_list)

在上传.py公司名称：

^{pr2}$

错误输出：

Fetching Cloud Container List... Got it!
All containers exist, none need to be added

Proceed to upload files? Enter [Y/y] for yes: y
Uploading /home/one/huh/one/green
Uploading /home/one/huh/one/red
Uploading /home/one/huh/two/white
Uploading /home/one/huh/one/blue
 Uploading /home/one/huh/two/yellow
done
Uploading /home/one/huh/two/purple
done
done
done
done
done
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 808, in __bootstrap_inner
  File "/usr/lib64/python2.7/threading.py", line 761, in run
  File "krunchuploader.py", line 97, in worker
  File "/usr/lib64/python2.7/Queue.py", line 168, in get
  File "/usr/lib64/python2.7/threading.py", line 332, in wait
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

更新：我放置了一个时间。睡觉（2） 在脚本的结尾，似乎已经解决了这个问题。我猜睡眠允许守护进程在脚本结束并关闭之前完成？我认为主进程必须等待守护进程完成。在

Answer 1

你可以用“毒丸”优雅地杀死工人。在将所有工作放入队列后，添加一个特殊对象，每个工人一个，工人识别并退出。您可以将线程设为非守护进程，以便Python在关闭进程之前等待它们完成。在

使worker识别poison并退出的一种简明方法是在for循环中使用^{}内置项的双参数形式：

def do_the_uploads(file_list):
    def worker():
        for item in iter(q.get, poison):
            upload_file(item)

    poison = object()
    num_workers = 5
    q = Queue()
    for i in range(num_workers):
        t = Thread(target=worker)
        t.start()

    for item in file_list:
        q.put(item)

    for i in range(num_workers):
        q.put(poison)