生产者/消费者 - 通过列表获取队列

Question

我想从数据库获取数据，以便输入到机器学习模型中。但是我的生产者（Producer）每次只把一行数据放到队列里，而消费者（Consumer）也只是一行一行地取数据。我的机器学习模型需要2000行数据才能处理。

我希望生产者和消费者每次能放入和取出多行数据，而不是一行一行地来。

这是我用Python写的代码：

def read_data():
    
    torque_data_queryset = Torque.get_torque_data()
    return list(torque_data_queryset)


def producer(queue):
    print('Producer: Running')
    data = read_data()
    for item in data:
       
        time.sleep(1)
        
        queue.put(item)
        print(f'> Producer added {item}')
    
    queue.put(None)
    print('Producer: Done')

# consumer task
def consumer(queue):
    print('Consumer: Running')
    while True:
        
        item = queue.get()
        print(item)
        
        if item is None:
            break
        
        time.sleep(1)
        
        print(f'> Consumer got {item}')
    print('Consumer: Done')

def thread_function():
    
    queue = Queue()
    
    consumer_thread = Thread(target=consumer, args=(queue,))
    consumer_thread.start()
    
    producer_thread = Thread(target=producer, args=(queue,))
    producer_thread.start()
    
    producer_thread.join()
    consumer_thread.join()
    
def thread_result(request):
    
    thread_function()
    return render(request, 'async_processing_result.html', {'message': 'Threads completed successfully!'})

Answer 1

你可以把一组对象放到队列（queue.Queue）里。消费者会一次性取出整个列表。

比如说：

import threading
import queue


def consumer(q: queue.Queue):
    while (value := q.get()) is not None:
        print(*value, sep="\n")


def producer(q: queue.Queue):
    # create a list of dictionaries
    data = [{"int": i, "chr": chr(i + ord("a"))} for i in range(10)]
    q.put(data)
    q.put(None)


def main():
    q = queue.Queue()
    threads = []
    for func in producer, consumer:
        (t := threading.Thread(target=func, args=(q,))).start()
        threads.append(t)
    for t in threads:
        t.join()


if __name__ == "__main__":
    main()

输出结果：

{'int': 0, 'chr': 'a'}
{'int': 1, 'chr': 'b'}
{'int': 2, 'chr': 'c'}
{'int': 3, 'chr': 'd'}
{'int': 4, 'chr': 'e'}
{'int': 5, 'chr': 'f'}
{'int': 6, 'chr': 'g'}
{'int': 7, 'chr': 'h'}
{'int': 8, 'chr': 'i'}
{'int': 9, 'chr': 'j'}