如何在从MongoDB获取数据的同时加速Kafka?

2024-04-25 23:48:35 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在从MongoDB获取数据并将其放入Kafka。357 responses/sec是完成获取和发布的速率。在

如何改进从MongoDB获取数据:

from kafka import KafkaProducer
from kafka.errors import KafkaError
import json
import pymongo
from pymongo import MongoClient
import sys

try:
  client = MongoClient('my_uri')
  db = client["xxx-dev"]
except Exception as e:
    print e
producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
producer = KafkaProducer(retries=5)
id = 1
for response in db.Response.find():
    try:        
        future = producer.send('collect-production-response', bytes(response))
    except Exception as e:
        print e
    id  += 1
    if(id >= 100000):
        print "Done 100k"
        producer.flush()
        sys.exit()

Tags: producerkafkafromimportclientiddbresponse