将avro文件索引为批量elasticsearch

Question

我写了这个简短的脚本

from elasticsearch import Elasticsearch
from fastavro import reader

es = Elasticsearch(['someIP:somePort'])
with open('data.avro', 'rb') as fo:
    avro_reader = reader(fo)
    for record in avro_reader:
        es.index(index="my_index", body=record)

它工作得非常好。每个记录都是一个json，Elasticsearch可以索引json文件。但是，有没有一种方法可以批量实现，而不是一个接一个地进行for循环呢？因为这很慢

Answer 1

有两种方法可以做到这一点

1. 使用Elasticsearch批量API和requestspython
2. 使用Elasticsearch python库，该库在内部调用相同的批量API

    from elasticsearch import Elasticsearch
    from elasticsearch import helpers
    from fastavro import reader
    
    es = Elasticsearch(['someIP:somePort'])
    
    with open('data.avro', 'rb') as fo:
        avro_reader = reader(fo)
        records = [
            {
                "_index": "my_index",
                "_type": "record",
                "_id": j,
                "_source": record
            }
            for j,record in enumerate(avro_reader)
            ]
        helpers.bulk(es, records)