fastavro将json文件转换为avro文件

Question

对avro&；Python

我正在尝试使用fastavro库进行到avro的简单转换，因为本机ApacheAvro库的速度有点太慢

我想：

1.获取一个json文件 2.将数据转换为avro

我的问题是，我的json似乎不是正确的“记录”格式，无法转换为avro。我甚至尝试将我的json放入一个字符串变量中，使其看起来与他们在站点@https://fastavro.readthedocs.io/en/latest/writer.html上的语法相似：

    {u'station': u'011990-99999', u'temp': 22, u'time': 1433270389},
    {u'station': u'011990-99999', u'temp': -11, u'time': 1433273379},
    {u'station': u'012650-99999', u'temp': 111, u'time': 1433275478},

这是我的密码：

from fastavro import json_writer, parse_schema, writer
import json

key = "test.json"
schemaFileName = "test_schema.avsc"
with open(r'C:/Path/to/file' + schemaFileName) as sc:
    w = json.load(sc)

schema = parse_schema(w)

with open(r'C:/Path/to/file/' + key) as js:
    x=json.load(js)

with open('C:/Path/to/file/output.avro', 'wb') as out:
    writer(out, schema,x, codec='deflate')

以下是我得到的输出：

  File "avropython.py", line 26, in <module>
    writer(out, schema,x, codec='deflate')
  File "fastavro\_write.pyx", line 608, in fastavro._write.writer
ValueError: "records" argument should be an iterable, not dict

我的json文件和模式，分别是：

  "joined": false,
  "toward": {
    "selection": "dress",
    "near": true,
    "shoulder": false,
    "fine": -109780201.3804388,
    "pet": {
      "stood": "saddle",
      "live": false,
      "leather": false,
      "tube": false,
      "over": false,
      "impossible": true
    },
    "higher": false
  },
  "wear": true,
  "asleep": "door",
  "connected": true,
  "stairs": -1195512399.5000324
}

{
  "name": "MyClass",
  "type": "record",
  "namespace": "com.acme.avro",
  "fields": [
    {
      "name": "joined",
      "type": "boolean"
    },
    {
      "name": "toward",
      "type": {
        "name": "toward",
        "type": "record",
        "fields": [
          {
            "name": "selection",
            "type": "string"
          },
          {
            "name": "near",
            "type": "boolean"
          },
          {
            "name": "shoulder",
            "type": "boolean"
          },
          {
            "name": "fine",
            "type": "float"
          },
          {
            "name": "pet",
            "type": {
              "name": "pet",
              "type": "record",
              "fields": [
                {
                  "name": "stood",
                  "type": "string"
                },
                {
                  "name": "live",
                  "type": "boolean"
                },
                {
                  "name": "leather",
                  "type": "boolean"
                },
                {
                  "name": "tube",
                  "type": "boolean"
                },
                {
                  "name": "over",
                  "type": "boolean"
                },
                {
                  "name": "impossible",
                  "type": "boolean"
                }
              ]
            }
          },
          {
            "name": "higher",
            "type": "boolean"
          }
        ]
      }
    },
    {
      "name": "wear",
      "type": "boolean"
    },
    {
      "name": "asleep",
      "type": "string"
    },
    {
      "name": "connected",
      "type": "boolean"
    },
    {
      "name": "stairs",
      "type": "float"
    }
  ]
}

如果有人能帮助我，我将不胜感激

Answer 1

正如错误ValueError: "records" argument should be an iterable, not dict中提到的，问题是当调用writer时，记录的参数需要是iterable。解决此问题的一种方法是将最后一行更改为writer(out, schema, [x], codec='deflate')

或者，有一个schemaless_writer可以用来只写一条记录：https://fastavro.readthedocs.io/en/latest/writer.html#fastavro._write_py.schemaless_writer