推送RethinkDB文档到RQ队列时出现UnpickleError

Question

我从RethinkDB数据库中获取的一个文档里有一个时间戳，这个时间戳是用Python的 datetime.datetime 对象表示的，并且有一个特别的 tzinfo 值，类型是 rethinkdb.ast.RqlTzinfo。

当我把这个文档放到RQ任务队列时，任务在尝试解码这个时间戳时出现了一个叫做UnpickleError的错误。

feeds = r.table('feeds') \
    .filter(lambda feed: feed['last_fetched'] \
        < (r.now() - config.FEED_REFRESH_INTERVAL)) \
    .pluck('id', 'feed_url', 'last_fetched').run(db)

q = Queue('feed_fetcher', connection=Redis())

for feed in feeds:
    print(feed['last_fetched']) # 2014-09-08 18:35:22.735000+00:00
    result = q.enqueue(fetch_feed, feed)

RQ工作者输出的信息：

18:48:34 *** Listening on feed_fetcher...
19:05:01 feed_fetcher: app.feedupdater.tasks.fetch_feed({u'last_fetched': datetime.datetime(2014, 9, 8, 18, 35, 22, 735000, tzinfo=<rethinkdb.ast.RqlTzinfo object at 0x7fdf508ed410>), u'id': u'ccd57063-a61a-4255-af67-94c244ff6bbb', u'feed_url': u'http://feeds.feedburner.com/hacker-news-feed-200?format=xml'}) (0002de8d-978c-4aeb-944e-2559458d114c)
Traceback (most recent call last):
  File "/usr/local/bin/rqworker", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/rq/scripts/rqworker.py", line 100, in main
    w.work(burst=args.burst)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 358, in work
    self.execute_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 422, in execute_job
    self.main_work_horse(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 457, in main_work_horse
    success = self.perform_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 473, in perform_job
    job.func_name,
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 225, in func_name
    self._unpickle_data()
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 193, in _unpickle_data
    self._func_name, self._instance, self._args, self._kwargs = unpickle(self.data)
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 50, in unpickle
    raise UnpickleError('Could not unpickle.', pickled_string, e)
rq.exceptions.UnpickleError: (u'Could not unpickle.', TypeError('__init__() takes exactly 2 arguments (1 given)', <class 'rethinkdb.ast.RqlTzinfo'>, ()))

我找到了一种解决办法，就是在放入队列之前把时间戳替换成一个时间戳的数字（epoch time），但是这只是一个临时解决方案，而且让事情变得有点复杂，因为在把这个数字再转回 datetime 对象之前，我还得处理时区的问题，然后再把它放回数据库。

问题出在哪里呢？是RethinkDB的 RqlTzinfo 对象有问题，还是RQ的解码实现有问题？这是一个真正的bug，还是只是我实现得不好？

时间戳任务队列时区处理 bug排查 rq RethinkDB 数据库文档 unpickleerror

推送RethinkDB文档到RQ队列时出现UnpickleError

1 个回答

撰写回答