Python交互代理ibapi非常非常

Question

我正在试用新的Python交互式代理API，但在第一步就遇到了一些严重的速度问题。。。在

以下代码（见下文）时间

0:00:08.832813直到接收完数据

0:00:36.000785直到应用程序完全断开连接。。。在

为什么这么慢？ 最好的加速方法是什么？在

from ibapi import wrapper
from ibapi.client import EClient
from ibapi.utils import iswrapper #just for decorator
from ibapi.common import *
from ibapi.contract import *
import datetime
from datetime import timedelta


class DataApp(wrapper.EWrapper, EClient):
    def __init__(self):
        wrapper.EWrapper.__init__(self)
        EClient.__init__(self, wrapper=self)

    @iswrapper
    def historicalData(self, reqId: TickerId, date: str, open: float, high: float,
                            low: float, close: float, volume: int, barCount: int,
                            WAP: float, hasGaps: int):
        super().historicalData(reqId, date, open, high, low, close, volume,
                                barCount, WAP, hasGaps)
        print("HistoricalData. ", reqId, " Date:", date, "Open:", open,
               "High:", high, "Low:", low, "Close:", close, "Volume:", volume)

    @iswrapper
    def historicalDataEnd(self, reqId: int, start: str, end: str):
        super().historicalDataEnd(reqId, start, end)
        print("HistoricalDataEnd ", reqId, "from", start, "to", end)
        print(datetime.datetime.now()-startime)
        self.done = True # This ends the messages loop - this was not in the example code...

    def get_data(self):        
        self.connect("127.0.0.1", 4002, clientId=10)
        print("serverVersion:%s connectionTime:%s" % (self.serverVersion(),
                                                self.twsConnectionTime()))

        cont = Contract()
        cont.symbol = "ES"
        cont.secType = "FUT"
        cont.currency = "USD"
        cont.exchange = "GLOBEX"
        cont.lastTradeDateOrContractMonth = "201706"
        self.reqHistoricalData(1, cont, datetime.datetime.now().strftime("%Y%m%d %H:%M:%S"),
                               "1800 S", "30 mins", "TRADES", 0, 1, [])
        self.run()        
        self.disconnect()
        print(datetime.datetime.now()-startime)

global starttime
startime = datetime.datetime.now()
DA = DataApp()
DA.get_data()

我还试着不停地运行它的后台，以便只在提交请求的时候就用

^{pr2}$

但它也非常慢。 有什么建议吗

Answer 1

当app.done == False时，app.run()是一个无限循环，但当app.done设置为True时，它不会立即停止。（我不知道为什么）。在

我所做的是编写一个新方法，而不是使用app.run()。在

我的解决方案是：

import time
from ibapi import (decoder, reader, comm)

把这个函数放到你的客户机类中。在

^{pr2}$

用法很简单。只需使用app.getMessage()而不是app.run()

Answer 2

当前的API（2019年5月）通过移除recvMsg中的锁来提高速度，正如@BenM在他的回答中所建议的那样。但是，根据请求的类型，它仍然可能很慢。在

删除他们的大部分日志记录有帮助，大概是因为某些消息类型相当大。我最初尝试使用logging库将其过滤掉，但是直接删除代码在速度方面效果更好，我把这归因于在传递给logging之前生成更大字符串所需的额外处理。在

将Queue替换为deque：可能还值得将client.py中的Queue替换为collections.deque，因为deque快了10倍。我已经在其他地方做过这样的加速，但还没有达到这个目的。deque应该是线程安全的，没有锁，就像这里使用的那样：“Deques支持线程安全、内存高效的附件，并从deque的两边弹出”（来自文档）。在

Queue vs deque speed comparison

Answer 3

我建议您修改ibapi模块中connection类中的连接套接字锁。这个建议来自github上的heshiming；如果您可以访问私有交互式代理repo，那么可以访问这里的讨论https://github.com/InteractiveBrokers/tws-api/issues/464

我这样做了，它大大提高了性能。在

Heshiming建议减少套接字锁对象的超时时间，每次发送或接收消息时都会调用该对象。 要修改套接字锁，请转到ibapi的site packages文件夹并在中修改connect函数连接.py，正在更改“self.socket.settimeout（1） “收件人”self.socket.settimeout（0.01）”。这里是48号线连接.py对于我的版本。在

如果你看不到黑石明的帖子，我已经把它放在了这篇文章的底部。在

备选方案：另一个有趣的解决方案是利用异步事件循环实现异步事件循环。我没有这样做，但看起来很有希望。请参阅Ewald组合的示例https://github.com/erdewit/tws_async

和世明点评：

The implementation of Connection /ibapi/connection.py has a Lock object shared in both sendMsg and recvMsg. Since connect, self.socket.settimeout(1) is called, therefore the underlying self.socket.recv(4096) only times out once per second.

Such implementation creates a performance problem. Since the lock is shared, the socket cannot send data while receiving. In the scenario where the message received is less than 4k bytes long, the recvMsg function will wait for 1 second before releasing the lock, making subsequent sendMsg wait. In my experiment, most messages appear to be shorter than 4k bytes. In other words, this imposes a cap of one recvMsg per second.

There are couple strategies to mitigate this. One can reduce the receive buffer to a number much less than 4k, or reduce the socket timeout to something like 0.001 second to make it block less.

Or according to http://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid , the socket itself is actually thread-safe. Thus no locks are necessary.

I tried all three strategies. Removing the lock works the best. And reducing the timeout to 0.001 works in similar ways.

I can only vouch for linux/unix platforms, and I haven't tried it on Windows. Would you consider to change the implementation to improve this?