ScrapydWeb:dockercompose中的连接被拒绝

Question

我尝试运行几个scrapyd服务，以便在本地主机上有一个简单的集群，但只有第一个节点起作用。对于另外两个，我得到以下错误

scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_3', port=6802): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060b8ef50>: Failed to establish a new connection: [Errno 111] Connection refused'))
scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_2', port=6801): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060a1e650>: Failed to establish a new connection: [Errno 111] Connection refused'))

我有以下docker-compose.yml文件：

version: '3'
services:
  scrapyd_node_1:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6800:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_2:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6801:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_3:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6802:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped


  scrapydweb:
    build:
      context: .
      dockerfile: ./crawlers/scrapydweb/Dockerfile
    environment:
      USERNAME: "test"
      PASSWORD: "test"
      SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"
    links:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3
    ports:
      - "5000:5000"
    depends_on:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3

    restart: unless-stopped

我的docker compose文件有什么问题

Answer 1

问题在于：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"

尝试将其更改为：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6800,scrapyd_node_3:6800"

说明：

例如，当您定义docker服务scrapyd_node_2时，您将端口定义为：

    ports:
      - "6801:6800"

这意味着，contanier的端口6800映射到主机上的端口6801。因此，当您想用主机名scrapyd_node_2声明节点时，应该使用它的端口=scrapyd_node_2:6800