Docker容器中的Python应用程序在应用程序失败时不会停止/删除Docker容器

Question

我有一个Python应用程序，它轮询队列以获取新数据，并将其插入TimescaleDB数据库（TimescaleDB是PostgreSQL的扩展）此应用程序必须始终保持运行。

问题是，Python程序可能会不时失败，我希望Docker Swarm重新启动容器。但是，即使在发生故障后，容器仍会继续运行为什么我的容器没有出现故障，然后由Docker Swarm重新启动？

Python应用程序如下所示：

def main():
    try:
        conn = get_db_conn()
        insert_data(conn)
    except Exception:
        logger.exception("Error with main inserter.py function")
        send_email_if_error()
        raise
    finally:
        try:
            conn.close()
            del conn
        except Exception:
            pass

        return 0


if __name__ == "__main__":
    main()

Dockerfile如下所示：

FROM python:3.8-slim-buster

# Configure apt and install packages
RUN apt-get update && \
    apt-get -y --no-install-recommends install cron nano procps

# Install Python requirements.
RUN pip3 install --upgrade pip && \
    pip3 install poetry==1.0.10

COPY poetry.lock pyproject.toml /
RUN poetry config virtualenvs.create false && \
  poetry install --no-interaction --no-ansi

# Copy everything to the / folder inside the container
COPY . /

# Make /var/log the default directory in the container
WORKDIR /var/log

# Start Python app on container startup
CMD ["python3", "/inserter/inserter.py"]

Docker编写文件：

version: '3.7'
services:
  inserter13:
    # Name and tag of image the Dockerfile creates
    image: mccarthysean/ijack:timescale
    depends_on: 
      - timescale13
    env_file: .env
    environment: 
      POSTGRES_HOST: timescale13
    networks:
      - traefik-public
    deploy:
      # Either global (exactly one container per physical node) or
      # replicated (a specified number of containers). The default is replicated
      mode: replicated
      # For stateless applications using "replicated" mode,
      # the total number of replicas to create
      replicas: 2
      restart_policy:
        on-failure # default is 'any'

  timescale13:
    image: timescale/timescaledb:2.3.0-pg13
    volumes: 
      - type: volume
        source: ijack-timescale-db-pg13
        target: /var/lib/postgresql/data # the location in the container where the data are stored
        read_only: false
      # Custom postgresql.conf file will be mounted (see command: as well)
      - type: bind
        source: ./postgresql_custom.conf
        target: /postgresql_custom.conf
        read_only: false
    env_file: .env
    command: ["-c", "config_file=/postgresql_custom.conf"]
    ports:
      - 0.0.0.0:5432:5432
    networks:
      traefik-public:
    deploy:
      # Either global (exactly one container per physical node) or
      # replicated (a specified number of containers). The default is replicated
      mode: replicated
      # For stateless applications using "replicated" mode,
      # the total number of replicas to create
      replicas: 1
      placement:
        constraints:
          # Since this is for the stateful database,
          # only run it on the swarm manager, not on workers
          - "node.role==manager"
      restart_policy:
        condition: on-failure # default is 'any'


# Use a named external volume to persist our data
volumes:
  ijack-timescale-db-pg13:
    external: true

networks:
  # Use the previously created public network "traefik-public", shared with other
  # services that need to be publicly available via this Traefik
  traefik-public:
    external: true

我用于构建“inserter.py”容器映像的“Docker compose.build.yml”文件：

version: '3.7'
services:
  inserter:
    # Name and tag of image the Dockerfile creates
    image: mccarthysean/ijack:timescale
    build:
      # context: where should docker-compose look for the Dockerfile?
      # i.e. either a path to a directory containing a Dockerfile, or a url to a git repository
      context: .
      dockerfile: Dockerfile.inserter
    environment: 
      POSTGRES_HOST: timescale

我运行的Bash脚本，它使用Docker Swarm构建、推送和部署数据库和插入器容器：

#!/bin/bash

# Build and tag image locally in one step. 
# No need for docker tag <image> mccarthysean/ijack:<tag>
echo ""
echo "Building the image locally..."
echo "docker-compose -f docker-compose.build.yml build"
docker-compose -f docker-compose.build.yml build

# Push to Docker Hub
# docker login --username=mccarthysean
echo ""
echo "Pushing the image to Docker Hub..."
echo "docker push mccarthysean/ijack:timescale"
docker push mccarthysean/ijack:timescale

# Deploy to the Docker swarm and send login credentials 
# to other nodes in the swarm with "--with-registry-auth"
echo ""
echo "Deploying to the Docker swarm..."
echo "docker stack deploy --with-registry-auth -c docker-compose.prod13.yml timescale13"
docker stack deploy --with-registry-auth -c docker-compose.prod13.yml timescale13

当Python inserter程序失败时（可能是数据库连接问题或其他问题），它会向我发送电子邮件警报，然后引发错误并失败。此时，我预计Docker容器将失败，并使用Docker Swarm的restart_policy: on-failure重新启动。但是，在出现错误后，当我键入docker service ls时，我会看到以下0/2 replicas：

ID                  NAME                                        MODE                REPLICAS            IMAGE                                         PORTS
u354h0uj4ug6        timescale13_inserter13                      replicated          0/2                 mccarthysean/ijack:timescale
o0rbfx5n2z4h        timescale13_timescale13                     replicated          1/1                 timescale/timescaledb:2.3.0-pg13              *:5432->5432/tcp

当它处于健康状态时（大多数情况下），它会显示2/2副本。为什么我的容器没有出现故障，然后由Docker Swarm重新启动

Answer 1

我找到了答案，并更新了我的问题，以提供有关try: except:失败例程的更多详细信息

下面是发生的错误（正如您将看到的，实际上是两个顺序错误）：

Here's the error information: 
Traceback (most recent call last):
  File "/inserter/inserter.py", line 357, in execute_sql
    cursor.execute(sql, values)
psycopg2.errors.AdminShutdown: terminating connection due to administrator command SSL connection has been closed unexpectedly


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/inserter/inserter.py", line 911, in main
    insert_alarm_log_rds(
  File "/inserter/inserter.py", line 620, in insert_alarm_log_rds
    rc = execute_sql(
  File "/inserter/inserter.py", line 364, in execute_sql
    conn.rollback()
psycopg2.InterfaceError: connection already closed

如您所见，第一个psycopg2.errors.AdminShutdown错误是在我的第一个try: except:例程中提出的。然而，这之后是第二个psycopg2.InterfaceError，实际上发生在我的finally:清理代码中，接着是一个pass语句和一个return 0，因此我猜前面的错误没有被重新引发，代码以错误代码0结束，而不是刺激重启所需的非零

@edijon关于需要非零出口代码的评论帮助我解决了这个问题

我需要在finally:例程中重新引发错误，如下所示：

def main():
    try:
        conn = get_db_conn()
        insert_data(conn)
    except Exception:
        logger.exception("Error with main inserter.py function")
        send_email_if_error()
        raise
    finally:
        try:
            conn.close()
            del conn
        except Exception:
            # previously the following was just 'pass' 
            # and I changed it to 'raise' to ensure errors
            # cause a non-zero error code for Docker's 'restart_policy'
            raise

        # The following was previously "return 0"
        # which caused the container not to restart...
        # Either comment it out, or change it to return non-zero
        return 1


if __name__ == "__main__":
    main()