计划启动EC2实例并在i中运行python脚本

Question

我试图在AWS中调度python脚本，但是我不希望实例一直运行。因此，尝试自动化以下过程：

1. 在特定时间启动EC2实例
2. 在其中运行python脚本
3. 作业完成后停止EC2实例。

我无法将此脚本直接作为Lambda函数运行，因为该脚本执行一些需要更多RAM的并行处理，因此选择一个更大的AWS实例，而不是将其作为Lambda函数编写。另外，不要让这个实例一直运行，因为它很昂贵。

到目前为止，我遵循Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com并创建了一个Lambda函数来在特定时间启动和停止实例，但是在实例启动后，我找不到运行python脚本的方法。

有人能指点我正确的方向吗？

Answer 1

我的应用程序每天运行一个实例@13:39 UST，处理完成后self关闭。它用在下面

1. 使用云监视事件规则的计划lambda函数

Cloud watch Event/rules config

1. lambda触发器将启动一个实例（使用硬编码id）

&13；

&13；

import boto3
def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='ap-south-1')
    ec2.start_instances(InstanceIds=['i-xxxxxxx'])
    print('started your instances: ' + str('i-xxxxxx'))
    return

；

；

；

1. 这将触发一个运行cron以执行Python脚本的实例

   @reboot python/home/Init.py

2. 脚本完成后，python作业将使用下面的代码段关闭自己

&13；

&13；

import boto.ec2
import boto.utils
import logging
logger=logging.getLogger()
def stop_ec2():
    conn = boto.ec2.connect_to_region("ap-south-1") # or your region
    # Get the current instance's id
    my_id = boto.utils.get_instance_metadata()['instance-id']
    logger.info(' stopping EC2 :'+str(my_id))
    conn.stop_instances(instance_ids=[my_id])

；

；

；

Answer 2

我在使用本文中的解决方案启动和停止实例时遇到问题。然后我按照https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/上的说明操作，这非常简单。基本上：

1. 转到https://console.aws.amazon.com/iam/home#/home，在左侧单击“策略”，然后单击“创建策略”。然后单击JSON选项卡。然后复制粘贴此项以创建新策略：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:Start*",
        "ec2:Stop*"
      ],
      "Resource": "*"
    }
  ]
}

1. 转到https://console.aws.amazon.com/iam/home#/home并在左侧选择角色。确保选择Lambda作为AWS服务，并附加在步骤1中创建的策略。

2. 然后转到Lambda控制台，单击Create Function。选择Python3.7，然后单击“权限”旁边的下拉列表，使用现有角色并附加在步骤2中创建的IAM角色。

3. 使用此代码：

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

1. 启动EC2实例，键入which python找到python的路径并将其写下来。然后，输入crontab -e编辑CRON作业。不要使用sudo…因为有时sudo在您没有使用它来运行Python文件时会把事情弄糟。在我的例子中，我有一个存储密码的pgpass文件，sudo看不见，但是删除sudo是有效的！
2. 在crontab编辑器中，在注释行之后键入@reboot /path/to/python /path/to/file.py例如，对于我来说，这是@reboot /home/init/python /home/init/Notebooks/mypredictor.py
3. 在Python文件的末尾，需要停止实例。你可以这样做：

import boto3
region = 'us-west-1' # Dont use the specific, like instead of us-east-1d just write us-east-1
instances = ['i-xxxxxxxxxxxx']
ec2 = boto3.client('ec2', region_name=region)

ec2.stop_instances(InstanceIds=instances)

Answer 3

对于提出这个问题的未来开发人员来说，一种新的解决方法是：

1. 使用包含AmazonEC2RoleforSSM策略的角色创建EC2
2. 创建lambda来执行唤醒、运行命令、关机
3. 使用Cloudwatch事件触发lambda

所以：

1. 按以下步骤操作：https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

2. 使用以下lambda骨架：

import time
import boto3

REGION_NAME = 'us-east-1'

WORKING_DIRECTORY = '<YOUR WORKING DIRECTORY, IF ANY>'

COMMAND = """
    echo "Hello, world!"
    """

INSTANCE_ID = '<YOUR INSTANCE ID>'


def start_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.start_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 16 ('running'), then proceed, else, wait 5 seconds and try again
        if state['Code'] == 16:
            break
        else:
            time.sleep(5)

    print('EC2 started')


def stop_ec2():
    ec2 = boto3.client('ec2', region_name=REGION_NAME)
    ec2.stop_instances(InstanceIds=[INSTANCE_ID])

    while True:
        response = ec2.describe_instance_status(InstanceIds=[INSTANCE_ID], IncludeAllInstances=True)
        state = response['InstanceStatuses'][0]['InstanceState']

        print(f"Status: {state['Code']} - {state['Name']}")

        # If status is 80 ('stopped'), then proceed, else wait 5 seconds and try again
        if state['Code'] == 80:
            break
        else:
            time.sleep(5)

    print('Instance stopped')


def run_command():
    client = boto3.client('ssm', region_name=REGION_NAME)

    time.sleep(10)  # I had to wait 10 seconds to "send_command" find my instance 

    cmd_response = client.send_command(
        InstanceIds=[INSTANCE_ID],
        DocumentName='AWS-RunShellScript',
        DocumentVersion="1",
        TimeoutSeconds=300,
        MaxConcurrency="1",
        CloudWatchOutputConfig={'CloudWatchOutputEnabled': True},
        Parameters={
            'commands': [COMMAND],
            'executionTimeout': ["300"],
            'workingDirectory': [WORKING_DIRECTORY],
        },
    )

    command_id = cmd_response['Command']['CommandId']
    time.sleep(1)  # Again, I had to wait 1s to get_command_invocation recognises my command_id

    retcode = -1
    while True:
        output = client.get_command_invocation(
            CommandId=command_id,
            InstanceId=INSTANCE_ID,
        )

        # If the ResponseCode is -1, the command is still running, so wait 5 seconds and try again
        retcode = output['ResponseCode']
        if retcode != -1:
            print('Status: ', output['Status'])
            print('StdOut: ', output['StandardOutputContent'])
            print('StdErr: ', output['StandardErrorContent'])
            break

        print('Status: ', retcode)
        time.sleep(5)

    print('Command finished successfully') # Actually, 0 means success, anything else means a fail, but it didn't matter to me
    return retcode


def lambda_handler(event, context):
    retcode = -1
    try:
        start_ec2()
        retcode = run_command()
    finally:  # Independently of what happens, try to shutdown the EC2
        stop_ec2()

    return retcode

1. 按以下步骤操作：https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html