在Amazon EMR上运行mrjob，t2.micro不受支持

Question

我试着在亚马逊电子病历上运行mrjob脚本。当我使用实例c1.medium时，它工作得很好，但是，当我将instance更改为t2.micro时，它有一个错误。完整的错误消息如下所示。

C:\Users\Administrator\MyIpython>python word_count.py -r emr 111.txt using configs in C:\Users\Administrator.mrjob.conf creating new scratch bucket mrjob-875a948553aab9e8 using s3://mrjob-875a948553aab9e8/tmp/ as our scratch dir on S3 creating tmp directory c:\users\admini~1\appdata\local\temp\word_count.Administr ator.20150731.013007.592000 writing master bootstrap script to c:\users\admini~1\appdata\local\temp\word_cou nt.Administrator.20150731.013007.592000\b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as de scribed at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protoc ols

creating S3 bucket 'mrjob-875a948553aab9e8' to use as scratch space Copying non-input files into s3://mrjob-875a948553aab9e8/tmp/word_count.Administ rator.20150731.013007.592000/files/ Waiting 5.0s for S3 eventual consistency Creating Elastic MapReduce job flow Traceback (most recent call last): File "word_count.py", line 16, in MRWordFrequencyCount.run() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\job.py", line 461, in run mr_job.execute() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\job.py", line 479, in execute super(MRJob, self).execute() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\launch.py", line 153, in execute self.run_job() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\launch.py", line 216, in run_job runner.run() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\runner.py", line 470, in run self._run() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\emr.py", line 881, in _run self._launch() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\emr.py", line 886, in _launch self._launch_emr_job() File "F:\Program Files\Anaconda\lib\site-packages\mrjob\emr.py", line 1593, in _launch_emr_job persistent=False) File "F:\Program Files\Anaconda\lib\site-packages\mrjob\emr.py", line 1327, in _create_job_flow self._job_name, self._opts['s3_log_uri'], **args) File "F:\Program Files\Anaconda\lib\site-packages\mrjob\retry.py", line 149, i n call_and_maybe_retry return f(*args, **kwargs) File "F:\Program Files\Anaconda\lib\site-packages\mrjob\retry.py", line 71, in call_and_maybe_retry result = getattr(alternative, name)(*args, **kwargs) File "F:\Program Files\Anaconda\lib\site-packages\boto\emr\connection.py", lin e 581, in run_jobflow 'RunJobFlow', params, RunJobFlowResponse, verb='POST') File "F:\Program Files\Anaconda\lib\site-packages\boto\connection.py", line 12 08, in get_object raise self.ResponseError(response.status, response.reason, body) boto.exception.EmrResponseError: EmrResponseError: 400 Bad Request
Sender ValidationError Instance type 't2.micro' is not supported c3ee1107-3723-11e5-8d8e-f1011298229d

这是我的配置文件详细信息

runners:
  emr:
    aws_access_key_id: xxxxxxxxxxx
    aws_secret_access_key: xxxxxxxxxxxxx
    aws_region: us-east-1
    ec2_key_pair: EMR
    ec2_key_pair_file: C:\Users\Administrator\EMR.pem
    ssh_tunnel_to_job_tracker: false
    ec2_instance_type: t2.micro
    num_ec2_instances: 2

Answer 1

EMR不支持t2实例类型。如果你担心钱，spot实例是一个非常划算的选择：现在m1.xlarge低于0.05美元/小时，m1.medium每小时0.01美元（无论如何比t2.micro便宜）支持的类型如下（来自EMR webapp控制台的屏幕截图：