需要从ipfi抓取amazons3部分

Question

我试图从下面的站点获取ip，我能够用python beautiful soup获得整个站点，并使用python regex模块获得ip4地址 但我遇到了一个问题，我只需要ipv4的ip是在节说“S3”任何帮助将不胜感激

https://ip-ranges.amazonaws.com/ip-ranges.json

akamai_feed = urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read() 
soup = BeautifulSoup(akamai_feed, 'html.parser')
ip_addr = re.findall(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.\d{1,3}.\b', soup.get_text())

Answer 1

正则表达式在这里是不必要的，因为IPs可能不需要验证，如果我正确理解这个问题，我们只想获得"ip_prefix"值，但是如果您希望使用正则表达式实现这一点，这可能就足够了：

"ip_prefix": "(.+?)"

Demo

测试

# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility

import re

regex = r"\"ip_prefix\": \"(.+?)\""

test_str = ("{\n"
    "  \"syncToken\": \"1560279544\",\n"
    "  \"createDate\": \"2019-06-11-18-59-04\",\n"
    "  \"prefixes\": [\n"
    "    {\n"
    "      \"ip_prefix\": \"18.208.0.0/13\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.245.0/24\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.194.0.0/15\",\n"
    "      \"region\": \"ap-northeast-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.155.0.0/16\",\n"
    "      \"region\": \"eu-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.196.0.0/15\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"99.78.170.0/23\",\n"
    "      \"region\": \"ap-southeast-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.94.22.0/24\",\n"
    "      \"region\": \"us-gov-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.255.112/28\",\n"
    "      \"region\": \"us-west-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"13.210.0.0/15\",\n"
    "      \"region\": \"ap-southeast-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.94.17.0/24\",\n"
    "      \"region\": \"eu-central-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.154.0/23\",\n"
    "      \"region\": \"eu-west-3\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.212.0/22\",\n"
    "      \"region\": \"ap-southeast-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.239.0.240/28\",\n"
    "      \"region\": \"eu-west-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.241.0.0/16\",\n"
    "      \"region\": \"us-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"184.169.128.0/17\",\n"
    "      \"region\": \"us-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"216.182.224.0/21\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n\n"
    "...")

matches = re.finditer(regex, test_str, re.MULTILINE)

for matchNum, match in enumerate(matches, start=1):

    print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))

    for groupNum in range(0, len(match.groups())):
        groupNum = groupNum + 1

        print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))

# Note: for Python 2.7 compatibility, use ur"" to prefix the regex and u"" to prefix the test string and substitution.

Answer 2

此提要是Json文件，因此可以使用Python标准库中的json模块：

from urllib.request import urlopen
import json

akamai_feed = json.loads( urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read() )

for prefix in akamai_feed['prefixes']:
    if prefix['service'] == 'S3':
        print(prefix['ip_prefix'])

印刷品：

...

52.95.163.0/24
52.95.145.0/24
52.92.40.0/21
52.219.32.0/21
52.95.136.0/23
52.219.62.0/23
52.95.175.0/24

... and so on

Answer 3

该网站包含一个作为JSON的嵌套字典，因此您需要首先读取JSON数据：

>>> import urllib.request
>>> import json
>>> akamai_feed = urllib.request.urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read()
>>> akamai_json = json.loads(akamai_feed)

现在您已经有了实际的数据，可以使用filter()函数进行筛选：

>>> list(filter(lambda _: _['service'] == 'S3', akamai_json['prefixes']))
[{'ip_prefix': '52.95.154.0/23', 'region': 'eu-west-3', 'service': 'S3'}, {'ip_prefix': '52.219.64.0/22', 'region': 'ap-south-1', 'service': 'S3'}, ...]

这将为您提供'service'为'S3'的词典列表。或者，您也可以使用列表：

>>> [_ for _ in json.loads(akamai_feed)['prefixes'] if _['service'] == 'S3']
[{'ip_prefix': '52.95.154.0/23', 'region': 'eu-west-3', 'service': 'S3'}, {'ip_prefix': '52.219.64.0/22', 'region': 'ap-south-1', 'service': 'S3'}, ...]

如果你只对IP地址感兴趣，那么

>>> [_['ip_prefix'][:-3] for _ in json.loads(akamai_feed)['prefixes'] if _['service'] == 'S3']
['52.95.154.0', '52.219.64.0', ...]

如果/xx可以是任意数量的数字，或者如果IP地址是IPv4字符串以外的格式，那么regex将有助于过滤字符串。你知道吗