如何在每日时间序列对象上迭代webscraping脚本，以便从webpag创建数据的每日时间序列

Question

谢谢你看我的问题。我用BeautifulSoup和Pandas创建了一个脚本，它从美联储的网站上搜集预测数据。预测每季度一次（约3个月）。我想写一个脚本，创建一个每日时间序列，每天检查一次美联储网站，如果发布了新的预测，脚本会将其添加到时间序列中。如果没有更新，那么脚本只会将时间序列附加上最后一个有效的、更新的投影。你知道吗

从我最初的挖掘来看，似乎有一些外部源代码我可以用来每天“触发”脚本，但我更愿意将所有内容都保留为纯python。你知道吗

我为完成刮片而编写的代码如下：

from bs4 import BeautifulSoup
import requests
import re
import wget
import pandas as pd 

# Starting url and the indicator (key) for links of interest
url = "https://www.federalreserve.gov/monetarypolicy/fomccalendars.htm" 
key = '/monetarypolicy/fomcprojtabl'

# Cook the soup
page = requests.get(url)
data = page.text
soup = BeautifulSoup(data)

# Create the tuple of links for projection pages
projections = []
for link in soup.find_all('a', href=re.compile(key)):
    projections.append(link["href"])

# Create a tuple to store the projections 
decfcasts = []
for i in projections:
    url = "https://www.federalreserve.gov{}".format(i)
    file = wget.download(url)
    df_list = pd.read_html(file)
    fcast = df_list[-1].iloc[:,0:2]
    fcast.columns = ['Target', 'Votes']
    fcast.fillna(0, inplace = True)
    decfcasts.append(fcast)

到目前为止，我编写的代码将所有内容放在一个元组中，但是没有数据的时间/日期索引。我一直在考虑编写伪代码，我猜它看起来像

Create daily time series object
    for each day in time series:
        if day in time series = day in link:
            run webscraper
        other wise, append time series with last available observation

至少，这是我的想法。最后的时间序列可能会看起来相当“笨拙”，因为会有很多天有相同的观测，然后当一个新的投影出来时，会有一个“跳跃”，然后会有更多的重复，直到下一个投影出来。你知道吗

显然，任何帮助都是非常感谢的。不管怎样，提前谢谢你！你知道吗

Answer 1

我为你编辑了代码。现在它从url获取日期。日期在数据帧中保存为句点。只有当数据帧（从pickle还原）中不存在日期时，才会对其进行处理和追加。你知道吗

from bs4 import BeautifulSoup
import requests
import re
import wget
import pandas as pd

# Starting url and the indicator (key) for links of interest
url = "https://www.federalreserve.gov/monetarypolicy/fomccalendars.htm"
key = '/monetarypolicy/fomcprojtabl'

# Cook the soup
page = requests.get(url)
data = page.text
soup = BeautifulSoup(data)

# Create the tuple of links for projection pages
projections = []
for link in soup.find_all('a', href=re.compile(key)):
    projections.append(link["href"])

# past results from pickle, when no pickle init empty dataframe
try:
    decfcasts = pd.read_pickle('decfcasts.pkl')
except FileNotFoundError:
    decfcasts = pd.DataFrame(columns=['target', 'votes', 'date'])


for i in projections:

    # parse date from url
    date = pd.Period(''.join(re.findall(r'\d+', i)), 'D')

    # process projection if it wasn't included in data from pickle
    if date not in decfcasts['date'].values:

        url = "https://www.federalreserve.gov{}".format(i)
        file = wget.download(url)
        df_list = pd.read_html(file)
        fcast = df_list[-1].iloc[:, 0:2]
        fcast.columns = ['target', 'votes']
        fcast.fillna(0, inplace=True)

        # set date time
        fcast.insert(2, 'date', date)
        decfcasts = decfcasts.append(fcast)

# save to pickle
pd.to_pickle(decfcasts, 'decfcasts.pkl')