当我将最后一行附加到文本文件中时,我的代码正在复制它
下面是一个输入文件示例
[headline - https://www.rfa.org/cantonese/news/us-pompeo-06012020072305.html]
This
some-text-here
[date - https://www.rfa.org/cantonese/news/us-pompeo-06012020072305.html]
https://www.rfa.org/cantonese/news/us-pompeo-06012020072305.html/US-Pompeo.mp3
[headline - https://www.rfa.org/cantonese/news/htm/tw-uk-06012020113435.html]
Is
some-text-here
[date - https://www.rfa.org/cantonese/news/htm/tw-uk-06012020113435.html]
https://www.rfa.org/cantonese/news/htm/tw-uk-06012020113435.html/tw-su.mp3
[headline - https://www.rfa.org/cantonese/news/wang-06012020103828.html]
Test
some-text-here
[date - https://www.rfa.org/cantonese/news/wang-06012020103828.html]
https://www.rfa.org/cantonese/news/wang-06012020103828.html/wang.mp3
[headline - https://www.rfa.org/cantonese/news/us-wang-06012020135251.html]
TEST
some-text-here
[headline - https://www.rfa.org/cantonese/news/htm/hk-chan-06012020073718.html]
TEST
some-text-here
[headline - https://www.rfa.org/cantonese/news/ear/ear-state-06012020035108.html]
TEST
some-text-here
[date - https://www.rfa.org/cantonese/news/ear/ear-state-06012020035108.html]
https://www.rfa.org/cantonese/news/ear/ear-state-06012020035108.html/EarState.mp3
[headline - https://www.rfa.org/cantonese/news/htm/hk-innocent-06012020114634.html]
TEST
some-text-here
但是,出于某种原因,输出文件总是附加最后一行两次。我为ear-state-06012020035108.txt
得到这个
TEST
some-text-here
some-text-here
您可以看到这里的一些文本正在被复制。我只是想把所有的文本都放到一个文本文件中。然而,出于某种原因,它只在底部添加最后一行两次
这是我的密码
import pandas as pd
import os
import re
import requests
def tabulate_headlines(text="", filename="", verbose=0):
"""Tabulates the information from either a text string (text) or from the
text read from a file (filename) and returns a dataframe.
Example:
# Case: from text
df = tabulate_headlines(text=s, filename="", verbose=1)
# Case: from a file
df = tabulate_headlines(text="", filename="input.txt", verbose=1)
"""
## Read text from input file
if not text and (filename is not None):
with open(filename, "r") as f:
text = f.read()
if text is not None:
## Define regex patterns for
# - headline-text and corresponding source-id
# - headline_url
# - date_url
# - mp3_url
# - source_id from (headline, date and mp3)
headline_text_pat = r"\n?\[headline - https://.*/(.*?)\.html\]\n((.*\n)+?)\n"
headline_pat = r".*\[headline - (https://.*?\.html?)[,\]]"
date_pat = r".*\[date - (https://.*?\.html?)[,\]]"
mp3_pat = r".*\n(https://.*?\.html/.*?\.mp3)\s*\n"
source_id_pat = r"https://.*/(.*?)\.html" # headline, date
source_id_pat_mp3 = r"https://.*/(.*?).html/.*?\.mp3" # mp3
## Compile regex-patterns for speed
headline_text_pat = re.compile(headline_text_pat)
headline_pat = re.compile(headline_pat)
date_pat = re.compile(date_pat)
mp3_pat = re.compile(mp3_pat)
source_id_pat = re.compile(source_id_pat)
source_id_pat_mp3 = re.compile(source_id_pat_mp3)
## Extract headlines (texts)
# and store in a pandas.Series object: headlines
headline_data = headline_text_pat.findall(text + '\n[')
headline_texts = []
source_ids = []
for headline in headline_data:
source_id, headline_text = headline[0], ''.join(headline[1:]).strip()
headline_texts.append(headline_text)
source_ids.append(source_id)
# Save as a pandas.Series object: headlines
headlines = pd.Series(data=headline_texts, index=source_ids)
## Extract the urls (for headline, date, mp3)
headline_urls = headline_pat.findall(text)
date_urls = date_pat.findall(text)
mp3_urls = mp3_pat.findall(text)
## Make temporary dataframes
df_headline = pd.DataFrame({'headline_url': headline_urls})
df_date = pd.DataFrame({'date_url': date_urls})
df_mp3 = pd.DataFrame({'mp3_url': mp3_urls})
## Process temporary dataframes to
df_headline['source_id'] = (df_headline['headline_url']
.str.replace(source_id_pat, r"\1", regex=True))
df_date['source_id'] = (df_date['date_url']
.str.replace(source_id_pat, r"\1", regex=True))
df_mp3['source_id'] = (df_mp3['mp3_url']
.str.replace(source_id_pat_mp3, r"\1", regex=True))
df_headline.set_index('source_id', inplace=True)
df_date.set_index('source_id', inplace=True)
df_mp3.set_index('source_id', inplace=True)
## Combine headlines, dates and mp3s together
df = pd.concat([df_headline, df_date, df_mp3], axis=1)
df['source_id'] = df.index
df['headline_text'] = headlines
df.reset_index(drop=True, inplace=True)
else:
df = None
if verbose>0:
print(df)
return df
def download_mp3(url, filename='out.mp3', output_dir=''):
if not filename.endswith('.mp3'):
filename += '.mp3'
r = requests.get(url, allow_redirects=True)
# update filename (add path if applicable)
filename = update_filename(filename=filename,
output_dir=output_dir)
# write to mp3 file
with open(filename, 'wb') as f:
f.write(r.content)
def write_headline(text, filename='out.txt', output_dir=''):
if not filename.endswith('.txt'):
filename += '.txt'
# update filename (add path if applicable)
filename = update_filename(filename=filename,
output_dir=output_dir)
# write to txt file
with open(filename, 'w') as f:
f.write(text)
def update_filename(filename, output_dir=''):
if output_dir:
output_dir = makedir(path=output_dir)
filename = os.path.join(output_dir, filename)
return filename
def makedir(path):
path = os.path.abspath(path)
if not os.path.exists(path):
os.makedirs(path)
return path
## Define output directory
OUTPUT_DIR_HEADLINES = 'output/headlines'
OUTPUT_DIR_AUDIO = 'output/audio'
## Extract data and tabulate as a pandas dataframe
df = tabulate_headlines(text="", filename="test.txt", verbose=1)
## Determine target rows
target_rows_headlines = ~df.headline_text.isna()
target_rows_mp3 = ~df.mp3_url.isna()
## Write headlines to .txt files
print('Total headlines: {}'.format(target_rows_headlines.sum()))
_ = (df
.loc[target_rows_headlines, ["source_id", "headline_text"]]
.apply(lambda row: write_headline(text = row['headline_text'],
filename = row['source_id'],
output_dir=OUTPUT_DIR_HEADLINES),
axis=1))
print('Writing headlines to .txt files: COMPLETE')
## Save audio to .mp3 files
print('Total audio files: {}'.format(target_rows_mp3.sum()))
_ = (df
.loc[target_rows_mp3, ["source_id", "mp3_url"]]
.apply(lambda row: download_mp3(url = row['mp3_url'],
filename = row['source_id'],
output_dir=OUTPUT_DIR_AUDIO),
axis=1))
print('Downloading audio to .mp3 files: COMPLETE')
问题在于你的正则表达式使得标题{}
如果对示例文件运行此正则表达式,您将看到捕获组1存储URl的结尾,组2捕获接下来的两行,组3捕获这两行中的最后一行
因此,问题不在于在文本文件的最后一行追加多次,而在于源数据中
输出
因此,您可以看到,由于您的正则表达式,您的数据中实际上有两次“此处有一些文本”,因此您的append只执行您要求的操作,并将其追加,您需要查看您的正则表达式并将其更改为不再像这样拾取
相关问题 更多 >
编程相关推荐