使用函数和for循环将多个文件的文本与列表进行比较

Question

我的最终目标是创建一个遍历多个文件的for循环，并使用一个额外的for循环将术语索引与数据帧进行比较。为了使这更有趣，我还包括了一个函数，因为我可能必须将相同的原理应用于同一数据帧中的另一个变量。有一些问题。你知道吗

1. 我不确定在这种情况下是否应该使用regex，或者一个简单的in语句是否足够。你知道吗
2. 我使用的方法效率不高（更不用说它不起作用了）。我希望有类似于isin语句的东西，但是列表中的每个单词都需要对照数据帧的一行进行检查。然而，我不知道如何应用它，当我试图这样做。。。你知道吗

df:    
     'headline'                                                'source'
     targets is making better stars in the bucks               target news
     more diamonds than rocks in saturn rings                  wishful thinking
     diamond in the rough employees take too many naps         refresh sleep

data:
      'company'
      targets 
      stars in the bucks
      wallymarty
      velocity global
      diamond in the rough

ccompanies = data['company'].tolist() #convert into list 
def find(x): #function to compare df['headline'] against list of companies
    result = []
    companies = set(ccompanies) #edit based on comment, saves time
    for i in companies:
        if i in x:
            result.append(x)
    return result

matches = df['headline'].apply(find)

所需的输出将是与公司匹配的标题列表： targets is making better stars in the bucks diamond in the rough employees take too many naps

编辑：我的脚本已被编辑，所以现在它的工作，并显示标题。但是，输出显示的不是所需的输出，而是数据帧的所有行，只填充了适用的行。你知道吗

Answer 1

在纯熊猫没有迭代和转换成一个列表。你知道吗

首先，将data与df连接起来，这样每个被比较的公司名称的标题都是“复制”的。临时列“key”用于促进此联接。你知道吗

In [60]: data_df = data.to_frame()

In [61]: data_df['key'] = 1

In [63]: df['key'] = 1

In [65]: merged = pd.merge(df, data_df, how='outer', on='key').drop('key', axis=1)

merged看起来像这样。如您所见，根据data的大小，使用此方法可能会得到一个巨大的数据帧。你知道吗

In [66]: merged
Out[66]:
                                             headline            source               company
0         targets is making better stars in the bucks       target news               targets
1         targets is making better stars in the bucks       target news    stars in the bucks
2         targets is making better stars in the bucks       target news            wallymarty
3         targets is making better stars in the bucks       target news       velocity global
4         targets is making better stars in the bucks       target news  diamond in the rough
5            more diamonds than rocks in saturn rings  wishful thinking               targets
6            more diamonds than rocks in saturn rings  wishful thinking    stars in the bucks
7            more diamonds than rocks in saturn rings  wishful thinking            wallymarty
8            more diamonds than rocks in saturn rings  wishful thinking       velocity global
9            more diamonds than rocks in saturn rings  wishful thinking  diamond in the rough
10  diamond in the rough employees take too many naps     refresh sleep               targets
11  diamond in the rough employees take too many naps     refresh sleep    stars in the bucks
12  diamond in the rough employees take too many naps     refresh sleep            wallymarty
13  diamond in the rough employees take too many naps     refresh sleep       velocity global
14  diamond in the rough employees take too many naps     refresh sleep  diamond in the rough

然后在标题中查找文本。如果找到，则在新的“找到”列中输入True，否则输入False。你知道吗

In [67]: merged['found'] = merged.apply(lambda x: x['company'] in x['headline'], axis=1)

然后删除未找到匹配项的标题：

In [68]: found_df = merged.drop(merged[merged['found']==False].index)

In [69]: found_df
Out[69]:
                                             headline         source               company  found
0         targets is making better stars in the bucks    target news               targets   True
1         targets is making better stars in the bucks    target news    stars in the bucks   True
14  diamond in the rough employees take too many naps  refresh sleep  diamond in the rough   True

如有必要，仅对标题和公司进行总结

In [70]: found_df[['headline', 'company']]
Out[70]:
                                             headline               company
0         targets is making better stars in the bucks               targets
1         targets is making better stars in the bucks    stars in the bucks
14  diamond in the rough employees take too many naps  diamond in the rough

快捷方式：步骤67，直到可以使用此命令总结结束

merged.drop(merged[merged.apply(lambda x: x['company'] in x['headline'], axis=1) == False].index)[['headline', 'source']]

Answer 2

... should be using regex in this case or if a simple in statement is sufficient?

使用in很好，因为您显然已经规范化为.lower()，并且删除了标点符号。你知道吗

你真的应该尝试使用更有意义的标识符。例如，通常的习惯用法是for company in companies:，而不是i。你知道吗

你知道如何使用.tolist()，很好。但是您确实希望创建set而不是list，以支持有效的in测试。这是O（1）散列查找和嵌套循环之间的区别，用于列表的线性扫描。你知道吗

这没什么意义：

        for i in ccompanies:
            i = [x]

你开始迭代，但是i本质上变成了一个常量？不清楚你要干什么。你知道吗

如果您将这个项目进行得更深入一点，您可能会考虑将公司与NLTK进行匹配 或者来自scikit learn的TFIDF矢量器， 或https://pypi.org/project/fuzzywuzzy/