如何用已计算的TFIDF分数计算余弦相似度

Question

我需要用已经计算的TFIDF分数计算文档之间的余弦相似度。在

通常我会使用TFIDFVectorizer来创建文档/术语的矩阵，计算TFIDF的分数。我不能应用它，因为它将重新计算TFIDF分数。这是不正确的，因为文档已经进行了大量的预处理，包括单词包和IDF过滤（我不解释原因，时间太长）。在

说明性输入CSV文件：

Doc, Term,    TFIDF score
1,   apples,  0.3
1,   bananas, 0.7
2,   apples,  0.1
2,   pears,   0.9
3,   apples,  0.6
3,   bananas, 0.2
3,   pears,   0.2

我需要生成通常由TFIDFVectorizer生成的矩阵，例如：

^{pr2}$

。。。这样我就可以计算文档之间的余弦相似度。在

我使用的是Python2.7，但欢迎提供其他解决方案或工具的建议。我很难切换到python3。在

编辑：

这不是真的要转移numpy数组。它涉及到将TFIDF分数映射到文档/术语矩阵，使用标记化的术语，缺少的值填充为0。在

Answer 1

如果您可以使用pandas先在一个数据帧中读取整个csv文件，它会变得更容易。在

import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder

df = pd.read_csv('sample.csv', index_col=None, skipinitialspace=True)

# Converting the text Term to column index
le = LabelEncoder()
df['column']=le.fit_transform(df['Term'])

# Converting the Doc to row index
df['row']=df['Doc'] - 1

# Rows will be equal to max index of document
num_rows = max(df['row'])+1

# Columns will be equal to number of distinct terms
num_cols = len(le.classes_)

# Initialize the array with all zeroes
tfidf_arr = np.zeros((num_rows, num_cols))

# Iterate the dataframe and set the appropriate values in tfidf_arr
for index, row in df.iterrows():
    tfidf_arr[row['row'],row['column']]=row['TFIDF score']

一定要仔细阅读评论，如果不理解任何东西。在

Answer 2

一个低效的黑客，我会离开这里，以防它帮助别人。欢迎提出其他建议。在

def calculate_cosine_distance():
    unique_terms = get_unique_terms_as_list()

    tfidf_matrix = [[0 for i in range(len(unique_terms))] for j in range(TOTAL_NUMBER_OF_BOOKS)]

    with open(INPUT_FILE_PATH, mode='r') as infile:
        reader = csv.reader(infile.read().splitlines(), quoting=csv.QUOTE_NONE)

        # Ignore header row
        next(reader)

        for rows in reader:
            book = int(rows[0]) - 1 # To make it a zero-indexed array
            term_index = int(unique_terms.index(rows[1]))
            tfidf_matrix[book][term_index] = rows[2]

    # Calculate distance between book X and book Y
    print cosine_similarity(tfidf_matrix[0:1], tfidf_matrix)

def get_unique_terms_as_list():
    unique_terms = set()
    with open(INPUT_FILE_PATH, mode='rU') as infile:
        reader = csv.reader(infile.read().splitlines(), quoting=csv.QUOTE_NONE)
        # Skip header
        next(reader)
        for rows in reader:
            unique_terms.add(rows[1])

        unique_terms = list(unique_terms)
    return unique_terms

Answer 3

我建议使用scipy.sparse中的稀疏矩阵

from scipy.sparse import csr_matrix, coo_matrix
from sklearn.metrics.pairwise import cosine_similarity

input="""Doc, Term,    TFIDF score
1,   apples,  0.3
1,   bananas, 0.7
2,   apples,  0.1
2,   pears,   0.9
3,   apples,  0.6
3,   bananas, 0.2
3,   pears,   0.2"""

voc = {}

# sparse matrix representation: the coefficient
# with coordinates (rows[i], cols[i]) contains value data[i]
rows, cols, data = [], [], []

for line in input.split("\n")[1:]: # dismiss header

    doc, term, tfidf = line.replace(" ", "").split(",")

    rows.append(int(doc))

    # map each vocabulary item to an int
    if term not in voc:
        voc[term] = len(voc)

    cols.append(voc[term])
    data.append(float(tfidf))

doc_term_matrix = coo_matrix((data, (rows, cols)))

# compressed sparse row matrix (type of sparse matrix with fast row slicing)
sparse_row_matrix = doc_term_matrix.tocsr()

print("Sparse matrix")
print(sparse_row_matrix.toarray()) # convert to array

# compute similarity between each pair of documents
similarities = cosine_similarity(sparse_row_matrix)

print("Similarity matrix")
print(similarities)

输出：

^{pr2}$