使用SentenceTransformers编码后，嵌入的校验和不同？

Question

我正在使用SentenceTransformers库来计算一些嵌入（embeddings）。但是，当我对句子进行编码并计算它们的嵌入值时，得到的结果却不一样，尤其是当我检查它们的值的总和时。例如：

输入：


RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
random.seed(RANDOM_SEED)
tf.random.set_seed(RANDOM_SEED)
torch.manual_seed(RANDOM_SEED)


transformer_models = [
    'M-CLIP/M-BERT-Distil-40', 
                     ]

sentences = df['content'].tolist()


for transformer_model in tqdm(transformer_models, desc="Transformer Models"):
    tqdm.write(f"Processing with Transformer Model: {transformer_model}")
    model = SentenceTransformer(transformer_model)
    embeddings = model.encode(sentences)
    print(f"Embeddings Checksum for {transformer_model}:", np.sum(embeddings))

输出：

Embeddings Checksum for M-CLIP/M-BERT-Distil-40: 1105.9185

或者

Embeddings Checksum for M-CLIP/M-BERT-Distil-40: 1113.5422

我注意到这种情况发生在我重启并清空jupyter notebook的输出，然后重新运行整个笔记本的时候。有没有人知道怎么解决这个问题？

另外，我尝试在计算嵌入之前和之后设置随机种子：

import torch
import numpy as np
import random
import tensorflow as tf
from sentence_transformers import SentenceTransformer
from tqdm.auto import tqdm

RANDOM_SEED = 42

# Setting seeds
np.random.seed(RANDOM_SEED)
random.seed(RANDOM_SEED)
tf.random.set_seed(RANDOM_SEED)
torch.manual_seed(RANDOM_SEED)

# Ensuring PyTorch determinism
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

transformer_models = ['M-CLIP/M-BERT-Distil-40']

sentences = df['content'].tolist()

for transformer_model in tqdm(transformer_models, desc="Transformer Models"):
    # Set the seed again right before loading the model
    np.random.seed(RANDOM_SEED)
    random.seed(RANDOM_SEED)
    tf.random.set_seed(RANDOM_SEED)
    torch.manual_seed(RANDOM_SEED)

    tqdm.write(f"Processing with Transformer Model: {transformer_model}")
    model = SentenceTransformer(transformer_model, device='cpu')  # Force to use CPU

    embeddings = model.encode(sentences, show_progress_bar=False)  # Disable progress bar and parallel tokenization
    print(f"Embeddings Checksum for {transformer_model}:", np.sum(embeddings))

但是我仍然遇到同样不一致的情况。

更新

我现在尝试的一个方法是把所有计算出的嵌入存储在文件里。可是，我觉得很奇怪的是，做这个的时候我得到的结果却不一样。有没有人之前遇到过这种情况？

jupyter notebook sentence transformers random seed sentence embeddings model consistency embedding storage output verification computational reproducibility

使用SentenceTransformers编码后，嵌入的校验和不同？

1 个回答

撰写回答