在Docker容器中加载微调的simpletransformer模型时出错

Question

我正在使用torch.save()和torch.load()命令保存和加载模型

在Docker容器中加载经过微调的简单变压器模型时，我遇到了无法解决的错误：

 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/xlm_roberta/tokenization_xlm_roberta.py", line 161, in __setstate__
    self.sp_model.Load(self.vocab_file)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece.py", line 177, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "/home/jupyter/.cache/huggingface/transformers/9df9ae4442348b73950203b63d1b8ed2d18eba68921872aee0c3a9d05b9673c6.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8": No such file or directory Error #2

如果有人对此有任何想法，请告诉我

我正在使用：

• 火炬==1.7.1+cu101
• 句子变换器0.3.9
• simpletransformers 0.51.15
• 变压器4.4.2
• tensorflow 2.2.0

Answer 1

我建议使用state_dict对象——Python字典，因为它们可以很容易地保存、更新和恢复，为以后恢复模型提供了灵活性。以下是使用state_dict保存模型的推荐保存/加载方法：

保存

torch.save(model.state_dict(), PATH)

加载

model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()