whatlangid 1.0.9

依赖性

可以使用requirements.txt文件安装依赖项：

$ pip install -r requirements.txt

安装

来自Github

$ pip install  --extra-index-url=https://test.pypi.org/simple/ git+https://github.com/bung87/whatlangid

来自PYPI

$ pip install  --extra-index-url=https://test.pypi.org/simple/ whatlangid

基本用法

使用whatlangid

>>>fromwhatlangidimportWhatLangId>>>wtl=WhatLangId()>>>wtl.predict_lang("Mother")'en'>>>wtl.predict_lang("தாய்")'ta'>>>wtl.predict_lang("അമ്മ")'ml'>>>wtl.predict_lang("पिता")'hi'>>>wtl.predict_pro(["English sentence","അമ്മ"])[('en',0.8848170638084412),('ml',0.9535570740699768)]

还支持批预测

>>>wtl.predict_lang(["അമ്മ","पिता","teacher"])['ml','hi','en']

高级使用

wtl = WhatLangId(custom_model=abs_path)

使用更快更精确的bin版本模型，但是 文件大小为126MB

python -m whatlangid.use_bin

支持的语言

支持176种语言。相应语言的iso代码 如下所示。

af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk
ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga
gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km
kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms
mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu
rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr
tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh

车型培训详情

使用fasttext建立的量化模型。更多细节请参见 快速文本blog

参考值

WhatLangId由FastText和langid供电

@article{bojanowski2016enriching,
  title={Enriching Word Vectors with Subword Information},
  author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.04606},
  year={2016}
}

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}