Testing performance of NER models for Russian

Abstract

This paper describes an experiment in testing 3 NER models (spaCy, Stanza and DeepPavlov) for Russian. The models were tested on WikiAnn Russian subset. Standard evaluation metrics (Precision, Recall and F-score) were complemented with 2 inter-annotator agreement measures (Fleiss’ κ and Krippendorff’s α). DeepPavlov performed better than the other 2 models in terms of all 3 standard performance measures. Stanza performed better than spaCy in terms of precision, while they shared the same Fmeasure score. Both Fleiss’ κ and Krippendorff’s α resulted in an only moderate agreement, showing, that our selected 3 NER models only moderately agree in terms of annotation of Russian named entities