Language identification for Lithuanian, Russian and Azeri languages

Abstract

Language identification is an important part of Natural Language Processing, because most of techniques are language sensitive, and therefore in multi-language systems language should be identified before further processing steps. Techniques and tools for more popular languages are well defined and are available in commercial and open source tools, but are not defined for less popular languages. In this work we investigate techniques for Lithuanian, Russian and Azeri (Azjerbaijani) languages. Corpora for these and similar languages (Latvian, Ukrainian, Belarusian, Turkish and Turkmen (Turkman) were collected and prepared. Selected approaches were trained. Results were evaluated using precision, recall and F-score