frequency dictionary created by "Language Reactor"

[Grander_Gard]

Where can I find the entire list of words from the frequency dictionary created by “Language Reactor”?

[joan_LanguageProcess]

Hi @Grander_Gard,

If you click on your settings gear () then click “set vocabulary level”, it’ll give you a list of words that may be helpful to you:

Other than that, there are no other lists of words that I can think of, and you would need to ask the LR team for a specific language of frequency info — if they are willing to share that since it goes into their algorithm.

I do know that they pull frequency lists from opensubtitles[.org]:

@joan_LanguageProcess

From what I can tell it’s from OpenSubtitles which source their info from http://www.opensubtitles.org/

Source:

@David_Wilkinson

the frequency lists come from subtitles (http://opus.nlpl.eu/OpenSubtitles-v2018.php ). I processed 10,000 subtitles for each language, tokenised the subs into words, then lemmatised. 5,000… was not really enough to have a good list for top 10,000 frequent words… 10,000 subtitles is kind of ok, but more is better I think, I just got a bit impatient to get the feature online, I will run more subs through the script sometime soon.

I hope this is somewhat helpful!

[David_Wilkinson]

There’s also this page, it lists the top 8000 words by frequency, as lemmatised by the NLP code.

The lemmatisation is done by software libraries and is not perfect… the lists start to get noisier above 8000 with incorrect lemmatisations, so we don’t show them.

Word frequency is based 40% on movie subtitles, 40% on Youtube subtitles, 20% on other web content, something like that. The idea was to focus on spoken language, because, well, speaking with people is fun.

The NLP code and the lists should be open-sourced soon.