Source of the Corpus used

[bharatingermany]

Dear fellow learners,

I would like to know how was the frequency list made? Generally all frequency list / dictionaries have the data on which the list is made mentioned. I saw David mention somewhere that it is based on NLP. I do not understand it.

If anybody can give me some more context, that would be lovely!

Thank you for your time!
Bharat

[joan_LanguageProcess]

Hi @bharatingermany,

Here is what I can tell you from the deep dive I’ve done in the forum in the past:

I know that they pull frequency lists from opensubtitles[.org]:

@joan_LanguageProcess

From what I can tell it’s from https://opus.nlpl.eu/OpenSubtitles-v2018.php which source their info from http://www.opensubtitles.org/

Source:

@David_Wilkinson

the frequency lists come from subtitles (http://opus.nlpl.eu/OpenSubtitles-v2018.php ). I processed 10,000 subtitles for each language, tokenised the subs into words, then lemmatised. 5,000… was not really enough to have a good list for top 10,000 frequent words… 10,000 subtitles is kind of ok, but more is better I think, I just got a bit impatient to get the feature online, I will run more subs through the script sometime soon.

And after Google it “NLP” refers to a type of programing language.

I hope some of this is helpful.