What is a language detector?
It is a tool that identifies which language a text is written in. For example, detect("Gracias") returns "es" for Spanish.
Efficient language detector (Nito-ELD or ELD) is a fast and accurate language detector, is one of the fastest non compiled detectors, while its accuracy is within the range of the heaviest and slowest detectors.
It's 100% JavaScript (vanilla), easy installation and no dependencies.
ELD is also available in PHP (v3), C Library (v3, includes Python package & executable), and an outdated Python (v1) implementation. ELD JavaScript is v2.
- For Node.js
$ npm install eld- For Web, just download or clone the files
git clone https://github.com/nitotm/efficient-language-detector-js
Importing a static, fixed size eld database. Options: 'eld/large', 'eld/medium', 'eld/small', 'eld/extrasmall'
- At Node.js
import { eld } from 'eld/large' // use .mjs extension for version <18- At Node.js REPL
const { eld } = await import('eld/large')- At the Web Browser
<script type="module" charset="utf-8">
import { eld } from './src/entries/static.large.js' // Update path.
// './src/entries/dynamic.js' for dynamic eld
</script>- To load a pre-built minified version (iife), it is not a module. Included at /minified (GitHub)
<script src="minified/eld.xs.min.js" charset="utf-8"></script>If we use dynamic 'eld', we need to load() a database to initialize.
Available sizes: 'large', 'medium', 'small' & 'extrasmall'
- Node.js example (Works also with all options displayed at static import)
import { eld } from 'eld' // use .mjs extension for version <18
const detector = eld.newInstance() // Isolated settings instance (Introduced in v2.1.0)
await detector.load('large') // Unavailable for static eld with preloaded database
// await eld.load('large') We can also load a database at global instancedetect() expects a UTF-8 string, and returns an object, with a language variable, with a ISO 639-1 code or empty string
const detector = eld.newInstance() // Isolated configuration instance. (Instances introduced in v2.1.0)
console.log( detector.detect('Hola, cómo te llamas?') )
// { language: 'es', getScores(): {'es': 0.5, 'et': 0.2}, isReliable(): true }
// returns { language: string, getScores(): Object, isReliable(): boolean }
console.log( detector.detect('Hola, cómo te llamas?').language )
// 'es'
// We can also use eld directly, with shared global settings per entry point type.
// Dynamic import is one global instance, and each static size is a global instance.
eld.detect('Hola, cómo te llamas?') // 'es'- To reduce the languages to be detected, there are 2 options, they only need to be executed once. (Check available languages below)
let languagesSubset = ['en', 'es', 'fr', 'it', 'nl', 'de']
// Option 1
// Setting setLanguageSubset(), detect() executes normally but finally filters the excluded languages
detector.setLanguageSubset(languagesSubset) // Returns an Object with the subset validated languages
// to remove the subset
detector.setLanguageSubset(false)
// The global `eld` instance has all methods and configuration options a `newInstance()` instance has
eld.setLanguageSubset(languagesSubset)
// Option 2 ( NOT available for static eld, with preloaded DB size )
// The optimal way to regularly use the same subset, is using saveSubset() to download a new database
detector.saveSubset(languagesSubset) // ONLY for the Web Browser
// We can load any Ngrams database saved at src/ngrams/, including subsets. Returns true if success
await detector.load('medium')
// eld.load('file').then((loaded) => { if (loaded) { } })- Also, we can get the current status of eld: languages, database type and subset
console.log( detector.info() )Build and minify static size example, with esbuild + terser. With npm package installed:
npx esbuild --bundle --format=esm eld/large --outfile=eld.large.js
terser eld.large.js --compress --mangle --output eld.large.min.js
Using folder path:
npx esbuild --bundle --format=esm src/entries/static.large.js > eld.large.js
For non-module iife browser scripts:
npx esbuild --bundle --format=iife --global-name=__eld_module src/entries/static.extrasmall.js > eld.xs.js --footer:js="globalThis.eld = __eld_module.default;"
For a client side solution, I included at /minified (GitHub) an iife bundle file size XS, which still performs great for sentences.
The XS version weights 940kb, when gzipped it's only 264kb.
I compared ELD with a different variety of detectors.
| URL | Version | Core Language |
|---|---|---|
| https://github.com/nitotm/efficient-language-detector-js/ | 2.0.0 | Javascript |
| https://github.com/nitotm/efficient-language-detector/ | 1.0.0 | PHP |
| https://github.com/pemistahl/lingua-py | 1.3.2 | Python |
| https://github.com/CLD2Owners/cld2 | Aug 21, 2015 | C++ |
| https://github.com/google/cld3 | Aug 28, 2020 | C++ |
| https://github.com/wooorm/franc | 6.1.0 | Javascript |
Benchmarks: Tweets: 700KB, short sentences of 140 chars max.; Big test: 10MB, sentences in all 60 languages supported; Sentences: 8MB, this is the Lingua sentences test, minus unsupported languages.
Short sentences is what ELD and most detectors focus on, as very short text is unreliable, but I included the Lingua Word pairs 1.5MB, and Single words 800KB tests to see how they all compare beyond their reliable limits.
These are the results, first, accuracy and then execution time.
1. Lingua could have a small advantage as it participates with 54 languages, 6 less.
2. CLD2 and CLD3, return a list of languages, the ones not included in this test where discarded, but usually they return one language, I believe they have a disadvantage.
Also, I confirm the results of CLD2 for short text are correct, contrary to the test on the Lingua page, they did not use the parameter "bestEffort = True", their benchmark for CLD2 is unfair.
The RAM memory usage for each DB size is XS: 37MB, S: 54MB, M: 71MB, L: 138MB.
These are the ISO 639-1 codes of the 60 supported languages for Nito-ELD
'am', 'ar', 'az', 'be', 'bg', 'bn', 'ca', 'cs', 'da', 'de', 'el', 'en', 'es', 'et', 'eu', 'fa', 'fi', 'fr', 'gu', 'he', 'hi', 'hr', 'hu', 'hy', 'is', 'it', 'ja', 'ka', 'kn', 'ko', 'ku', 'lo', 'lt', 'lv', 'ml', 'mr', 'ms', 'nl', 'no', 'or', 'pa', 'pl', 'pt', 'ro', 'ru', 'sk', 'sl', 'sq', 'sr', 'sv', 'ta', 'te', 'th', 'tl', 'tr', 'uk', 'ur', 'vi', 'yo', 'zh'
Full name languages:
Amharic, Arabic, Azerbaijani (Latin), Belarusian, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, English, Spanish, Estonian, Basque, Persian, Finnish, French, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Armenian, Icelandic, Italian, Japanese, Georgian, Kannada, Korean, Kurdish (Arabic), Lao, Lithuanian, Latvian, Malayalam, Marathi, Malay (Latin), Dutch, Norwegian, Oriya, Punjabi, Polish, Portuguese, Romanian, Russian, Slovak, Slovene, Albanian, Serbian (Cyrillic), Swedish, Tamil, Telugu, Thai, Tagalog, Turkish, Ukrainian, Urdu, Vietnamese, Yoruba, Chinese
Donate / Hire
If you wish to Donate for open source improvements, Hire me for private modifications / upgrades, or to Contact me, use the following link: https://linktr.ee/nitotm