A simple python 3.11 script to generate child-parent lemma mappings for import into Lute.
This uses spaCy-stanza, a wrapper around Stanza (formerly StanfordNLP) to find lemma. Stanza has models for over 60 languages, see this page for the complete list.
For example, given the following Spanish input file (in demo/es_input.txt):
perros
perro
perras
vives
vivimos
vivieron
muchacho
muchacha
coches
vez
The script generates a file containing the following commented file, where the first column is the root or parent form, and the second is the child form from the input file:
### 3 single child parents:
coche coches
perra perras
perro perros
### 1 multi-child parents:
vivir vives
vivir vivieron
vivir vivimos
Only cases where the lemma form differs from the original term are included, so this doesn't show the terms muchacho, vez, que, etc.
(If you know Spanish, you'll see that some of the above aren't really useful ... but the spaCy pipeline is often very, very good.)
- python3.11 (perhaps will work with earlier versions, untested)
Use pip3.11:
python3.11 -m venv .env
source .env/bin/activate
pip3.11 install -r requirements.txt
$ source .env/bin/activate
# Ignore warnings with -W ignore
#
# 1st arg: Stanza language code of the terms (link below)
# 2nd arg: path to input file
# 3rd arg: path to the output file
#
$ python3.11 -W ignore main.py es demo/es_input.txt output.txt
Opening library ...
Done.
Loading pipeline ...
Done. Processing 1 batches.
1 of 1
File generated: output_1.txt
Please remove any mappings you don't want from this file before importing it.
Stanza language codes: Stanza has models for over 60 languages, see this page for the complete list.