Skip to content

Latest commit

 

History

History
139 lines (123 loc) · 4.34 KB

Readme.md

File metadata and controls

139 lines (123 loc) · 4.34 KB
Raw

KFTT

Morphosyntactic tagger for Polish, the winner of PolEval 2020 Task 2: Morphosyntactic tagging of Middle, New and Modern Polish; successor of KRNNT.

Model Accuracy Acc on known Acc on ign Acc on manual known
KFTT train+devel wo_morf 95.63% 95.95% 81.91% 67.30%
KFTT train+devel 95.73% 96.07% 81.02% 67.81%
KRNNT train 92.07% 93.00% 48.63%

KFTT train+devel accuracy on different parts of the test corpus:

Corpus Period Accuracy Acc on known Acc on ign Acc on manual
KorBa — a corpus of 17th and 18th century Middle 94.35% 94.83% 79.43% 73.87%
a corpus of 19th century New 96.94% 97.15% 83.24% 78.39%
1M subcorpus of the National Corpus of Polish NKJP Modern 97.37% 97.48% 87.78% 84.07%

PolEval 2020 results reproduction

Install PyTorch with a compatible version of CUDA with your drivers.

pip install torch

Install requirements:

pip install -r requirements.txt

Download test data:

NAME="test"

mkdir input_data
cd input_data
wget http://poleval.pl/task2/${NAME}-plain.tar.gz
tar -xf ${NAME}-plain.*
cd ..

Convert DAGs to JSONL format:

mkdir data
python3 dag_to_jsonl.py "input_data/${NAME}-plain/*" data/${NAME}-plain.jsonl poleval2020-${NAME}

Prepare data for tokenization:

python3 jsonl_to_tsv_segmentation_every_char.py data/${NAME}-plain.jsonl data/${NAME}-plain.segmentation.tsv.char

Download tokenization models:

mkdir models
cd models
wget https://github.com/kwrobel-nlp/kftt/releases/download/v0.1/ModelB+allF+CRF.pt
wget https://github.com/kwrobel-nlp/kftt/releases/download/v0.1/ModelB+CRF.pt
cd ..

Choose tokenization model:

MODEL_DIR="models/ModelB+allF+CRF.pt"

or (wo_morf):

MODEL_DIR="models/ModelB+CRF.pt"

Tokenize:

INPUT="data/${NAME}-plain.segmentation.tsv.char"
OUTPUT=${INPUT}.`basename "$MODEL_DIR"`
python3 predict_segmentation.py ${MODEL_DIR} $INPUT "$OUTPUT"
# Tokenization: 14.21 seconds

Prepare data for tagging:

python3 test_data_tagging.py data/${NAME}-plain.jsonl $OUTPUT data/${NAME}-plain.segmentation.tsv.char ${OUTPUT}.tagging
echo ${OUTPUT}.tagging
# data/test-plain.segmentation.tsv.char.ModelB+allF+CRF.pt.tagging
mkdir test_data_temp
cp ${OUTPUT}.tagging test_data_temp/test.txt

Download model:

cd models
wget https://github.com/kwrobel-nlp/kftt/releases/download/v0.1/train+dev.zip
unzip train+dev.zip
cd ..

Tag:

export BERT_MODEL=models/train+dev
time python3 run_ner_predict.py --data_dir ./test_data_temp/ \
--model_type xlmroberta \
--labels models/train+dev/labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $BERT_MODEL \
--max_seq_length 512 \
--per_gpu_eval_batch_size 1 \
--do_predict
# Tagging: 16.61 seconds

Copy results:

cp ${OUTPUT_DIR}/test_predictions_long.txt ${OUTPUT}.tagged

Join results and split to separate files:

mkdir predictions
python3 split_to_separate_files.py data/${NAME}-plain.jsonl ${OUTPUT}.tagged predictions --sep $'\t'

Evaluate:

python3 poleval-eval.py predictions input_data/${NAME}-disamb 2>/dev/null

Results:

Poleval 2020 Task 2

Evaluating files in: predictions
against 252 gold standard files in: input_data/test-disamb


Accuracy (Your score!): 0.9572980397053314

Tokens total:           40045
Correct tokens:         38335
Unknown tokens:         901 (2.25%)
Correct unknown:        730
Accuracy on unknown:    0.8102108768035516
Known tokens:           39144 (97.75%)
Accuracy on known:      0.960683629675046
Manual tokens:          1482 (known 581 + ign 901)
Correct manual:         1124
Accuracy on manual:     0.7584345479082322
Accuracy manual known:  0.6781411359724613
Accuracy manual ign:    0.8102108768035516