notebooks/TextPreprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/github/LUMII-AILab/NLP_Course/blob/main/notebooks/TextPreprocessing.ipynb\" target=\"_new\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
      ],
      "metadata": {
        "id": "x-UrgOqGM3r3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Rīkkopas teksta priekšapstrādei | Toolkits for text preprocessing\n",
        "\n",
        "**LV**: Šajā nodarbībā turpināsim izmantot *Python* moduļus darbam ar regulārajām izteiksmēm, un aplūkosim NLP rīkkopas [NLTK](https://www.nltk.org/), [spaCy](https://spacy.io/) un [Stanza](https://stanfordnlp.github.io/stanza/), kā arī [Hugging Face](https://github.com/huggingface/tokenizers/) BPE bibliotēku.\n",
        "\n",
        "Nākamajās nodarbībās redzēsim, ka šeit aplūkotās daudzfunkcionālās rīkkopas ir izmantojamas ne vien teksta priekšapstrādei, bet arī teksta tālākajai analīzei."
      ],
      "metadata": {
        "id": "krvncS3Glj-g"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install the packages\n",
        "!pip install nltk spacy spacy-transformers stanza\n",
        "\n",
        "# Import the packages\n",
        "import nltk\n",
        "import spacy\n",
        "import stanza"
      ],
      "metadata": {
        "id": "YXA9WEdHmvUB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Darbs ar NLTK datu kopām | Working with NLTK datasets\n",
        "\n",
        "**LV**: NLTK ietver daudzveidīgu teksta korpusu un leksisko resursu kolekciju. Šos korpusus u.c. resursus iespējams lejuplādēt un strādāt ar tiem, izmantojot `nltk.corpus` pakotni. Vairāk informācijas: https://www.nltk.org/data.html\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: NLTK includes a diverse set of preprocessed text corpora and lexical resources that can be downloaded and manipulated using the `nltk.corpus` package. More info: https://www.nltk.org/data.html"
      ],
      "metadata": {
        "id": "cU8ze2lsTuOQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Open the NLTK Downloader CLI\n",
        "nltk.download()\n",
        "\n",
        "# Download the Brown corpus (see https://en.wikipedia.org/wiki/Brown_Corpus)\n",
        "nltk.download('brown')\n",
        "\n",
        "# Import the Brown corpus\n",
        "from nltk.corpus import brown"
      ],
      "metadata": {
        "id": "peaPz53jKAyz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the size of the corpus\n",
        "total_size = len(brown.words())\n",
        "print(\"Total number of words:\", total_size)\n",
        "\n",
        "# List the text categories (genres) in the corpus\n",
        "for cat in brown.categories():\n",
        "    doc_per_cat = len(brown.fileids(categories=cat))\n",
        "    words_per_cat = len(brown.words(categories=cat))\n",
        "    cat_percentage = words_per_cat / total_size\n",
        "\n",
        "    print(f'\\t{cat:10}\\t{doc_per_cat}\\t{words_per_cat}\\t{cat_percentage:.2%}')"
      ],
      "metadata": {
        "id": "xa0iGN2sLBuz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LV**: Piezīme: salīdziniet līdzsvarotā *Brown* (1964!!) korpusa žanru proporcijas un līdzvarotā latviešu valodas korpusa [LVK2022](https://korpuss.lv/id/LVK2022) [žanru proporcijas](https://nosketch.korpuss.lv/#wordlist?corpname=LVK2022&tab=attribute&onecolumn=1&wlattr=doc.section&wlminfreq=1&include_nonwords=1&showresults=1)."
      ],
      "metadata": {
        "id": "ofohRnK3pgIA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Work with a sub-corpus\n",
        "print(\"humor:\", brown.fileids(categories='humor'))\n",
        "print(\"cr01:\", len(brown.words('cr01')), '\\n')\n",
        "\n",
        "# NLTK supports reading a corpus as a raw/annotated text..\n",
        "print(\"Raw text with POS tags:\", brown.raw('cr01').split('\\n\\n')[0], '\\n')\n",
        "\n",
        "# ..as well as a list of words, sentences, or paragraphs\n",
        "print(\"Words:\", brown.words('cr01'))\n",
        "print(\"Sentences:\", brown.sents('cr01'))\n",
        "print(\"Paragraphs:\", brown.paras('cr01'))"
      ],
      "metadata": {
        "id": "CheE1cfgjmul"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dalīšana tekstvienībās un teikumos | Tokenization and sentence splitting"
      ],
      "metadata": {
        "id": "4q4zPBBZThUp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### RegEx\n",
        "\n",
        "**LV**: Vispirms aplūkosim, kā varam mēģināt risināt tekstvienību un teikumu segmentēšanu, izmantojot tikai regulārās izteiksmes."
      ],
      "metadata": {
        "id": "hEAOZk3hb954"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's begin with the built-in re module\n",
        "import re\n",
        "\n",
        "# We will need BeautifulSoup again\n",
        "!pip install beautifulsoup4\n",
        "from bs4 import BeautifulSoup"
      ],
      "metadata": {
        "id": "mK1YYrcgjagF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Download some non-NLTK corpora:\n",
        "!wget -O \"Rainis.txt\" https://repository.clarin.lv/repository/xmlui/bitstream/handle/20.500.12574/41/rainis_v20180716.txt\n",
        "!wget -O \"Romeo.txt\" https://www.gutenberg.org/cache/epub/1112/pg1112.txt\n",
        "\n",
        "# See also https://www.gutenberg.org"
      ],
      "metadata": {
        "id": "F6k6O5A8oDRh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Alternatively, upload files from your local file system\n",
        "from google.colab import files\n",
        "files.upload()"
      ],
      "metadata": {
        "id": "RLW1bSBkwCnn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Read and clean up the downloaded corpora\n",
        "\n",
        "with open(\"Romeo.txt\", mode='r', encoding='utf-8') as file_en:\n",
        "    text_en = file_en.read()\n",
        "\n",
        "with open(\"Rainis.txt\", mode='r', encoding='utf-8') as file_lv:\n",
        "    text_lv = file_lv.read().split('</doc>')[0]\n",
        "    text_lv = BeautifulSoup(text_lv, 'html.parser').text\n",
        "\n",
        "text_en = text_en.strip()\n",
        "text_lv = text_lv.strip()"
      ],
      "metadata": {
        "id": "jYLWWe6uXQWo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Normalize white spaces (incl. line breaks)\n",
        "text_en = re.sub(r'\\s+', ' ', text_en)\n",
        "text_lv = re.sub(r'\\s+', ' ', text_lv)\n",
        "\n",
        "print(text_en)\n",
        "print(text_lv)"
      ],
      "metadata": {
        "id": "016CQsl5EQ8t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Simplified (and lossy) tokenization\n",
        "\n",
        "words_en = re.split(r'\\W+', text_en)\n",
        "print(len(words_en), words_en)\n",
        "\n",
        "words_lv = re.split(r'\\W+', text_lv)\n",
        "print(len(words_lv), words_lv)\n",
        "\n",
        "words_en = re.findall('[A-z]+', text_en) # vs. \\w+\n",
        "print(len(words_en), words_en) # FIXME: https://en.wikipedia.org/wiki/ASCII#Character_set\n",
        "\n",
        "words_lv = re.findall('[A-Za-zĀāČčĒēĢģĪīĶķĻļŅņŌōŖŗŠšŪūŽž]+', text_lv)\n",
        "print(len(words_lv), words_lv)"
      ],
      "metadata": {
        "id": "zm2fE8RXo-on"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LV**: Šādi izgūstam vārdus, bet pazaudējam citas tekstvienības un sadalām kompleksus vārdus (piem., \"Covid-19\").\n",
        "\n",
        "Mēģināsim uzlabot šablonu un izmantosim ārējo `regex` moduli, kas ir savietojams ar iebūvēto `re`, var uzlabot ātrdarbību, kā arī nodrošina papildu iespējas, piemēram, ērti definējamas *Unicode* rakstzīmju klases: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category."
      ],
      "metadata": {
        "id": "l3soSF83Bl4v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install regex\n",
        "\n",
        "import regex"
      ],
      "metadata": {
        "id": "plMKo3R6YrUQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "words_en = regex.findall('\\p{L}+|\\p{P}', text_en)\n",
        "words_lv = regex.findall('\\p{L}+|\\p{P}', text_lv)\n",
        "\n",
        "print(text_en + '\\n' + ' '.join(words_en) + '\\n' + str(len(words_en)) + '\\n')\n",
        "print(text_lv + '\\n' + ' '.join(words_lv) + '\\n' + str(len(words_lv)) + '\\n')\n",
        "\n",
        "# TODO: wee'l, o'th, peoples', eye-sight, sald-sāpīgs, zil-ziediņi, sarkan-zili-zaļi"
      ],
      "metadata": {
        "id": "9ovaEaUs7ll0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LV**: Mēģināsim dalīt tekstu teikumos un rindkopās, izmantojot vienkāršas regulārās izteiksmes.\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: An attempt to split a text into sentences and paragraph using simple regular expressions."
      ],
      "metadata": {
        "id": "cpCA2a4hPltV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: Re-load text_en and text_lv\n",
        "\n",
        "# Naive assumptions:\n",
        "# (1) the full-stop characters indicate end-of-sentence;\n",
        "# (2) line breaks indicate end-of-paragraph.\n",
        "\n",
        "# Sentence splitting\n",
        "text_en = regex.sub(r'(?<=[.!?])[ ]+', '[SENT]', text_en)\n",
        "text_lv = regex.sub(r'(?<=[.!?])[ ]+', '[SENT]', text_lv)\n",
        "\n",
        "# Paragraph splitting\n",
        "text_en = regex.sub(r'\\n+([ ]+\\n+)?', '[PARA]', text_en)\n",
        "text_lv = regex.sub(r'\\n+([ ]+\\n+)?', '[PARA]', text_lv)\n",
        "\n",
        "# Normalization of the remaining white spaces\n",
        "text_en = regex.sub(r'\\s+', ' ', text_en)\n",
        "text_lv = regex.sub(r'\\s+', ' ', text_lv)\n",
        "\n",
        "print(text_en)\n",
        "print(text_lv)"
      ],
      "metadata": {
        "id": "q-1X48tXphxm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### NLTK\n",
        "\n",
        "**EN**: The NLTK sentence splitter (`sent_tokenize`) and tokenizer (`word_tokenize`) use pre-trained models and heuristics that take into account the complexity and irregularities of natural language text.\n",
        "\n",
        "Before using these functions, ensure you have downloaded the *Punkt* tokenizer models. This NLTK data package includes a pre-trained model for English and some other languages.\n",
        "\n",
        "*Punkt* uses an unsupervised algorithm to build a tokenization and sentence splitting model. It must be trained on a large plain-text corpus in the target language.\n",
        "\n",
        "See https://www.nltk.org/api/nltk.tokenize.punkt.html"
      ],
      "metadata": {
        "id": "thuCr0KGUD9d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('punkt')\n",
        "\n",
        "# To list the available Punkt models:\n",
        "\n",
        "import os\n",
        "\n",
        "punkt_path = os.path.join(nltk.data.find('tokenizers/punkt'), '')\n",
        "punkt_files = [f for f in os.listdir(punkt_path) if f.endswith('.pickle')]\n",
        "\n",
        "print([f.replace('.pickle', '') for f in punkt_files])"
      ],
      "metadata": {
        "id": "yIAXhgHJpw-7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "short_text_en = '''\n",
        "    Punkt knows that the periods in Mr. Smith and Johann S. Bach\n",
        "    do not mark sentence boundaries. And sometimes sentences\n",
        "    start with non-capitalized words. i is a good variable\n",
        "    name.\n",
        "    You may copy it, give it away or re-use it under the terms\n",
        "    of the Project Gutenberg License on-line\n",
        "    at www.gutenberg.org. If you're not in the US,\n",
        "    you'll have to check the laws where you are located\n",
        "    before using this e-Book.\n",
        "    '''\n",
        "\n",
        "short_text_lv = '''\n",
        "    TĀLAS NOSKAŅAS ZILĀ VAKARĀ\n",
        "    Vēlreiz garā tuvajiem mīļā dzimtenē sirsnīgus sveicienus!\n",
        "    Daudz simtu jūdžu tāļumā,\n",
        "    aiz tīreļiem, purviem un siliem,\n",
        "    guļ mana dzimtene diendusā.\n",
        "    Tā aizsegta debešiem ziliem,\n",
        "    zil-saulainiem debešu palagiem\n",
        "    pret dvesmām un strāvām, un negaisiem...\n",
        "    Piemērs tapis 2024. gadā. Šis u.c. piemēri.\n",
        "    '''"
      ],
      "metadata": {
        "id": "Hzm1eKh3XwGg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import sent_tokenize, word_tokenize\n",
        "\n",
        "sents_en = sent_tokenize(short_text_en)\n",
        "\n",
        "for s in sents_en:\n",
        "    s = regex.sub(r'\\s+', ' ', s.strip())\n",
        "    w = word_tokenize(s)\n",
        "    print(s + '\\n' + ' '.join(w) + '\\n')"
      ],
      "metadata": {
        "id": "rUjGBMC_uz_s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# No Punkt model for Latvian (and many other languages) -\n",
        "# let's try a model for some nearest supported language.\n",
        "\n",
        "sents_lv = sent_tokenize(short_text_lv, language='polish')\n",
        "\n",
        "for s in sents_lv:\n",
        "    s = regex.sub(r'\\s+', ' ', s.strip())\n",
        "    w = word_tokenize(s, language='polish')\n",
        "    print(s + '\\n' + ' '.join(w) + '\\n')\n",
        "\n",
        "# TODO: a potential mini-project - train and use a Punkt model for a new language"
      ],
      "metadata": {
        "id": "uOefAhZ7A_gj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### spaCy"
      ],
      "metadata": {
        "id": "YYR3KFe2g5CJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# First, we must download language models trained for the languages of interest.\n",
        "# See https://spacy.io/usage/models\n",
        "\n",
        "!python -m spacy download en_core_web_sm\n",
        "!python -m spacy download lt_core_news_sm\n",
        "!python -m spacy download xx_sent_ud_sm\n",
        "\n",
        "nlp_en = spacy.load(\"en_core_web_sm\")\n",
        "nlp_lt = spacy.load(\"lt_core_news_sm\")\n",
        "nlp_xx = spacy.load(\"xx_sent_ud_sm\")\n",
        "\n",
        "# Assumption: LT should be the closest one to LV.\n",
        "# Compare to the multilingual (XX) models.\n",
        "# Compare to the large models (lg and rtf)!\n",
        "\n",
        "# PS. The xx_ent_wiki_sm model does not include a component for setting sentence\n",
        "# boundaries by default - the sentencizer has to be added to the nlp_xx pipeline\n",
        "# before using it: nlp_xx.add_pipe('sentencizer').\n",
        "# Still, the xx_sent_ud_sm model is more accurate for LV sentence splitting."
      ],
      "metadata": {
        "id": "JCm4V6-7C8C4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install spacy-transformers\n",
        "!python -m spacy download en_core_web_trf\n",
        "nlp_en = spacy.load(\"en_core_web_trf\")"
      ],
      "metadata": {
        "id": "i-0RelIRUB4e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download lt_core_news_lg\n",
        "nlp_lt = spacy.load(\"lt_core_news_lg\")"
      ],
      "metadata": {
        "id": "VVxYqBY5Y-An"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Pass a plain-text to the respective language model,\n",
        "# which returns a processed spaCy document.\n",
        "doc_en = nlp_en(short_text_en) # TODO: vs. en-trf\n",
        "\n",
        "# For each sentence in the EN text, tokenize it and pretty-print\n",
        "for sent in doc_en.sents:\n",
        "    sent_text = ' '.join(tok.text for tok in sent)\n",
        "    print(regex.sub(r'\\s+', ' ', sent_text.strip()))"
      ],
      "metadata": {
        "id": "UkMaL10Ig3wf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Try Latvian with the multilingual model\n",
        "doc_lv = nlp_xx(short_text_lv) # TODO: vs. lt-lg\n",
        "\n",
        "for sent in doc_lv.sents:\n",
        "    sent_text = ' '.join(tok.text for tok in sent)\n",
        "    print(regex.sub(r'\\s+', ' ', sent_text.strip()))"
      ],
      "metadata": {
        "id": "pwrDz46_ARzQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Stanza"
      ],
      "metadata": {
        "id": "HTrruniedB2M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the necessary models\n",
        "# See https://stanfordnlp.github.io/stanza/performance.html\n",
        "\n",
        "stanza.download('en')\n",
        "stanza.download('lv')"
      ],
      "metadata": {
        "id": "uPQ3sbbodFj5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create NLP pipelines\n",
        "nlp_en = stanza.Pipeline('en')\n",
        "nlp_lv = stanza.Pipeline('lv')"
      ],
      "metadata": {
        "id": "3X7Iomz_e_r8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Process the texts\n",
        "doc_en = nlp_en(short_text_en)\n",
        "doc_lv = nlp_lv(short_text_lv)"
      ],
      "metadata": {
        "id": "mDYQ3Mglfoqg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "twSVVH-WaMSp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the EN sentences and tokens\n",
        "for i, sent in enumerate(doc_en.sentences):\n",
        "    print(i, ' '.join(tok.text for tok in sent.tokens))"
      ],
      "metadata": {
        "id": "8Mb8Vg5AmDWN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the LV sentences and tokens\n",
        "for i, sent in enumerate(doc_lv.sentences):\n",
        "    print(i, ' '.join(tok.text for tok in sent.tokens))"
      ],
      "metadata": {
        "id": "3PeJ8QvhmBDz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## BPE tokenizācija\n",
        "\n",
        "*HuggingFace* BPE tokenizācijas bibliotēka: https://github.com/huggingface/tokenizers/"
      ],
      "metadata": {
        "id": "xa1-4lOGhiwM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tokenizers\n",
        "\n",
        "from tokenizers import Tokenizer\n",
        "from tokenizers.models import BPE\n",
        "from tokenizers.trainers import BpeTrainer\n",
        "from tokenizers.pre_tokenizers import Whitespace"
      ],
      "metadata": {
        "id": "RdBMFD2xiy_n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize a tokenizer with the BPE model\n",
        "tokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\n",
        "\n",
        "# Use a pre-tokenizer to split the input into words\n",
        "tokenizer.pre_tokenizer = Whitespace()\n",
        "\n",
        "# Create a trainer for the tokenizer\n",
        "trainer = BpeTrainer(vocab_size=10000, special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\"])\n",
        "# TODO: the default 30k vocab size might be sub-optimal for small corpora\n",
        "\n",
        "# Train the tokenizer\n",
        "tokenizer.train(files=[\"Rainis.txt\"], trainer=trainer)\n",
        "# TODO: try and compare with Romeo.txt\n",
        "\n",
        "print(\"Vocabulary size:\", tokenizer.get_vocab_size())\n",
        "\n",
        "# Save the tokenizer on disk\n",
        "tokenizer.save(\"Rainis_BPR_tokenizer.json\")"
      ],
      "metadata": {
        "id": "OCISYVGOhr9y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load a pre-trained tokenizer\n",
        "tokenizer = Tokenizer.from_file(\"Rainis_BPR_tokenizer.json\")\n",
        "\n",
        "test = tokenizer.encode(short_text_lv) # TODO: try with short_text_en\n",
        "print(test.tokens) # Compare to the Rainis' word frequency list at Korpuss.lv\n",
        "print(test.ids)\n",
        "\n",
        "print()\n",
        "\n",
        "test = tokenizer.encode(\"Šis teksts ir latviešu valodā.\")\n",
        "print(test.tokens) # See Rainis.txt\n",
        "\n",
        "test = tokenizer.encode(\"Цей текст українською мовою.\")\n",
        "print(test.tokens) # See Rainis.txt"
      ],
      "metadata": {
        "id": "Hbk40n67mRht"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Celmošana | Stemming"
      ],
      "metadata": {
        "id": "6nFhzDGSkErb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# NLTK PorterStemmer for English\n",
        "\n",
        "from nltk.stem.porter import PorterStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "stemmer = PorterStemmer()\n",
        "\n",
        "test_sentence_en = \"These boys running were fastest off-road runners when they smiled.\"\n",
        "\n",
        "stemmed_tokens = [stemmer.stem(t) for t in word_tokenize(test_sentence_en)]\n",
        "\n",
        "print(' '.join(stemmed_tokens))"
      ],
      "metadata": {
        "id": "b4gn2HRBkZUE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# NLTK SnowballStemmer supports 15+ languages\n",
        "\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "print(SnowballStemmer.languages)"
      ],
      "metadata": {
        "id": "9cEB8PxXmdlV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "stemmer = SnowballStemmer('english')\n",
        "stemmed_tokens = [stemmer.stem(t) for t in word_tokenize(test_sentence_en)]\n",
        "print(\"EN:\", ' '.join(stemmed_tokens))\n",
        "\n",
        "test_sentence_se = \"Dessa fiskar smakar gott.\"\n",
        "\n",
        "stemmer = SnowballStemmer('swedish')\n",
        "stemmed_tokens = [stemmer.stem(t) for t in word_tokenize(test_sentence_se)]\n",
        "print(\"SE:\", ' '.join(stemmed_tokens))"
      ],
      "metadata": {
        "id": "3XQtgatgmiA0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Lemmatizācija"
      ],
      "metadata": {
        "id": "mDpnXZkWkjAJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Lemmatization with NLTK\n",
        "\n",
        "nltk.download('wordnet')\n",
        "\n",
        "from nltk.stem.wordnet import WordNetLemmatizer\n",
        "from nltk.corpus import wordnet\n",
        "\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "lemmatized_tokens = [lemmatizer.lemmatize(w) for w in word_tokenize(test_sentence_en)]\n",
        "\n",
        "print(test_sentence_en)\n",
        "print(' '.join(lemmatized_tokens))\n",
        "\n",
        "# TODO: POS-tagging first..."
      ],
      "metadata": {
        "id": "Dzi08jDPksk8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Corresponding POS tags (WordNet tags)\n",
        "pos_tags = ['s', 'n', 'v', 's', 'a', 'a', 'n', 'r', 's', 'v']\n",
        "# 's' tags are incorrect - used instead of None\n",
        "\n",
        "for w, p in zip(word_tokenize(test_sentence_en), pos_tags):\n",
        "    lemma = lemmatizer.lemmatize(w, pos=p)\n",
        "    print(f\"{w:10}\\t{lemma}\")"
      ],
      "metadata": {
        "id": "FbrTDdfozaKY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Lemmatization with Spacy\n",
        "\n",
        "# Load the multilingual tokenizer, tagger, lemmatizer, etc.\n",
        "#nlp_xx = spacy.load(\"xx_sent_ud_sm\")\n",
        "\n",
        "# Process the EN sentence\n",
        "for token in nlp_en(test_sentence_en):\n",
        "    print(f\"{token.text}\\t{token.lemma_}\")\n",
        "\n",
        "print()\n",
        "\n",
        "# Process the SE sentence\n",
        "for token in nlp_xx(test_sentence_se):\n",
        "    print(f\"{token.text}\\t{token.lemma_}\")"
      ],
      "metadata": {
        "id": "sbs_1Dvsk6kk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Lemmatization with Stanza\n",
        "\n",
        "stanza.download('en')\n",
        "stanza.download('sv')\n",
        "\n",
        "nlp_en = stanza.Pipeline(lang='en')\n",
        "nlp_sv = stanza.Pipeline(lang='sv')"
      ],
      "metadata": {
        "id": "5xgM5zJC53YW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc_en = nlp_en(test_sentence_en)\n",
        "\n",
        "for s in doc_en.sentences:\n",
        "    for w in s.words:\n",
        "        print(f\"{w.text}\\t{w.lemma}\")"
      ],
      "metadata": {
        "id": "WOuo7cxu4667"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc_sv = nlp_sv(test_sentence_se)\n",
        "\n",
        "for s in doc_sv.sentences:\n",
        "    for w in s.words:\n",
        "        print(f\"{w.text}\\t{w.lemma}\")"
      ],
      "metadata": {
        "id": "p-2nGeY_6LRa"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}