notebook/ver.0.7/eg_with_myanmar_names_fasttext_freeze.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6f0e9fb3-5cb3-4769-8214-8bb74126d94a",
   "metadata": {},
   "source": [
    "# Laphet (Version 0.7) with Myanmar Names Dataset\n",
    "## Training with fasttext_freeze    \n",
    "\n",
    "Shell script နဲ့ပဲ MLP, Bi-LSTM, Transformer, BERT, GPT အခြေခံတဲ့ language modelအကုန်ဆောက်ပြီး၊ name generation, testing အဆင့်ဆင့် လုပ်သွားပါမယ်။   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1982968b-4ddf-4d5e-9703-c12aa09c774e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ye/exp/name-lm/lib\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3546849c-7dbd-4fe3-950c-677006cf1c41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash\n",
      "\n",
      "# Updated for Laphet LM Toolkit Version 0.7\n",
      "# Last updated: 27 Jan 2025\n",
      "\n",
      "# Create the output and log directories if they don't exist\n",
      "mkdir -p model/name/\n",
      "mkdir -p output/name/\n",
      "mkdir -p log/name/\n",
      "\n",
      "# Function to train, generate text, and test a language model\n",
      "task() {\n",
      "  local model_type=$1\n",
      "  local model_file=\"./model/name/${model_type}.ftfz.model\"\n",
      "  local output_file=\"./output/name/${model_type}_ftfz_gen_texts.txt\"\n",
      "  local log_file=\"./log/name/${model_type}.ftfz.log\"\n",
      "  local train_data=\"./data/myRoman/train_name.txt\"\n",
      "  local dev_data=\"./data/myRoman/dev_name.txt\"\n",
      "  local test_data=\"./data/myRoman/test_name.txt\"\n",
      "  local start_name=\"./data/myRoman/start_names.txt\"\n",
      "\n",
      "  {\n",
      "    echo \"Training ${model_type^} language model:\";\n",
      "    time python -u laphet.py --model_type $model_type --train --data $train_data \\\n",
      "      --dev_file $dev_data --model $model_file --seq_len 50 --epochs 10 --batch_size 32 \\\n",
      "      --lr 0.0001 --embedding_method fasttext_freeze \\\n",
      "      --fasttext_model ./fasttext-model/myfasttext_v1.bin --embed_dim 100;\n",
      "\n",
      "    echo \"Text generation:\";\n",
      "    time python -u laphet.py --model_type $model_type --generate --model $model_file \\\n",
      "      --seq_len 50 --prompt \"ရဲ\" --no_of_generation 10 \\\n",
      "      --embedding_method fasttext_freeze\n",
      "\n",
      "    echo \"Batch text generation from file:\";\n",
      "    time python -u laphet.py --model_type $model_type --generate --model $model_file \\\n",
      "      --seq_len 2 --input $start_name --no_of_generation 5 --output $output_file \\\n",
      "      --embedding_method fasttext_freeze;\n",
      "\n",
      "    echo \"Testing:\";\n",
      "    time python -u laphet.py --model_type $model_type --test --model $model_file \\\n",
      "      --test_file $test_data --seq_len 50 --batch_size 64 --embedding_method fasttext_freeze 2>&1;\n",
      "  } | tee \"$log_file\"\n",
      "}\n",
      "\n",
      "# Run tasks for each model type in the specified order\n",
      "task mlp\n",
      "task bilstm\n",
      "task transformer\n",
      "task bert\n",
      "task gpt\n",
      "\n",
      "echo \"All tasks completed!\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!cat ./train_test_name_ftfz.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16472c2d-9e6f-4795-b2ee-3ad324a523d3",
   "metadata": {},
   "source": [
    "## Fasttext Model\n",
    "\n",
    "ဒီ test run မှာ သုံးမယ့် fasttext embedding model က [Ngapi](https://github.com/ye-kyaw-thu/NgaPi) code တုန်းက သုံးခဲ့တဲ့ မော်ဒယ်နဲ့ အတူတူပါပဲ။ စာကြောင်းတွေကို syllable unit ဖြတ်ထားပြီး 100 dimension နဲ့ ဆောက်ထားတဲ့ မော်ဒယ်ပါ။  \n",
    "\n",
    "Model download link:\n",
    "[https://huggingface.co/ye-nlp/mySyllableFastText-Version1](https://huggingface.co/ye-nlp/mySyllableFastText-Version1)\n",
    "\n",
    "myfasttext_v1.bin ဖိုင်ဆိုက်က ကြီးလို့ HuggingFace ရဲ့ personal account မှာပဲတင်ပေးထားလိုက်တယ်။\n",
    "\n",
    "**embedding model က word နဲ့ ဖြတ်ထားရင် training, dev, test ဒေတာတွေကိုလည်း word unit နဲ့ ဖြတ်ပေးထားရပါမယ်။ fasttext embedding model ကို syllable unit နဲ့ ဖြတ်ဆောက်ထားရင် training, dev, test ဒေတာတွေကလည်း syllable ဖြတ်ထားမှ embedding မော်ဒယ်ကို apply လုပ်သွားမှာပါ။ ထိုနည်းလည်းကောင်း dimension command line argument ကိုလည်း ဆောက်ခဲ့တဲ့ မော်ဒယ်နဲ့ ညှိပေးထားရပါလိမ့်မယ်။   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "22f149ef-6af9-484a-93ca-9701c46dc823",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 ye ye 766M Jan 27 16:37 ./fasttext-model/myfasttext_v1.bin\n",
      "-rw-rw-r-- 1 ye ye 3.9M Jan 27 16:37 ./fasttext-model/myfasttext_v1.vec\n"
     ]
    }
   ],
   "source": [
    "!ls -lh ./fasttext-model/myfasttext*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e2ca0c-111d-428f-a81e-5b5ddd73b60e",
   "metadata": {},
   "source": [
    "training လုပ်တဲ့အခါမှာ .bin ဖိုင်ကို သုံးပါမယ်။  \n",
    ".vec ဖိုင်ကတော့ text ဖိုင်မို့လို့ syllable တစ်လုံးချင်းစီရဲ့ vector information ကို ဖတ်လို့ရပါတယ်။  \n",
    "အရင်ဆုံး ဖိုင်ရဲ့ ထိပ်ဆုံး စာကြောင်း ၁၀ကြောင်းကို လေ့လာကြည့်ရအောင်။  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "739d2cf3-3903-423c-a659-f60376a5d23a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3462 100\n",
      "အ -0.020812677 0.014222041 0.14707279 -0.1842381 -0.061052196 -0.057658885 -0.03292892 0.04936385 -0.025804132 0.0734657 -0.1159874 0.016686555 0.35442472 -0.14443621 0.05544194 0.12364729 -0.023075808 0.09519182 -0.045482293 0.029036311 -0.2193143 0.18958911 0.18115446 0.2333122 0.1195823 -0.008966078 0.411379 -0.08511002 0.047515854 -0.09509543 0.08582706 0.03881732 -0.055743393 0.03412059 -0.16639902 -0.28278458 -0.032576233 0.10848193 -0.020877324 -0.0009929277 -0.06336136 -0.14327583 -0.031124137 -0.08016338 -0.024092454 0.07601534 0.065982565 0.17565352 -0.10458869 0.18354851 0.1652404 0.09512043 -0.083493486 0.24238697 -0.17059515 0.010160673 -0.09070958 0.1116292 -0.037382837 0.2967968 -0.14358653 0.04922141 -0.22735184 0.01362576 -0.07403269 0.07475961 -0.19550946 -0.08912936 0.06503683 -0.14732634 -0.042020865 0.16236758 0.0344707 0.110176384 -0.04351531 0.02504056 -0.017584132 -0.019345423 0.14525683 0.018760469 -0.11468452 -0.16687019 0.12602636 -0.11606391 -0.05526773 0.0042534173 0.12149795 -0.09459259 -0.10891135 -0.2414366 0.073258676 -0.0504788 0.15406191 -0.024610322 0.015399458 -0.1084087 0.104009554 -0.05159897 -0.21134643 -0.029999733\n",
      "</s> 0.12382664 -0.012683906 0.041837506 -0.06987271 0.06381257 0.06208927 -0.23126368 0.048813067 0.08809406 0.039452787 -0.22591248 0.10878711 0.24657933 -0.09025508 -0.20819944 0.11296403 -0.0023698541 0.13694344 0.046330955 -0.06459741 -0.011776806 0.0020629307 0.008789067 0.26550153 -0.022911312 -0.04013552 0.0279873 0.017961724 -0.0054563517 -0.026302485 0.08658377 -0.102739446 -0.0013639664 0.20668857 0.004934055 -0.34088972 0.2774442 0.08034218 0.16066907 -0.21243756 0.0037596314 0.058731575 -0.049128115 -0.0017906616 0.094759576 0.062752426 -0.058349565 0.05442252 0.02527177 0.28109205 -0.17506446 -0.24817337 -0.2785353 0.036080763 -0.07960388 -0.19838692 -0.09586938 0.032010466 -0.12380073 0.13872214 -0.096330285 0.08983679 0.10626658 0.06370716 0.091440745 -0.0612045 0.0082077 0.0670313 -0.017856117 -0.21964775 0.05315741 0.0147147095 -0.1176217 0.06689351 0.045692213 -0.027084365 -0.12495102 -0.038943823 0.19913141 0.10390068 0.11612357 0.051901486 0.17593633 0.18200813 -0.24298528 0.14925897 0.012650026 0.18348004 0.019809598 -0.4322835 -0.13959268 -0.20270965 0.15223348 -0.13839447 0.2104736 -0.09130107 -0.1292427 0.19842337 -0.32999054 -0.07565341\n",
      "မ -0.037138704 -0.13569695 0.05291237 -0.23952962 0.054309398 0.08480413 -0.12574023 -0.10288342 0.26802805 -0.023060258 -0.19200593 0.048837014 0.13388237 -0.10156472 -0.27330744 0.06254955 -0.018021502 0.10738753 -0.08699856 -0.11971502 -0.02427984 0.00821751 0.011763198 0.22179054 0.0405163 -0.064645365 0.21057953 -0.20626605 0.052216925 0.06893001 -0.083571464 0.062132645 -0.03155385 0.1259717 -0.14362349 -0.29381698 -0.024351114 0.09014881 -0.02725529 -0.084978566 -0.034774803 -0.046036735 -0.14072639 0.124127895 -0.096729 0.16550906 0.078789495 0.18324527 0.059834424 0.07983999 -0.22204566 -0.0071226573 -0.22143735 0.14136066 -0.18776268 0.111439355 -0.034212455 0.29030272 0.062479146 0.13921164 -0.011364475 0.16420348 -0.18552603 0.2543297 -0.0671265 -0.1643539 -0.1521545 0.011511192 -0.14606753 -0.18639424 -0.091895506 -0.038037926 -0.013226733 -0.020700991 -0.1304299 -0.10411659 -0.123490945 -0.121870376 0.025200529 -0.0728125 0.02042448 0.0319191 0.16621827 -0.0046351263 -0.07936244 -0.02828452 -0.07946257 -0.013573024 0.0023430604 -0.2846415 0.004855334 -0.1849191 0.26290923 0.06771301 0.33243358 -0.018295549 0.14210498 0.07429257 -0.16856532 -0.097810596\n",
      "ပါ -0.087525435 -0.005859427 0.27533922 -0.02983224 -0.039899968 0.049487114 -0.28226385 0.17397852 0.23602515 0.00026850402 -0.25732568 -0.12463488 0.08218895 -0.03002106 -0.11644097 0.11331933 0.16041115 0.2823048 0.0058015343 -0.008242266 -0.11977239 -0.16394351 -0.11037723 0.23661324 -0.08783595 0.09576425 0.299117 0.003330242 0.076135896 -0.13499334 -0.089114785 0.10768247 -0.102942936 0.3175604 -0.06907454 -0.26737332 0.19487469 0.027710423 0.1234796 -0.27797264 -0.036803883 0.07294426 -0.19873166 -0.06823772 -0.111085676 0.123619474 0.28799415 0.13046129 -0.09571865 0.33422554 -0.04376065 -0.2152585 -0.35091978 0.013717361 -0.1434667 -0.2915153 -0.10823727 0.048322715 -0.12467216 0.2949836 -0.18683922 0.20867643 -0.096441984 0.16219223 0.038168233 -0.20211723 -0.061449233 0.10116825 -0.10512587 -0.18431446 -0.0693818 0.022668384 -0.018073462 0.12479973 -0.0061559975 -0.021700172 -0.057297543 -0.051328972 0.18727888 0.15065491 0.001404902 -0.0042740926 0.17474222 -0.043993764 -0.036044426 -0.06262929 -0.017722342 -0.1013968 0.16547391 -0.31008813 -0.01926384 0.019985143 -0.077733725 -0.16357918 0.00094194524 0.031407647 0.01720908 0.1017039 -0.21075726 0.054990195\n",
      "က 0.005586397 0.0035553724 0.043903466 -0.025929756 0.06820154 -0.083473936 0.17526793 0.025444353 -0.010341607 0.012500547 -0.06879764 -0.022912554 0.19046825 -0.09192437 -0.065295674 -0.0124853905 -0.05358848 0.14326698 -0.021700917 -0.008445582 -0.054013252 0.019059066 0.09665468 0.22297117 -0.042007785 0.0563076 0.42052028 -0.13233158 0.22632253 -0.013369214 0.006161142 -0.07444376 -0.22130388 -0.016274836 -0.11054453 -0.21350713 -0.012493099 0.1371899 0.043665145 -0.021117441 -0.068028174 -0.04764892 0.010197798 0.035215147 -0.040331736 0.10699217 0.20265245 0.24131921 0.09565459 0.1330294 0.012323478 0.13944894 -0.04777919 0.14558257 -0.009986118 -0.11140704 0.076862484 0.059137706 -0.15367246 0.1172542 -0.039038908 0.0800514 -0.16803804 0.18416142 -0.046993427 -0.09745307 -0.039417863 -0.049574837 -0.055808738 -0.009096504 -0.020848284 0.19776654 0.15017495 -0.013011406 -0.09384664 0.021649998 -0.13679193 -0.13376203 -0.022702852 0.07466504 -0.111704424 -0.077716 -0.06293553 0.14709842 -0.0823473 0.17380673 -0.022408854 0.00034506433 -0.19590604 -0.08407478 0.14193982 0.0018730997 0.27397293 -0.07504713 0.024243716 0.19164222 0.12516136 -0.0275762 -0.31775403 -0.32855007\n",
      "ကို -0.14035897 -0.20247672 0.1522462 -0.1592438 -0.04335637 -0.10129674 0.0289356 0.05071132 0.0889965 -0.00135214 -0.16407943 0.04276938 0.216871 -0.16061687 -0.07371625 0.13684046 -0.16825767 0.09134262 0.024781587 0.13245153 -0.24981527 0.023143714 0.1167979 0.15466931 -0.071643785 0.089075245 0.37485343 -0.09469601 0.120481886 -0.23762815 -0.004829833 -0.10042644 -0.09305838 0.12268869 -0.18135175 -0.22339243 0.070216134 0.10949483 0.11248172 -0.13408911 -0.03096802 0.03587532 -0.0010756084 0.025387011 -0.22672552 0.036987174 0.045254923 0.15705995 0.03835295 0.21106799 0.07006357 0.024549434 -0.116572715 0.2365292 -0.0064354967 -0.27123913 -0.04282965 0.046844672 -0.0775909 0.32956553 0.039943602 0.05636056 -0.24621639 0.23866582 -0.16478834 0.10037286 -0.03135763 -0.06522189 0.047627356 0.06016024 -0.03588742 -0.00063734397 0.04921951 0.08579688 -0.05137061 -0.06604894 -0.1606001 0.05543119 0.1622271 0.055894613 0.009143041 0.10874284 0.15345165 0.11906961 -0.041506276 0.01619666 -0.22009136 -0.007732209 -0.019190013 -0.3299635 0.00041829105 -0.07723701 0.16245684 0.03912612 0.11016564 -0.008738305 0.16529053 0.059800748 -0.14611718 -0.098225154\n",
      "တယ် -0.120471336 -0.018285215 0.4533828 0.056147113 -0.063245565 0.053406563 -0.4467594 0.0973573 0.015560304 -0.0075013405 -0.25234514 -0.058425974 0.17098938 -0.23994355 -0.1640212 0.27626455 0.07986635 -0.119295485 -0.0050384915 -0.11835963 -0.0015431303 -0.12790558 0.018034093 0.47423318 0.09921052 0.04763681 0.30432615 -0.08247365 0.12788221 -0.12415005 0.10119429 0.11169708 -0.0354309 0.25630134 -0.13584085 -0.4440856 0.24584708 0.11716656 0.13736127 -0.12940744 -0.011258109 0.0792069 -0.16058801 -0.044098046 -0.290039 0.12187289 0.26664132 0.25621757 -0.043904074 0.501729 -0.024915602 -0.21647009 -0.10493446 -0.030752532 0.065629736 -0.4260809 -0.06419717 0.009782156 0.07937679 0.16467118 -0.15726167 0.038465332 -0.15767308 -0.021764314 0.04305435 -0.18023114 -0.24379285 -0.1954492 -0.09574732 -0.059373423 0.23023994 0.10441988 0.109726496 0.06956385 -0.012988484 0.14208709 -0.11922363 -0.18472713 0.1308994 0.26774776 -0.12838815 0.18306153 0.12675257 0.09087505 0.008301395 0.0007042864 -0.24252665 -0.2013461 0.06217788 -0.20095687 -0.09003493 -0.11727926 0.017054338 -0.09944423 -0.103094645 0.14915675 0.09124262 0.13907568 -0.37772167 -0.110662974\n",
      "နေ 0.048886072 -0.011819106 0.2975276 -0.0012909761 0.1251401 -0.028090375 -0.095248684 -0.0032416787 0.12339536 -0.0053562075 -0.22194068 -0.12946838 0.25370762 -0.21229751 -0.05537299 0.037339985 -0.041386064 0.06714039 -0.18557784 0.083090864 -0.09687073 0.023984643 0.075407386 0.22928321 0.024193095 -0.09934579 0.28638634 0.09949465 -0.01919273 -0.13266549 -0.09923762 0.074267074 -0.08712657 0.13067636 -0.20511693 -0.36451292 0.28716242 -0.19483468 0.009250808 0.013777632 -0.11485828 -0.007749319 -0.025535172 0.030564219 -0.17353758 0.11538676 0.13587792 0.2224195 -0.04777381 0.2729874 0.01848652 -0.095303684 -0.22759122 0.24076937 -0.05552526 -0.16929159 0.032177873 0.05935123 0.053086184 0.20018654 0.040758073 -0.0027874839 -0.0646288 0.21544237 -0.046396684 0.15841709 -0.116035566 -0.075150385 0.094882146 -0.06554534 -0.030115541 -0.034544855 -0.051330484 0.057215557 0.033348918 0.17677371 -0.24633877 -0.06970134 0.18098429 0.05667135 0.001614477 -0.0012523048 -0.04237651 -0.04455544 -0.09252065 0.19407815 -0.022971122 0.04657116 0.07657422 -0.11040982 -0.1414829 0.01112334 0.3202184 -0.0647468 0.033276737 -0.07872544 0.1207119 -0.05889726 -0.2059288 -0.13019425\n",
      "ရ 0.07895931 0.021591492 -0.00974522 -0.1954571 0.03715715 0.19192086 -0.16420113 -0.048558295 0.180944 -0.039754268 -0.21430117 0.015529577 0.24136762 0.01799596 -0.04952926 0.083865896 -0.07847873 0.113229685 -0.1036828 0.14505091 -0.15163416 0.13571873 -0.036331203 0.26130563 0.022977222 -0.017934687 0.14438695 -0.15869294 0.16259226 0.10760036 -0.057033528 0.04902984 -0.16630113 0.04820086 -0.03314672 -0.096001126 0.065662146 0.01041087 0.072664514 -0.03414534 -0.0017841728 0.02442407 -0.1967488 -0.10730509 -0.04234231 -0.048308313 0.08167268 0.14075202 -0.04791197 0.095709346 -0.00039508834 -0.028387519 -0.12931135 0.07175967 -0.2577539 -0.09668729 0.019835126 0.08390751 -0.14297912 0.3197953 -0.3027686 0.2652532 -0.10551424 0.14783493 0.03529992 0.04610823 -0.0746163 0.032330126 0.015001297 -0.23282209 -0.10659642 0.09913879 -0.058567397 0.107539795 0.16343871 0.045880172 -0.36562073 -0.1150111 0.079135254 -0.081988245 -0.14062381 0.21396255 0.0070283376 0.09891861 -0.15961233 -0.014829058 -0.040204123 -0.041554965 0.064516455 -0.17597377 -0.17241801 -0.27929875 0.18177998 -0.17023832 0.058134444 0.050726775 0.02103829 0.16006836 0.0056325197 -0.13922656\n"
     ]
    }
   ],
   "source": [
    "!head ./fasttext-model/myfasttext_v1.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fff5f83a-9efa-45dd-89c2-d83f580aabd0",
   "metadata": {},
   "source": [
    "Vector ဖိုင်ရဲ့ နောက်ဆုံးပိုင်းကိုလည်း တချက် ကြည့်ကြည့်ရအောင်။  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "758ed2af-ea04-4fa3-944f-0146247a6f0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "လဗ္ဘေ 0.1830714 0.22337396 -0.21023625 -0.045918267 0.26395327 0.015270137 -0.27974412 -0.0021621864 0.36375126 0.25287792 -0.3485732 0.31900573 0.76218 -0.584555 0.2602841 0.2150892 0.015093825 0.40335682 -0.0021270078 -0.0102439625 -0.27154714 -0.24530803 -0.30234715 0.8159268 -0.33370203 0.16331537 -0.08953913 -0.15671305 0.073804915 0.21522263 0.37621176 -0.4310723 -0.07006551 0.19596152 0.26038486 -0.34378362 -0.29537085 0.2091432 0.28675535 0.13887826 0.22702347 -0.4010916 0.23250727 -0.37237856 0.3132686 -0.61562485 -0.258343 0.14362666 -0.036417037 -0.5316609 0.029325252 0.17908096 -0.42861366 0.21407056 -0.19330813 0.17727767 0.2880356 -0.028847774 -0.08432743 0.3579867 0.035279304 0.4984944 0.051906973 0.46864054 -0.03723403 -0.00057201437 0.34042454 0.23796026 -0.16938731 -0.42765442 0.023206659 0.11070449 0.43708295 -0.18668526 -0.28554085 0.1419076 -0.3099975 -0.31298527 -0.082222976 -0.35318598 0.26455504 -0.077361144 0.28766176 -0.2408238 -0.009982903 0.060138658 0.02775521 0.0734009 -0.24349132 0.13439271 0.35626617 -0.41808903 -0.067227684 0.034056876 -0.21464583 0.2795677 -0.1496724 -0.001305461 0.24437293 -0.02882282\n",
      "တဗ္ဗံ 0.104341894 0.19125508 0.13042389 0.05782834 0.14539906 0.05196335 -0.4010302 -0.0027961615 0.20104143 0.5516942 -0.377521 0.13971536 0.6555644 -0.5689611 0.039594356 0.24281874 -0.0017498931 0.59031504 -0.51997936 0.09937805 -0.20728718 -0.08825751 -0.09554912 0.90628475 -0.4524254 -0.0925517 -0.17027739 -0.41588265 -0.07899862 0.3201917 0.085372254 -0.10582247 -0.058678895 0.4305951 0.26684707 -0.3368737 -0.18324028 0.27057683 0.07485149 0.30443588 -0.12275765 -0.46582893 0.0936307 -0.16258475 0.23796187 -0.6366647 -0.23609927 0.056917246 -0.071175635 -0.35178915 0.1887115 0.17389835 -0.45525342 0.13894099 -0.34651613 0.1562809 0.23965026 -0.10183493 -0.18022542 0.25411746 0.041324563 0.6663802 -0.020847818 0.47111678 0.003833798 0.1815742 0.6562791 0.40081605 -0.24699491 -0.18109284 0.13477024 0.06575044 0.21532612 -0.16761641 -0.6348698 0.16235743 -0.33110952 -0.4635248 -0.30593476 -0.2975592 -0.04197147 0.077429645 0.24227196 -0.069967195 -0.11728464 -0.15494616 -0.1308245 0.09756822 -0.44089395 0.26098505 0.26574934 -0.27187836 0.15250646 0.23580317 -0.2323804 0.25942865 -0.3661681 -0.0639105 0.6016198 -0.1822776\n",
      "စုက္က 0.20521365 0.09438536 -0.0264808 0.13624255 0.14474016 -0.03511443 0.053657748 -0.39148733 0.17815137 0.15941456 -0.31022483 0.15797757 0.30702072 0.021061974 -0.079983346 -0.08149047 0.10008772 0.2109896 -0.017679496 -0.067980796 -0.21406952 -0.043480843 0.13753532 0.5135827 -0.32427675 0.22002983 0.25263333 -0.27144453 0.50699496 -0.078765735 -0.14859308 0.015984282 -0.27738523 0.23826288 0.07659247 -0.4837527 -0.11766404 0.09542528 -0.104246795 -0.23281375 -0.24899693 -0.17460649 0.22916813 -0.17444521 -0.016369514 -0.16771527 -0.12921186 0.111947216 -0.20651351 0.016459227 -0.02513503 -0.16615811 -0.34267616 0.36001614 -0.38686684 -0.12285696 -0.09103212 -0.21198098 0.20011766 0.16460988 -0.08295597 -0.038331605 -0.35282165 -0.10300383 0.2651734 -0.09020426 -0.0653542 -0.077462964 -0.0649436 0.10342922 0.06834048 -0.21099071 0.0051885867 -0.07131982 0.03668426 0.002504321 -0.41786012 -0.019796338 -0.2614099 -0.63675374 -0.075255126 -0.09076484 -0.29956767 0.049946487 -0.09959397 -0.054288324 0.17055716 0.05418735 -0.11283664 0.15269355 0.3066885 -0.12543567 0.017158978 -0.279656 0.15705477 0.007428575 0.10790944 0.076289125 0.20892316 0.012310111\n",
      "လိုတ် -0.08159438 -0.20415902 0.08390559 -0.23229901 0.17286961 -0.046563964 -0.07453947 -0.07180177 0.21920165 -0.07911644 -0.27955252 -0.11124981 0.31730735 -0.06747491 -0.25757474 0.08881594 0.02707368 -0.023569206 0.12820752 0.0693024 -0.105688356 0.0827992 0.004581297 0.27952996 0.06366298 0.03809198 0.23756494 -0.16370168 0.014405136 0.101150535 -0.052265797 -0.0024518033 -0.16780704 0.0840801 -0.1519936 -0.22041984 0.042650286 0.16544747 0.062777676 0.060937647 -0.0190463 0.05588926 -0.19766779 0.082015045 0.090781614 -0.036571644 0.079534404 0.0291194 -0.015827712 0.03523335 -0.057715848 -0.08483462 -0.17581896 -0.01810925 0.006911449 -0.17935926 0.037554838 -0.03391355 0.18425168 0.14735813 -0.008329449 0.25559965 -0.028825384 0.3703595 0.11743349 -0.20505811 0.052377116 0.019328142 -0.05090016 0.08268529 0.09549898 -0.014900943 -0.011471555 0.03565844 0.06655583 -0.089241095 -0.15974851 -0.11255197 -0.005470011 0.12715787 -0.075542584 0.009496979 0.07481394 0.19817366 -0.1335583 0.13124664 -0.027208515 0.005170904 0.18741743 -0.24395773 0.04886307 0.0015358826 0.1001949 -0.13496271 0.13095634 -0.12854894 -0.005651166 0.08019046 -0.15187743 -0.19446106\n",
      "ဏုပ္ပတ္တိ 0.3501218 0.089501746 0.047717825 0.056334473 -0.28651705 -0.07172957 -0.53005856 0.19796218 0.27060035 -0.27476907 0.06414387 0.13052648 0.18087743 0.3805313 0.14581233 0.36809355 -0.13729829 0.2917144 -0.22279517 -0.23711899 -0.24374458 0.026259076 0.0121544 0.039941616 -0.25216064 0.26695874 -0.011435159 -0.22070435 0.267071 -0.09881403 0.13359104 0.07627199 -0.11827025 -0.12857144 -0.10465869 -0.44881585 -0.34499747 0.099971846 0.14436224 -0.073556855 -0.1781766 -0.44428462 0.205006 -0.22097915 -0.066753335 -0.49147233 -0.17217974 0.22944547 -0.023725104 -0.3640966 0.03791675 -0.3185305 -0.07893197 -0.18784721 0.08002684 0.5451927 0.6577787 0.06713744 -0.61491555 0.3444274 -0.056561 0.40221798 -0.0077686673 0.33652842 -0.09437028 0.1907099 0.025995536 0.082187556 -0.4249811 -0.11514057 0.07528894 0.32158607 0.04553038 -0.46819073 -0.25386876 0.07705777 0.3082277 -0.060321737 -0.07269798 -0.41349214 0.04830942 -0.27053112 0.36000076 0.026555449 -0.1886671 0.18298982 -0.19947037 -0.14815599 -0.24087936 -0.3910367 0.3775682 -0.12904929 -0.12892447 -0.30745527 -0.45360485 -0.402538 -0.29686493 0.08447591 0.09767929 -0.5436254\n",
      "ဝှတ် 0.02758753 -0.15060519 0.14588787 -0.043258354 0.3405714 -0.01742934 -0.18019146 -0.020686675 0.21251664 0.09934935 -0.27755767 0.060743447 0.30125418 -0.23129933 -0.15978675 0.14479063 -0.0034814496 0.025731066 0.08803448 0.31623507 -0.1399345 0.19704355 -0.03097078 0.41997027 -0.15364106 0.1153584 0.21238808 -0.23511061 0.14599136 -0.025738763 0.18608963 0.014891977 -0.24958628 0.26489416 -0.12570451 -0.3569646 -0.0069069266 0.06377732 0.20491958 -0.19011545 0.011684109 -0.040676728 -0.22436604 0.16927488 -0.064592525 0.036799617 0.011346676 0.22024024 0.10157181 -0.023126602 -0.20438683 0.12155544 -0.18126822 0.18481195 0.14200135 -0.024017464 0.05219787 -0.25642163 0.15153566 0.0012593378 0.14028844 -0.12778367 -0.09779356 0.13523449 0.20480473 0.048122864 0.23353304 0.0712795 0.02926962 0.3613201 0.008634497 0.032550633 -0.14960386 -0.077190764 -0.31261346 -0.10371513 -0.17810982 -0.25970915 0.036655236 -0.0017336983 -0.1416196 -0.16660058 -0.019589085 0.22240618 0.072392516 0.10111122 -0.15118693 -0.06683828 -0.10874457 0.054666154 0.07962736 0.1843549 0.06512564 -0.1267258 -0.00013410368 -0.057222627 -0.2669123 -0.23537938 -0.077235736 -0.24162766\n",
      "ပဏ္ဍိတ် 0.0025043502 -0.060064316 -0.06813489 0.30839548 0.06430216 -0.113338314 -0.06638431 0.06941075 0.09951032 0.17169707 -0.2391629 0.15856552 0.11117749 -0.09454439 0.03018526 0.011150449 -0.048310757 0.36279157 -0.12869504 -0.049381465 -0.17166963 -0.31276476 0.23749802 0.22095466 -0.25056428 0.33512762 0.22583468 -0.1278927 0.3693094 0.15029365 0.09613863 0.07775229 0.15843852 0.15890059 -0.063755915 -0.18066671 -0.1275286 0.124549404 -0.1329327 -0.38371295 0.030525394 -0.28563446 -0.10947442 -0.2360589 -0.026754754 -0.18372989 -0.3147558 0.075161144 0.12536447 -0.14165257 0.27402952 0.13388348 -0.2879775 -0.15354446 -0.06693468 0.2623265 0.44279662 -0.09946116 -0.27976093 0.152054 -0.25790054 0.17472774 -0.084198155 0.43096784 -0.042418867 -0.08193529 0.267759 0.1602906 -0.44913876 -0.037324484 0.030120147 -0.16256492 -0.06934526 -0.26430917 -0.35658216 0.17871025 -0.15837069 -0.28825358 -0.13675393 -0.32872695 0.23964392 -0.100895405 0.08881996 -0.18317753 -0.29040632 -0.020396985 -0.009924469 0.1313332 -0.006028076 0.01859618 0.39343694 0.050504122 0.39792293 -0.25913554 0.048784677 -0.051296525 0.028461726 0.03865738 -0.124041386 -0.2742954\n",
      "ဝေ့စ် 0.23391351 0.12337126 0.2734134 -0.28896248 -0.41600576 0.121599086 -0.076327786 -0.39324117 0.55085456 -0.086254425 -0.61185914 0.061210483 0.7160943 -0.22875571 -0.23023061 -0.14831327 -0.18326917 -0.38169178 0.42031354 0.23869596 -0.01710078 0.46364063 0.08974114 0.46426997 0.021778928 0.12262411 0.2903504 -0.29043195 0.42067945 -0.27386087 0.16680756 -0.39108822 -0.07689704 0.23875578 0.41634604 -0.55327183 0.4108625 -0.11565947 0.3319508 0.0039666733 0.3152592 0.066436574 -0.4386897 0.18710916 0.0033058706 0.077781886 0.085583195 0.036545344 0.06687406 0.27792835 -0.054621954 0.37697712 -0.014155657 -0.4846648 0.69187677 0.061245915 0.34485063 0.026134381 0.23928532 0.19078575 -0.24015456 -0.13988778 0.40994498 0.23778518 0.041180104 -0.064155795 0.08515805 0.20663483 0.31581345 0.47018296 0.7155985 -0.07775527 -0.32899857 0.30599692 0.08820922 0.13410524 -0.3475354 -0.078497835 0.093686916 -0.07813864 -0.37873736 -0.105501965 -0.08444989 -0.0017433128 -0.18402274 0.08527053 -0.041258708 0.11692058 -0.34836394 -0.024061138 0.16846007 0.26734293 0.114269905 0.42951083 0.04326676 0.24877658 -0.17724453 -0.194292 -0.5371715 -0.43304682\n",
      "ကုက္ကုစ္စ 0.20503645 0.15331632 -0.15970503 0.058487777 0.0704556 0.040250007 -0.057481445 -0.19448644 0.020324072 0.1995402 -0.2651161 0.08778776 0.492468 -0.4066327 0.21996026 0.005628406 0.13258024 0.30738834 -0.22437084 -0.056189228 -0.04698509 -0.102357104 -0.2062008 0.6472072 -0.166485 -0.016427102 -0.0023509995 -0.38404915 0.050420344 0.054055043 0.036625206 -0.105569884 -0.0617861 0.26237494 0.20200852 -0.23733056 -0.19193488 -0.118142016 0.06812437 -0.0043337005 -0.16681324 -0.33212236 0.11405078 -0.2597478 0.056086473 -0.18921973 -0.12715465 0.089052096 0.066308506 -0.008295855 0.19987626 0.2084764 -0.29262674 0.03353294 -0.18312646 0.17448477 0.22600481 -0.12456949 -0.03204798 0.119025365 -0.045889482 0.29576102 -0.17617704 0.24285811 -0.08315565 -0.026916811 0.23069657 0.20583649 -0.10801239 -0.21444231 0.06975542 -0.021206027 -0.048869215 -0.11130478 -0.1625409 0.067400046 -0.36800623 -0.30127418 -0.22657417 -0.39045128 0.07454491 -0.14564534 -0.05906972 0.02826118 -0.23768945 -0.008176513 0.01061538 0.1910325 -0.16913289 0.25863695 0.20419958 -0.12588626 0.1592865 -0.11190364 0.062089868 -0.09833311 -0.19556148 0.13379087 0.16463295 -0.2665381\n",
      "ဝိစ္စ -0.1554334 0.024039373 -0.037652735 -0.08088405 0.114527725 -0.04556298 -0.12290039 0.027864732 -0.012984173 -0.14834854 -0.17392732 0.06465291 0.349792 -0.40594053 -0.18451852 0.015245712 -0.20358084 0.10143587 -0.09219152 -0.00018691421 0.0982844 -0.07419096 0.016758386 0.19423854 -0.28043476 0.07282302 0.031581007 -0.41939217 -0.09923038 0.023396729 -0.038892724 -0.018286848 0.052159194 -0.002797238 0.067191646 -0.26512334 -0.14520617 0.041473567 0.032450594 0.041059792 -0.24224874 -0.1667705 -0.07869364 0.037051402 0.093970105 0.09377576 -0.12896098 0.100080885 -0.0014752866 0.20813134 -0.21149127 -0.028489823 -0.0072847083 -0.25623962 -0.2686316 0.32496127 0.15802495 -0.13391279 -0.033091173 0.35086453 -0.04957518 0.2369229 -0.13104267 0.22426327 0.12991577 0.0077484106 0.018055715 0.12655622 -0.16785504 -0.38114867 -0.072523266 0.012895191 -0.13524209 -0.115947194 -0.0045707147 0.25616738 -0.22794566 -0.3789474 0.3271211 -0.16168465 -0.060372643 -0.19626084 0.051042553 -0.29317805 -0.21256202 0.059377223 -0.012677662 0.11013025 -0.11636968 -0.13076161 -0.120241344 0.12137503 0.5251231 -0.17200543 0.34827095 -0.07514057 -0.13512692 -0.11572279 -0.14420164 -0.36492413\n"
     ]
    }
   ],
   "source": [
    "!tail ./fasttext-model/myfasttext_v1.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e169cbe9-5f55-485e-b02f-8ee4ee3557af",
   "metadata": {},
   "source": [
    "Bash shell script ကို စ run ပြီ။  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d9d994c0-6648-4469-bf37-a1e269437b28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Mlp language model:\n",
      "Epoch 1/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 179.94it/s]\n",
      "Epoch 1, Training Loss: 3.3194\n",
      "Epoch 1/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 522.86it/s]\n",
      "Epoch 1, Validation Loss: 1.0696\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0696\n",
      "Epoch 2/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 194.29it/s]\n",
      "Epoch 2, Training Loss: 1.0595\n",
      "Epoch 2/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 535.03it/s]\n",
      "Epoch 2, Validation Loss: 1.0569\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0569\n",
      "Epoch 3/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 193.37it/s]\n",
      "Epoch 3, Training Loss: 1.0536\n",
      "Epoch 3/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 517.16it/s]\n",
      "Epoch 3, Validation Loss: 1.0549\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0549\n",
      "Epoch 4/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 193.06it/s]\n",
      "Epoch 4, Training Loss: 1.0527\n",
      "Epoch 4/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 518.69it/s]\n",
      "Epoch 4, Validation Loss: 1.0545\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0545\n",
      "Epoch 5/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 190.45it/s]\n",
      "Epoch 5, Training Loss: 1.0524\n",
      "Epoch 5/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 510.38it/s]\n",
      "Epoch 5, Validation Loss: 1.0544\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0544\n",
      "Epoch 6/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 193.35it/s]\n",
      "Epoch 6, Training Loss: 1.0523\n",
      "Epoch 6/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 533.59it/s]\n",
      "Epoch 6, Validation Loss: 1.0544\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0544\n",
      "Epoch 7/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 188.54it/s]\n",
      "Epoch 7, Training Loss: 1.0522\n",
      "Epoch 7/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 517.79it/s]\n",
      "Epoch 7, Validation Loss: 1.0544\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0544\n",
      "Epoch 8/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 187.23it/s]\n",
      "Epoch 8, Training Loss: 1.0522\n",
      "Epoch 8/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 518.37it/s]\n",
      "Epoch 8, Validation Loss: 1.0544\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0544\n",
      "Epoch 9/10 (Training): 100%|█████████████████| 852/852 [00:04<00:00, 193.80it/s]\n",
      "Epoch 9, Training Loss: 1.0521\n",
      "Epoch 9/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 540.83it/s]\n",
      "Epoch 9, Validation Loss: 1.0544\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0544\n",
      "Epoch 10/10 (Training): 100%|████████████████| 852/852 [00:04<00:00, 189.41it/s]\n",
      "Epoch 10, Training Loss: 1.0521\n",
      "Epoch 10/10 (Validation): 100%|████████████████| 32/32 [00:00<00:00, 524.95it/s]\n",
      "Epoch 10, Validation Loss: 1.0543\n",
      "Best model saved at ./model/name/mlp.ftfz.model with validation loss: 1.0543\n",
      "\n",
      "real\t0m47.621s\n",
      "user\t0m50.043s\n",
      "sys\t0m2.498s\n",
      "Text generation:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Generated Text 1: ရဲ လု ကျော် ကျွင်း တူး ဖန်း ဘုတ် ခြူး ကွေ့ ဆော် လှိုဏ်း ဣန္တာ ဆုန်း မြင့့်် ချင် မွှေး ရှင် နား နာ ဗျာလ် တက္ခ သော့ ဘက် ပိုး ချက်စ် စိမ်း စိ ကျို ကွေး လေး သိဉ္စည်း ဘူး ညိန်း ဇဉ် ချွတ် ဗင် ဝါ တင့် ဆောင်း ဗွီ နဲမ် လွမ် အမ်း အောင် တူး ပေ ကျိန်း ဇန် လာဒ် ဒေ့ဗ် ဌေး\n",
      "Generated Text 2: ရဲ ယိမ်း ဿာ ကျွင်း ဖု သင်္ကြန် ယူ ကင်း ငြိမ့် ကြံ နေး ပြည် ဒိမ့် တေ့ အား လဲ နွဲ့ ဖူး ဒါးလ် ပေါ့ စက္ကန့် ဇာရ် ထည် နွမ်း ကျော့် လျို မောင် ဘို လှိုဏ်း မာ့ သန္တာ ရှဲ တည် ရှီ ထွဏ်း ရှဲလ် တီ ဟြေ ယန်း လှီး ရှယ် ကွပ် ဒေါင်း ဟဲ ဂိတ် နိုင်း နှဲမ် ကွ ညား ငြိမ်း ရွှေ့\n",
      "Generated Text 3: ရဲ ဖူး ဆဲ ထောင် သြ ဓ ခွမ် အုန်း ဒို ငယ် ယက် ကြွက် သတ် ချိုင်း နှင်း အိမ် အဂ္ဂါ သိင်္ဂီ ကိ [PAD] ဆပ် သုန် တုပ် တာ ပြန် အာ ခြာ မြန် ဝယ်လ် ဘူ ချိုင်း ကျူး ရှိုင်းန်း မေ ကြံ ဆွေး ခေါလ် အိ စ တိမ်း ဘို့ တက္ခ ရစ် တွဲ ဂန္ဓာ ဆွဲ ဇေ ဆွိ မြင့့်် လွှင် ဝါ\n",
      "Generated Text 4: ရဲ လုတ် ပု လော သန္တာ ဥဂ္ဂါ ဘူး ရင်း မွန်း ဟမ် ဗန်း မျှော် ဗီ ဒေါ် ရီ ထ အို တြီ ဆွိ ခြိမ့် ကျက် ထ နွန် လဒ် ပုံ စောမ် သိင်္ဃ လိုင်း အင်း အုံ ကပ် ငြိမ် အက္ခ ငြား ပုံ့ သဉ္ဇူ ကောက် လင်း ပီ ကော် နွံ ယိမ်း မြော် အမ်း ထန် ဖိ ခေါက် ရှယ် ထာရ် မိုး ရှန်\n",
      "Generated Text 5: ရဲ စစ် ကီ နာ ဒေါင် ကဲ အိဏ်း ဆိုင်း ရှည် န ကတ္တီ မှိုင် ရှီ ရွယ် ပုံ ကျွန် ကွန်း ကြား ရော့ဒ် မြို့ သက် ဇို ထွေ နန် ရှီ ရွာ ဒေါ ဂျမ် ထာသ် အဉ္ဇ နန္ဒ နန္ဒာ ကက် ဒင်း တြီ ကျောင် ဗို ဖျား ဘတ် ဂျော် ထက် ယဉ်း နက် ဥတ္တ က ရက် ဘင်း လွန်း လောင်ဝ် လိန်း ဂိတ်\n",
      "Generated Text 6: ရဲ ဂျန် ပဉ္စ ခွါ ဣန္ဒ ဣ [PAD] ကျေ ရွေ့ ဝဏ္ဏ ခီ ဆွမ့် ဘတ် ရိုင်း ဝင် သျှမ် နှဲမ် ဖျား ငိုက် ဒြာ တင်း ကျွဲ တောင်း ခယ် ဂျူး မီး ညိမ့် ဗြ ဗေ ရှဲမ်း ချုံ ပြန်း ရေ ကျော မုံ တုတ် ဃာ ကွဲ ဒါး မွှေး ဖာ ရို သျှား တ ကြေး ကျူး ကိုလ် ထိပ် ချိုး စည် လာန်\n",
      "Generated Text 7: ရဲ ကေ ရင် အမ္မ ဆားရ် နှာ အိန္ထ ဘွယ် ဝိုင် လဒ် ယွန်း သျှင်း ငယ် ပပ်ဖ် မဲန် ကျား ဖိန် ဏာ ဒေး ဆွတ် ညင် မိုင်း ရင်း ဇွဲ ဂျယ် အုံ ဝီ ချက်စ် ဠာ ချက်စ် မိုင်းလ် ထူး နိုး ဗျာ သာ အုပ် ပွန်း ပိ ခံ ဘို့ ညင် ဒန့် ဆပ် ဂေး စန် ကမ္မ ရင်န် အဲလ် ဥ ဖူး တ\n",
      "Generated Text 8: ရဲ နဲမ် ချစ် အဉ္စ ချူး လစ် ဒါ နိုက် ထပ် လှေး ရိုး မှူး ဒိမ်း မာ့ ဆွန်း မောင် ယံ မင် နာ ချင် ရပ် ဟိန်း သောင် တုန်း မှုန်း ဘ ကျုံး ဒါး မို့စ် ကြူ ချုံ ပြား ဂိတ် ကြောင်း သိဉ္စည်း ကြုတ် ဘက် မွှေး ဒါးလ် ရပ် စွမ် အက္ခ ခဏ် ချစ် ကျင့် ကြင်း ဒေါင့် လွန် တိ ကမ်း သောက်\n",
      "Generated Text 9: ရဲ ဖန် လွှမ်း ဝုန် ဟမ်း မွှန်း သိုင်း ကယ်လ် ဒိ ကေး ဟေး ခေး အဲင်န် နန္ဒာ လွေ ဗျာလ် ရှာမ် ကြွယ် ရား ကစ် တိုင်း ထင် ဝေး လှိုင် ရှိန်း မော သွင့် လု ဒူး ကြိုး ကျွန်း ကျော် မဲန် ထင် မျိုး နွန်း ထီး ချင် ဂွမ် ချန်း စင်း ခတ္တာ ညွန့် မိုး ကြိုက် တင် ဟုန်း စိုး အီ နှဲမ်း ခင်\n",
      "Generated Text 10: ရဲ ချယ်လ် အာဖ် ပြန်း ယောင် နှစ် ကျန် ဖွယ် ချယ် ချော် ချုံး ကျုံး မောင်း ရိ မ နှင်း ကံ့ လှိုဏ်း ဇင်း ကြွမ် မဲန် ပူး ဗိုင်း ဖြူး ဘ ဝူ ရှိ ရယ် တယ်လ် ကျုံး တူး ရွှေ ချစ် သွယ် ဟိုး ခြည် ဆန့် ဓု ရှိန်း မား ပွန်း ဇမ္ဗူ နား ဆောင်း မိ ကြိုင် ဗျက် ထု ဆို ဟေ သ\n",
      "\n",
      "real\t0m2.454s\n",
      "user\t0m5.291s\n",
      "sys\t0m2.373s\n",
      "Batch text generation from file:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Random Prompt Generated: စွန်\n",
      "Generated texts saved to ./output/name/mlp_ftfz_gen_texts.txt\n",
      "\n",
      "real\t0m1.949s\n",
      "user\t0m4.731s\n",
      "sys\t0m2.434s\n",
      "Testing:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:429: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Testing: 100%|██████████| 16/16 [00:00<00:00, 52.68it/s]\n",
      "Average Perplexity on Test Data: 1.1143\n",
      "Average Cross-Entropy on Test Data: 0.1082\n",
      "\n",
      "real\t0m1.860s\n",
      "user\t0m4.697s\n",
      "sys\t0m2.381s\n",
      "Training Bilstm language model:\n",
      "Epoch 1/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 55.88it/s]\n",
      "Epoch 1, Training Loss: 0.5049\n",
      "Epoch 1/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 165.46it/s]\n",
      "Epoch 1, Validation Loss: 0.3247\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.3247\n",
      "Epoch 2/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 56.59it/s]\n",
      "Epoch 2, Training Loss: 0.3191\n",
      "Epoch 2/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 169.85it/s]\n",
      "Epoch 2, Validation Loss: 0.3145\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.3145\n",
      "Epoch 3/10 (Training): 100%|██████████████████| 852/852 [00:14<00:00, 56.83it/s]\n",
      "Epoch 3, Training Loss: 0.3120\n",
      "Epoch 3/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 164.84it/s]\n",
      "Epoch 3, Validation Loss: 0.3101\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.3101\n",
      "Epoch 4/10 (Training): 100%|██████████████████| 852/852 [00:14<00:00, 57.13it/s]\n",
      "Epoch 4, Training Loss: 0.3082\n",
      "Epoch 4/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 162.88it/s]\n",
      "Epoch 4, Validation Loss: 0.3075\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.3075\n",
      "Epoch 5/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 56.32it/s]\n",
      "Epoch 5, Training Loss: 0.3002\n",
      "Epoch 5/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 169.39it/s]\n",
      "Epoch 5, Validation Loss: 0.2922\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.2922\n",
      "Epoch 6/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 56.32it/s]\n",
      "Epoch 6, Training Loss: 0.2844\n",
      "Epoch 6/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 160.48it/s]\n",
      "Epoch 6, Validation Loss: 0.2776\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.2776\n",
      "Epoch 7/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 55.79it/s]\n",
      "Epoch 7, Training Loss: 0.2651\n",
      "Epoch 7/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 161.36it/s]\n",
      "Epoch 7, Validation Loss: 0.2566\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.2566\n",
      "Epoch 8/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 56.55it/s]\n",
      "Epoch 8, Training Loss: 0.2488\n",
      "Epoch 8/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 161.98it/s]\n",
      "Epoch 8, Validation Loss: 0.2377\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.2377\n",
      "Epoch 9/10 (Training): 100%|██████████████████| 852/852 [00:15<00:00, 56.74it/s]\n",
      "Epoch 9, Training Loss: 0.2185\n",
      "Epoch 9/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 167.24it/s]\n",
      "Epoch 9, Validation Loss: 0.1906\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.1906\n",
      "Epoch 10/10 (Training): 100%|█████████████████| 852/852 [00:14<00:00, 56.86it/s]\n",
      "Epoch 10, Training Loss: 0.1720\n",
      "Epoch 10/10 (Validation): 100%|████████████████| 32/32 [00:00<00:00, 163.73it/s]\n",
      "Epoch 10, Validation Loss: 0.1571\n",
      "Best model saved at ./model/name/bilstm.ftfz.model with validation loss: 0.1571\n",
      "\n",
      "real\t2m36.066s\n",
      "user\t2m37.459s\n",
      "sys\t0m3.117s\n",
      "Text generation:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "All logits are -Inf after filtering! Fallback to raw logits.\n",
      "Generated Text 1: ရဲ တန် ဟူ ဌေး နှီး နှီး ဣန္တာ ရှဲလ် ရှဲလ် ဌေး ဣန္တာ နှီး ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 2: ရဲ ဌေး ဌေး တန် ဣန္တာ ဌေး နှီး နှီး ရှဲလ် ရှဲလ် ရှဲလ် ရှဲလ် တန် နှီး ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 3: ရဲ ရှဲလ် ဌေး ဣန္တာ တန် မျာ ရှဲလ် ဌေး ဿန်း ရှဲလ် ရှဲလ် နှီး ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 4: ရဲ ဟူ ဣန္တာ ရှဲလ် ဟူ ဟူ ဟူ ဿန်း ဟူ ရှဲလ် ဿန်း ရှဲလ် ဣန္တာ ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 5: ရဲ မျာ ဟူ ဣန္တာ ဟူ တန် တန် တန် ဣန္တာ ဿန်း ရှဲလ် ရှဲလ် ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 6: ရဲ မျာ ဿန်း ဌေး နှီး ဟူ ရှဲလ် မျာ မျာ တန် ရှဲလ် ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 7: ရဲ နှီး ရှဲလ် တန် ဿန်း ရှဲလ် ရှဲလ် ဿန်း ရှဲလ် တန် ဣန္တာ ဟူ ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 8: ရဲ ဣန္တာ ဌေး မျာ တန် နှီး ရှဲလ် မျာ ဟူ ရှဲလ် ဿန်း မျာ ရှဲလ် နှီး [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 9: ရဲ ဌေး နှီး မျာ မျာ ဌေး ဌေး ရှဲလ် ရှဲလ် ရှဲလ် ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Generated Text 10: ရဲ ဟူ ဿန်း ဟူ ဌေး ရှဲလ် ရှဲလ် ရှဲလ် ဟူ ရှဲလ် ရှဲလ် တန် မျာ ရှဲလ် [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "\n",
      "real\t0m2.359s\n",
      "user\t0m5.072s\n",
      "sys\t0m2.402s\n",
      "Batch text generation from file:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Random Prompt Generated: ထော်\n",
      "Generated texts saved to ./output/name/bilstm_ftfz_gen_texts.txt\n",
      "\n",
      "real\t0m2.158s\n",
      "user\t0m4.901s\n",
      "sys\t0m2.460s\n",
      "Testing:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:429: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Testing: 100%|██████████| 16/16 [00:00<00:00, 41.75it/s]\n",
      "Average Perplexity on Test Data: 1.1671\n",
      "Average Cross-Entropy on Test Data: 0.1544\n",
      "\n",
      "real\t0m2.135s\n",
      "user\t0m4.907s\n",
      "sys\t0m2.402s\n",
      "Training Transformer language model:\n",
      "Epoch 1/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 220.13it/s]\n",
      "Epoch 1, Training Loss: 0.7299\n",
      "Epoch 1/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 632.45it/s]\n",
      "Epoch 1, Validation Loss: 0.3432\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.3432\n",
      "Epoch 2/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 249.30it/s]\n",
      "Epoch 2, Training Loss: 0.2345\n",
      "Epoch 2/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 623.22it/s]\n",
      "Epoch 2, Validation Loss: 0.1231\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.1231\n",
      "Epoch 3/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 248.89it/s]\n",
      "Epoch 3, Training Loss: 0.0966\n",
      "Epoch 3/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 628.36it/s]\n",
      "Epoch 3, Validation Loss: 0.0648\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0648\n",
      "Epoch 4/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 232.91it/s]\n",
      "Epoch 4, Training Loss: 0.0543\n",
      "Epoch 4/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 619.85it/s]\n",
      "Epoch 4, Validation Loss: 0.0429\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0429\n",
      "Epoch 5/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 237.47it/s]\n",
      "Epoch 5, Training Loss: 0.0358\n",
      "Epoch 5/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 628.94it/s]\n",
      "Epoch 5, Validation Loss: 0.0312\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0312\n",
      "Epoch 6/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 234.60it/s]\n",
      "Epoch 6, Training Loss: 0.0257\n",
      "Epoch 6/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 582.23it/s]\n",
      "Epoch 6, Validation Loss: 0.0242\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0242\n",
      "Epoch 7/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 230.69it/s]\n",
      "Epoch 7, Training Loss: 0.0195\n",
      "Epoch 7/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 626.21it/s]\n",
      "Epoch 7, Validation Loss: 0.0189\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0189\n",
      "Epoch 8/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 237.34it/s]\n",
      "Epoch 8, Training Loss: 0.0153\n",
      "Epoch 8/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 604.33it/s]\n",
      "Epoch 8, Validation Loss: 0.0158\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0158\n",
      "Epoch 9/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 232.56it/s]\n",
      "Epoch 9, Training Loss: 0.0123\n",
      "Epoch 9/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 574.81it/s]\n",
      "Epoch 9, Validation Loss: 0.0138\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0138\n",
      "Epoch 10/10 (Training): 100%|████████████████| 852/852 [00:03<00:00, 231.54it/s]\n",
      "Epoch 10, Training Loss: 0.0100\n",
      "Epoch 10/10 (Validation): 100%|████████████████| 32/32 [00:00<00:00, 639.25it/s]\n",
      "Epoch 10, Validation Loss: 0.0121\n",
      "Best model saved at ./model/name/transformer.ftfz.model with validation loss: 0.0121\n",
      "\n",
      "real\t0m38.959s\n",
      "user\t0m41.095s\n",
      "sys\t0m2.529s\n",
      "Text generation:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Generated Text 1: ရဲ ကြော ရန် ဆွင်း သျှန် သျှန် ခူး လွမ်း ကြွက် ဘွယ် သျှန် သျှန် မှီ သျှန် ဆွင် ဖြိုး ခေါင်း တော် မှုံ မိုး ဂဲလ် ရန် ဒု လက် ခူး လွမ်း သျှန် သျှန် သျှန် လမ်း နာမ် လို သျှန် ဆွတ် ကြွက် စောလ် လက်စ် ဆွမ်း သျှန် ကြွက် ဓ ခူး လွမ်း မြီ စွယ် ရွှေ့ ဇဉ် သန့် လဲ့ သျှန် သျှန်\n",
      "Generated Text 2: ရဲ မောင် နှော ချ ကြွက် သျှန် သျှန် ခူး သျှန် ဆူး မြို့ သျှန် အိန္မာ ခူး သျှန် ဘင် ယော် ဇ္ဈိ သဒ္ဒါ ဖောင်း ထဲ သျှန် ဘု နီးလ် သျှန် ခူး ခူး သျှန် သျှန် ကွေး န ချောင် သျှန် စည် လွမ်း သျှန် ခီ င ယဉ်း သျှန် ဂဂ် လျင့် သျှန် သင့် တိန့် သျှန် စော ဒါးလ် ခင် သျှန် သျှန်\n",
      "Generated Text 3: ရဲ ညွတ် ခူး သျှန် သျှန် သျှန် ခူး ကြွက် ခူး သျှန် သျှန် သျှန် ရက် ခူး သျှန် ဂျူး ကြင်း လျန် သျှန် ဘူး ရော့ခ် သျှန် ဗြ ကိန္န သျှန် သျှန် သျှန် သျှန် ခူး ကိမ်း ပ တြီ သျှန် သော ခွမ်း သျှန် ဒို့ဗ် အိန္ဒြ သျှန် သျှန် ကြိုင်း သျှန် လွမ်း ရှယ် ထူး သျှန် ချွတ် ဉာဏ် ဒန် သျှန် သျှန်\n",
      "Generated Text 4: ရဲ ဂုန် ကြွက် ခူး ချ သျှန် ခူး သျှန် ခူး သျှန် သျှန် သျှန် မြော် ခူး သျှန် နာ့ ဠု တွန်း ချင် ယန်း ရှား ချ စွပ် နှောင်း သျှန် ကြွက် ကမ် ရန် သျှန် အင်း ယျန် သိမ့် ခူး ဆွန် သျှန် သျှန် ကျန်း ကမ် သျှန် ရန် နည် သျှန် ခူး သျှန် အပ် သျှန် ဂျူ တေး ချိန် လာလ် လဲမ့်\n",
      "Generated Text 5: ရဲ ညွန့် ယ ပြည့် သက် ရဲမ် သျှန် ချ သျှန် အာဖ် နန်း သျှန် ဧည့် သျှန် ပို ဘယ် ယျ ဿီ ဖာ ထီး ဒိမ် ရန် ဓ ဗန်း သျှန် သျှန် သျှန် သျှန် သျှန် ကုမ္မာ အပ္ပ ကီး သျှန် ဘီလ် ရန် သျှန် ဆ စုက္က ဗုံ သျှန် ချူ သျှန် သျှန် အံ သျှန် ရင်း ဂို ဖြူး ဒေါင် ဒေါင် ဒိန်\n",
      "Generated Text 6: ရဲ ဗာ ခွမ်း သျှန် သျှန် သျှန် ရန် ခူး သျှန် ရှုံး သျှန် သျှန် ခ သျှန် သိင်္ဂီ ရစ် စောလ် မာန် သျှန် မာ အီး ကြွက် ဆိုင်း စိုက် သျှန် သျှန် သျှန် သျှန် သျှန် ညင် ဘီ သိဏ်း သျှန် ဟမ် သျှန် သျှန် လှဲ ထပ် လွှမ်း သျှန် ရှန် သျှန် သျှန် လေ စိုင်း သျှန် စဉ့် အဲင်န် က ဘာ ဆန်း\n",
      "Generated Text 7: ရဲ ဟမ် သျှန် သျှန် သျှန် သျှန် ခူး ခူး ခူး သျှန် သျှန် သျှန် တိမ်း သျှန် ရွတ် ဒို့ဗ် ချိုင်း အိုင် သုဒ္ဓေါ ထီ ဒေ သျှန် ဟ အိုမ် အိန္ဒု ရန် သျှန် သျှန် သျှန် ဇ္ဈိ မြဉ္ဇူ ဖြူ လွမ်း လွမ်း သျှန် သျှန် သော့ခ် ကျစ် ဂင့် ကြွက် ချင် ပယ်လ် ခွမ်း နွမ်း လှောင် သုံး အိန္တ ရမ်း နှာ သျှန် သျှန်\n",
      "Generated Text 8: ရဲ လု ချ ချ ခူး ခူး သျှန် လွမ်း ခူး သျှန် သျှန် သျှန် အေး သျှန် ဂျို ဒွပ် အောလ် ခေါက် လယ် လိုင်း ဆုန် ခွမ်း ဗုံ ပမ် သျှန် သျှန် သျှန် သျှန် သျှန် ရောင် ချိတ် ဂျယ်လ် သျှန် မုံ ခူး ခူး ဝိန် ညိမ်း ယူ သျှန် ဟော ခါး သျှန် ရစ္စ သျှန် ရ ရိုင်း ဖုန် ကျို သျှန် သျှန်\n",
      "Generated Text 9: ရဲ သီး လွမ်း သျှန် သျှန် သျှန် ရန် သျှန် လွမ်း ရေ မန်း သျှန် ယယ် ခူး သျှန် ဗီ ရှည် လွမ်း သျှန် ရိပ် ရှေး ရန် မေ သု ရန် ရန် သျှန် သျှန် သျှန် စွပ် လောင်ဝ် ဆွေ သျှန် ဂို လွမ်း သျှန် ဖူး ဆောင် လှိုဏ်း ခူး လျှပ် သျှန် ခွမ်း တွမ် တိုက် သျှန် ဒါ ပု သူ မွန် သျှန်\n",
      "Generated Text 10: ရဲ မ ဟောက် မွိုင် ခူး သျှန် သျှန် ခူး ခူး သျှန် သျှန် သျှန် မြ ချ ကွက် ဂွိ ဆန့် စွပ် မင်္ဂ လူ လုံ ရန် ဝံ နွမ် သျှန် သျှန် သျှန် သျှန် သျှန် ကီး ရံ မယ် သျှန် နျူး ချ သျှန် ဇံ ထွေး သျှန် သျှန် အုတ် ဖို့ ခူး သျှန် ခွန် ရန် မြင် လှိုင်း ဒွေး ယောင် ရည်\n",
      "\n",
      "real\t0m2.613s\n",
      "user\t0m5.420s\n",
      "sys\t0m2.386s\n",
      "Batch text generation from file:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Random Prompt Generated: ကန်\n",
      "Generated texts saved to ./output/name/transformer_ftfz_gen_texts.txt\n",
      "\n",
      "real\t0m2.169s\n",
      "user\t0m4.962s\n",
      "sys\t0m2.419s\n",
      "Testing:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:429: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Testing: 100%|██████████| 16/16 [00:00<00:00, 50.61it/s]\n",
      "Average Perplexity on Test Data: 1.0106\n",
      "Average Cross-Entropy on Test Data: 0.0105\n",
      "\n",
      "real\t0m1.923s\n",
      "user\t0m4.683s\n",
      "sys\t0m2.351s\n",
      "Training Bert language model:\n",
      "Epoch 1/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 217.84it/s]\n",
      "Epoch 1, Training Loss: 0.7893\n",
      "Epoch 1/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 648.30it/s]\n",
      "Epoch 1, Validation Loss: 0.3514\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.3514\n",
      "Epoch 2/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 233.74it/s]\n",
      "Epoch 2, Training Loss: 0.2517\n",
      "Epoch 2/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 598.07it/s]\n",
      "Epoch 2, Validation Loss: 0.1352\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.1352\n",
      "Epoch 3/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 230.09it/s]\n",
      "Epoch 3, Training Loss: 0.1056\n",
      "Epoch 3/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 646.80it/s]\n",
      "Epoch 3, Validation Loss: 0.0681\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0681\n",
      "Epoch 4/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 237.93it/s]\n",
      "Epoch 4, Training Loss: 0.0579\n",
      "Epoch 4/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 628.70it/s]\n",
      "Epoch 4, Validation Loss: 0.0439\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0439\n",
      "Epoch 5/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 231.92it/s]\n",
      "Epoch 5, Training Loss: 0.0374\n",
      "Epoch 5/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 630.48it/s]\n",
      "Epoch 5, Validation Loss: 0.0318\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0318\n",
      "Epoch 6/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 238.60it/s]\n",
      "Epoch 6, Training Loss: 0.0267\n",
      "Epoch 6/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 626.77it/s]\n",
      "Epoch 6, Validation Loss: 0.0254\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0254\n",
      "Epoch 7/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 229.65it/s]\n",
      "Epoch 7, Training Loss: 0.0202\n",
      "Epoch 7/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 627.93it/s]\n",
      "Epoch 7, Validation Loss: 0.0203\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0203\n",
      "Epoch 8/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 237.97it/s]\n",
      "Epoch 8, Training Loss: 0.0159\n",
      "Epoch 8/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 615.93it/s]\n",
      "Epoch 8, Validation Loss: 0.0166\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0166\n",
      "Epoch 9/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 234.59it/s]\n",
      "Epoch 9, Training Loss: 0.0128\n",
      "Epoch 9/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 637.90it/s]\n",
      "Epoch 9, Validation Loss: 0.0139\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0139\n",
      "Epoch 10/10 (Training): 100%|████████████████| 852/852 [00:03<00:00, 234.39it/s]\n",
      "Epoch 10, Training Loss: 0.0105\n",
      "Epoch 10/10 (Validation): 100%|████████████████| 32/32 [00:00<00:00, 633.83it/s]\n",
      "Epoch 10, Validation Loss: 0.0125\n",
      "Best model saved at ./model/name/bert.ftfz.model with validation loss: 0.0125\n",
      "\n",
      "real\t0m39.376s\n",
      "user\t0m41.451s\n",
      "sys\t0m2.481s\n",
      "Text generation:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Generated Text 1: ရဲ ဂျန် ဝိုက် ဝိုက် ရှား ပီ ရှား လှေ စေး ညှာ လူး စု အော လွှင် ပါး ရင့် နေး ယန် တုတ် ဟ ဧည့် ခွေး ကျ နမ် ခေါက် ကျော် ဖောင်း ကန်း မှု ပဲန်း ဗန် စွာ ဒမ်း မျက် ယာန် ဘောက် ရစ္စ ဗွန် ဟိဏ်း ချားလ် ဂူး ယား ဆွန် ဗုံ လျံ ဘတ် ဗျော ရုံ အင်း ရွန်း ည\n",
      "Generated Text 2: ရဲ သဲ ဟိမ်း ဝိုက် ရ ဆောင်း ကစ္စ ဝိုက် ကျီ ဇိန်း သိမ် လတ် ခြိမ့် ဒါ ဒင့် မာ့ နိုး ခြယ် [PAD] ရင်န် ရေ့စ် စန် ထွန်း ဘင်း ဇူ မဉ္ဇူ ထဲ ပေါ့ ပွန် မဲန် ဂိုးလ် ထွဏ်း ပေါက် ထ လဲမ့် ဆွယ် ယယ် ဏ ကြည် အိန် တုပ် မိ ချိန် ဇေါင်း ရီ ကယ်လ် အစ် ရံ ပုံ ဆီး ခြူ\n",
      "Generated Text 3: ရဲ ဟိန်း အာ က ဝန်း သိုင်း ကျိမ်း ဝိုက် ရှီ တိ နဲ ဘဲလ် လွီ အုံ ဋာ လျှို လျို ဆင်း တုံ ဂျူး ဘိ နွဲ့ ပပ်ဖ် ဝေ တည် ပန်း ထန်း စေး ဖား ဗျာလ် နဲမ် တေ တုပ် ခေါင် ဆိန်း ဘေ အား ပွင့်် ညက် မှီ နွယ် ဟို နွေ လာန် ဂါး ညွှန်း လမ့် ဆောင်း ပါရ် တုတ် ဟေ\n",
      "Generated Text 4: ရဲ ပိုက် ပီ ရှား လှေ ရှား ရှား ဘောမ် မြန် ဇွန်း သန့် ကမ္ဘာ ဟန် ရံ ရွယ် လီ ဏိ တေ ရှဲမ်း ဒူး ခုံ ရံ ညို့ အောင်း လာလ် ရွှေ့ ဿ ဂုန် ညွန့် ပုမ်း ကွန့် စွမ် ရည် မြေး ကဉ္စ ဟူ ထာ ဂဂ္ဂါး ကျုံး အံ လှေ ယျ ဓိ သတ် သို့ ထီး ကြော ရှိန်း ကော ချော တီးလ်\n",
      "Generated Text 5: ရဲ [PAD] ကျင်း ပီ ရှား လှေ ကျင်း ပီ ယျန် ဝင်း ပုံ သင့် ပေါင် ငွေ့ ဗြာ ဇာ ရန်း လဲမ့် ဆူ လှိုင် နတ် ဆေး န ဒေ လမ် ဒြာ ဖြူ ယက် လည်း ချောင် သိဏ်း အို မှန် ပေါက် အေး အားလ် ဗန် ကျောက် ထဲမ်း ဂိမ် ဝါး ကိန်း ကောက် လတ် ဿီ က စော လွမ်း နွန် ငွေ့ ပြီး\n",
      "Generated Text 6: ရဲ ဖွယ် ကျိမ်း ရှား ဝိုက် ရ နန်းဒ် မွန် ဇမ် မ ဂျို ထိ မြူး ဣ ဂန္တ ငိုင် စိုး သတ္တိ ဖုန်း ဏိ ထု ချောက် ရှိ ရုတ်စ် ဟောက် ကွေး ဂီ အဲ့ ယ ကျင့် ခုံ ကျော့ သော့ခ် နောင့် ကော့ ဂေး ဇောင်း ဇင် ပေါ ထောင် ပါရ် ဗိုလ် [PAD] မုံ နန့် ထပ် အီ ကမ္ဘာ ရှည် ပြီ သား\n",
      "Generated Text 7: ရဲ ခြိမ့် လှေ ရှား ရှား ရှား ရ ဇွန် ဒ ကင် ခွန်း အဉ္ဇ လျှမ် ဖြိုး အိန် ကျွမ်း လက် ကောင်း ဘင်း နွန်း လာဒ် စက် ဟန် ကျူ ဂန့် လန်း ချိုး မဲန်း ဇန္ဒား ကျီ ရော့ခ် ဘင် စည်း ကင် ခြာ စွန်း ဿန်း သိန်း မောင်း ထဲ မွေ ဆယ် ကတ် မြို့ နှဲမ် သျွှန်း ထွန်း ကျောင်း ထာရ် ပံ ဂ\n",
      "Generated Text 8: ရဲ ယု ကျဲရ် ရှား ပီ ရှား နှင်း သင့် ရံ ထန် သိမ့် ဒြာ ပန် ရုံ ဒူ ကိမ် ဘင် ကေး တက် လာန်း အဲ့ ပြည် ဂန္တ နွေ ညို့ ဂျူ ဖော် မား ဒု ပြိုင် ကြောင် ထာသ် မား အူလ္လာ ကြင်း မွေး အဂ္ဂ ဟေး ဒွန်း တုံး နှော မှာ ကိုလ် ရွိုင် သိပ္ပံ ထယ် ရှာမ် ဇိ ပေါင် ညွန့် သိမ့်\n",
      "Generated Text 9: ရဲ ဖား ရှား နှင်း လွှမ်း မွန် ကူ ဝိုက် ဆွေ ပင် ရန်း ဆယ် ရွာ သင်္ခ ဆောင်း ရစ္စ ဆည်း နပ် ဒင့် ပေါ ထွန်း ရို နန္ဒ ကမ် တင့် ရှိတ် ရှမ် ရင်း ဂိုးလ် ကောင်း ခံ့ တောင် ပြုံ ဒယ်လ် ဆည်း လိမ်း ဓီ ထောင် နံ့ စိန် ဒွန့် ယျာ ဖီ ဆွန် သူ ဒင်း ယ လိန် ကျိန်း ကိုး ဆံ\n",
      "Generated Text 10: ရဲ ချယ် ဟူ ရှား နှင်း အ ရှား ရှား ဇိတ် ငြိမ့် နိုး မြင် မျက် ဟင်း ဗွန် အိဇ္ဇာ ချား ကျွမ်း လျာ ဿန်း လွီ ထဲ ဒိုင် အောင် ဟင်း ကျေး ကျေ အာ ဆယ် ပါးရ် ရှမ်း စန္ဒ ရွယ် ဈာန် ဟမ် ဂေါင် ဥတ္တ ကောန် ရေ့ဒ်ဗ် သျွှန်း ဆွေး နိုင်း ထဲ တန် လိန်း ယံ ဈာန် ရဲမ် လူး ခြိမ့် ဌေ\n",
      "\n",
      "real\t0m2.626s\n",
      "user\t0m5.424s\n",
      "sys\t0m2.385s\n",
      "Batch text generation from file:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Random Prompt Generated: သိင်္ခ\n",
      "Generated texts saved to ./output/name/bert_ftfz_gen_texts.txt\n",
      "\n",
      "real\t0m2.120s\n",
      "user\t0m4.923s\n",
      "sys\t0m2.404s\n",
      "Testing:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:429: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Testing: 100%|██████████| 16/16 [00:00<00:00, 47.98it/s]\n",
      "Average Perplexity on Test Data: 1.0112\n",
      "Average Cross-Entropy on Test Data: 0.0111\n",
      "\n",
      "real\t0m1.903s\n",
      "user\t0m4.754s\n",
      "sys\t0m2.361s\n",
      "Training Gpt language model:\n",
      "Epoch 1/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 252.07it/s]\n",
      "Epoch 1, Training Loss: 0.4694\n",
      "Epoch 1/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 753.96it/s]\n",
      "Epoch 1, Validation Loss: 0.0344\n",
      "Best model saved at ./model/name/gpt.ftfz.model with validation loss: 0.0344\n",
      "Epoch 2/10 (Training): 100%|█████████████████| 852/852 [00:02<00:00, 288.98it/s]\n",
      "Epoch 2, Training Loss: 0.0141\n",
      "Epoch 2/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 777.23it/s]\n",
      "Epoch 2, Validation Loss: 0.0062\n",
      "Best model saved at ./model/name/gpt.ftfz.model with validation loss: 0.0062\n",
      "Epoch 3/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 267.90it/s]\n",
      "Epoch 3, Training Loss: 0.0038\n",
      "Epoch 3/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 763.84it/s]\n",
      "Epoch 3, Validation Loss: 0.0035\n",
      "Best model saved at ./model/name/gpt.ftfz.model with validation loss: 0.0035\n",
      "Epoch 4/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 273.20it/s]\n",
      "Epoch 4, Training Loss: 0.0018\n",
      "Epoch 4/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 774.92it/s]\n",
      "Epoch 4, Validation Loss: 0.0027\n",
      "Best model saved at ./model/name/gpt.ftfz.model with validation loss: 0.0027\n",
      "Epoch 5/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 269.67it/s]\n",
      "Epoch 5, Training Loss: 0.0010\n",
      "Epoch 5/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 744.24it/s]\n",
      "Epoch 5, Validation Loss: 0.0031\n",
      "Epoch 6/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 276.34it/s]\n",
      "Epoch 6, Training Loss: 0.0006\n",
      "Epoch 6/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 729.34it/s]\n",
      "Epoch 6, Validation Loss: 0.0020\n",
      "Best model saved at ./model/name/gpt.ftfz.model with validation loss: 0.0020\n",
      "Epoch 7/10 (Training): 100%|█████████████████| 852/852 [00:03<00:00, 276.34it/s]\n",
      "Epoch 7, Training Loss: 0.0004\n",
      "Epoch 7/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 787.27it/s]\n",
      "Epoch 7, Validation Loss: 0.0026\n",
      "Epoch 8/10 (Training): 100%|█████████████████| 852/852 [00:02<00:00, 288.93it/s]\n",
      "Epoch 8, Training Loss: 0.0003\n",
      "Epoch 8/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 758.14it/s]\n",
      "Epoch 8, Validation Loss: 0.0029\n",
      "Epoch 9/10 (Training): 100%|█████████████████| 852/852 [00:02<00:00, 287.04it/s]\n",
      "Epoch 9, Training Loss: 0.0002\n",
      "Epoch 9/10 (Validation): 100%|█████████████████| 32/32 [00:00<00:00, 789.00it/s]\n",
      "Epoch 9, Validation Loss: 0.0029\n",
      "Epoch 10/10 (Training): 100%|████████████████| 852/852 [00:03<00:00, 270.66it/s]\n",
      "Epoch 10, Training Loss: 0.0001\n",
      "Epoch 10/10 (Validation): 100%|████████████████| 32/32 [00:00<00:00, 787.65it/s]\n",
      "Epoch 10, Validation Loss: 0.0026\n",
      "\n",
      "real\t0m33.605s\n",
      "user\t0m35.813s\n",
      "sys\t0m2.544s\n",
      "Text generation:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Generated Text 1: ရဲ ရိမ် ငဲ ခက်ခ် ရှိုး ငိုင် ညင် ဂ ယဉ်း နန့် စွမ် ဖို့ နောင့် ဖွာ ကိုင် ဦး ငိုင် ယောင် ထန် မာန် စိမ်း ဘန့် ဒ သွဲ့ ခူး ခွာ ဂိတ် ဓီ ထု ပို ဆံ ညွတ် အု ချိုင်း ထွာ ရှုံး အိမ်း ရှုံး နမ် မန့် သုဒ္ဓေါ ကြွက် ဟောက် ဖတ် ခုပ် ဒိမ့် ရှန်း သို လိ ဝ ဘွမ်\n",
      "Generated Text 2: ရဲ ဝိုင်း တို ကြယ် ထိုင် မော် ဘောမ် သဉ္ဇူ ယု ဓ ရေ့စ် ရုံး ဖား ရို့စ် ထန်း သုံး ပဲန် ငြိမ့် လိ ရောင်းန် ချုပ် သိင်္ဂီ လျို သိမ့် မှုန်း စဉ့် ပြိုင် ကြင် ဟုန် ခံ ရော့ဒ် ပေါ် ဂျမ် ထန်း ဂိတ် ဗုတ် ပဲန်း အောင် တိုင်း ကယ် ဥက္ကာ သျှန္တီ ဘို မီးရ် ပေါက် ဆူ အိဏ်း ကွေး ဟွေ လှန် သင်္ခ\n",
      "Generated Text 3: ရဲ ပြန့် ကိန်း ယိန်း ရွမ်း နှောင်း ကျစ် ဿန်း ပိ ထန် ဆွမ့် ချောင် နှင်း ညွှန့် ဂျိုး ဆွေး ပါ့ ဝါး သက် ယား အင် ဗဲလ် ဂျိုး ထွဋ် ကျွဲ ပေါ ငု သုံး ဏ ပွန် စိမ် ရစ္စ အစ် ဇုံ မတ် ပမ် ဂျား ဖျား ရိုင်းန် နဲမ် ဒင့် ဇင့် ကွတ် အာဖ် တိန့် ဇာ ကုမ္မာ ကယ် အက္ခ တန် ထွဏ်း\n",
      "Generated Text 4: ရဲ ဇမ့် ဖျား ရစ္စ ဆိုက် ရော် သျှား ခွာ အဉ္ဇူ ထူး ဆည်း ဒင့် ရော့ဒ် ဒုလ် ညိမ်း လျန် ထော သျှန် စို ကုန် လော် ဂေ ဆွဲ သိမ့် ပြား ပယ်လ် ညောင် ရိ ဟ ရင်န် ဂျား တေ ဂျိုး လက္ခ ရှန်း ချို ဟွေ ကြယ် ဆန္ဒ စီး နု စော် ဟယ် ဋာ ဗ ယု ဂျာလ် ဒေါင်း ထန်း မှိုင်း အောလ်\n",
      "Generated Text 5: ရဲ တုတ် ရေး ရီး ဖောင် ရှင် ထား လပ် ချောင်း ဂျူ ဇေ ကိုလ် ရိုး နယ် နှီး ဇူ ဟူ ဒွတ် မှူး ဗွန် အဲန်း အုပ် ကက် ဂဂ္ဂါ ညက် ဇမ်း နာမ် ဝီ ကျွန် င ဂျွန်း ဂင့် မှုံ ကန့် ရင့် ဟိန်း လျင် မွန်း မင်္ဂ ရစ် နန္ဒီ နန်းဒ် သင်း ယုန် ရိုင်းန် ကွေ့ ကွန်း ကျ အဲန် ကီလ် ဉာဏ်\n",
      "Generated Text 6: ရဲ အောလ် နှဲမ် ရှိတ် ဒီ ဟိမ်း လွှင် ခေါင် ဿ မူး မြဉ္ဇူ ချေ အဉ္ဇူ ကယ်လ် ကိမ် ဖန် ဆောင် မြေ့ ပြာ ဟန် တ ပြန် ကွဲ ကိုက် စန္ဒီ ရွက် ကောလ် ဂီ ရို့စ် နို ကွိဇ် မွေ အိင်္ခ ငဲ ချိန် မြဉ္ဇူ ဟိုး ဒို့ဗ် မွေ့ ရှုံး ခြင်း အဉ္စ ဟုမ် သိန်း ဝါး ဂင့် ဆင်း နွယ် ဂွိ သိ နွေး\n",
      "Generated Text 7: ရဲ ဗြ ပွန်း ခြိမ့် ခေါ ထား ဒင့် ပု ဖော မျိုး ကျွဲ ဇိတ် ခါ စင်း စံ ထွေး စန္ဒာ ဓီ အက်စ် ဗို အီး လစ် ခေါင်း စွမ်း ကစ္စ အိန္ဒြ ဖျား သောင် ချင် သိင်္ဂီ ဖန်း ရဲ့ ညွတ် ဟို စံ ညာ မိုင်း မြဲ ဂျီ သွင့် ဆည်း န အု ငိုက် သံ သာ ဒ ဂုန် ဇမ့် ထပ် မူး\n",
      "Generated Text 8: ရဲ သုန် ဇင်း ပပ်ဖ် ထာသ် ကံ့ ဟာ ငုံ ရား ယန် လျန်း ဉာဏ် ဂမ် ကွေ့ ဖား ဒူ ရော့ဒ် လှဲန် တိုင်း အယ် ကျေ နော နု ရစ္စ ဂဂ္ဂါ ကြား လျန်း လှေး ကမ်း ချိတ် ကာ ကစ် အင် ဗေ ဟိဏ်း ထက် ဇမ်း သုဒ္ဓေါ စိုင်း သီ ကိ မွေ့ နာ့ ပွင့်် အက် ခြင်း သင်္ကြန် နန္ဒီ မူ ချုံ ဆားရ်\n",
      "Generated Text 9: ရဲ ဘုတ် ငွေ့ ဘုန်း [UNK] ဇော ပါ့ ဋာ သတ္တိ ဝေ လက် ဂျေ ဥတ္တ ကင်းမ် ကွေ့ ဗုံ မိန်း ငြိမ် ဂျ ညှင်း နောင့် ဟွ လမ်း ကဉ္စ ဝဏ် ဘန့် ကြံ့ ဓီ ထိုး တွဲ ကစ် ထောင် စိုင်း ဂေါ် ဖန်း လွိုင်း မျာ ပွ လှီး ကိန်း ယုံ ဂျန် နှော ရက် အိမ်း ဇမ့် မြေ ကျိမ်း အိုက် မြဲ ဗျက်\n",
      "Generated Text 10: ရဲ အိဇ္ဇာ ဇေါင်း သျှန် အိဏ် ဂျီ နော် စိမ်း လပ် ထိ သက္က တီလ် ဘုဏ်း နတ် ပေါ ကျွင်း ကမ်း ငွေ ဂတ် နန္ဒီ ရုဏ် ထွယ် ဝင် မြူ ကျောက် ခါ စွယ် ဆွင်း အွန် ညောင် အန့် ခဏ် ကွတ် မွင်း မိုး တီလ် နှိုင်း ကြုံ ကြော့ ကြုံ ရယ် ကြံ ဂျမ်း သား လစ် သီး ထယ် မိုရ် သဒ္ဒါ ပြုံ ပြိုင်\n",
      "\n",
      "real\t0m2.506s\n",
      "user\t0m5.306s\n",
      "sys\t0m2.316s\n",
      "Batch text generation from file:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:228: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Random Prompt Generated: ဂျယ်လ်\n",
      "Generated texts saved to ./output/name/gpt_ftfz_gen_texts.txt\n",
      "\n",
      "real\t0m2.105s\n",
      "user\t0m4.861s\n",
      "sys\t0m2.430s\n",
      "Testing:\n",
      "/home/ye/exp/name-lm/lib/laphet.py:429: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  checkpoint = torch.load(args.model)\n",
      "Testing: 100%|██████████| 16/16 [00:00<00:00, 53.20it/s]\n",
      "Average Perplexity on Test Data: 1.0015\n",
      "Average Cross-Entropy on Test Data: 0.0015\n",
      "\n",
      "real\t0m1.870s\n",
      "user\t0m4.743s\n",
      "sys\t0m2.344s\n",
      "All tasks completed!\n"
     ]
    }
   ],
   "source": [
    "!./train_test_name_ftfz.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96c18a92-865a-4a1a-ac55-b5ab24fcfade",
   "metadata": {},
   "source": [
    "## Model, Vocab Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d4f46c75-6286-420f-a33c-b1068ee85730",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 ye ye 3.4M Jan 29 00:25 ./model/name/bert.ftfz.model\n",
      "-rw-rw-r-- 1 ye ye  88M Jan 29 00:23 ./model/name/bilstm.ftfz.model\n",
      "-rw-rw-r-- 1 ye ye 3.4M Jan 29 00:25 ./model/name/gpt.ftfz.model\n",
      "-rw-rw-r-- 1 ye ye 3.7M Jan 29 00:21 ./model/name/mlp.ftfz.model\n",
      "-rw-rw-r-- 1 ye ye 3.4M Jan 29 00:24 ./model/name/transformer.ftfz.model\n"
     ]
    }
   ],
   "source": [
    "!ls -lh ./model/name/*ftfz*model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "da8d47d4-cb74-4dfe-b61a-d2ffacc252ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1443   2886  24556 ./model/name/bert.ftfz.model.vocab\n",
      "  1443   2886  24556 ./model/name/bilstm.ftfz.model.vocab\n",
      "  1443   2886  24556 ./model/name/gpt.ftfz.model.vocab\n",
      "  1443   2886  24556 ./model/name/mlp.ftfz.model.vocab\n",
      "  1443   2886  24556 ./model/name/transformer.ftfz.model.vocab\n",
      "  7215  14430 122780 total\n"
     ]
    }
   ],
   "source": [
    "!wc ./model/name/*ftfz*vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2020dbf8-5609-4a56-8750-c2fc58b8d1b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ပယ်\t0\n",
      "လို့\t1\n",
      "ချဲ့\t2\n",
      "အိန္မာ\t3\n",
      "ကျူး\t4\n",
      "သည်း\t5\n",
      "ရှိတ်\t6\n",
      "ရွိုင်\t7\n",
      "သွယ်\t8\n",
      "နယ်\t9\n"
     ]
    }
   ],
   "source": [
    "!head ./model/name/gpt.ftfz.model.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b719c7f6-8b7e-412d-939b-0dced5795c81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "လင်္ကာ\t0\n",
      "ကျို\t1\n",
      "ဇေ\t2\n",
      "ယန်း\t3\n",
      "ဒိန်\t4\n",
      "စုံ\t5\n",
      "မတ်\t6\n",
      "ဂျေ\t7\n",
      "ဇိန်း\t8\n",
      "တည်\t9\n"
     ]
    }
   ],
   "source": [
    "!head ./model/name/transformer.ftfz.model.vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f75d25-3d16-488f-9f8d-e00c502a6d3a",
   "metadata": {},
   "source": [
    "မော်ဒယ် တစ်ခုချင်းစီရဲ့ generated name တွေကို လေ့လာကြည့်ရအောင်။   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ee7b3e37-dc7f-4457-a358-8481f7755ee0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ကျော် လန်း မွှန်း\n",
      "ကျော် ဠု ထား\n",
      "ကျော် ဝန် တာ\n",
      "ကျော် ကျ ကြွမ်\n",
      "ကျော် ကင်း မြင့့််\n",
      "မ မ တန် မင်္ဂ\n",
      "မ မ ချင်း ဿီ\n",
      "မ မ ဥမ္မာ ဋာ\n",
      "မ မ ကျီး ဟယ်\n",
      "မ မ ယော ခွေ\n",
      "အေး ဂို လှဲ\n",
      "အေး ဆီ ဟွ\n",
      "အေး မြန် ခု\n",
      "အေး လျှား သိမ်း\n",
      "အေး ညွန် ရေး\n",
      "လှ လှ ဇက် ခါ\n",
      "လှ လှ ရှောင် စဲ\n",
      "လှ လှ ကွမ်း အိမ်း\n",
      "လှ လှ ချက်ထ် နေ\n",
      "လှ လှ ဆုန်း နွေ\n",
      "စွန် ချက်ထ် လိ\n",
      "စွန် ဆာ ရစ်\n",
      "စွန် လွေ ရွန်း\n",
      "စွန် ဒေါင်း သင့်\n",
      "စွန် အဲန်း ကြ\n",
      "မြ အေး ဟ ခေါင်\n",
      "မြ အေး အိန္တာ ဠု\n",
      "မြ အေး သန်း ပြောင်း\n",
      "မြ အေး ဝံ ဠု\n",
      "မြ အေး လမ် ဟောမ်း\n",
      "သ နိုက် ဿဏ်\n",
      "သ အဲ ပွန်း\n",
      "သ ဟေ သု\n",
      "သ ခမ်း မွမ်\n",
      "သ ရိုး ဆိုး\n",
      "မောင် နာ့ သောင်\n",
      "မောင် ကွဲ အော်\n",
      "မောင် သား ခြူး\n",
      "မောင် ဂျမ် ခြူး\n",
      "မောင် ဇမ်း ထိန်\n",
      "မြင့် မြင့် အိင်္ခ ဗို\n",
      "မြင့် မြင့် လင်း ထွတ်\n",
      "မြင့် မြင့် အဲ ကမ်\n",
      "မြင့် မြင့် ဟိန်း ပယ်\n",
      "မြင့် မြင့် [PAD] တီးလ်\n",
      "ရွှေ ဒတ် မွေး\n",
      "ရွှေ အုံ ပွန်\n",
      "ရွှေ အု ပွား\n",
      "ရွှေ မူလ္လာ ရုန်း\n",
      "ရွှေ ဝီ ရုံ\n",
      "အဂ္ဂ သျှမ် ခြာ\n",
      "အဂ္ဂ ရှင်း နှိုင်း\n",
      "အဂ္ဂ လမ်း သော\n",
      "အဂ္ဂ ပဉ္စ နောင့်\n",
      "အဂ္ဂ ရီး ဗြာ\n",
      "ဥက္ကာ မြေ့ သွင့်\n",
      "ဥက္ကာ စိ ရိုင်းန်\n",
      "ဥက္ကာ ထီ ခါရ်\n",
      "ဥက္ကာ လျန် ထပ်\n",
      "ဥက္ကာ ရော ထ\n",
      "သိင်္ဂီ ခမ်း ဇင်\n",
      "သိင်္ဂီ လဲန်း နေး\n",
      "သိင်္ဂီ မြဲ ကြည်\n",
      "သိင်္ဂီ ညို့ လျှန်\n",
      "သိင်္ဂီ ရုံး ဂန်\n",
      "မေ ဝတ် ဒွန့်\n",
      "မေ အဏ္ဏ လုတ်\n",
      "မေ ရှိန်း ဆဲ\n",
      "မေ ကျောက် ဖူး\n",
      "မေ ခြင်း ထော\n",
      "ခိုင် သို ပျံ့\n",
      "ခိုင် ဘယ် တွာ\n",
      "ခိုင် ခွာ ဗွန်\n",
      "ခိုင် ဂျော် တီးလ်\n",
      "ခိုင် ဖွား နောင့်\n"
     ]
    }
   ],
   "source": [
    "!cat ./output/name/mlp_ftfz_gen_texts.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8bac8ee6-ee98-4d1f-b891-e1459a1b264e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ကျော် ဿန်း ဟူ\n",
      "ကျော် နှီး ဿန်း\n",
      "ကျော် ရှဲလ် ဣန္တာ\n",
      "ကျော် ဣန္တာ မျာ\n",
      "ကျော် ဟူ ရှဲလ်\n",
      "မ မ ဿန်း နှီး\n",
      "မ မ ဟူ နှီး\n",
      "မ မ ရှဲလ် ဣန္တာ\n",
      "မ မ ဣန္တာ ရှဲလ်\n",
      "မ မ ရှဲလ် ဿန်း\n",
      "အေး ရှဲလ် ဌေး\n",
      "အေး မျာ ဿန်း\n",
      "အေး ဟူ ဣန္တာ\n",
      "အေး ရှဲလ် ဌေး\n",
      "အေး ရှဲလ် ရှဲလ်\n",
      "လှ လှ မျာ တန်\n",
      "လှ လှ မျာ ရှဲလ်\n",
      "လှ လှ မျာ ဿန်း\n",
      "လှ လှ မျာ ရှဲလ်\n",
      "လှ လှ ရှဲလ် မျာ\n",
      "ထော် ဟူ ရှဲလ်\n",
      "ထော် မျာ ရှဲလ်\n",
      "ထော် ဌေး ဌေး\n",
      "ထော် ရှဲလ် မျာ\n",
      "ထော် ရှဲလ် ဟူ\n",
      "မြ အေး ရှဲလ် နှီး\n",
      "မြ အေး တန် တန်\n",
      "မြ အေး မျာ ဟူ\n",
      "မြ အေး နှီး နှီး\n",
      "မြ အေး ဟူ နှီး\n",
      "သ မျာ မျာ\n",
      "သ ဌေး နှီး\n",
      "သ မျာ ဿန်း\n",
      "သ ဿန်း မျာ\n",
      "သ ရှဲလ် ဿန်း\n",
      "မောင် ဟူ နှီး\n",
      "မောင် ဣန္တာ ရှဲလ်\n",
      "မောင် တန် နှီး\n",
      "မောင် ဣန္တာ နှီး\n",
      "မောင် ဿန်း တန်\n",
      "မြင့် မြင့် ရှဲလ် ရှဲလ်\n",
      "မြင့် မြင့် ဣန္တာ တန်\n",
      "မြင့် မြင့် တန် ရှဲလ်\n",
      "မြင့် မြင့် မျာ ဌေး\n",
      "မြင့် မြင့် ဿန်း ဿန်း\n",
      "ရွှေ မျာ ရှဲလ်\n",
      "ရွှေ ဣန္တာ ဿန်း\n",
      "ရွှေ တန် ဣန္တာ\n",
      "ရွှေ ဌေး ဿန်း\n",
      "ရွှေ မျာ ဟူ\n",
      "အဂ္ဂ ဣန္တာ နှီး\n",
      "အဂ္ဂ ရှဲလ် နှီး\n",
      "အဂ္ဂ ဟူ နှီး\n",
      "အဂ္ဂ ရှဲလ် ဟူ\n",
      "အဂ္ဂ နှီး တန်\n",
      "ဥက္ကာ တန် နှီး\n",
      "ဥက္ကာ မျာ ဌေး\n",
      "ဥက္ကာ ရှဲလ် နှီး\n",
      "ဥက္ကာ ဌေး နှီး\n",
      "ဥက္ကာ မျာ မျာ\n",
      "သိင်္ဂီ တန် ဌေး\n",
      "သိင်္ဂီ ဿန်း တန်\n",
      "သိင်္ဂီ မျာ ဌေး\n",
      "သိင်္ဂီ ဿန်း နှီး\n",
      "သိင်္ဂီ နှီး မျာ\n",
      "မေ ဿန်း ဌေး\n",
      "မေ ဌေး ဌေး\n",
      "မေ နှီး ဿန်း\n",
      "မေ ရှဲလ် ဿန်း\n",
      "မေ ဣန္တာ နှီး\n",
      "ခိုင် ရှဲလ် ဌေး\n",
      "ခိုင် နှီး နှီး\n",
      "ခိုင် နှီး ရှဲလ်\n",
      "ခိုင် ဿန်း နှီး\n",
      "ခိုင် ရှဲလ် ဌေး\n"
     ]
    }
   ],
   "source": [
    "!cat ./output/name/bilstm_ftfz_gen_texts.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "093e9047-327d-452e-b616-62b1f5e36efb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ကျော် ပြိုင် ကြွက်\n",
      "ကျော် ဖြင့် ခူး\n",
      "ကျော် အိန္ဒြေ သက်\n",
      "ကျော် ဇိန်း ကြွက်\n",
      "ကျော် ကျုံး ခေါက်\n",
      "မ မ ရွမ် ခူး\n",
      "မ မ လူ ခယ်\n",
      "မ မ လှေ လွမ်း\n",
      "မ မ လာမ် ခွမ်း\n",
      "မ မ ချိ ချ\n",
      "အေး ညိမ်း ဖောင်း\n",
      "အေး ခေါ် သျှန်\n",
      "အေး လင် ရန်\n",
      "အေး ကျီ သျှန်\n",
      "အေး ဂေါ် ရန်\n",
      "လှ လှ ကယ် ရန်\n",
      "လှ လှ ချက်ထ် ကြွက်\n",
      "လှ လှ သျှန္တီ ရန်\n",
      "လှ လှ လောင်း ရန်\n",
      "လှ လှ ထွန်း ကိုင်\n",
      "ကန် သျှန် သျှန်\n",
      "ကန် ခူး သျှန်\n",
      "ကန် လွမ်း သျှန်\n",
      "ကန် သျှန် သျှန်\n",
      "ကန် သျှန် သျှန်\n",
      "မြ အေး ဒို့ဗ် သျှန်\n",
      "မြ အေး ထက် ညို\n",
      "မြ အေး လင်္ကာ ခူး\n",
      "မြ အေး ဇ ခေတ်\n",
      "မြ အေး သိင်္ဃ သျှန်\n",
      "သ ဒိန် ခူး\n",
      "သ ကစ် သျှန်\n",
      "သ ယာန် ကြွက်\n",
      "သ မီး လောဒ်\n",
      "သ သွေး ကွိဇ်\n",
      "မောင် သျှန်း သျှန်\n",
      "မောင် ဗန်း ခူး\n",
      "မောင် ဝိုင်း ခူ\n",
      "မောင် ရှောင် သီ\n",
      "မောင် ဟင်း ခူး\n",
      "မြင့် မြင့် ကျိမ်း သျှန်\n",
      "မြင့် မြင့် သေး သျှန်\n",
      "မြင့် မြင့် ကျီ ခူး\n",
      "မြင့် မြင့် တုံ သျှန်\n",
      "မြင့် မြင့် စီး ရန်\n",
      "ရွှေ မှု ရန်\n",
      "ရွှေ ထင် ညာ\n",
      "ရွှေ ရွက် သျှန်\n",
      "ရွှေ ကျုံ သျှန်\n",
      "ရွှေ စိမ် သျှန်\n",
      "အဂ္ဂ သျှန် သျှန်\n",
      "အဂ္ဂ သျှန် သျှန်\n",
      "အဂ္ဂ သျှန် သျှန်\n",
      "အဂ္ဂ ခူး သျှန်\n",
      "အဂ္ဂ သျှန် သျှန်\n",
      "ဥက္ကာ ဂျယ်လ် သျှန်\n",
      "ဥက္ကာ လက် ဟောက်\n",
      "ဥက္ကာ နော ခူး\n",
      "ဥက္ကာ ဣန္တာ သျှန်\n",
      "ဥက္ကာ ခံ သျှန်\n",
      "သိင်္ဂီ အိုင် သျှန်\n",
      "သိင်္ဂီ ထင် အဲလ်\n",
      "သိင်္ဂီ သတ် သျှန်\n",
      "သိင်္ဂီ ရော ခူး\n",
      "သိင်္ဂီ အိပ် ခူး\n",
      "မေ မိုး ကန်း\n",
      "မေ မာန် ဿန်း\n",
      "မေ စူ ချ\n",
      "မေ နက် သျှန်\n",
      "မေ မိုင်းလ် သျှန်\n",
      "ခိုင် ဂူး ခူး\n",
      "ခိုင် စန်း ပူး\n",
      "ခိုင် လန့် ချ\n",
      "ခိုင် အိမ် မှိုင်\n",
      "ခိုင် ညာ သျှန်\n"
     ]
    }
   ],
   "source": [
    "!cat ./output/name/transformer_ftfz_gen_texts.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b67b72f0-26cd-4b00-ae3f-b7271a07f8fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ကျော် ယော ပီ\n",
      "ကျော် ညို နန္ဒ\n",
      "ကျော် သိမ့် ဇော်\n",
      "ကျော် ကြိုး လှေ\n",
      "ကျော် ဇိ ရှား\n",
      "မ မ လဲ လှေ\n",
      "မ မ ပျို ရှား\n",
      "မ မ ဒေ သိဒ္ဓိ\n",
      "မ မ မြီ ရှား\n",
      "မ မ ပွင့်် သေ့\n",
      "အေး တောင်း လှေ\n",
      "အေး ညှင်း ရှား\n",
      "အေး ဣန္ဒြေ ကျိမ်း\n",
      "အေး နာ့ ကျင်း\n",
      "အေး ကုမ္မာ ဝိုက်\n",
      "လှ လှ ရွာ ပီ\n",
      "လှ လှ နွန် ရှား\n",
      "လှ လှ ချောင်း ပီ\n",
      "လှ လှ နှင်း သင်္ချာ\n",
      "လှ လှ ကျီး ရှား\n",
      "သိင်္ခ ရှား ရ\n",
      "သိင်္ခ ရှား ဝိုက်\n",
      "သိင်္ခ ရှား ရှား\n",
      "သိင်္ခ ရှား ဝိုက်\n",
      "သိင်္ခ ရှား ရှား\n",
      "မြ အေး ဘူ ရှား\n",
      "မြ အေး ဒေ ဘောမ်\n",
      "မြ အေး ချိတ် ရှား\n",
      "မြ အေး တုန် ရှား\n",
      "မြ အေး ရေ ကောက်\n",
      "သ ကျော့် ဟုမ်\n",
      "သ ယွန်း မြူ\n",
      "သ ဆွေ ဟင်\n",
      "သ နို သိဒ္ဓိ\n",
      "သ ယိမ့် ကျိမ်း\n",
      "မောင် သိင်္ဂီ ချွန်\n",
      "မောင် လာန်း လှေ\n",
      "မောင် ရော့ခ် ပီ\n",
      "မောင် နှဲမ် နှင်း\n",
      "မောင် ဖု ကျိမ်း\n",
      "မြင့် မြင့် ကောန် ရှား\n",
      "မြင့် မြင့် ဂွမ် လှေ\n",
      "မြင့် မြင့် နန္ဒ ပီး\n",
      "မြင့် မြင့် သင်္ချာ ကျိမ်း\n",
      "မြင့် မြင့် ဝိုင်း ဒမ်း\n",
      "ရွှေ ဓမ္မာ ပီ\n",
      "ရွှေ ဿာ ရှား\n",
      "ရွှေ ဟေ ခွမ်း\n",
      "ရွှေ ဂိုး ရှား\n",
      "ရွှေ ထွေ ပီ\n",
      "အဂ္ဂ ပီ သိမ်း\n",
      "အဂ္ဂ ရှား ရှား\n",
      "အဂ္ဂ ပီ သင်္ကြန်\n",
      "အဂ္ဂ ရ မှီ\n",
      "အဂ္ဂ ပီ အိမ့်\n",
      "ဥက္ကာ ဿန် လှေ\n",
      "ဥက္ကာ လှဲန် နှင်း\n",
      "ဥက္ကာ ရေး ရှား\n",
      "ဥက္ကာ အေ လွန်\n",
      "ဥက္ကာ ကြင် အဲမ်\n",
      "သိင်္ဂီ ဘွဲ ရှား\n",
      "သိင်္ဂီ ဗေ ရှား\n",
      "သိင်္ဂီ ဟြေ ဝိုက်\n",
      "သိင်္ဂီ မူး ပီ\n",
      "သိင်္ဂီ သဉ္စာ ချော\n",
      "မေ ချုံး ရှား\n",
      "မေ ခေါမ်း လှေ\n",
      "မေ အိုင်း ရှား\n",
      "မေ လိတ် လှေ\n",
      "မေ ဆပ် ပီ\n",
      "ခိုင် မီး ယွန်း\n",
      "ခိုင် သေး ပီ\n",
      "ခိုင် စေး ရှား\n",
      "ခိုင် ခွန်း ရှား\n",
      "ခိုင် ပွန်း ရှား\n"
     ]
    }
   ],
   "source": [
    "!cat ./output/name/bert_ftfz_gen_texts.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "82c35a4f-377f-4974-9b9e-85b8e8870565",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ကျော် ခေါက် ဗေ\n",
      "ကျော် ဂို ရွတ်\n",
      "ကျော် သမ္ဘူ ရ\n",
      "ကျော် ပိုင် ဝီ\n",
      "ကျော် ဝင့် ရယ်လ်\n",
      "မ မ ရိုင်း သီ\n",
      "မ မ မောရ် ဣ\n",
      "မ မ အိုမ် မြင်\n",
      "မ မ ဒေါင်း အဉ္ဇူ\n",
      "မ မ ဆိုင် အိန်\n",
      "အေး ခွာ ကေ\n",
      "အေး ခုံ လဲန်း\n",
      "အေး ပဲန် ဗ\n",
      "အေး လာန် သန္တာ\n",
      "အေး နေ မြင့်\n",
      "လှ လှ ထာ ဘင်း\n",
      "လှ လှ ငြိမ့် မင်္ဂ\n",
      "လှ လှ ကီး ဂန္ဓ\n",
      "လှ လှ လွှင် ကြွမ်\n",
      "လှ လှ ဇွန် ဒီ\n",
      "ဂျယ်လ် တေ အေ\n",
      "ဂျယ်လ် အိုက် ကျောက်\n",
      "ဂျယ်လ် ကျင် အူလ္လာ\n",
      "ဂျယ်လ် အဲန် ရစ္စ\n",
      "ဂျယ်လ် ထွေ ခွား\n",
      "မြ အေး အဲန် ခင်\n",
      "မြ အေး ရို မှုန်\n",
      "မြ အေး စမ်း လှောင်\n",
      "မြ အေး ထွန်း လွဲ\n",
      "မြ အေး စစ် လာဘ်\n",
      "သ ဖ ဒေ့ဗ်\n",
      "သ တွန်း သန္ဒြာ\n",
      "သ ချစ် ဂန္တ\n",
      "သ မြတ် ဒူး\n",
      "သ ရ ရှာ\n",
      "မောင် ဟွမ်း ဇမ့်\n",
      "မောင် ဇာ တန့်\n",
      "မောင် ခတ် ထာသ်\n",
      "မောင် ရက် အိန္တာ\n",
      "မောင် ဟ ဆ\n",
      "မြင့် မြင့် ဆား ချင်\n",
      "မြင့် မြင့် လိန်း ရင်န်\n",
      "မြင့် မြင့် ပစ် တန့်\n",
      "မြင့် မြင့် ဟူး အံ\n",
      "မြင့် မြင့် ကန်း မြန်\n",
      "ရွှေ ဘူး ခု\n",
      "ရွှေ ဒြာ နီ\n",
      "ရွှေ ကွမ်း တော\n",
      "ရွှေ သု ဝိ\n",
      "ရွှေ ကြော အိုး\n",
      "အဂ္ဂ တောင်း ပိုး\n",
      "အဂ္ဂ ကတ် ညို\n",
      "အဂ္ဂ လှန် တက္ခ\n",
      "အဂ္ဂ ဒင့် ယူ\n",
      "အဂ္ဂ ညွတ် သန်း\n",
      "ဥက္ကာ ငြိမ်း ဂျိုး\n",
      "ဥက္ကာ လမ်း စန္ဒီ\n",
      "ဥက္ကာ ကြောင်း လိန်\n",
      "ဥက္ကာ ဿီ တေး\n",
      "ဥက္ကာ သဲ ကြည်\n",
      "သိင်္ဂီ ဇေါင်း မီး\n",
      "သိင်္ဂီ ဆွိ ဟွေ\n",
      "သိင်္ဂီ ကီလ် ဇမ်း\n",
      "သိင်္ဂီ ဂျတ် ဂျယ်လ်\n",
      "သိင်္ဂီ ဖွေး ဟမ်း\n",
      "မေ ဘုန်း ငု\n",
      "မေ ပေါ လောင်\n",
      "မေ အုံး ချာ\n",
      "မေ ပီး ချူး\n",
      "မေ ကိုလ် ရှေး\n",
      "ခိုင် ရုဏ် ချင်း\n",
      "ခိုင် တင့် ဟဲ\n",
      "ခိုင် လာန် ဝါ\n",
      "ခိုင် အိုင်း မဉ္ဇူ\n",
      "ခိုင် သီ ဒွန့်\n"
     ]
    }
   ],
   "source": [
    "!cat ./output/name/gpt_ftfz_gen_texts.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d151de5-ffdb-4e87-8b41-fc1af427b8ff",
   "metadata": {},
   "source": [
    "Testing data နဲ့ evaluation လုပ်ထားတဲ့ perplexity, cross-entropy ရလဒ်တွေကို လေ့လာကြည့်ရအောင်။  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c8539c29-d9b8-4ce7-940b-b31b71f450f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> ./log/name/bert.ftfz.log <==\n",
      "Average Perplexity on Test Data: 1.0112\n",
      "Average Cross-Entropy on Test Data: 0.0111\n",
      "\n",
      "==> ./log/name/bilstm.ftfz.log <==\n",
      "Average Perplexity on Test Data: 1.1671\n",
      "Average Cross-Entropy on Test Data: 0.1544\n",
      "\n",
      "==> ./log/name/gpt.ftfz.log <==\n",
      "Average Perplexity on Test Data: 1.0015\n",
      "Average Cross-Entropy on Test Data: 0.0015\n",
      "\n",
      "==> ./log/name/mlp.ftfz.log <==\n",
      "Average Perplexity on Test Data: 1.1143\n",
      "Average Cross-Entropy on Test Data: 0.1082\n",
      "\n",
      "==> ./log/name/transformer.ftfz.log <==\n",
      "Average Perplexity on Test Data: 1.0106\n",
      "Average Cross-Entropy on Test Data: 0.0105\n"
     ]
    }
   ],
   "source": [
    "!tail -n 2 ./log/name/*ftfz*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d086776-9568-49cf-80e8-43dde4ce1ce0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}