Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tokenizer #64

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open

Add tokenizer #64

wants to merge 31 commits into from

Commits on Aug 20, 2022

  1. 高機能なトークナイザをターミナルに組み込み

    Ryosuke committed Aug 20, 2022
    Configuration menu
    Copy the full SHA
    76913c7 View commit details
    Browse the repository at this point in the history

  2. トークナイザのテストを追加

    Ryosuke committed Aug 20, 2022
    Configuration menu
    Copy the full SHA
    8a7e48d View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2022

  1. 関数名の変更

    Ryosuke committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    e00d4f5 View commit details
    Browse the repository at this point in the history

  2. 関数名変更の適応し忘れを修正

    Ryosuke committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    74a2b91 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2023

  1. fix memory leak

    @tomiy-0x62
    tomiy-0x62 committed Mar 5, 2023
    Configuration menu
    Copy the full SHA
    730baa0 View commit details
    Browse the repository at this point in the history

  2. 不要な変数を削除

    @tomiy-0x62
    tomiy-0x62 committed Mar 5, 2023
    Configuration menu
    Copy the full SHA
    97ad03f View commit details
    Browse the repository at this point in the history

  3. strcmpを==に置換

    @tomiy-0x62
    tomiy-0x62 committed Mar 5, 2023
    Configuration menu
    Copy the full SHA
    c196367 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. 無駄な行を1行にまとめるように修正

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    11e14a4 View commit details
    Browse the repository at this point in the history

  2. 無駄なif文を削除

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    55b952d View commit details
    Browse the repository at this point in the history

  3. stringの長さの取得方法を修正

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    4e2adf6 View commit details
    Browse the repository at this point in the history

  4. 引数を改行した箇所の先頭位置を修正

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    fee03ea View commit details
    Browse the repository at this point in the history

  5. 列挙子の意味をコメントに追加

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    41056cf View commit details
    Browse the repository at this point in the history

  6. メンバ変数の仕様をコメントに追加

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    998e053 View commit details
    Browse the repository at this point in the history

  7. fix typo

    @tomiy-0x62
    tomiy-0x62 committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    a756392 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. 関数名に含まれるTokenizerInnerStateの略称TisをTISに変更

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    bd6bf0f View commit details
    Browse the repository at this point in the history

  2. using c++17

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    06b5d3e View commit details
    Browse the repository at this point in the history

  3. TokenizerInnerStateにコンストラクタとunipue_prtのエイリアスを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    31c05bf View commit details
    Browse the repository at this point in the history

  4. 構造体のポンタをuniqueポインタに変更

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    66d2556 View commit details
    Browse the repository at this point in the history

  5. use dep file

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    701d772 View commit details
    Browse the repository at this point in the history

  6. TestTokenizeの戻り値をboolに変更

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    41f7811 View commit details
    Browse the repository at this point in the history

  7. テストの成功割合を表示

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    e3de427 View commit details
    Browse the repository at this point in the history

  8. 引数のインデントレベルを修正

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    2f279f6 View commit details
    Browse the repository at this point in the history

  9. tmp_tokenがstringのコンストラクタを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    3a4d3cd View commit details
    Browse the repository at this point in the history

  10. 引数を変更しない実装に変更し、その引数にconstを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    7522e04 View commit details
    Browse the repository at this point in the history

  11. verboseを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    630092b View commit details
    Browse the repository at this point in the history

  12. 関数ドキュメントを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    cf37d70 View commit details
    Browse the repository at this point in the history

  13. 関数ドキュメントを追加

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    3a45705 View commit details
    Browse the repository at this point in the history

  14. 変更しない引数をconst参照に変更

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    41e852b View commit details
    Browse the repository at this point in the history

  15. 削除された領域を指すポインタを生成する問題を修正

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5917e7d View commit details
    Browse the repository at this point in the history

  16. strchrを削除

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    b9164c0 View commit details
    Browse the repository at this point in the history

  17. cat の複数のファイル対応を削除

    @tomiy-0x62
    tomiy-0x62 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    ab09297 View commit details
    Browse the repository at this point in the history