Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tokenizers don't split words into sub-words #5756

Merged
merged 16 commits into from
May 5, 2020
Merged

Tokenizers don't split words into sub-words #5756

merged 16 commits into from
May 5, 2020

Commits on Apr 30, 2020

  1. take mean vec of sub-tokens for ConveRT

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    260b282 View commit details
    Browse the repository at this point in the history

  2. take mean vec of sub-tokens for HF models

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    a1f1aaa View commit details
    Browse the repository at this point in the history

  3. clean up doc strings

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    a23b279 View commit details
    Browse the repository at this point in the history

  4. remove unused method

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    5baa02e View commit details
    Browse the repository at this point in the history

  5. remove unused method

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    b878ad6 View commit details
    Browse the repository at this point in the history

  6. update doc strings

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    a7bf4c1 View commit details
    Browse the repository at this point in the history

  7. fix tests

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    fe1b77f View commit details
    Browse the repository at this point in the history

  8. add changelog

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    3588b5f View commit details
    Browse the repository at this point in the history

  9. remove unused imports

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    25a3635 View commit details
    Browse the repository at this point in the history

  10. update test

    @tabergma
    tabergma committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    562bff7 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2020

  1. add test for HFTransfomerNLP

    @tabergma
    tabergma committed May 4, 2020
    Configuration menu
    Copy the full SHA
    3d6653c View commit details
    Browse the repository at this point in the history

  2. fix import

    @tabergma
    tabergma committed May 4, 2020
    Configuration menu
    Copy the full SHA
    35b546d View commit details
    Browse the repository at this point in the history

  3. update tests

    @tabergma
    tabergma committed May 4, 2020
    Configuration menu
    Copy the full SHA
    9270738 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into fix-tokenization

    @tabergma
    tabergma committed May 4, 2020
    Configuration menu
    Copy the full SHA
    6ce0bea View commit details
    Browse the repository at this point in the history

  5. model breaking: upate version

    @tabergma
    tabergma committed May 4, 2020
    Configuration menu
    Copy the full SHA
    6a8f2b5 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2020

  1. Merge branch 'master' into fix-tokenization

    @tabergma
    tabergma authored May 5, 2020
    Configuration menu
    Copy the full SHA
    ae2ad82 View commit details
    Browse the repository at this point in the history