Skip to content

CRAN Release 0.2
Compare
Choose a tag to compare
@jwijffels jwijffels released this 14 Dec 23:07
· 3 commits to master since this release
0.2
93bb5f3

CHANGES IN sentencepiece VERSION 0.2

  • Fix wordpiece bug for 1-character words. (@jonthegeek, #4)
  • Upgraded to sentencepiece release v0.1.96

CHANGES IN sentencepiece VERSION 0.1.3

  • Fix wordpiece bug for 1-character words. (@jonthegeek, #4)
  • Fix Solaris installation issue related to incorrect usage of pointer as a function
  • Also download the binary model in sentencepiece_download_model as it can be loaded with word2vec::read.wordvectors
  • read_word2vec now uses word2vec::read.wordvectors from word2vec >= 0.2.0
  • added BPEembed and predict.BPEembed
  • allow subword regularisation by adding nbest and alpha option in sentencepiece_encode and changed sentencepiece_decode accordingly
  • Added txt_remove_
  • Upgrade sentencepiece to release v0.1.91 commit a32d7dc6ce6f383a65ad6e1cbe1983f94ab11932 which has subword regularisation for BPE

Contributors

jonthegeek
Assets 2
Loading