[crf] Wapiti

• Wapiti alternative

  • combine Wapiti, Wapiti Python wrapper and CQDB
  • process line by line not file
  • use CQDB for fast lookup feature string to id(label mode only)

• about Wapiti

  • it can retrain a pretrained model.
  • it implements f(y_t, y_t-1, obs_t) bigram features.
    • refer
    • but, if the number of labels is much larger, out of memory error might be issue.
  • in case, the training data is huge and you are using ‘B’ bigram feature, then training would be failed.
    • refer

    ...
    * Train the model with l-bfgs
      [   1] obj=3080220.85 act=1447258  err=45.81%/99.34% time=1997.97s/1997.97s
      [   2] obj=2061806.01 act=1446884  err=45.81%/99.34% time=5747.45s/7745.42s
      [   3] obj=1897392.82 act=989401   err=45.81%/99.34% time=4996.30s/12741.72s
      [   4] obj=1864936.55 act=1397073  err=45.81%/99.34% time=5314.24s/18055.96s
      [   5] obj=1862659.23 act=978958   err=45.81%/99.34% time=3562.89s/21618.85s
    * Compacting the model
        - Scan the model
        - Compact it
            1278 observations removed
            886932 features removed
    * Save the model
    ...
    

    • try with ‘-o 10 -w 10 -d devel.txt’ setting to come up with 'stop criterias'