Code

Add checkpointing for training#188

Code

Merged

tanaysoni merged 24 commits intomasterdeepset-ai/FARM:masterfrom train_checkpointdeepset-ai/FARM:train_checkpointCopy head branch name to clipboardJan 22, 2020

+453-53

Commits

Commits on Jan 17, 2020

• Add save and load of checkpoints during training

  tanaysoni

  committed
  e71cb9a

  Copy full SHA for e71cb9a

• Refactor train checkpointing

  tanaysoni

  committed
  e04c7be

  Copy full SHA for e04c7be

• Start implementation

  

  brandenchan

  authored and

  tanaysoni

  committed
  8c8a551

  Copy full SHA for 8c8a551

• latest hyperparams

  

  brandenchan

  authored and

  tanaysoni

  committed
  bc1dc0c

  Copy full SHA for bc1dc0c

• Add dataset checkpointing

  tanaysoni

  committed
  e49822b

  Copy full SHA for e49822b

• Fix vocab_size initialization in BertConfig

  tanaysoni

  committed
  d396e2d

  Copy full SHA for d396e2d

• Fix lm pretraining

  

  brandenchan

  authored and

  tanaysoni

  committed
  5e87492

  Copy full SHA for 5e87492

• Remove in-place key pop for opts dict in scheduler

  tanaysoni

  committed
  83a36b3

  Copy full SHA for 83a36b3

• Add docs and clean-up code

  tanaysoni

  committed
  1eb5e6a

  Copy full SHA for 1eb5e6a

• Fix steps iteration in resuming train from a checkpoint

  tanaysoni

  committed
  ade4722

  Copy full SHA for ade4722

• Add calculation of global_step when resuming trainining from a checkpoint

  tanaysoni

  committed
  7266a95

  Copy full SHA for 7266a95

• Add log_learning_rate param in the state_dict of Trainer

  tanaysoni

  committed
  1a19b7e

  Copy full SHA for 1a19b7e

• Fix resume step when starting to train from a checkpoint

  tanaysoni

  committed
  06b96d1

  Copy full SHA for 06b96d1

• rename and update basic example

  

  tholor

  authored and

  tanaysoni

  committed
  890cdae

  Copy full SHA for 890cdae

• Add additional states in train checkpoint

  tanaysoni

  committed
  30999b3

  Copy full SHA for 30999b3

• Fix Trainer init arguments

  tanaysoni

  committed
  9f11bc3

  Copy full SHA for 9f11bc3

• Set initial epoch and set to 0

  tanaysoni

  committed
  0227a65

  Copy full SHA for 0227a65

Commits on Jan 20, 2020

• Add docstrings

  tanaysoni

  committed
  ead1eb2

  Copy full SHA for ead1eb2

• Update init of Trainer to include the model

  tanaysoni

  committed
  207a92d

  Copy full SHA for 207a92d

Commits on Jan 22, 2020

• Change sigkill to sigterm for train checkpoint

  tanaysoni

  committed
  dc15186

  Copy full SHA for dc15186

• Update train_from_scratch example

  tanaysoni

  committed
  b9b139d

  Copy full SHA for b9b139d

• Type-cast str to Path

  tanaysoni

  committed
  1f037dc

  Copy full SHA for 1f037dc

• Remove duplicate model

  tanaysoni

  committed
  06f9037

  Copy full SHA for 06f9037

• Remove stray comment

  tanaysoni

  committed
  bb39c1e

  Copy full SHA for bb39c1e