A3C distributed modes

[kengz]

A3C distributed modes

• rework global_nets init to support different Hogwild distributed modes:
  • synced (CPU-only): periodically push grads and pull params from global network to local network
  • shared: global networks are always shared, so global net override local net. This works with GPU and is extremely fast, and seems to be more sample efficient.
• allow for global optimizer for distributed modes. add GlobalAdam and GlobalRMSProp optims
• add global net methods for init, set algorithm nets, and push grads in net_utils
• update specs accordingly
• divide max_tick if distributed to end up with the same max_tick in total
• add spec compatibility check

Network interface

• simplify global net init at algorithm level
• rewrite network interface method from training_step(self, loss, optim, lr_scheduler, lr_clock=None) to train_step(self, loss, optim, lr_scheduler, lr_clock=None, global_net=None)
• standardize network naming convention: must end with net. Optimizer and lr_scheduler names are net_name.replace('net', 'optim'), net_name.replace('net', 'lr_scheduler') respectively