customization of learning rate

[tkng]

Currently, Blueoil doesn't support customization of learning rate. It would be very nice if we can customize learning rate, since it is a very important hyperparameter.

For example, internally, we've used some learning rate scheduling, like tf.train.piecewise_constant, tf.train.polynomial_decay.

[tkng]

We discussed about concrete specification.

• a user should be able to choose from the following choices
  1. 2 times decay
  2. 3 times deday
  3. warm up + 3 times decay
  4. constant learning rate (current implementation, but this one should be possible to set initial value)

If a use choose option 1 and total epochs is 100, then learning rate should be scaled to 1/10, at 50 epochs and 99 epochs.

If a use choose option 2 and total epochs is 100, then learning rate should be scaled to 1/10, at 33, 66 epochs and 99 epochs.

Since accuracy is high immediately after we decreased the learning rate, I'd like to keep 1 decay until (num_epochs-1) epoch. Other decay timing is just equal divide.

Estimated effort for this issue would be 1 or 2 FTE days.

where should be modified?

• we need to modify blueoil_init.py, to make user possible to choose from these 4 options above.
• we need to modify generate_lmnet_config.py, to generate configuration python script for lmnet.

Sample of generated config lines for lmnet:

blueoil/lmnet/configs/core/classification/lmnet_cifar10.py

Line 76 in 614a18e

NETWORK.LEARNING_RATE_KWARGS = {

[lm-jira]

I wonder if this issue is assigned to anyone or not.
If not, can I try to solving it?

[tkng]

Thanks, I appreciate your initiative. However, I already askes to @lm-konda to solve the ticket and it's already sent as #18.

I'll propose some new issues within this week, so that you can try to new interesting tasks, please wait a moment.

[lm-konda]

I want to close this issue.
For details see the following.
#18