You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Presently asynchronous plotting/3D rendering is not supported in the part of rllab based on TensorFlow (sandbox.rocky.tf), but it is supported in the rllab code which uses Theano.
This means that when you turn plotting on for a Theano training session, the plot does not block the training process. The TensorFlow implementation runs the rendering loop directly in the training algorithm (rather than a worker), so it blocks. This makes training using TensorFlow much slower than Theano when plotting is turned on (they are about the same without plotting).
TensorFlow's notion of a session makes this tricky. I'm not 100% sure that there is a solution. If you figure out that it is impossible, or requires rewriting large parts of the repository, email me with what you tried and some explanations why.
Current behavior
3D plotting with MuJoCo in TensorFlow is synchronous, blocks the training process.
Desired behavior
3D plotting with MuJoCo in TensorFlow is asynchronous and does not block the training process (just as with Theano)
Fork this repository into you own Github account, and implement this feature in its own branch, based on the master branch. When you are done and would like a code review, send me an email with a link to your feature branch. DO NOT SUBMIT A PULL REQUEST TO THIS REPOSITORY.
Consider this a professional software engineering task, and provide a high-quality solution which does not break existing users, minimizes change, and is stable. Tests are welcome where appropriate. Please always use PEP8 style in your code, and format it using YAPF (with the PEP8 setting).
Notes
Testing the software requires a freely-available student license for MuJoCo available here. It takes a couple days to get approved, so do it early.
You can find examples of how to launch rllab in examples and sandbox/rocky/tf/launchers. Note that everything must run using the run_experiment_lite wrapper with the parameter n_parallel greater than 1 (this triggers multiprocess operation).
rllab currently has two parallel implementation of the neural network portions of the library. The original is written in Theano and is found in rllab/. The tree sandbox/rocky/tf re-implements classes from the original tree using TensorFlow, and is backwards-compatible with the Theano tree. We are working towards using only one NN library soon, but for now your implementation needs to work in both trees.
rllab is an upstream dependency to many projects, so it is important we do not break the existing APIs. Adding to APIs is fine as long as there is a good reason.
The text was updated successfully, but these errors were encountered:
Task
Presently asynchronous plotting/3D rendering is not supported in the part of rllab based on TensorFlow (sandbox.rocky.tf), but it is supported in the rllab code which uses Theano.
This means that when you turn plotting on for a Theano training session, the plot does not block the training process. The TensorFlow implementation runs the rendering loop directly in the training algorithm (rather than a worker), so it blocks. This makes training using TensorFlow much slower than Theano when plotting is turned on (they are about the same without plotting).
TensorFlow's notion of a session makes this tricky. I'm not 100% sure that there is a solution. If you figure out that it is impossible, or requires rewriting large parts of the repository, email me with what you tried and some explanations why.
Current behavior
3D plotting with MuJoCo in TensorFlow is synchronous, blocks the training process.
Desired behavior
3D plotting with MuJoCo in TensorFlow is asynchronous and does not block the training process (just as with Theano)
Pointers to relevant source
To run a basic training algorithm which has 3D plots using Theano (you will need to pass plot=True to TRPO):
https://github.com/ryanjulian/rllab/blob/master/examples/trpo_swimmer.py
To run a basic training algorithm with TensorFlow:
https://github.com/ryanjulian/rllab/blob/master/sandbox/rocky/tf/launchers/trpo_cartpole.py
Note: this does not use a MuJoCo environment/3D plotting, but you just need to change the environment to SwimmerEnv() and it will.
Asynchronous plotter API used by the Theano code
https://github.com/ryanjulian/rllab/tree/master/rllab/plotter
Theano and TF implementations of BatchPolopt (calls the plotter)
Theano: https://github.com/ryanjulian/rllab/blob/master/rllab/algos/batch_polopt.py
TF: https://github.com/ryanjulian/rllab/blob/master/sandbox/rocky/tf/algos/batch_polopt.py
rllab implementation of
Serializable
(used to transfer objects across multiprocessing queues): https://github.com/ryanjulian/rllab/blob/master/rllab/core/serializable.pySubmission instructions
Fork this repository into you own Github account, and implement this feature in its own branch, based on the
master
branch. When you are done and would like a code review, send me an email with a link to your feature branch. DO NOT SUBMIT A PULL REQUEST TO THIS REPOSITORY.Consider this a professional software engineering task, and provide a high-quality solution which does not break existing users, minimizes change, and is stable. Tests are welcome where appropriate. Please always use PEP8 style in your code, and format it using YAPF (with the PEP8 setting).
Notes
run_experiment_lite
wrapper with the parametern_parallel
greater than 1 (this triggers multiprocess operation).rllab/
. The treesandbox/rocky/tf
re-implements classes from the original tree using TensorFlow, and is backwards-compatible with the Theano tree. We are working towards using only one NN library soon, but for now your implementation needs to work in both trees.The text was updated successfully, but these errors were encountered: