Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: higher step precision for images etc. #1012

Open
agrinh opened this issue Mar 1, 2018 · 3 comments
Open

Feature request: higher step precision for images etc. #1012

agrinh opened this issue Mar 1, 2018 · 3 comments

Comments

@agrinh
Copy link

agrinh commented Mar 1, 2018

Images in Tensorboard are incredibly useful for us when training generative models as they allow an intuitive feel for what different scalar metrics mean. Currently the number of selectable steps are limited to 10 making it difficult to see transients and instabilities (common in e.g. GANs).

It would be great if we could get a method for selecting the step for an image with higher precision. E.g. a manual input for the step would be very helpful, especially if coupled with the ability to scroll around in its immediate area.

Ideally the suggested global step slider in #469 should support this higher precision step selection as well.

@chihuahua
Copy link
Member

chihuahua commented Mar 1, 2018

The reason TensorBoard samples events from disk and serves that sampled subset to the frontend (of say 10 steps per image tag) is because TensorBoard serves data stored in memory (python data structures), and storing entire log directories (all events) would be too large for memory.

Also, reading from disk to find events at particular steps would be too slow for many cases, especially within distributed file systems. I tried implementing a plugin a while back that reads from disk. Reading the event at the 10,000th step took I believe 30s from my local directory. Reading events at millions of steps was basically impossible.

We are developing a new backend based on SQL. This lets the user write to for instance SQLite databases, and TensorBoard would (relatively quickly) read from them. However, this effort will launch in probably a few months, and unfortunately, we might end up sampling in the end to conserve database space.

I think this issue warrants some more thought. Many have broached it, but we lack a robust solution.

@nfelt
Copy link
Contributor

nfelt commented Jun 15, 2018

FYI, PR #1138 added a --samples_per_plugin flag that can be used to set the number of samples retained on a per-plugin basis. So e.g. --samples_per_plugin=images=100 should set the image dashboard to retain 100 images per series.

Leaving this open to track the request to be able to manually input a step and scroll around near it (without necessarily having to keep 1000s of images in memory), which will probably need to wait until the SQL backend support is ready.

@davinnovation
Copy link

waiting for SQL backend support!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants