Skip to content

Commit

Permalink
Stop tensorflow from eating all GPU memory (#473)
Browse files Browse the repository at this point in the history
By default, tensorflow will consume all available GPU memory:
https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth

Say you are running on KERAS_BACKEND=jax, and jax and tf have both
been configured with GPU suport:
- keras_core will always import and initialize tensorflow
- tensorflow will use all available GPU memory
- jax will attempt any GPU allocation and immediately fail

Note this does not happen in colab because colab automatically exports
the environment variable:
TF_FORCE_GPU_ALLOW_GROWTH=true

We can do the same from keras-core.
  • Loading branch information
mattdangerw authored Jul 16, 2023
1 parent 5b27c15 commit 02c7f69
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions keras_core/backend/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,13 @@ def standardize_data_format(data_format):
_BACKEND = _backend


if _BACKEND != "tensorflow":
# If we are not running on the tensorflow backend, we should stop tensorflow
# from using all available GPU memory. See
# https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth
os.environ["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"


@keras_core_export(
[
"keras_core.config.backend",
Expand Down

0 comments on commit 02c7f69

Please sign in to comment.