Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Fix memory leaks in Gluon #18328

Merged
merged 2 commits into from
May 15, 2020
Merged

Fix memory leaks in Gluon #18328

merged 2 commits into from
May 15, 2020

Conversation

leezu
Copy link
Contributor

@leezu leezu commented May 15, 2020

Description

Previously the _BlockScope keeps references to the parameter ndarrays, preventing memory from being freed if a Block is not used anymore. Among other problems, this causes memory usage to increase constantly in unittests (due to testing different blocks and disposing them at the end of the test) until the garbage collector kicks in (which can be too late and the system can run OOM as the parameter arrays can be large).

@leezu leezu requested a review from szha as a code owner May 15, 2020 01:05
@mxnet-bot
Copy link

Hey @leezu , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [miscellaneous, website, centos-gpu, centos-cpu, sanity, unix-cpu, unix-gpu, windows-gpu, edge, windows-cpu, clang]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@leezu leezu mentioned this pull request May 15, 2020
Closed
@leezu
Copy link
Contributor Author

leezu commented May 15, 2020

@mxnet-bot run ci [centos-gpu, windows-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [centos-gpu, windows-gpu]

@sxjscience
Copy link
Member

@Jerryzcn I think this is also related to your previous benchmark.

@leezu leezu merged commit 3e676fc into apache:master May 15, 2020
@leezu leezu deleted the fixblockmemoryleaks branch May 15, 2020 17:01
@leezu
Copy link
Contributor Author

leezu commented May 15, 2020

@ciyongch should we backport this to 1.7?

@ciyongch
Copy link
Contributor

Hi @leezu , if the issue also appears in 1.7 then please help to backport to 1.7 and 1.x branches and tag me to the new PR, thanks!

@leezu
Copy link
Contributor Author

leezu commented May 16, 2020

This issue is present in all versions of Gluon. OK, let's backport the fix.

leezu added a commit to leezu/mxnet that referenced this pull request May 18, 2020
@leezu
Fix memory leaks in Gluon (apache#18328)
c89ac9b 
Fix leak of ndarray objects in the frontend due to reference cycle.

Backport of 3e676fc
leezu added a commit to leezu/mxnet that referenced this pull request May 18, 2020
@leezu
Fix memory leaks in Gluon (apache#18328)
2d74889 
Fix leak of ndarray objects in the frontend due to reference cycle.

Backport of 3e676fc
This was referenced May 18, 2020
Merged
Merged
@leezu
Copy link
Contributor Author

leezu commented May 18, 2020

@ciyongch I created the backport PRs

@ciyongch
Copy link
Contributor

Thanks @leezu to help backport the PR.

TaoLv pushed a commit that referenced this pull request May 19, 2020
@leezu
Fix memory leaks in Gluon (#18328) (#18358)
c4d9270 
Fix leak of ndarray objects in the frontend due to reference cycle.

Backport of 3e676fc
@apeforest
Copy link
Contributor

apeforest commented May 19, 2020 •
edited
Loading

@ChaiBapchya @access2rohit This may have also fixed our out-of-memory issue in large tensor nightly test when running them in sequence.

@zhreshold zhreshold mentioned this pull request May 20, 2020
Closed
@leezu leezu mentioned this pull request May 20, 2020
Merged
TaoLv pushed a commit that referenced this pull request May 27, 2020
@leezu
Fix memory leaks in Gluon (#18328) (#18359)
b523527 
Fix leak of ndarray objects in the frontend due to reference cycle.

Backport of 3e676fc
AntiZpvoh pushed a commit to AntiZpvoh/incubator-mxnet that referenced this pull request Jul 6, 2020
@leezu @AntiZpvoh
Fix memory leaks in Gluon (apache#18328)
f3d04a0 
Fix leak of ndarray objects in the frontend due to reference cycle.
rondogency added a commit to rondogency/incubator-mxnet that referenced this pull request Jul 10, 2020
@rondogency
Revert "Fix memory leaks in Gluon (apache#18328) (apache#18358)"
bac103c 
This reverts commit c4d9270.
szha pushed a commit that referenced this pull request Jul 12, 2020
@rondogency
Revert "Fix memory leaks in Gluon (#18328) (#18358)" (#18692)
c4c7b11 
This reverts commit c4d9270.
rondogency added a commit to rondogency/incubator-mxnet that referenced this pull request Jul 13, 2020
@rondogency
Revert "Fix memory leaks in Gluon (apache#18328) (apache#18359)"
581df09 
This reverts commit b523527.
chinakook pushed a commit to chinakook/mxnet that referenced this pull request Jul 24, 2020
@leezu @chinakook
Fix memory leaks in Gluon (apache#18328)
2cc44b4 
Fix leak of ndarray objects in the frontend due to reference cycle.
ChaiBapchya pushed a commit to ChaiBapchya/mxnet that referenced this pull request Aug 15, 2020
@leezu @ChaiBapchya
Fix memory leaks in Gluon (apache#18328) (apache#18359)
ee8c3c8 
Fix leak of ndarray objects in the frontend due to reference cycle.

Backport of 3e676fc
leezu added a commit that referenced this pull request Sep 18, 2020
@leezu
Revert "Fix memory leaks in Gluon (#18328) (#18359)"
c264f37 
This reverts commit b523527.
samskalicky pushed a commit that referenced this pull request Sep 19, 2020
@leezu
Revert "Fix memory leaks in Gluon (#18328) (#18359)" (#19181)
0496690 
This reverts commit b523527.
@szha szha mentioned this pull request Sep 20, 2020
Open
chinakook added a commit to chinakook/gluon-cv that referenced this pull request Nov 17, 2020
@chinakook
Fix yolo to support a memory leak fix
3f35aa6 
After this commit apache/mxnet#18328 , some memory leak were fixed.
Whitout this commit faster rcnn traning cannot be successfully closed.
These commits can be commit again after this yolo training fix.
apache/mxnet#18692
apache/mxnet@0496690
@chinakook chinakook mentioned this pull request Nov 17, 2020
Merged
zhreshold added a commit to dmlc/gluon-cv that referenced this pull request Nov 24, 2020
@chinakook @zhreshold
Fix all generator to support a memory leak fix (#1528)
8d5bdbb 
* Fix yolo to support a memory leak fix

After this commit apache/mxnet#18328 , some memory leak were fixed.
Whitout this commit faster rcnn traning cannot be successfully closed.
These commits can be commit again after this yolo training fix.
apache/mxnet#18692
apache/mxnet@0496690

* fix all generator error in windows when training with multiprocessing

* add pylint disable not-callable

* Fix pylint

* Fix pylint

Co-authored-by: Joshua Z. Zhang <cheungchih@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@sxjscience sxjscience sxjscience approved these changes

@szha szha Awaiting requested review from szha szha is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@leezu @mxnet-bot @sxjscience @ciyongch @apeforest