Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory while exporting data #363

Closed
claeis opened this issue Sep 3, 2020 · 2 comments
Closed

Out of memory while exporting data #363

claeis opened this issue Sep 3, 2020 · 2 comments
Milestone

Comments

@claeis
Copy link
Owner

claeis commented Sep 3, 2020

with --diableValidation the export works

with validation the export fails:

java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at ch.interlis.iox_j.validator.Validator.getScopedName(Validator.java:660)
at ch.interlis.iox_j.validator.Validator.validateAttrValue(Validator.java:3319)
at ch.interlis.iox_j.validator.Validator.validateObject(Validator.java:3052)
at ch.interlis.iox_j.validator.Validator.validate(Validator.java:351)
at ch.ehi.ili2db.toxtf.TransferToXtf.dumpObjHelper(TransferToXtf.java:1133)
at ch.ehi.ili2db.toxtf.TransferToXtf.dumpObject(TransferToXtf.java:1070)
at ch.ehi.ili2db.toxtf.TransferToXtf.doBasket(TransferToXtf.java:554)
at ch.ehi.ili2db.toxtf.TransferToXtf.doit(TransferToXtf.java:287)
at ch.ehi.ili2db.base.Ili2db.transferToXtf(Ili2db.java:2300)
at ch.ehi.ili2db.base.Ili2db.runExport(Ili2db.java:1720)
at ch.ehi.ili2db.base.Ili2db.run(Ili2db.java:225)
at ch.ehi.ili2db.AbstractMain.domain(AbstractMain.java:555)
at ch.ehi.ili2pg.PgMain.main(PgMain.java:70)
@claeis
Copy link
Owner Author

claeis commented Sep 3, 2020

There is no memory leak.

If there is enough heap space (5GB), the export with validation works.

I found two places (one in the validation code and one in the export code), where I can replace the heap based implementation with a file based one.

withValidation

@claeis
Copy link
Owner Author

claeis commented Sep 3, 2020

After the improvements:
afterImprovement

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant