"value too large"

[demon9514753]

I using postgreSQL to install UMCdb, when it copy "listitems.csv" and "numericitems.csv", the softwaer prompts error message: "psql:postgres-load-data-csv.sql:16: error: unable to extract files "listitems.csv" status: value too large" and "psql:postgres-load-data-csv.sql:18: error: unable to extract files "numericitems.csv" status: value too large".
What's wrong about it? What can I do to solve these problems?
Thank you !

[patrickthoral]

Since I don't use psql directly: @peiyaoli Could you comment on this? I'm assuming you manually created the database schema and did not use BIGINT?

Alternatively, you could use this notebook from Python: https://github.com/AmsterdamUMC/AmsterdamUMCdb/blob/master/setup-amsterdamumcdb/setup-amsterdamumc.ipynb. That should not return that error.

[demon9514753]

Thank you very much, I’ll try it. 从 Windows 版邮件发送 发件人: Patrick Thoral 发送时间: 2022年1月18日 5:05 收件人: AmsterdamUMC/AmsterdamUMCdb 抄送: demon9514753; Author 主题: Re: [AmsterdamUMC/AmsterdamUMCdb] "value too large" (Issue #54) Since I don't use psql directly: @peiyaoli Could you comment on this? I'm assuming you manually created the database schema and did not use BIGINT? Alternatively, you could use this notebook from Python: https://github.com/AmsterdamUMC/AmsterdamUMCdb/blob/master/setup-amsterdamumcdb/setup-amsterdamumc.ipynb. That should not return that error. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: ***@***.***>

[XiangBu]

I using postgreSQL to install UMCdb, when it copy "listitems.csv" and "numericitems.csv", the softwaer prompts error message: "psql:postgres-load-data-csv.sql:16: error: unable to extract files "listitems.csv" status: value too large" and "psql:postgres-load-data-csv.sql:18: error: unable to extract files "numericitems.csv" status: value too large". What's wrong about it? What can I do to solve these problems? Thank you !

I doubt it might have something to do with the RAM.

[patrickthoral]

@demon9514753 Were you able to fix this?

[wubuyun1987]

I have fixed it. The problem that “unable to extract files status: value too large" is because of a relatively small RAM. But it could be handled through compression by 7Z. Then the two csv files (listitems.csv 2.7G, numericitems.csv 78G) became smaller as 7z files (listitems.7z 0.16G, numericitems.7z 5.2G).
Then revise postgres-load-data-csv.sql as below:
"
-- listitems
\copy listitems FROM PROGRAM '7z e -so listitems.7z' DELIMITER ',' CSV HEADER NULL ''
-- numericitems
\copy numericitems FROM PROGRAM '7z e -so numericitems.7z' DELIMITER ',' CSV HEADER NULL ''
"
I have stalled UMCdb successfully using this method recently.