memory leak causes data corruption that check:construction not working properly #170

Yrp · 2020-10-22T08:51:24Z

Describe the bug

rosetta-cli may have a memory leak.
rosetta-cli exits after an exception, causing data corruption and the inability to continue running check:data and check:construction.

oom log

43040703.484867] Out of memory: Kill process 30248 (rosetta-cli) score 719 or sacrifice child
[43040703.486183] Killed process 30248 (rosetta-cli) total-vm:14133908kB, anon-rss:11682132kB, file-rss:124kB, shmem-rss:0kB
[43040703.488414] AliYunDun invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[43040703.489785] AliYunDun cpuset=/ mems_allowed=0
[43040703.490852] CPU: 0 PID: 19469 Comm: AliYunDun Tainted: G OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1
[43040703.492451] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS ab23bb1 04/01/2014
[43040703.493831] ffff88003620dee0 00000000b6cb212d ffff8803b99c38f8 ffffffff816a3db1
[43040703.495269] ffff8803b99c3988 ffffffff8169f1a6 ffffffff810e939c ffff8804213719f0
[43040703.496701] ffff880421371a08 0000000000000206 0000000000000206 ffff8803b99c3978
[43040703.498165] Call Trace:
[43040703.499175] [] dump_stack+0x19/0x1b
[43040703.500416] [] dump_header+0x90/0x229
[43040703.501667] [] ? ktime_get_ts64+0x4c/0xf0
[43040703.502944] [] ? delayacct_end+0x8f/0xb0
[43040703.504241] [] ? virtballoon_oom_notify+0x2a/0x70 [virtio_balloon]
[43040703.505720] [] oom_kill_process+0x254/0x3d0
[43040703.507035] [] ? oom_unkillable_task+0xcd/0x120
[43040703.508391] [] ? find_lock_task_mm+0x56/0xc0
[43040703.509721] [] out_of_memory+0x4b6/0x4f0
[43040703.511026] [] __alloc_pages_slowpath+0x5d6/0x724
[43040703.512401] [] __alloc_pages_nodemask+0x405/0x420
[43040703.513764] [] alloc_pages_current+0x98/0x110
[43040703.515107] [] __page_cache_alloc+0x97/0xb0
[43040703.516415] [] filemap_fault+0x170/0x410
[43040703.517721] [] ext4_filemap_fault+0x36/0x50 [ext4]
[43040703.519113] [] __do_fault+0x52/0xe0
[43040703.520494] [] do_read_fault.isra.44+0x4b/0x130
[43040703.521860] [] handle_mm_fault+0x691/0xfa0
[43040703.523205] [] ? __schedule+0x39d/0x8b0
[43040703.524562] [] __do_page_fault+0x154/0x450
[43040703.525928] [] trace_do_page_fault+0x56/0x150
[43040703.527385] [] do_async_page_fault+0x1a/0xd0
[43040703.528765] [] async_page_fault+0x28/0x30
[43040703.530186] Mem-Info:
[43040703.531328] active_anon:3942805 inactive_anon:74 isolated_anon:0
active_file:423 inactive_file:949 isolated_file:0
unevictable:0 dirty:19 writeback:11 unstable:0
slab_reclaimable:19893 slab_unreclaimable:9571
mapped:361 shmem:178 pagetables:10731 bounce:0
free:33895 free_pcp:7 free_cma:0

check:construction error log

badger 2020/10/22 11:44:39 INFO: Storing value log head: {Fid:1889 Len:35 Offset:17139043}
Error: unable to process blocks: unable to process block: unable to broadcast pending transactions: unable to handle broadcast failure: broadcast failed for transaction 0157ffe297e480a7337685c8455440071efa560874f008ea187ba7bf9c4c46fd with intent [
{
"operation_identifier": {
"index": 0
},
"type": "TransferContract",
"status": "",
"account": {
"address": "TLy2Gi9BvDy2DL1bKqXAPBpJqzD3UfY5Xw"
},
"amount": {
"value": "-163283103798002",
"currency": {
"symbol": "TRX",
"decimals": 6
}
}
},
{
"operation_identifier": {
"index": 1
},
"type": "TransferContract",
"status": "",
"account": {
"address": "THpBsNsjyhFJc9f7SduLwzRGPLMjWoHr4k"
},
"amount": {
"value": "163283103798002",
"currency": {
"symbol": "TRX",
"decimals": 6
}
}
}
]: unable to add block to storage 0000000000035359945ac819aceebcfcf521e9049941f829708dc3626305b285:217945: >unable to sync to 217952
Usage:
rosetta-cli check:construction [flags]

Flags:
-h, --help help for check:construction

Global Flags:
--configuration-file string Configuration file that provides connection and test settings.
If you would like to generate a starter configuration file (populated
with the defaults), run rosetta-cli configuration:create.
                               Any fields not populated in the configuration file will be populated with
                               default values.
 --cpu-profile string          Save the pprof cpu profile in the specified file
 --mem-profile string          Save the pprof mem profile in the specified file

Command Failed: unable to process blocks: unable to process block: unable to broadcast pending transactions: unable to handle broadcast failure: broadcast failed for transaction 0157ffe297e480a7337685c8455440071efa560874f008ea187ba7bf9c4c46fd with intent [
{
"operation_identifier": {
"index": 0
},
"type": "TransferContract",
"status": "",
"account": {
"address": "TLy2Gi9BvDy2DL1bKqXAPBpJqzD3UfY5Xw"
},
"amount": {
"value": "-163283103798002",
"currency": {
"symbol": "TRX",
"decimals": 6
}
}
},
{
"operation_identifier": {
"index": 1
},
"type": "TransferContract",
"status": "",
"account": {
"address": "THpBsNsjyhFJc9f7SduLwzRGPLMjWoHr4k"
},
"amount": {
"value": "163283103798002",
"currency": {
"symbol": "TRX",
"decimals": 6
}
}
}
]: unable to add block to storage 0000000000035359945ac819aceebcfcf521e9049941f829708dc3626305b285:217945: unable to sync to 217952

The text was updated successfully, but these errors were encountered:

patrick-ogrady · 2020-10-22T15:33:15Z

Thanks for posting this, @Yrp !

Would you mind including:

OS
RAM on your computer
All the logs printed by the rosetta-cli that start with [MEMORY] (this will help immensely when debugging):

Yrp · 2020-10-22T16:07:58Z

Thanks for posting this, @Yrp !

Would you mind including:

OS

RAM on your computer

All the logs printed by the rosetta-cli that start with [MEMORY] (this will help immensely when debugging):

OS
CentOS 7.4 64位
RAM on your computer
16G, there are two other JVM processes running at the time of the problem and this may be caused by the system's inability to allocate memory.
All the logs printed by the rosetta-cli that start with [MEMORY] (this will help immensely when debugging):

All the logs are here:
1015.log

patrick-ogrady · 2020-10-22T16:12:58Z

Would you mind uploading your configuration file too? Interesting memory blow up here:

[STATS] Transactions Confirmed: 507123 (Created: 507153, In Progress: 30, Stale: 0, Failed: 0) Addresses Created: 100
2020/10/22 11:21:55 waiting for available jobs...
[MEMORY] Heap: 2536.957611MB Stack: 1.406250MB System: 6268.450745MB GCs: 18
2020/10/22 11:22:05 waiting for available jobs...
[MEMORY] Heap: 3461.377029MB Stack: 1.437500MB System: 6268.450745MB GCs: 21
2020/10/22 11:22:15 waiting for available jobs...
[STATS] Transactions Confirmed: 507153 (Created: 507183, In Progress: 30, Stale: 0, Failed: 0) Addresses Created: 100
[MEMORY] Heap: 4475.607384MB Stack: 1.406250MB System: 6268.450745MB GCs: 23
2020/10/22 11:22:25 waiting for available jobs...
[MEMORY] Heap: 2674.809647MB Stack: 1.500000MB System: 6471.022057MB GCs: 26
2020/10/22 11:22:35 waiting for available jobs...
[MEMORY] Heap: 4660.202507MB Stack: 1.437500MB System: 6471.022057MB GCs: 28
2020/10/22 11:22:45 waiting for available jobs...
[MEMORY] Heap: 2526.077148MB Stack: 1.468750MB System: 6537.091400MB GCs: 31
2020/10/22 11:22:55 waiting for available jobs...
[MEMORY] Heap: 4056.239601MB Stack: 1.718750MB System: 6537.091400MB GCs: 34
2020/10/22 11:23:05 waiting for available jobs...
[MEMORY] Heap: 3413.235031MB Stack: 1.312500MB System: 6537.091400MB GCs: 37
2020/10/22 11:23:15 waiting for available jobs...
[MEMORY] Heap: 2868.626564MB Stack: 1.406250MB System: 6537.778900MB GCs: 39
2020/10/22 11:23:25 waiting for available jobs...
[MEMORY] Heap: 3396.237381MB Stack: 1.437500MB System: 6537.778900MB GCs: 42
2020/10/22 11:23:35 waiting for available jobs...
[MEMORY] Heap: 2114.777794MB Stack: 1.343750MB System: 6537.778900MB GCs: 44
2020/10/22 11:23:45 waiting for available jobs...
[MEMORY] Heap: 3444.396614MB Stack: 1.343750MB System: 6537.778900MB GCs: 49
2020/10/22 11:23:55 waiting for available jobs...
[MEMORY] Heap: 6541.630119MB Stack: 1.437500MB System: 7347.814148MB GCs: 51
2020/10/22 11:24:05 waiting for available jobs...
[MEMORY] Heap: 4583.915512MB Stack: 1.468750MB System: 10117.016533MB GCs: 53
2020/10/22 11:24:15 waiting for available jobs...
[MEMORY] Heap: 3439.226173MB Stack: 1.406250MB System: 10120.266533MB GCs: 55
2020/10/22 11:24:25 waiting for available jobs...
[MEMORY] Heap: 5054.867516MB Stack: 1.406250MB System: 10120.266533MB GCs: 57
2020/10/22 11:24:35 waiting for available jobs...
[MEMORY] Heap: 2166.962669MB Stack: 1.375000MB System: 10120.266533MB GCs: 60
2020/10/22 11:24:45 waiting for available jobs...
[MEMORY] Heap: 2893.007820MB Stack: 1.375000MB System: 10120.266533MB GCs: 63
2020/10/22 11:24:55 waiting for available jobs...
[MEMORY] Heap: 1779.845428MB Stack: 1.468750MB System: 10120.266533MB GCs: 65
2020/10/22 11:25:05 waiting for available jobs...
[MEMORY] Heap: 2523.953278MB Stack: 1.437500MB System: 10120.266533MB GCs: 69
2020/10/22 11:25:15 waiting for available jobs...
[MEMORY] Heap: 4306.934853MB Stack: 1.468750MB System: 10120.266533MB GCs: 71
2020/10/22 11:25:25 waiting for available jobs...
[MEMORY] Heap: 1696.654839MB Stack: 1.281250MB System: 10120.266533MB GCs: 74
2020/10/22 11:25:35 waiting for available jobs...
[MEMORY] Heap: 4576.543541MB Stack: 1.312500MB System: 10120.266533MB GCs: 78
2020/10/22 11:25:46 waiting for available jobs...
[MEMORY] Heap: 5791.045135MB Stack: 1.437500MB System: 10120.266533MB GCs: 80
2020/10/22 11:25:56 waiting for available jobs...
[MEMORY] Heap: 5695.983742MB Stack: 1.437500MB System: 10120.266533MB GCs: 82
2020/10/22 11:26:06 waiting for available jobs...
[MEMORY] Heap: 5056.272171MB Stack: 1.437500MB System: 10120.266533MB GCs: 85
2020/10/22 11:26:16 waiting for available jobs...
[MEMORY] Heap: 8066.467598MB Stack: 1.718750MB System: 10120.266533MB GCs: 87
2020/10/22 11:26:26 waiting for available jobs...
[MEMORY] Heap: 6008.201591MB Stack: 1.406250MB System: 10255.397408MB GCs: 89
2020/10/22 11:26:36 waiting for available jobs...
[MEMORY] Heap: 2810.633598MB Stack: 1.343750MB System: 10256.397408MB GCs: 92
2020/10/22 11:26:46 waiting for available jobs...
[MEMORY] Heap: 3496.660667MB Stack: 1.343750MB System: 10256.397408MB GCs: 94
2020/10/22 11:26:56 waiting for available jobs...
[MEMORY] Heap: 2821.861259MB Stack: 1.437500MB System: 10256.397408MB GCs: 96
2020/10/22 11:27:06 waiting for available jobs...
[MEMORY] Heap: 4788.906250MB Stack: 1.500000MB System: 10256.397408MB GCs: 98
2020/10/22 11:27:16 waiting for available jobs...
[MEMORY] Heap: 5414.800255MB Stack: 1.437500MB System: 10256.397408MB GCs: 100
2020/10/22 11:27:26 waiting for available jobs...
[MEMORY] Heap: 3017.419968MB Stack: 1.406250MB System: 10256.397408MB GCs: 103
2020/10/22 11:27:36 waiting for available jobs...
[MEMORY] Heap: 4395.506439MB Stack: 1.437500MB System: 10256.397408MB GCs: 105
2020/10/22 11:27:46 waiting for available jobs...
[MEMORY] Heap: 2450.604675MB Stack: 1.406250MB System: 10256.397408MB GCs: 107
2020/10/22 11:27:56 waiting for available jobs...
[MEMORY] Heap: 3214.833008MB Stack: 1.468750MB System: 10256.397408MB GCs: 110
2020/10/22 11:28:06 waiting for available jobs...
[MEMORY] Heap: 4024.512939MB Stack: 1.437500MB System: 10256.397408MB GCs: 112
2020/10/22 11:28:16 waiting for available jobs...
[MEMORY] Heap: 3399.471794MB Stack: 1.656250MB System: 10256.397408MB GCs: 114
2020/10/22 11:28:26 waiting for available jobs...
[MEMORY] Heap: 3708.812431MB Stack: 1.437500MB System: 10256.397408MB GCs: 117
2020/10/22 11:28:36 waiting for available jobs...
[MEMORY] Heap: 2176.987221MB Stack: 1.500000MB System: 10256.397408MB GCs: 119
2020/10/22 11:28:46 waiting for available jobs...
[MEMORY] Heap: 3664.523590MB Stack: 1.406250MB System: 10256.397408MB GCs: 122
2020/10/22 11:28:56 waiting for available jobs...
[MEMORY] Heap: 5326.265305MB Stack: 1.750000MB System: 10256.397408MB GCs: 124
2020/10/22 11:29:06 waiting for available jobs...
[MEMORY] Heap: 2485.713585MB Stack: 1.437500MB System: 10256.397408MB GCs: 127
2020/10/22 11:29:16 waiting for available jobs...
[MEMORY] Heap: 3094.558327MB Stack: 1.437500MB System: 10256.397408MB GCs: 130
2020/10/22 11:29:26 waiting for available jobs...
[MEMORY] Heap: 6690.759315MB Stack: 1.468750MB System: 10256.397408MB GCs: 132
2020/10/22 11:29:36 waiting for available jobs...
[MEMORY] Heap: 4512.033539MB Stack: 1.281250MB System: 10256.397408MB GCs: 136
2020/10/22 11:29:46 waiting for available jobs...
[MEMORY] Heap: 3162.579269MB Stack: 1.468750MB System: 10256.397408MB GCs: 138
2020/10/22 11:29:56 waiting for available jobs...
[MEMORY] Heap: 3594.258316MB Stack: 1.218750MB System: 10256.397408MB GCs: 140
2020/10/22 11:30:06 waiting for available jobs...
[MEMORY] Heap: 3007.972343MB Stack: 1.500000MB System: 10256.397408MB GCs: 143
2020/10/22 11:30:16 waiting for available jobs...
[MEMORY] Heap: 4723.361771MB Stack: 1.468750MB System: 10256.397408MB GCs: 145
2020/10/22 11:30:26 waiting for available jobs...
[MEMORY] Heap: 1885.506096MB Stack: 1.312500MB System: 10256.397408MB GCs: 148
2020/10/22 11:30:36 waiting for available jobs...
[MEMORY] Heap: 2652.148392MB Stack: 1.250000MB System: 10256.397408MB GCs: 151
2020/10/22 11:30:46 waiting for available jobs...
[MEMORY] Heap: 3597.987869MB Stack: 1.375000MB System: 10256.397408MB GCs: 153
2020/10/22 11:30:56 waiting for available jobs...
[MEMORY] Heap: 2880.349297MB Stack: 1.437500MB System: 10256.397408MB GCs: 155
2020/10/22 11:31:06 waiting for available jobs...
[MEMORY] Heap: 2957.822052MB Stack: 1.343750MB System: 10256.397408MB GCs: 158
2020/10/22 11:31:16 waiting for available jobs...
[MEMORY] Heap: 5900.992241MB Stack: 1.437500MB System: 10256.397408MB GCs: 159
2020/10/22 11:31:26 waiting for available jobs...
[MEMORY] Heap: 3727.712311MB Stack: 1.156250MB System: 10256.397408MB GCs: 163
2020/10/22 11:31:36 waiting for available jobs...
[MEMORY] Heap: 3354.384605MB Stack: 1.437500MB System: 10256.397408MB GCs: 165
2020/10/22 11:31:46 waiting for available jobs...
[MEMORY] Heap: 1886.527031MB Stack: 1.312500MB System: 10256.397408MB GCs: 168
2020/10/22 11:31:56 waiting for available jobs...
[MEMORY] Heap: 4322.060555MB Stack: 1.750000MB System: 10256.397408MB GCs: 170
2020/10/22 11:32:06 waiting for available jobs...
[MEMORY] Heap: 5631.936302MB Stack: 1.500000MB System: 10256.397408MB GCs: 172
2020/10/22 11:32:16 waiting for available jobs...
[MEMORY] Heap: 3234.222855MB Stack: 1.406250MB System: 10256.397408MB GCs: 176
2020/10/22 11:32:26 waiting for available jobs...
[MEMORY] Heap: 3267.672653MB Stack: 1.437500MB System: 10256.397408MB GCs: 179
2020/10/22 11:32:36 waiting for available jobs...
[MEMORY] Heap: 3053.515923MB Stack: 1.562500MB System: 10256.397408MB GCs: 182
2020/10/22 11:32:46 waiting for available jobs...
[MEMORY] Heap: 5360.563820MB Stack: 1.468750MB System: 10256.397408MB GCs: 185
2020/10/22 11:32:56 waiting for available jobs...
[MEMORY] Heap: 6802.360344MB Stack: 1.468750MB System: 10256.397408MB GCs: 187
2020/10/22 11:33:06 waiting for available jobs...
[MEMORY] Heap: 2743.813820MB Stack: 1.718750MB System: 10256.397408MB GCs: 189
2020/10/22 11:33:16 waiting for available jobs...
[MEMORY] Heap: 4155.927589MB Stack: 1.437500MB System: 10256.397408MB GCs: 192
2020/10/22 11:33:26 waiting for available jobs...
[MEMORY] Heap: 5571.764084MB Stack: 1.718750MB System: 10256.397408MB GCs: 194
2020/10/22 11:33:36 waiting for available jobs...
[MEMORY] Heap: 2035.469826MB Stack: 1.312500MB System: 10256.397408MB GCs: 197
2020/10/22 11:33:46 waiting for available jobs...
[MEMORY] Heap: 3829.794853MB Stack: 1.437500MB System: 10256.397408MB GCs: 199
2020/10/22 11:33:56 waiting for available jobs...
[MEMORY] Heap: 4494.318329MB Stack: 1.437500MB System: 10256.397408MB GCs: 201
2020/10/22 11:34:06 waiting for available jobs...
[MEMORY] Heap: 2371.550316MB Stack: 1.343750MB System: 10256.397408MB GCs: 204
2020/10/22 11:34:16 waiting for available jobs...
[MEMORY] Heap: 4028.542221MB Stack: 1.375000MB System: 10256.397408MB GCs: 206
2020/10/22 11:34:26 waiting for available jobs...
[MEMORY] Heap: 2413.688263MB Stack: 1.437500MB System: 10256.397408MB GCs: 208
2020/10/22 11:34:36 waiting for available jobs...
[MEMORY] Heap: 3751.008904MB Stack: 1.406250MB System: 10256.397408MB GCs: 211
2020/10/22 11:34:46 waiting for available jobs...
[MEMORY] Heap: 5368.612251MB Stack: 1.437500MB System: 10256.397408MB GCs: 214
2020/10/22 11:34:56 waiting for available jobs...
[MEMORY] Heap: 4642.024857MB Stack: 1.406250MB System: 10256.397408MB GCs: 217
2020/10/22 11:35:06 waiting for available jobs...
[MEMORY] Heap: 7043.959053MB Stack: 1.500000MB System: 10256.397408MB GCs: 219
2020/10/22 11:35:16 waiting for available jobs...
[MEMORY] Heap: 4227.219650MB Stack: 1.406250MB System: 10256.397408MB GCs: 221
2020/10/22 11:35:26 waiting for available jobs...
[MEMORY] Heap: 2162.055298MB Stack: 1.281250MB System: 10256.397408MB GCs: 224
2020/10/22 11:35:36 waiting for available jobs...
[MEMORY] Heap: 4038.807747MB Stack: 1.406250MB System: 10256.397408MB GCs: 226
2020/10/22 11:35:46 waiting for available jobs...
[MEMORY] Heap: 1556.686188MB Stack: 1.187500MB System: 10256.397408MB GCs: 229
2020/10/22 11:35:56 waiting for available jobs...
[MEMORY] Heap: 2747.283951MB Stack: 1.468750MB System: 10256.397408MB GCs: 232
2020/10/22 11:36:06 waiting for available jobs...
[MEMORY] Heap: 4110.576881MB Stack: 1.468750MB System: 10256.397408MB GCs: 234
2020/10/22 11:36:16 waiting for available jobs...
[MEMORY] Heap: 2761.419807MB Stack: 1.437500MB System: 10256.397408MB GCs: 237
2020/10/22 11:36:27 waiting for available jobs...
[MEMORY] Heap: 3884.565140MB Stack: 1.406250MB System: 10256.397408MB GCs: 241
2020/10/22 11:36:37 waiting for available jobs...
[MEMORY] Heap: 6244.286827MB Stack: 1.500000MB System: 10256.397408MB GCs: 243
2020/10/22 11:36:47 waiting for available jobs...
[MEMORY] Heap: 5528.342453MB Stack: 1.343750MB System: 10256.397408MB GCs: 245
2020/10/22 11:36:57 waiting for available jobs...
[MEMORY] Heap: 3750.548256MB Stack: 1.500000MB System: 10256.397408MB GCs: 247
2020/10/22 11:37:07 waiting for available jobs...
[MEMORY] Heap: 1427.640068MB Stack: 1.250000MB System: 10256.397408MB GCs: 250
2020/10/22 11:37:17 waiting for available jobs...
[MEMORY] Heap: 3122.388901MB Stack: 1.468750MB System: 10256.397408MB GCs: 253
2020/10/22 11:37:27 waiting for available jobs...
[MEMORY] Heap: 4909.477654MB Stack: 1.406250MB System: 10256.397408MB GCs: 255
2020/10/22 11:37:37 waiting for available jobs...
[MEMORY] Heap: 3358.217796MB Stack: 1.562500MB System: 10256.397408MB GCs: 259
2020/10/22 11:37:47 waiting for available jobs...
[MEMORY] Heap: 5166.278191MB Stack: 1.437500MB System: 10256.397408MB GCs: 262
2020/10/22 11:37:57 waiting for available jobs...
[MEMORY] Heap: 6251.215958MB Stack: 1.437500MB System: 10256.397408MB GCs: 264
2020/10/22 11:38:07 waiting for available jobs...
[MEMORY] Heap: 1513.662102MB Stack: 1.218750MB System: 10459.843719MB GCs: 265
2020/10/22 11:38:17 waiting for available jobs...
[MEMORY] Heap: 2755.411812MB Stack: 1.468750MB System: 10459.843719MB GCs: 266
2020/10/22 11:38:27 waiting for available jobs...
[MEMORY] Heap: 3265.693939MB Stack: 1.437500MB System: 10459.843719MB GCs: 269
2020/10/22 11:38:37 waiting for available jobs...
[MEMORY] Heap: 3861.916260MB Stack: 1.531250MB System: 10459.843719MB GCs: 271
2020/10/22 11:38:47 waiting for available jobs...
[MEMORY] Heap: 4404.640694MB Stack: 1.406250MB System: 10459.843719MB GCs: 274
2020/10/22 11:38:57 waiting for available jobs...
[MEMORY] Heap: 5453.725731MB Stack: 1.437500MB System: 10459.843719MB GCs: 277
2020/10/22 11:39:07 waiting for available jobs...
[MEMORY] Heap: 2275.793427MB Stack: 1.500000MB System: 10525.909157MB GCs: 279
2020/10/22 11:39:17 waiting for available jobs...
[MEMORY] Heap: 3631.300247MB Stack: 1.500000MB System: 10525.909157MB GCs: 284
2020/10/22 11:39:27 waiting for available jobs...
[MEMORY] Heap: 7032.191818MB Stack: 1.406250MB System: 10525.909157MB GCs: 286
2020/10/22 11:39:37 waiting for available jobs...
[MEMORY] Heap: 5691.436470MB Stack: 1.500000MB System: 10525.909157MB GCs: 288
2020/10/22 11:39:47 waiting for available jobs...
[MEMORY] Heap: 3740.111488MB Stack: 1.343750MB System: 10525.909157MB GCs: 292
2020/10/22 11:39:57 waiting for available jobs...
[MEMORY] Heap: 6390.945808MB Stack: 1.406250MB System: 10525.909157MB GCs: 294
2020/10/22 11:40:07 waiting for available jobs...
[MEMORY] Heap: 9634.360146MB Stack: 1.500000MB System: 10525.909157MB GCs: 295
2020/10/22 11:40:17 waiting for available jobs...
[MEMORY] Heap: 1723.369011MB Stack: 1.375000MB System: 10731.910042MB GCs: 296
2020/10/22 11:40:27 waiting for available jobs...
[MEMORY] Heap: 2267.520004MB Stack: 1.500000MB System: 10731.910042MB GCs: 296
2020/10/22 11:40:37 waiting for available jobs...
[MEMORY] Heap: 2100.307991MB Stack: 1.218750MB System: 10733.035042MB GCs: 298
2020/10/22 11:40:47 waiting for available jobs...
[MEMORY] Heap: 3836.317039MB Stack: 1.437500MB System: 10733.035042MB GCs: 300
2020/10/22 11:40:57 waiting for available jobs...
[MEMORY] Heap: 7349.499176MB Stack: 1.468750MB System: 10733.035042MB GCs: 302
2020/10/22 11:41:07 waiting for available jobs...
[MEMORY] Heap: 6304.722359MB Stack: 1.781250MB System: 10733.035042MB GCs: 305
2020/10/22 11:41:17 waiting for available jobs...
[MEMORY] Heap: 8274.542038MB Stack: 1.406250MB System: 10733.035042MB GCs: 307
2020/10/22 11:41:27 waiting for available jobs...
[MEMORY] Heap: 1756.419502MB Stack: 1.281250MB System: 10733.035042MB GCs: 310
2020/10/22 11:41:37 waiting for available jobs...
[MEMORY] Heap: 5281.729538MB Stack: 1.781250MB System: 10733.035042MB GCs: 313
2020/10/22 11:41:47 waiting for available jobs...
[MEMORY] Heap: 5429.138359MB Stack: 1.468750MB System: 10733.035042MB GCs: 316
2020/10/22 11:41:57 waiting for available jobs...
[MEMORY] Heap: 4806.623604MB Stack: 1.406250MB System: 10733.035042MB GCs: 318
2020/10/22 11:42:07 waiting for available jobs...
[MEMORY] Heap: 3470.874359MB Stack: 1.312500MB System: 10733.035042MB GCs: 321
2020/10/22 11:42:17 waiting for available jobs...
[MEMORY] Heap: 5599.000710MB Stack: 1.437500MB System: 10733.035042MB GCs: 323
2020/10/22 11:42:27 waiting for available jobs...
[MEMORY] Heap: 5257.462555MB Stack: 1.375000MB System: 10733.035042MB GCs: 325
2020/10/22 11:42:37 waiting for available jobs...
[MEMORY] Heap: 3670.337288MB Stack: 1.406250MB System: 10733.035042MB GCs: 327
2020/10/22 11:42:47 waiting for available jobs...
[MEMORY] Heap: 4703.336205MB Stack: 1.437500MB System: 10733.035042MB GCs: 329
2020/10/22 11:42:57 waiting for available jobs...
[MEMORY] Heap: 2411.756599MB Stack: 1.156250MB System: 10733.035042MB GCs: 333
2020/10/22 11:43:07 waiting for available jobs...
[MEMORY] Heap: 3868.894302MB Stack: 1.468750MB System: 10733.035042MB GCs: 335
2020/10/22 11:43:17 waiting for available jobs...
[MEMORY] Heap: 4386.837456MB Stack: 1.437500MB System: 10733.035042MB GCs: 337
2020/10/22 11:43:27 waiting for available jobs...
[MEMORY] Heap: 3549.995132MB Stack: 1.656250MB System: 10733.035042MB GCs: 339
2020/10/22 11:43:37 waiting for available jobs...
[MEMORY] Heap: 3490.940010MB Stack: 1.031250MB System: 10733.035042MB GCs: 342
2020/10/22 11:43:42 Syncing 217497-217952
2020/10/22 11:43:42 increasing syncer concurrency to 5 (projected new cache size: 0.011873 MB)
2020/10/22

patrick-ogrady · 2020-10-22T16:15:36Z

It would also help if you could run rosetta-cli with the --mem-profile mem.pprof flag (in addition to your existing flags) and then upload the result. This will be the fastest way to find the cause of this.

Yrp · 2020-10-22T16:44:15Z

the configuration file is here

Have re-run rosetta-cli check:construction, tomorrow I'll see if the old data still exists!

patrick-ogrady · 2020-10-22T16:49:10Z

On an unrelated note, you may want to use the newly created Construction DSL instead of the json-based workflow specification for construction tests (much easier to read and maintain): https://github.com/coinbase/rosetta-cli/blob/master/examples/configuration/ethereum.ros

This is how you link it to your config file: https://github.com/coinbase/rosetta-cli/blob/72051b6a51d638403835956090b2ec0268e2e129/examples/configuration/ethereum.json#L25

Yrp · 2020-10-22T16:53:59Z

On an unrelated note, you may want to use the newly created Construction DSL instead of the json-based workflow specification for construction tests (much easier to read and maintain): https://github.com/coinbase/rosetta-cli/blob/master/examples/configuration/ethereum.ros

This is how you link it to your config file:

https://github.com/coinbase/rosetta-cli/blob/72051b6a51d638403835956090b2ec0268e2e129/examples/configuration/ethereum.json#L25

It's amazing. I will replace it with the new DSL sometime.

patrick-ogrady · 2020-11-18T01:35:29Z

Were you able to collect any additional data here @Yrp? we haven't been able to reproduce this locally 😢

Yrp added the bug Something isn't working label Oct 22, 2020

patrick-ogrady mentioned this issue Dec 1, 2020

[chore] Update rosetta-sdk-go #193

Merged

6 tasks

patrick-ogrady closed this as completed in #193 Dec 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

memory leak causes data corruption that check:construction not working properly #170

memory leak causes data corruption that check:construction not working properly #170

Yrp commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020 •

edited

Loading

Yrp commented Oct 22, 2020 •

edited

Loading

patrick-ogrady commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020 •

edited

Loading

Yrp commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020

Yrp commented Oct 22, 2020

patrick-ogrady commented Nov 18, 2020

memory leak causes data corruption that check:construction not working properly #170

memory leak causes data corruption that check:construction not working properly #170

Comments

Yrp commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020 • edited Loading

Yrp commented Oct 22, 2020 • edited Loading

patrick-ogrady commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020 • edited Loading

Yrp commented Oct 22, 2020

patrick-ogrady commented Oct 22, 2020

Yrp commented Oct 22, 2020

patrick-ogrady commented Nov 18, 2020

patrick-ogrady commented Oct 22, 2020 •

edited

Loading

Yrp commented Oct 22, 2020 •

edited

Loading

patrick-ogrady commented Oct 22, 2020 •

edited

Loading