Testing RPC pay, connected wallet-cli client ID mis-identified as inactive for negative days value, Monerod crashes with "Aborted" #6555
Comments
|
Please run with gdb, ie: gdb ./monerod Then after it does: bt full And post the (probably multi page) output here. |
|
Also, what's the time on your machine running monerod, and the one running the wallet ? |
I got 2005 lines in my Putty window before the error with gdb ./monerod... Below is the last, what seems like relevant bit. Here is a paste bin of the full 2005 lines for neatness of this post, but I can dig out the entire log file if needed.
Then bt full produces
|
Because I only have wallet-cli v0.16 on the same device as the node, this was produced locally. GMT so 1 hour ahead of what the Monerod quotes in it's logs. I can do this (connect) from a different device (windows) but it would be with wallet-cli (v0.15.0.5) |
|
You're missing the interesting part from gdb, it's not here nor in the paste. Note the message telling you to type for more, it's a multi page output. |
My bad, a habbit of using ctrl+c to copy, so ending it. This time: 3 outputs for you:
Because of the bt full "Backtrace stopped: previous frame identical to this frame (corrupt stack?)" when trying to view next page I repeated this to try and get a better output. This error was produced both times. Big headers for easier scrolling... Log 3 Putty output of Monerod until point of crash ~2000 lines:2020-05-19 17:43:36.877 T BlockchainLMDB::block_rtxn_start Thread 6 "monerod" received signal SIGABRT, Aborted. bt full(gdb) bt full Next page#24 0x76440578 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6 |
|
Helpful, thanks. I think I know what the problem is, I'll make a patch. |
TBH I'm kinda thankful it's actually a real error. After I submitted it I had the gut feeling this was going to be another lesson in why not to mess around with single board computers. Haha |
|
Please try the last patch in https://github.com/moneromooo-monero/bitmonero/tree/rpl |
|
Sure, and just to clarify how I'm doing it: I've removed my old Monero files Downloaded v0.16 again with Replaced the files I build with And will repeat my testing to see what the new outcome is. Edit: 20/05 pm |
The latest patch built without error, sync was fine and continued to be stable for many hours. Problems occurred again when connecting the wallet-cli (same start commands as above with local ip now changed ending 127): monerod output (wallet connected)came home from work to find the monerod terminal screen filling with:
I used ctrl+c to stop the node bt fullThread 1 "monerod" received signal SIGINT, Interrupt. Wallet-cli
Program terminated with signal SIGKILL, Killed. bt full(gdb) bt full The wallet-cli was taking a very long time to start RPC Mining so I left it whilst at work. I will repeat this again to see if this a repeatable fault, or was just unlucky this first time. |
|
Looks like some weird other problem got to you first. Doesn't seem related. |
|
Yeah I wasn't around to see it fail so I'm going to see if it repeats |
|
@moneromooo-monero Whatever has been changed seems to prevent any commands being passed from the wallet to to daemon. At the wallet command line neither Both commands process for approx 20mins before stating "Killed" Output:
|
|
Works for me. "Killed" is the OOM killer. Looks likely that your wallet is trying to use more RAM that the system is able to give it. Try again, run start_mining_for_rpc, wait a minute, then run: |
|
@moneromooo-monero Unfortunately I'm then back to a variation of the original problem. SetupRunning your modified Monero v.16 node on an Odroid XU4 Armbian. Node runs happily. Wallet-cli v0.15.0.5 Windows PC plenty of resources, no RAM issue, can send commands and receive cli feedback fine. IssueThe variation on the original issue is that despite checking and double checking internet time settings and zones on the node and wallet, both are sync'd with internet time, the payment of mining credits is sporadic due to the node falsely marking the connected wallet as inactive for a now positive number of days. This seems to cause the node to delete the wallets held credits. Symptoms - I have heavily edited the following logs so they are chronological and show what I hope are relevant linesWallet-cliMonero 'Carbon Chamaeleon' (v0.15.0.5-release) Use the "help" command to see the list of available commands. Starting refresh... Monerod2020-05-22 16:33:21.611 I Erasing <5fd351948851957b719888c347108a4d12cbad117915354d5e786764bd59c232> with 190 credits, inactive for 9429 days SENT ANOTHER COMMAND via WALLET "rpc_payment_info" approx here2020-05-22 16:37:21.614 I Erasing <5fd351948851957b719888c347108a4d12cbad117915354d5e786764bd59c232> with 439 credits, inactive for 6478 days Wallet-cliRPC client ID: <5fd351948851957b719888c347108a4d12cbad117915354d5e786764bd59c232> Monerod2020-05-22 16:44:21.618 I Erasing <5fd351948851957b719888c347108a4d12cbad117915354d5e786764bd59c232> with 80 credits, inactive for 1764 days Edited for some embarrassingly bad spelling |
|
But it doesn't crash anymore, right ? :) Please paste the output of this command on the machines running both daemon and wallet: |
|
No more crashes. All good on that front. Monerod on Armbian pinodexmr@PiNodeXMR:~$ date +%s Windows Wallet-cli Edit: added date command for windows to avoid major discrepancy Edit2: + Human readable pinodexmr@PiNodeXMR:~$ date +%c Edit3: I put the windows wallet-cli v0.15.0.5 and wallet file+keys on a USB drive and ran the same from my laptop. The same inactive for an irrational number of days with erase issue exists. Occasional 2020-05-22 18:09:28.430 W Very stale nonce from monerod Edit4: I've seen on monerod several negative values for "inactive" for days scroll past, so I think the original issue still exists Is it relevant that it is erasing at 1 minute intervals? Does that give a clue? 2020-05-22 18:12:26.941 I Erasing with 249 credits, inactive for -9381 days 2020-05-22 18:13:26.943 I Erasing with 34 credits, inactive for -10232 days 2020-05-22 18:14:26.944 I Erasing with 0 credits, inactive for -10801 days 2020-05-22 18:15:26.945 I Erasing with 0 credits, inactive for -11840 days 2020-05-22 18:16:27.318 I Erasing with 50 credits, inactive for -12187 days |
|
I think I found what's going on, please try this patch: Apply by saving the patch in a file (ie, FILENAME) and: patch -p1 < FILENAME |
|
Apologies again I need a little more clarification on how to apply this patch. I've created the file. Called it FILENAME (because it's as good as any) with:
Pasted in the contents of the file from the link and saved it. Changed permission of /home/pinodexmr/FILENAME to 777 to prevent issues. Then I'm not applying it right, trying to use: ./monerod patch -pl < /home/pinodexmr/FILENAME unrecognised option '-pl' So I'm not understanding correctly, I've tried some variations (thats a -p(one) I'm just copy/pasting from your comment. |
|
Instead of
do
Edit: oops see mooos comment |
|
p1, not pl. Like one. |
|
Thanks for that. I'm learning all sorts today. It asked a few questions but I think I've answered them as intended:
Testing again results in2020-05-22 19:27:44.227 I Erasing <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> with 0 credits, inactive for -12041 days 2020-05-22 19:28:44.228 I Erasing <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> with 0 credits, inactive for -12774 days 2020-05-22 19:28:57.720 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> sends nonce: 594, current 2020-05-22 19:28:59.336 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> credited for 50, now 50 2020-05-22 19:30:44.229 I Erasing <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> with 0 credits, inactive for -13830 days 2020-05-22 19:31:09.974 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> credited for 50, now 50 2020-05-22 19:31:25.290 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> sends nonce: 4574, current 2020-05-22 19:31:26.804 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> credited for 50, now 149 2020-05-22 19:31:43.023 I client <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> credited for 50, now 199 2020-05-22 19:31:44.230 I Erasing <8b14107818971ee6bc275572d2deb7b33266073f54a49388178b8d93abeac4c5> with 199 credits, inactive for -14966 days Because I have lots of monero related commands in crontabs for getting statistics I have removed all of those to (as of about an hour ago before you made that patch). That should remove more external variables my end. However did I apply the patch correctly? Editing to try to remove awful formatting |
|
Sorry, I forgot to say: please run with --log-level 0,daemon.rpc.payment:DEBUG |
|
So below is a short log of that debug command. It's raw/un-edited but I'd say uneventful. One Error at timestamp 21:04:25.756 of: E [192.168.1.108:51203 INC] Failed to on_get_blocks() And two: W Very stale nonce I do notice though that for the short time my wallet-cli ID does own some credits, it doesn't seem to be spending them to sync the wallet. Should it be spending them more proactively? The Log:pinodexmr@PiNodeXMR:~/monero/build/release/bin$ ./monerod --rpc-bind-ip=0.0.0.0 --rpc-bind-port=18083 --rpc-restricted-bind-port=18089 --confirm-external-bind --rpc-payment-allow-free-loopback --rpc-ssl disabled --rpc-payment-address=43HoAhqx9q3MR1crAjpQtYVhvzQhZgqPwSWVQMmPvYmr18qVUEjCHcsEasuCxS486rWSSg1gbGqanet67NWRsh1bQL9KkB9 --rpc-payment-credits=50 --rpc-payment-difficulty=1000 --log-level 0,daemon.rpc.payment:DEBUG |
|
The patch does not seem to have been applied. |
|
That's inconvenient as I copy/pasted the output and once I resolved the file paths it seemed happy. I'm happy to re-install your version of monero again with the two edited .cpp files? Or I can try and re-apply the same patch again? Edit: Re-reading that this morning sounded as if my tone was off. What I mean is that it's inconvenient when these things don't perform as expected, but I'm happy to keep pursuing/testing solutions when required. I hope it's not too complicated a fix. Can the edit from the patch be added manually? |
|
I pushed the patch to the https://github.com/moneromooo-monero/bitmonero/commits/rpl branch now. You can copy src/rpc/rpc_payment.* from that branch. |
|
This seems to have solved it. Thank you. Sorry for the delay. Until I checked today I wasn't aware of your last message for some reason. So I compiled Monero again with your patch to the two files this morning and it has been successfully running for several hours without issue this afternoon.
I can see the fix performing the update_time function each minute.
I will leave it running to reach it's mining/credit target, and for a few days after but all seems good. Thanks again. |
|
Thanks for the logs. |
|
Right, that makes sense for the wild difference in time before this fix. My last question regarding this before I think this can be closed would be about the application of this fix. Are the two files you made changes to likely to get worked into the master Monero release? If they are 32bit exclusive then for context you may recognise my username from the PiNodeXMR project and I'm asking this because I have quite a few users looking forward to running their nodes on single board computers (SBC) with the RPC_pay flag. The project as of recently allows builds on any Armbian supported SBCs (not just Raspberry Pis) however an official Armbian 64bit OS isn't always available even though the board has 64bit architecture (in the case of some Odroids), so they may take a couple of years to catch up. As a work around now there is this fix I could tweak my install scripts to detect a 32bit OS and so pull in your files before building Monero? If this can't be worked into the master Monero release? |
|
The patch will be merged to master. In the meantime, you can certainly apply this patch to anything that might need it. |
|
Will do, and thanks again for the fix. |
A bit of a lengthy title I know put I think it's still to the point.
I've been trying this out with regards to the rpc payment features. and the --rpc-payment-allow-free-loopback which was added to solve issue #3038
I'm having some stability issues I can't work out. I start the node with:
./monerod --rpc-bind-ip=0.0.0.0 --rpc-bind-port=18083 --rpc-restricted-bind-port=18089 --confirm-external-bind --rpc-payment-allow-free-loopback --rpc-ssl disabled --rpc-payment-address=43HoAhqx9q3MR1crAjpQtYVhvzQhZgqPwSWVQMmPvYmr18qVUEjCHcsEasuCxS486rWSSg1gbGqanet67NWRsh1bQL9KkB9 --rpc-payment-credits=5 --rpc-payment-difficulty=100 --log-level=3Where the un-restricted port 18083 is kept behind my firewall and used by other scripts to produce stats for a user interface (print_pl, status, version, etc,)
Port 18089 restricted and port forwarded through router for external wallet access requiring payment.
The
--rpc-payment-allow-free-loopbackflag is great in that it now permits the node to start with both an unrestricted and restricted port. So it solved that initial issue.The stability issue I have is that when a wallet-cli client submits hashes, I get what appear to be "timing" issues. Sometimes monerod will run for 5 minutes, sometimes up to 45mins before this happens. I've increased the log-level to try and get a better idea of whats causing it but I'm out of my depth, only the last couple of lines seem relevant.
Monerod (Monero 'Carbon Chamaeleon' (v0.16.0.0-4987161fa)):
It's weird my currently connected wallet-cli client (in this case id <99...) is reported as not being seen for 56.5 years? Doesn't erase it's credits, adds another 5 credits to it's balance, then terminates the node with single "Aborted" phrase.
On another occasion (so different client ID <a8a...):
Wallet (CLI) (Monero 'Carbon Chamaeleon' (v0.16.0.0-4987161fa)) started with:
./monero-wallet-cli --daemon-address=192.168.1.116:18089 --trusted-daemon start_mining_for_rpcErrors with:
Other Notes:
Using
rpc_payment_infoin the wallet cli I see my client ID, and it matched the inactive one from the daemon log each time. I've seen the above crash repeated about 6 times, each requires a manual restart of both daemon and wallet.The daemon is 100% sync'd
I compiled Monero 0.16 from source, this is running on a Raspberry Pi, the build produced no errors. The logs above are from the wallet cli 0.16 also.
This was initially noticed when connecting to the Raspberry Pi node (v0.16) from my Windows wallet-cli (v0.15.0.5) over LAN and then also an external connection. To remove the possible version conflict I then produced the above logs/test running the node and wallet on the same device (v0.16) to mitigate the different clocks on different devices.
I've included my start commands in this because I never rule out user error?
The text was updated successfully, but these errors were encountered: