[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

petzsch · 2024-03-27T08:59:08Z

Background

I am running LND as part of my BTCPay Server setup together with lnbits side by side deployed via Docker. When I tried to send me some funds from getAlby I noticed that the payment wouldn't go through. After Checking bitcoin-lncli.sh getinfo, I found:

    "synced_to_chain":  false,
    "synced_to_graph":  true,

Your environment

version of lnd: 0.17.4-beta commit=basedon-v0.17.4-beta-fresh-btcpay
which operating system (uname -a on *Nix): Ubuntu 22.04 on ARM64 (RPI4 board)
version of btcd, bitcoind, or other backend: Bitcoin Core 26.0
any other relevant environment details: Deployed in Docker

Steps to reproduce

I've reported the loss of synced to chain in the BTCPayServer Telegram group before and was told to enable profile logging. So here is als the info I can provide. (collected from within the container via curl http://localhost:9736/debug/pprof/goroutine?debug=2

profile.txt

Expected behaviour

The node should stay synced to chain.

Actual behaviour

Can't receive payments, sync to chain lost.

The text was updated successfully, but these errors were encountered:

petzsch · 2024-03-27T08:59:39Z

linked to this issue which should have been fixed: #8250

bitromortac · 2024-03-27T09:42:06Z

Could you give more details concerning what backend mode is being used? Is it a pruned bitcoind node? Is bitcoind synced and what block height is reported in lncli getinfo?

petzsch · 2024-03-27T09:49:23Z

yes, it's a pruned bitcoin core: prune=50000
the bitcoind is synced according to bitcoin-cli.sh getblockchaininfo

{
  "chain": "main",
  "blocks": 836504,
  "headers": 836504,
  "bestblockhash": "000000000000000000005cf4bbf979da3b6ae50d69b28c71e52c7ac6922bdb60",
  "difficulty": 83947913181361.55,
  "time": 1711532474,
  "mediantime": 1711529318,
  "verificationprogress": 0.9999986208168137,
  "initialblockdownload": false,
  "chainwork": "0000000000000000000000000000000000000000712365f8c0ac006aef879e1c",
  "size_on_disk": 52403960028,
  "pruned": true,
  "pruneheight": 809561,
  "automatic_pruning": true,
  "prune_target_size": 52428800000,
  "warnings": ""
}

lncli getinfo returns the correct block height as well:

{
    "version":  "0.17.4-beta commit=basedon-v0.17.4-beta-fresh-btcpay",
    "commit_hash":  "3da3d28064d2712400f1b579bd4dfa34e04fd227",
    "identity_pubkey":  "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c",
    "alias":  "btcpay.host",
    "color":  "#3399ff",
    "num_pending_channels":  0,
    "num_active_channels":  2,
    "num_inactive_channels":  0,
    "num_peers":  2,
    "block_height":  836504,
    "block_hash":  "000000000000000000005cf4bbf979da3b6ae50d69b28c71e52c7ac6922bdb60",
    "best_header_timestamp":  "1711532474",
    "synced_to_chain":  false,
    "synced_to_graph":  true,
    "testnet":  false,
    "chains":  [
        {
            "chain":  "bitcoin",
            "network":  "mainnet"
        }
    ],
    "uris":  [
        "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c@152.53.16.125:9735",
        "02fed6c495383abf57e1742bd6ef3a966fba62a0014278d89dc65d078f4e38bb3c@qdf2intfsade3jxh3ci7zoxkfvslqnl5pqtoizaztq3v6bmji3spkyad.onion:9735"
    ],
    "features":  {
        "0":  {
            "name":  "data-loss-protect",
            "is_required":  true,
            "is_known":  true
        },
        "5":  {
            "name":  "upfront-shutdown-script",
            "is_required":  false,
            "is_known":  true
        },
        "7":  {
            "name":  "gossip-queries",
            "is_required":  false,
            "is_known":  true
        },
        "9":  {
            "name":  "tlv-onion",
            "is_required":  false,
            "is_known":  true
        },
        "12":  {
            "name":  "static-remote-key",
            "is_required":  true,
            "is_known":  true
        },
        "14":  {
            "name":  "payment-addr",
            "is_required":  true,
            "is_known":  true
        },
        "17":  {
            "name":  "multi-path-payments",
            "is_required":  false,
            "is_known":  true
        },
        "19":  {
            "name":  "wumbo-channels",
            "is_required":  false,
            "is_known":  true
        },
        "23":  {
            "name":  "anchors-zero-fee-htlc-tx",
            "is_required":  false,
            "is_known":  true
        },
        "27":  {
            "name":  "shutdown-any-segwit",
            "is_required":  false,
            "is_known":  true
        },
        "30":  {
            "name":  "amp",
            "is_required":  true,
            "is_known":  true
        },
        "31":  {
            "name":  "amp",
            "is_required":  false,
            "is_known":  true
        },
        "45":  {
            "name":  "explicit-commitment-type",
            "is_required":  false,
            "is_known":  true
        },
        "2023":  {
            "name":  "script-enforced-lease",
            "is_required":  false,
            "is_known":  true
        }
    },
    "require_htlc_interceptor":  false,
    "store_final_htlc_resolutions":  false
}

What other "details concerning what backend mode" do you need?

bitromortac · 2024-03-27T10:20:00Z

There are a few goroutines that hang since some time (3590 minutes) related to gossiper, which may be due to

goroutine 1146033 [select, 3593 minutes]:
github.com/btcsuite/btcwallet/chain.(*BitcoindConn).GetBlock(0x40001620b0, 0x400feffaa0)
	/go/pkg/mod/github.com/btcsuite/btcwallet@v0.16.10-0.20240127010340-16b422a2e8bf/chain/bitcoind_conn.go:485 +0x1d8
github.com/btcsuite/btcwallet/chain.(*BitcoindClient).GetBlock(0x400170da68?, 0x0?)
	/go/pkg/mod/github.com/btcsuite/btcwallet@v0.16.10-0.20240127010340-16b422a2e8bf/chain/bitcoind_client.go:144 +0x20
github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock(0x400170da70, 0x400feffaa0, 0x4002b4feb0)
	/go/src/github.com/lightningnetwork/lnd/blockcache/blockcache.go:55 +0x1cc
github.com/lightningnetwork/lnd/lnwallet/btcwallet.(*BtcWallet).GetBlock(0x1?, 0x1?)
	/go/src/github.com/lightningnetwork/lnd/lnwallet/btcwallet/blockchain.go:139 +0x168
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTx(0x400043ac80, 0x400f818450)
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1810 +0x64
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTxWrapper.func1()
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1774 +0x34
created by github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTxWrapper
	/go/src/github.com/lightningnetwork/lnd/routing/router.go:1773 +0xc8

This could be related to the issue you posted, will investigate further, thank you for the profile. It may help you to try a restart.

petzsch · 2024-03-27T10:23:28Z

restart of lnd usually fixes the issue. Last time I ran into this was march 13th. (had to restart to activate the profile logging)

Since the node is not activly used (personal lnbits wallet not much more), I waited with the restart in case you need further details from me.

If you are certain, you don't need anything more from the hung node, I would restart it now.

bitromortac · 2024-03-27T10:26:29Z

Ok, then it could make sense to let it run to perhaps extract more info. Could you check the logs if anything strange happened 3593 minutes (~60 hours) ago?

petzsch · 2024-03-27T10:42:24Z

Bit overwhelmed looking at 5,2MB of log files. Hope it's ok for me to post it here:

lnd-container-log.zip

ziggie1984 · 2024-03-27T16:08:02Z

Thank you for the profile and the log files this helped me a lot tracking down the issue.
In general your bitcoin peers are just 4 (normally bitcoind as at least 8 outbound peers iirc) according to the logs and the profile dump. Those peers have a very high latency which causes the PrunedBlock worker of LND to run in an edge case where it would retry the same block query over and over again. This should be fixed with the linked PR, but you can also make sure you don't run into this case when yours have a reliable PingTime, you can check it with

bitcoin-cli getpeerinfo | jq '.[] | select(.inbound == false) | .pingtime'

the ping times should be at least less than 1 sec.

petzsch · 2024-03-27T16:11:54Z

here is what it currently looks like:

root@server:~# bitcoin-cli.sh getpeerinfo | jq '.[] | select(.inbound == false) | .pingtime'
0.158792
0.011204
0.162383
0.059948
0.089443
0.032051
1.169742
0.021967
0.026476
0.482593

ziggie1984 · 2024-03-27T16:20:37Z

can you share your logs within this state, I wonder if you still have a lot of these log lines:

Query(???) from peer ????.onion:8333 failed, rescheduling: did not get response before timeout

petzsch · 2024-03-27T16:53:40Z

several per minute:

2024-03-27 16:40:29.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:40:47.658 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:01.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:19.659 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:33.789 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:41:51.661 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:05.790 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:23.661 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:37.791 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:42:55.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:09.791 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:27.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:41.792 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:43:59.662 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:13.793 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:31.663 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:44:45.794 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:03.663 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:17.794 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:35.664 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:45:49.796 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:07.665 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:21.796 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:39.666 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:46:53.798 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:11.666 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:25.799 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:43.667 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:47:57.802 [WRN] LNWL: Query(656) from peer zmtd72b4nsa22ocez2hbjsxxrdfuxxyghg3zjtqjob5sgnjpustat4qd.onion:8333 failed, rescheduling: did not get response before timeout
2024-03-27 16:48:15.668 [WRN] LNWL: Query(654) from peer 6zpkgoruozapgjzbtfralawslmcrnf56rugbkis4ehnttvv6tw6okrqd.onion:8333 failed, rescheduling: did not get response before timeout

according to docker logs btcpayserver_lnd_bitcoin | grep ".onion:8333 failed, rescheduling"

Btw. I made a mistake above. This isn't running on a RPI4 - I mixxed it up with my home node - but rather on a KVM instance on a Ampere Altra Max (ARM64). To be specific this very VM. -> not sure if that is relevant information.

ziggie1984 · 2024-03-27T17:38:31Z

Ok unfortunately it seems like you already run in the same issue again, because the same query is retried all the time and normally that should only happen twice max. You are currently not in a deadlock because you probably still have some free workers available but its just a question of time that it hits again, next release will fix your issue.

ziggie1984 · 2024-03-27T17:41:00Z

are you interested in running a patch with the fix based on your current version 17.4 ? @petzsch

petzsch · 2024-03-27T17:44:48Z

I'd have to figure out how the docker image for lnd that btcpay uses is build. Other then that, I'd be happy to test the patch for you.

EDIT: I think I found there lnd repo and the tag used for the version I'm running: https://github.com/btcpayserver/lnd/tree/basedon-v0.17.4-beta

petzsch · 2024-03-27T18:28:48Z

I think I'm ready for the diff/patch file. Alternativly I could make you collaborator on my lnd fork so you could apply it directly. whatever makes most sense to you. 😄

ziggie1984 · 2024-03-27T19:40:30Z

make me a collaborator thats faster.

EDIT:

This PR is based on 17.4 and includes your fix: #8596

petzsch · 2024-03-28T16:28:49Z

successfully included your commit in my fork, currently building the docker image, but ran into an error that seems to be btcpay build environment specific... I'll try to sort it out with @rockstardev if he's available. Will get back here when I got the docker container running.

petzsch · 2024-03-28T17:08:09Z

got the docker build issues sorted out. Docker image is running, my 2 public channels are back online ans sync to chain looks ok:

    "synced_to_chain":  true,
    "synced_to_graph":  true,

Let me know if you need any logs or so.

ziggie1984 · 2024-03-28T17:14:55Z

Cool, so let's see whether you ever run into the sync issue again, you can look into the logs occasionally and check if the queries repeat themselves with the same index number, this is a sign that something is not right. But apart from that you should be good to go now.

ziggie1984 · 2024-04-03T14:27:49Z

@petzsch how does your system look like, we are about to merge the fix, any problems so far?

petzsch · 2024-04-03T14:31:21Z

No losses of sync so far.

Here the log output, looks good so far to me:

root@server:~# docker logs btcpayserver_lnd_bitcoin | grep ".onion:8333 failed, rescheduling"
panic: runtime error: index out of range [21] with length 21

goroutine 339 [running]:
github.com/lightninglabs/neutrino/query.(*workQueue).Swap(0x400da07978?, 0x40004d8968?, 0xf291e0?)
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workqueue.go:33 +0xc0
container/heap.Remove({0x149e060, 0x40005cedb0}, 0x15)
        /usr/local/go/src/container/heap/heap.go:71 +0x54
github.com/lightninglabs/neutrino/query.(*peerWorkManager).workDispatcher(0x4006a907b0)
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workmanager.go:455 +0x1558
created by github.com/lightninglabs/neutrino/query.(*peerWorkManager).Start
        /go/pkg/mod/github.com/ziggie1984/neutrino@v0.15.1-0.20240327153143-f2b82122ab75/query/workmanager.go:140 +0x70
2024-04-03 10:52:32.594 [WRN] LNWL: Query(0) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.718 [WRN] LNWL: Query(1) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.733 [WRN] LNWL: Query(2) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:32.790 [WRN] LNWL: Query(3) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.597 [WRN] LNWL: Query(4) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.720 [WRN] LNWL: Query(5) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.735 [WRN] LNWL: Query(6) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:38.791 [WRN] LNWL: Query(7) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.598 [WRN] LNWL: Query(8) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.721 [WRN] LNWL: Query(9) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.737 [WRN] LNWL: Query(10) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:44.799 [WRN] LNWL: Query(11) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.600 [WRN] LNWL: Query(12) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.723 [WRN] LNWL: Query(13) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.739 [WRN] LNWL: Query(14) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:50.800 [WRN] LNWL: Query(15) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.602 [WRN] LNWL: Query(16) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.724 [WRN] LNWL: Query(17) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.741 [WRN] LNWL: Query(18) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:52:56.803 [WRN] LNWL: Query(19) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:53:02.604 [WRN] LNWL: Query(20) from peer 5gpjmdnojjulidhqcbjzdzzhvrkq2v4o23vea3fl6r5mtemgbh5nmqid.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 10:53:02.726 [WRN] LNWL: Query(21) from peer rziiwvt7wwkpmhoj5fqzsyem5i2tgb6ot77qnujqy6ewc3fvmlxtk2id.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.568 [WRN] LNWL: Query(0) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.570 [WRN] LNWL: Query(1) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:34.578 [WRN] LNWL: Query(3) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.570 [WRN] LNWL: Query(95) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.572 [WRN] LNWL: Query(96) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:40.580 [WRN] LNWL: Query(97) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.572 [WRN] LNWL: Query(189) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.572 [WRN] LNWL: Query(188) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:46.582 [WRN] LNWL: Query(190) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.573 [WRN] LNWL: Query(277) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.573 [WRN] LNWL: Query(276) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:52.583 [WRN] LNWL: Query(278) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.574 [WRN] LNWL: Query(362) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.574 [WRN] LNWL: Query(363) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:53:58.585 [WRN] LNWL: Query(364) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.575 [WRN] LNWL: Query(453) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.575 [WRN] LNWL: Query(452) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:04.586 [WRN] LNWL: Query(454) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.577 [WRN] LNWL: Query(525) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.577 [WRN] LNWL: Query(526) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:10.588 [WRN] LNWL: Query(527) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.593 [WRN] LNWL: Query(600) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.593 [WRN] LNWL: Query(601) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:16.594 [WRN] LNWL: Query(599) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.594 [WRN] LNWL: Query(670) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.594 [WRN] LNWL: Query(669) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:22.595 [WRN] LNWL: Query(671) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(741) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(743) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:28.596 [WRN] LNWL: Query(742) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(816) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(817) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:34.597 [WRN] LNWL: Query(818) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.598 [WRN] LNWL: Query(891) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.598 [WRN] LNWL: Query(890) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:40.599 [WRN] LNWL: Query(892) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(964) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(965) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:46.610 [WRN] LNWL: Query(963) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1038) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1036) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:52.612 [WRN] LNWL: Query(1037) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.612 [WRN] LNWL: Query(1108) from peer 3l5gmo3x3zosskiyhtusw4bjp4zp4t6qqo3htuvpoyumu7ekxd5oqaqd.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.613 [WRN] LNWL: Query(1110) from peer elscqsi626cggqw3cuq2catfmz6zqhifqnc76d4jyubm4karfuttsnad.onion:8333 failed, rescheduling: did not get response before timeout
2024-04-03 11:54:58.615 [WRN] LNWL: Query(1109) from peer v5kazszlgg2xovmowi5eaaa6woftlscjkm2odscwerhvba3pqzfn74yd.onion:8333 failed, rescheduling: did not get response before timeout

EDIT: Had to restart the container this morning for some update on the docker-compose stack. but also before no issues.

ziggie1984 · 2024-04-03T23:22:15Z

@petzsch can you update your version, I refinied the PR #8596

can you use this branch, I am updating the PR to be rebased on master, but this branch will be based on 17.4: ziggie1984:fix-pruned-node-query (https://github.com/ziggie1984/lnd/tree/fix-pruned-node-query)

petzsch · 2024-04-04T16:16:18Z

Added you to my fork of lnd with the btcpay related stuff in it. If it's not to much to ask, it would be awesome if you could apply your changes to: https://github.com/petzsch/lnd/tree/patch-for-issue-8593 directly.

I will build the docker image afterwards and let it run on my machine.

ziggie1984 · 2024-04-04T16:25:33Z

Done, you should have the newest version.

petzsch · 2024-04-04T17:26:07Z

thx, the build went through and I'm now running the version provided.

petzsch added bug Unintended code behaviour needs triage labels Mar 27, 2024

bitromortac added pruning sync labels Mar 27, 2024

ziggie1984 self-assigned this Mar 27, 2024

saubyk removed the needs triage label Mar 27, 2024

ziggie1984 mentioned this issue Mar 27, 2024

query: fix retry query case. lightninglabs/neutrino#297

Merged

ziggie1984 mentioned this issue Mar 27, 2024

fix pruned node query #8596

Closed

AbelLykens mentioned this issue Mar 28, 2024

[bug]: v0.17.4-beta neutrino will not start if neutrino fails: node crashed #8598

Closed

saubyk added this to the v0.18.0 milestone Mar 28, 2024

saubyk added this to lnd v0.18 Mar 28, 2024

saubyk moved this to 👀 In review in lnd v0.18 Mar 28, 2024

saubyk changed the title ~~[bug]: LND 0.17.4-beta still looses sync to chain after long times of inactivity as it seems~~ [bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems Apr 2, 2024

guggero mentioned this issue Apr 11, 2024

[bug]: Walletbalance of default account is 0, after import of external xpub #8109

Closed

guggero closed this as completed in lightninglabs/neutrino#297 Apr 25, 2024

github-project-automation bot moved this from 👀 In review to ✅ Done in lnd v0.18 Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

petzsch commented Mar 27, 2024

petzsch commented Mar 27, 2024 •

edited

Loading

bitromortac commented Mar 27, 2024

petzsch commented Mar 27, 2024

bitromortac commented Mar 27, 2024 •

edited

Loading

petzsch commented Mar 27, 2024

bitromortac commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024 •

edited

Loading

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024 •

edited

Loading

petzsch commented Mar 28, 2024

petzsch commented Mar 28, 2024

ziggie1984 commented Mar 28, 2024

ziggie1984 commented Apr 3, 2024

petzsch commented Apr 3, 2024 •

edited

Loading

ziggie1984 commented Apr 3, 2024 •

edited

Loading

petzsch commented Apr 4, 2024

ziggie1984 commented Apr 4, 2024

petzsch commented Apr 4, 2024

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

[bug]: LND 0.17.4-beta still loses sync to chain after long times of inactivity as it seems #8593

Comments

petzsch commented Mar 27, 2024

Background

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

petzsch commented Mar 27, 2024 • edited Loading

bitromortac commented Mar 27, 2024

petzsch commented Mar 27, 2024

bitromortac commented Mar 27, 2024 • edited Loading

petzsch commented Mar 27, 2024

bitromortac commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024

petzsch commented Mar 27, 2024 • edited Loading

petzsch commented Mar 27, 2024

ziggie1984 commented Mar 27, 2024 • edited Loading

petzsch commented Mar 28, 2024

petzsch commented Mar 28, 2024

ziggie1984 commented Mar 28, 2024

ziggie1984 commented Apr 3, 2024

petzsch commented Apr 3, 2024 • edited Loading

ziggie1984 commented Apr 3, 2024 • edited Loading

petzsch commented Apr 4, 2024

ziggie1984 commented Apr 4, 2024

petzsch commented Apr 4, 2024

petzsch commented Mar 27, 2024 •

edited

Loading

bitromortac commented Mar 27, 2024 •

edited

Loading

petzsch commented Mar 27, 2024 •

edited

Loading

ziggie1984 commented Mar 27, 2024 •

edited

Loading

petzsch commented Apr 3, 2024 •

edited

Loading

ziggie1984 commented Apr 3, 2024 •

edited

Loading