Eclair v0.8.0 issues

[DerEwige]

I've upgraded my node to Eclair v0.8.0 and run into a few problems along the way.

I would like to discuss, what problems I encountered and if they are worth opening an issue.

1.) Error spam during initial synch with peer

eclair.2022-12-01_22.log:2022-12-01 22:58:55,252 ERROR f.a.e.c.TransportHandler CON n:03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601 - cannot deserialize ByteVector(130 bytes, 0x01026aaece577d5ee90f6ea2756f22cbba0e620363ac873d3d79d8831dee14d5816c7f8a891b47d838f789301ee6b2bf51942833662bd6b578b1efbb58c20cdc27816fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d619000000000008292a0004ad00006388ccb00000010f0000000000011fce00000b240000000e): messageFlags: expected constant BitVector(1 bits, 0x8) but got BitVector(1 bits, 0x0)
eclair.2022-12-01_22.log:2022-12-01 22:58:55,907 ERROR f.a.e.c.TransportHandler CON n:03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601 - cannot deserialize ByteVector(130 bytes, 0x0102013705e594b20bea2ef212cf9f84013323de07fa07a5f8f515df4205c23ada6f6ecd24c06be1221c5d4e7b236c761bdef5a1f1e5b63480eb112ccf9597d5b9c26fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000083d780000b700015df635d00001009000000000000003e8000003e800000001): messageFlags: expected constant BitVector(1 bits, 0x8) but got BitVector(1 bits, 0x0)
eclair.2022-12-01_22.log:2022-12-01 22:58:55,910 ERROR f.a.e.c.TransportHandler CON n:03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601 - cannot deserialize ByteVector(130 bytes, 0x01024d77b955573527208ac391cf0aeffdfd5efbd99f365ef59f050e373dfbbec69337023a2ce66439cf9c56255486785af0760a08fec51cefce57e0b85ba3c594f46fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d619000000000008409700047800005df69f480000009000000000000003e8000003e800000001): messageFlags: expected constant BitVector(1 bits, 0x8) but got BitVector(1 bits, 0x0)

I got thousands of these errors after the upgrade and had to enable sync-whitelist to stop the flood.

2.) Extremely high CPU and RAM usage
The CPU usage is usually very high during start-up and then drops fast afterwards.
After the first start-up it constantly stayed at 200% (2 CPU x 100%) and the Memory usage kept climbing steadily. After like 20 minutes it reached 4 GB and I shut down the node.
I suspected that the node is doing something on the DB and because my DB is so large has problem and runs out of Memory.
I went into DB. Deleted all expired invoices by hand. Increased the Memory for my Node to 8 GB and started it up again.
After another 10 minutes of high resources usage the node finally returned to normal levels.

3.) A lot of changed classed that broke all of my plugins.
Changed class names:

fr.acinq.bitcoin.ByteVector32 -> fr.acinq.bitcoin.scalacompat.ByteVector32
fr.acinq.bitcoin.Crypto.PublicKey -> fr.acinq.bitcoin.scalacompat.Crypto.PublicKey
fr.acinq.eclair.channel.RES_GETINFO -> fr.acinq.eclair.channel.RES_GET_CHANNEL_INFO
fr.acinq.eclair.payment.relay.Relayer.UsableBalance -> fr.acinq.eclair.payment.relay.Relayer.ChannelBalanc
fr.acinq.eclair.payment.PaymentRequest -> fr.acinq.eclair.payment.Bolt11Invoice

Incompatible code changes:
The hiding of the ShortChannelID behind RealScidStatus

Channel.shortChannelId() 
->
 ((RealScidStatus.Final) Channel.shortIds().real()).realScid()

The change of the Constructor for PublicKey

new PublicKey(ByteVector.fromValidHex(rs.getString("NextNodeID"), ByteVector.fromValidHex$default$2()) ) 
-> 
new PublicKey(new fr.acinq.bitcoin.PublicKey(fr.acinq.bitcoin.PublicKey.compress(ByteVector.fromValidHex(rs.getString("NextNodeID"), ByteVector.fromValidHex$default$2()).toArray())));

Those changes where all fairly straight forward but it still took me about 3 hours to identify all the correct solutions and change it in all the files.

I have one open problem an this regard. Currently I am unable to create a Channel Update and publish it by hand. I get this error:

2022-12-02 09:00:00,582 ERROR f.a.e.io.PeerConnection SYN n:03423790614f023e3c0cdaa654a3578e919947e4c3a14bf5044e7c787ebd11af1a - private channel update for scid=764005x553x1 was incorrectly added to staggered broadcast
I have not yet figured out, what exactly I have to change in my code.

[t-bast]

Thanks for the feedback! Let's go through these issues one by one.

1.) Error spam during initial synch with peer

This is because we deprecated support for channel updates that don't contain an htlc_maximum_msat field. This was deprecated in the spec a while ago already, implementations have changed their behavior a long time ago to stop creating such updates, but some still relay channel updates from very old, non-updated nodes that don't contain this field. Everyone will drop those updates at some point, but obviously someone has to go first. That's what you see in the logs, but this is completely harmless, you can simply ignore those (but they are wasting bandwidth for no good reason). There is no perfect solution here until your peers have also updated their implementation to be spec-compliant and stop relaying these updates, using a sync-whitelist until they do is a good trade-off.

2.) Extremely high CPU and RAM usage

That very likely linked to the previous point: we have to do a first pass on the DB to read all previously stored channel updates and drop those that don't contain an htlc_maximum_msat field. I'm surprised that it is that expensive on your machine though, because the network DB isn't that big...we have some work to do on improving the start-up performance, that's on our backlog for the next release. It's probably worth opening an issue for that one to make sure it's tracked.

3.) A lot of changed classed that broke all of my plugins.

We migrated to a new library for all bitcoin-related tasks that is shared with our mobile wallet: https://github.com/acinq/lightning-kmp
This is the reason behind most of the build changes. It does require work to update plugins, but there's no good way around that: we cannot guarantee bytecode stability at that point, there is too much work ongoing at the eclair-core level to be able to provide such guarantees.

Incompatible code changes: the hiding of the ShortChannelID behind RealScidStatus

That is linked to the scid-alias spec proposal, that has been added by all implementations. This was a painful change, but required to work with the updated lightning specification. It forces nodes to deal with the fact that channels can be used before they're fully confirmed, which better matches reality (otherwise code could think it's doing the right thing while it's not - now the compiler forces you to handle it correctly).

I have one open problem an this regard. Currently I am unable to create a Channel Update and publish it by hand. I get this error:

A new flag was added to channel updates (linked to the scid-alias proposal) to ensure that channel updates for private channels don't get sent to other nodes (because if they were, you would be doxxing private channels). Is your channel private or public? If it's public, you simply need to correctly set the dontForward = false inside the update's messageFlags:

eclair/eclair-core/src/main/scala/fr/acinq/eclair/wire/protocol/LightningMessageTypes.scala

Line 446 in 3b2eb96

case class MessageFlags(dontForward: Boolean)

If the channel is private, you're not emitting your channel update correctly (and probably previously leaked private channel data to the whole network): if that's the case, you should use the LocalChannelUpdate event:

eclair/eclair-core/src/main/scala/fr/acinq/eclair/channel/ChannelEvents.scala

Line 52 in 3b2eb96

case class LocalChannelUpdate(channel: ActorRef, channelId: ByteVector32, shortIds: ShortIds, remoteNodeId: PublicKey, channelAnnouncement_opt: Option[ChannelAnnouncement], channelUpdate: ChannelUpdate, commitments: AbstractCommitments) extends ChannelEvent {

[DerEwige]

If the channel is private, you're not emitting your channel update correctly (and probably previously leaked private channel data to the whole network): if that's the case, you should use the LocalChannelUpdate event:

I think my problem is that I creat a LocalChannelUpdate and then try to publish it.
That looks to not be the correct way, but was easy and worked in the last version.

Which would be the correct Class to use instead of LocalChannelUpdate for public channels?

That very likely linked to the previous point: we have to do a first pass on the DB to read all previously stored channel updates and drop those that don't contain an htlc_maximum_msat field. I'm surprised that it is that expensive on your machine though, because the network DB isn't that big...we have some work to do on improving the start-up performance, that's on our backlog for the next release. It's probably worth opening an issue for that one to make sure it's tracked.

Is there not also a mechano now to auto deleete expired invoices? Might this have started directly at start-up too?
Or do I need to enable this in the config? (did not check yet)

[t-bast]

Which would be the correct Class to use instead of LocalChannelUpdate for public channels?

Ok so you're confirming that you are using public channels only? Using LocalChannelUpdate is the correct way for that, but you have to make sure you set isPrivate = false when you call makeChannelUpdate:

eclair/eclair-core/src/main/scala/fr/acinq/eclair/router/Announcements.scala

Line 121 in 3b2eb96

def makeChannelUpdate(chainHash: ByteVector32, nodeSecret: PrivateKey, remoteNodeId: PublicKey, shortChannelId: ShortChannelId, cltvExpiryDelta: CltvExpiryDelta, htlcMinimumMsat: MilliSatoshi, feeBaseMsat: MilliSatoshi, feeProportionalMillionths: Long, htlcMaximumMsat: MilliSatoshi, isPrivate: Boolean = false, enable: Boolean = true, timestamp: TimestampSecond = TimestampSecond.now()): ChannelUpdate = {

I'm surprised because that's the default value, so you would need to override it to true to make this fail?

Is there not also a mechano now to auto deleete expired invoices? Might this have started directly at start-up too?

Right, good catch, that's enabled by default, that's probably why it took so long if you have a lot of invoices. Since you have a very specific use-case around invoice generation, you probably want to update the default values for this invoice purger, see the release notes for that feature here: https://github.com/ACINQ/eclair/blob/master/docs/release-notes/eclair-v0.8.0.md#expired-incoming-invoices-now-purged-if-unpaid

[DerEwige]

Ok so you're confirming that you are using public channels only? Using LocalChannelUpdate is the correct way for that, but you have to make sure you set isPrivate = false when you call makeChannelUpdate:

Yes I currently use only public Channels.
But I try to make my code work with private channels to, so I don't use default values but take the current values from the channel.

Also I found the issue.

In the function you want isPrivate and on the old Channel I can only ask isPublic
And I just used Channel.isPublic() instead of !Channel.isPublic() in the function.

Simple mistake from my side.
Works again now. Thanks for the help

Right, good catch, that's enabled by default, that's probably why it took so long if you have a lot of invoices. Since you have a very specific use-case around invoice generation, you probably want to update the default values for this invoice purger, see the release notes for that feature here:

I normally purge the db by hand one a week or so.
But I did not do this before the upgrade so there were about 2 million invoice the be purged.

The restarts after this initial purge worked fine. So I think even 24 hours default would work for me.
But I still changed it down to 2 hours to keep the db small

[dennisreimann]

Can you push a release-0.8.0 to Docker Hub? Right now only latest is available.

https://hub.docker.com/r/acinq/eclair/tags?page=1&ordering=last_updated

[DerEwige]

@t-bast pinging you for this

[t-bast]

This has been published today by @sstone

[dennisreimann]

Thank you!

[sstone]

We automatically publish latest docker images whenever master is updated but we don't tag them when a new release is published, we have to do that manually for now.