BLS: invalid signature messages from drand client due to erroneously contacting devnet

[CluEleSsUK]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

I’m not actually running lotus, but have been informed of the error by operators. It was first reported by why on 2023-01-31, so likely versions from v1.19.0 onwards are affected (see: https://filecoinproject.slack.com/archives/CP50PPW2X/p1675119689004949)

Repro Steps

No response

Describe the Bug

Periodic BLS: invalid signature errors are being emitted from the drand client due to inclusion of devnet endpoints in the production config.
The incentinet config (erroneously) lists the drand devnet (which has no liveness or support guarantees) as its endpoint: https://github.com/filecoin-project/lotus/blob/v1.20.3/build/drand.go#L71-L77

Additionally, it seems that blocks from 0-51000 were generated using this devnet:

lotus/build/params_mainnet.go

Lines 18 to 21 in dcb49dc

	var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
	0: DrandIncentinet,
	UpgradeSmokeHeight: DrandMainnet,
	}

The BLS signature errors happen when the drand client tries speedtesting different endpoints in the background, and tries to validate devnet beacons using the mainnet public key (as their public keys are different).

It doesn't appear to actually be breaking anything, but it may scare operators when they see errors in their logs.

Logging Information

2023-03-14T07:33:21.337Z    INFO    drand   log/log.go:109      {"level": "info", "optimizing_client": "endpoint down when speed tested", "client": "HTTP(\"https://dev2.drand.sh/\").(+verifier)", "err": "verification of {[86 125 71 133 18 42 90 62 117 169 188 153 17 215 234 128 125 216 95 247 107 120 220 79 240 107 7 87 18 137 134 7] 1 [148 106 123 169 26 108 161 249 246 115 6 255 106 108 243 56 5 188 80 63 213 39 16 122 235 16 216 149 107 104 102 80 40 95 6 9 24 5 146 23 9 73 161 160 6 248 209 98 22 78 207 89 61 219 72 224 18 159 150 102 101 212 182 45 104 230 164 65 4 221 160 102 165 225 19 151 73 245 136 23 161 247 216 173 202 174 80 64 154 232 211 189 21 75 38 228]} failed: bls: invalid signature"}
2023-03-14T07:38:21.350Z    INFO    drand   log/log.go:109      {"level": "info", "optimizing_client": "endpoint down when speed tested", "client": "HTTP(\"https://dev1.drand.sh/\").(+verifier)", "err": "verification of {[86 125 71 133 18 42 90 62 117 169 188 153 17 215 234 128 125 216 95 247 107 120 220 79 240 107 7 87 18 137 134 7] 1 [148 106 123 169 26 108 161 249 246 115 6 255 106 108 243 56 5 188 80 63 213 39 16 122 235 16 216 149 107 104 102 80 40 95 6 9 24 5 146 23 9 73 161 160 6 248 209 98 22 78 207 89 61 219 72 224 18 159 150 102 101 212 182 45 104 230 164 65 4 221 160 102 165 225 19 151 73 245 136 23 161 247 216 173 202 174 80 64 154 232 211 189 21 75 38 228]} failed: bls: invalid signature"}
2023-03-14T07:38:21.362Z    INFO    drand   log/log.go:109      {"level": "info", "optimizing_client": "endpoint down when speed tested", "client": "HTTP(\"https://dev2.drand.sh/\").(+verifier)", "err": "verification of {[86 125 71 133 18 42 90 62 117 169 188 153 17 215 234 128 125 216 95 247 107 120 220 79 240 107 7 87 18 137 134 7] 1 [148 106 123 169 26 108 161 249 246 115 6 255 106 108 243 56 5 188 80 63 213 39 16 122 235 16 216 149 107 104 102 80 40 95 6 9 24 5 146 23 9 73 161 160 6 248 209 98 22 78 207 89 61 219 72 224 18 159 150 102 101 212 182 45 104 230 164 65 4 221 160 102 165 225 19 151 73 245 136 23 161 247 216 173 202 174 80 64 154 232 211 189 21 75 38 228]} failed: bls: invalid signature"}
2023-03-14T07:43:21.379Z    INFO    drand   log/log.go:109      {"level": "info", "optimizing_client": "endpoint down when speed tested", "client": "HTTP(\"https://dev1.drand.sh/\").(+verifier)", "err": "verification of {[86 125 71 133 18 42 90 62 117 169 188 153 17 215 234 128 125 216 95 247 107 120 220 79 240 107 7 87 18 137 134 7] 1 [148 106 123 169 26 108 161 249 246 115 6 255 106 108 243 56 5 188 80 63 213 39 16 122 235 16 216 149 107 104 102 80 40 95 6 9 24 5 146 23 9 73 161 160 6 248 209 98 22 78 207 89 61 219 72 224 18 159 150 102 101 212 182 45 104 230 164 65 4 221 160 102 165 225 19 151 73 245 136 23 161 247 216 173 202 174 80 64 154 232 211 189 21 75 38 228]} failed: bls: invalid signature"}
2023-03-14T07:43:21.392Z    INFO    drand   log/log.go:109      {"level": "info", "optimizing_client": "endpoint down when speed tested", "client": "HTTP(\"https://dev2.drand.sh/\").(+verifier)", "err": "verification of {[86 125 71 133 18 42 90 62 117 169 188 153 17 215 234 128 125 216 95 247 107 120 220 79 240 107 7 87 18 137 134 7] 1 [148 106 123 169 26 108 161 249 246 115 6 255 106 108 243 56 5 188 80 63 213 39 16 122 235 16 216 149 107 104 102 80 40 95 6 9 24 5 146 23 9 73 161 160 6 248 209 98 22 78 207 89 61 219 72 224 18 159 150 102 101 212 182 45 104 230 164 65 4 221 160 102 165 225 19 151 73 245 136 23 161 247 216 173 202 174 80 64 154 232 211 189 21 75 38 228]} failed: bls: invalid signature"}

[CluEleSsUK]

Tagging you @jennijuju because I don't have the magical powers of assignment ;;

[jsoares]

This is not the first time that drand past network config causes confusion. See eg #4125 (but different cause and solution).

The bug description is accurate but I would suggest that the listing of wrong endpoints is a distraction, even if it triggered the log entries. Rather, we should be ignoring endpoints for past networks, and probably not require them to remain in the config in the first place. There is no point in speed-checking an endpoint from which you won't need fast randomness in the present or future, and they may no longer be active at all.

The ChanInfo must, of course, remain, so that we can validate old on-chain randomness.

[jsoares]

As pointed out by @ribasushi, the incentinet points were actually removed at one point (so not required, good), and then re-added in #9654 (comment).