Very delayed Router fast path transfers (12-18min)

[alexwhte]

Another (small) tx stuck: https://connextscan.io/tx/0xa2366577259abb0da2a873870b620a251a6556991a5ae1eceb18f866b1324b09?source=search
Transfer | Connextscan
Transfer | Connextscan
Explorer for the Connext protocol
Image
This should be treated as ultra critical, including for token launch
if our routers aren't bidding properly, then this will mean our crosschain airdrop totally fails
cc/ @alexblanq
By the way, this has happened every single time i have used Connext in the past two weeks. Completely unusable
not sure how this is going to work for partners like fuji btw. The above bridge transaction is to GC to fill my wallet so I could play with the Fujii flow. I specifically picked GC because lots and lots of liquidity. I'm willing to bet rn that we see exact same issue with needing to go through slow path when I finally get liquidity and try to use Fuji's dapp
We likely need to escalate this into a p0 to investigate
arjun — Today at 1:51 PM
Completed. Odd?
eddie | MST — Today at 1:51 PM
can't glean from logs exactly what happened here, looks like routers indeed did not bid on this initially
arjun — Today at 1:51 PM
Why did it take 18 minutes?
eddie | MST — Today at 1:51 PM
but yeah it completed just now
arjun — Today at 1:51 PM
Yea this is very weird and bad
@preetham | PST cc^
Preetham | PST — Today at 2:09 PM
Looking into this
arjun — Today at 2:09 PM
Another: https://connextscan.io/tx/0x108521bee91ec023fa7fc3a0e42d60d0862a77f8e5d3de576fca4be6ed2a7eb6

[preethamr]

Routers that bid for https://connextscan.io/tx/0x108521bee91ec023fa7fc3a0e42d60d0862a77f8e5d3de576fca4be6ed2a7eb6:

https://app.datadoghq.com/logs?query=region%3Aus-east-2%20%40requestContext.transferId%3A0x108521bee91ec023fa7fc3a0e42d60d0862a77f8e5d3de576fca4be6ed2a7eb6%20-service%3A%28router-publisher%20OR%20router-subscriber%29&agg_q=%40bid.router&cols=host%2Cservice%2C%40bid.router&index=&messageDisplay=inline&sort_m=&sort_t=&stream_sort=time%2Cdesc&top_n=10&top_o=top&viz=sunburst&x_missing=true&from_ts=1673810905780&to_ts=1673897305780&live=true

[preethamr]

Router that bid for 0x9ee68898913a7c062f893c57969afdc5dfd8cd06b51c3ad4f3229a92b1bfae48:

https://app.datadoghq.com/logs?query=region%3Aus-east-2%20%40requestContext.transferId%3A0x9ee68898913a7c062f893c57969afdc5dfd8cd06b51c3ad4f3229a92b1bfae48%20-service%3A%28router-publisher%20OR%20router-subscriber%29%20&agg_q=%40bid.router&cols=host%2Cservice%2C%40bid.router&index=&messageDisplay=inline&sort_m=&sort_t=&stream_sort=time%2Cdesc&top_n=10&top_o=top&viz=sunburst&x_missing=true&from_ts=1673810905780&to_ts=1673897305780&live=true

[preethamr]

Likely root cause:
Sequencer does not update auctions data structure, on subsequent bids if this check is false (which is the case when there is a pending execution):

monorepo/packages/agents/sequencer/src/lib/operations/execute/fastpath.ts

Line 81 in 7e80436

if (status === ExecStatus.None) {

This is where we update auctions:

monorepo/packages/agents/sequencer/src/lib/operations/execute/fastpath.ts

Lines 97 to 108 in 7e80436

	// Update and/or create the auction instance in the cache if necessary.
	const res = await cache.auctions.upsertAuction({
	transferId,
	origin: transfer.xparams!.originDomain,
	destination: transfer.xparams!.destinationDomain!,
	bid,
	});
	logger.info("Updated auction", requestContext, methodContext, {
	new: res === 0,
	auction: await cache.auctions.getAuction(transferId),
	status: await cache.auctions.getExecStatus(transferId),
	});