Support new Lotus API + calibration network #529

jsign · 2020-07-20T14:31:05Z

This PR targets the ntwk-calibration Lotus branch, which is a newer codebase with breaking APIs, this means it can't coexist supporting testnet nor current nerpa. ntwk-calibration code will be the next nerpa network.

The command make up now prints a message saying that current Testnet isn't supported since is still running a Lotus version with old broken APIs. Whenever is updated we can re-enable that. The message suggests running make nerpa-up so to help the user connect to some network.

Apart from changes in this repo, there were many changes in others. In particular, upgrading textileio/lotus-devnet and textileio/lotus-build. In the process of upgrading lotus-devnet which is a pillar of the CI, we've discovered multiple bugs which ended up being resolved: e.g: here, here, and here.

Considering that lotus@master is still targeting testnet, I made textileio/lotus-devnet have multiple branches per network. So, the newest one which Powergate is using now is lotus-devnet@ntwk-calibration which is the localnet running the latest code of lotus. Whenever master has the latest code, we might abandon that branch.

While updating Powergate to the latest Lotus APIs, I took advantage and did other changes:

Avoid panicking scenarios in test favoring test failing.
Gas limits were unset in sending txn since now new logic about gas-limits exists in Lotus. This means that max-gas and gas-price will be estimated by the Lotus node using stat data.
New Lotus code for localnet seems heavier. I split the integration tests in multiple packages to we can cache passing results better to rerun if some flaky-test fails. The CI is a bit painful sometimes. I also created a helper artifact that retries a test N times before failing, this reduced a lot flakyness and need to rerun.
As a side note, the localnet with bigsectors is very unstable because some extra changes are needed in sector-storage. For more info see this and the mentioned PR there. I guess eventually that would be totally fixed. Not much to do now.
The network has defined now a minimum deal duration which is 180 days. This is now a constant for Powergate that is used to check deal duration validations.
I changed the chain speed for localnet to be faster since now deals takes more epoch to finish. Now with one block per 500ms, they take less than 2 min (with big sectors).

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign · 2020-08-10T19:26:46Z

cmd/powbench/runner/runner.go

-				DealMinDuration: 1000,
+				DealMinDuration: util.MinDealDuration,


Now min deal duration isn't arbitrary for Filecoin, so it's a constant.

jsign · 2020-08-10T19:27:18Z

deals/module/deals.go

+	if minDuration < util.MinDealDuration {
+		return nil, fmt.Errorf("duration %d should be greater or equal to %d", minDuration, util.MinDealDuration)
+	}


Early check that the duration is >= the minimum defined by spec. If it isn't, fail fast.

jsign · 2020-08-10T19:27:50Z

deals/module/deals.go

@@ -116,7 +119,8 @@ func (m *Module) Store(ctx context.Context, waddr string, dataCid cid.Cid, piece
 		}
 		params := &api.StartDealParams{
 			Data: &storagemarket.DataRef{
-				Root: dataCid,
+				TransferType: storagemarket.TTGraphsync,


Necessary to have deals working in new Lotus version.

jsign · 2020-08-10T19:29:10Z

deals/module/deals_test.go

-			require.NoError(t, err)
-			require.Len(t, final, nm)
+			t.Parallel()
+			tests.RunFlaky(t, func(t *tests.FlakyT) {


tests.RunFlaky is a helper that runs a tests and consider that fails if fails 5 times in a row.

jsign · 2020-08-10T19:32:00Z

ffs/integrationtest/config/config_test.go

@@ -0,0 +1,279 @@
+package config


Integration tests got partitioned in their own packages. This to allow if some test fail, only invalidate a subset of them in the test cache. This was more relevant before I did the tests.RunFlaky thing, but I kept this change since provides also better organization; the integration test file was very big.

jsign · 2020-08-10T19:41:10Z

tests/flaky.go

@@ -0,0 +1,65 @@
+package tests


The FlakyT helper I created to provide some retry at the integration test level.
Might be useful in other projects.

jsign · 2020-08-10T19:43:33Z

tests/ldevnet.go

 // LaunchDevnetDocker launches the devnet docker image.
-func LaunchDevnetDocker(t *testing.T, numMiners int, ipfsMaddr string, mountVolumes bool) *dockertest.Resource {
+func LaunchDevnetDocker(t TestingTWithCleanup, numMiners int, ipfsMaddr string, mountVolumes bool) *dockertest.Resource {


This *testing.T to TestinTWithCleanup replacement happened in some places.
*testing.T is a concrete type, and we want LaunchDevnetDocker to work with *testing.T but also work with our "FlakyT" runner. If things would still use *testing.T and call Fail() that would mess up retries.

TestingTWithCleanup has the same interface as *testing.T so the user shouldn't be worried about anything extra. Just do whatever usually would do, and FlakyT will do whatever it needs.

jsign · 2020-08-10T19:46:20Z

tests/ldevnet.go

-	if err != nil {
-		panic(fmt.Sprintf("couldn't create ipfs-pool: %s", err))
-	}
+	require.NoError(t, err)


TestingTWithCleanup is compatible with require, so can be used as "nothing happened".

jsign · 2020-08-10T19:51:07Z

wallet/module/wallet.go

-			From:     m.masterAddr,
-			To:       addr,
-			Value:    types.BigInt{Int: m.iAmount},
-			GasLimit: 1000,
-			GasPrice: types.NewInt(0),
+			From:  m.masterAddr,
+			To:    addr,
+			Value: types.BigInt{Int: m.iAmount},


A lot happened here while tracking different releases of Lotus code.
To make the story short: this way the Lotus node will decide the most reasonable Gas price considering recent history of gas prices.

jsign · 2020-08-11T13:00:02Z

docker/Makefile

@@ -1,19 +1,10 @@
+COMPAT_MSG="The current version of Powergate support newer versions than Testnet network. You might be interested in using the Calibration network (make calibration-up)"


Some message that gets printed if make up is executed. Will eventually go away when Testnet resets to latest code.

asutula

Lots of changes, but it all makes sense. Probably the crazier changes were in the other repos we depend on here. Nice work.

asutula · 2020-08-11T15:57:02Z

deals/module/deals_test.go

-			require.NoError(t, err)
-			require.Len(t, final, nm)
+			t.Parallel()
+			tests.RunFlaky(t, func(t *tests.FlakyT) {


asutula · 2020-08-11T15:58:59Z

ffs/integrationtest/config/config_test.go

@@ -0,0 +1,279 @@
+package config


jsign · 2020-08-11T16:07:56Z

Lots of changes, but it all makes sense. Probably the crazier changes were in the other repos we depend on here. Nice work.

You can be totally sure about that 😅

jsign added this to the Sprint 41 milestone Jul 20, 2020

jsign self-assigned this Jul 20, 2020

jsign force-pushed the jsign/networkbutter branch from aa15e7d to d5c4adf Compare July 20, 2020 22:01

jsign changed the title ~~Support Butterfly network~~ Support next nerpa network Jul 22, 2020

jsign force-pushed the jsign/networkbutter branch 3 times, most recently from d93c590 to 1077b6b Compare July 23, 2020 13:36

weekly-digest bot mentioned this pull request Jul 24, 2020

Weekly Digest (17 July, 2020 - 24 July, 2020) #537

Closed

jsign force-pushed the jsign/networkbutter branch 2 times, most recently from 20c333b to c72b696 Compare July 30, 2020 17:24

jsign modified the milestones: Sprint 41, Sprint 44 Jul 30, 2020

jsign force-pushed the jsign/networkbutter branch from d412b0d to 9030d80 Compare July 30, 2020 22:39

weekly-digest bot mentioned this pull request Jul 31, 2020

Weekly Digest (24 July, 2020 - 31 July, 2020) #542

Closed

jsign force-pushed the jsign/networkbutter branch from 9030d80 to d85379e Compare July 31, 2020 17:23

jsign modified the milestones: Sprint 44, Sprint 42 Jul 31, 2020

jsign force-pushed the jsign/networkbutter branch 5 times, most recently from cb64afc to 76252c6 Compare August 6, 2020 17:36

weekly-digest bot mentioned this pull request Aug 7, 2020

Weekly Digest (31 July, 2020 - 7 August, 2020) #563

Closed

jsign force-pushed the jsign/networkbutter branch 2 times, most recently from 7a12351 to 76430af Compare August 8, 2020 20:57

jsign added 4 commits August 10, 2020 09:16

rebase

14fe237

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

print msg when trying using testnet

17837d8

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update to next branch

8be63a9

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

rebase

2ad136d

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign added 13 commits August 10, 2020 09:16

more parallel

6f6cc27

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

try flaky stragety

9172d64

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix

3439e15

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix flakyness on general test

2c7a0e5

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

improve test deflakyness

c631beb

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fixes

9c495b0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

more improvements

5079925

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix retry message

6b1d1ef

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

reorg

7d72f66

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

faster devnet

336e12c

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

slow down

e0d0260

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

move flaky to own file

58c1467

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix all

f562c74

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign force-pushed the jsign/networkbutter branch from ca8d3d8 to bfa79fc Compare August 10, 2020 12:18

update to 8.8.0

49a9acb

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign force-pushed the jsign/networkbutter branch from bfa79fc to 49a9acb Compare August 10, 2020 12:20

jsign changed the title ~~Support next nerpa network~~ Support new Lotus API + calibration network Aug 10, 2020

jsign added 2 commits August 10, 2020 16:25

support calibration network

583c27b

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

calib changes and localnet speed

ca62a41

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign modified the milestones: Sprint 42, Sprint 43 Aug 11, 2020

jsign commented Aug 11, 2020

View reviewed changes

jsign marked this pull request as ready for review August 11, 2020 13:00

jsign requested a review from asutula August 11, 2020 13:00

asutula approved these changes Aug 11, 2020

View reviewed changes

jsign merged commit 5ee1531 into master Aug 11, 2020

jsign deleted the jsign/networkbutter branch August 11, 2020 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support new Lotus API + calibration network #529

Support new Lotus API + calibration network #529

jsign commented Jul 20, 2020 •

edited

Loading

jsign Aug 10, 2020

jsign Aug 10, 2020

jsign Aug 10, 2020

jsign Aug 10, 2020

asutula Aug 11, 2020

jsign Aug 10, 2020

asutula Aug 11, 2020

jsign Aug 10, 2020

jsign Aug 10, 2020

jsign Aug 10, 2020

jsign Aug 10, 2020

jsign Aug 11, 2020

asutula left a comment

asutula Aug 11, 2020

asutula Aug 11, 2020

jsign commented Aug 11, 2020

		DealMinDuration: 1000,
		DealMinDuration: util.MinDealDuration,

		@@ -1,19 +1,10 @@
		COMPAT_MSG="The current version of Powergate support newer versions than Testnet network. You might be interested in using the Calibration network (make calibration-up)"

Support new Lotus API + calibration network #529

Support new Lotus API + calibration network #529

Conversation

jsign commented Jul 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign commented Aug 11, 2020

jsign commented Jul 20, 2020 •

edited

Loading