Limit maximum number of DAG links to traverse

[hannahhoward]

Goals

Set reasonable limits on number of links traversed while calculating commP & doing go-graphsync transfers

Blockers:

• update to tagged go-ipld-prime 0.12.3
• merge Expose MaxTraversalLinks option for generating CommP go-fil-markets#633
• Add traversals: limit link load count for SelectiveCar use as well #7428
• update to tagged go-fil-markets
• rebase on master and change the base for this PR when Update to graphsync to v0.10.0, enable seperate storage and retrieval transfer limits #7405 merges with tagged go-graphsync
• Validate the error condition locally by setting the environment variable low. The goals here are: 1) Confirm the limit works 2) See what the actual error message in the logs is when we hit the limit. Paste the error message in this PR.
• Sanity check by running a manual storage and retrieval deal against production

Implementation

• update go-graphsync and go-car and go-fil-markets
• set a config.MaxTraversalLinks
• allow it to be overridden via ENV
• pass to initialization code for markets & go-graphsync

[rvagg]

Is this the only place that graphsync is initiated? It's the only one I could find but do we have other avenues, like go-datatransfer? Otherwise we're relying on unbounded graphsync

[rvagg]

#7428 was intended to be in here as well, there's a couple of SelectiveCar uses locally that aren't deferred via gfm.

[codecov]

Codecov Report

Merging #7427 (6d29552) into feat/update-graphsync-0.10.0 (a4e2fce) will increase coverage by 0.55%.
The diff coverage is 89.28%.

❗ Current head 6d29552 differs from pull request most recent head 97c2cdb. Consider uploading reports for the commit 97c2cdb to get more accurate results

@@                       Coverage Diff                        @@
##           feat/update-graphsync-0.10.0    #7427      +/-   ##
================================================================
+ Coverage                         38.94%   39.50%   +0.55%     
================================================================
  Files                               616      616              
  Lines                             65297    65319      +22     
================================================================
+ Hits                              25432    25806     +374     
+ Misses                            35462    35010     -452     
- Partials                           4403     4503     +100     

Impacted Files	Coverage Δ
node/config/def.go	97.38% <33.33%> (-1.29%)	⬇️
node/impl/client/client.go	40.73% <83.33%> (+0.53%)	⬆️
node/modules/client.go	74.03% <100.00%> (ø)
node/modules/graphsync.go	73.07% <100.00%> (+9.91%)	⬆️
node/modules/storageminer.go	62.93% <100.00%> (+0.43%)	⬆️
markets/loggers/loggers.go	89.28% <0.00%> (-10.72%)	⬇️
extern/sector-storage/sched_resources.go	75.00% <0.00%> (-9.38%)	⬇️
chain/stmgr/call.go	67.87% <0.00%> (-3.64%)	⬇️
node/hello/hello.go	63.63% <0.00%> (-3.41%)	⬇️
extern/sector-storage/sched.go	83.05% <0.00%> (-0.83%)	⬇️
... and 30 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4e2fce...97c2cdb. Read the comment docs.

[rvagg]

Rebased onto current #7405 which has been rebased to master.

[rvagg]

Pulled in go-fil-markets@v1.13.0 which pulls in the newly tagged go-data-transfer@v1.11.0 which therefore incorporates #7441 into this branch.

[rvagg]

So, the easiest test of this I think is to start the daemon with LOTUS_MAX_TRAVERSAL_LINKS=10 lotus daemon

Importing a file of decent size (in this case just the built lotus binary) using: lotus client generate-car <input> <output> yields:

ERROR: failed to write CAR to output file: traversal budget exceeded: budget for links reached zero while on path "Links/10/Hash" (link: "bafk2bzacecdzw7j4fcqxwcgad7zb4rehej7uqa4a2263dhoglkcgt465p32qy")

There are some other places that are a bit more difficult to run manual tests on without an operating storage network.

[dirkmc]

Version Compatibility Testing

Old: lotus v1.11.3
New: This branch

• New client / Old server
  • Storage ✅
  • Retrieval ✅
• New client / New server
  • Storage ✅
  • Retrieval ✅
• Old client / New server
  • Storage ✅
  • Retrieval ✅