Splitting lotus codebase into multiple modules to improve reusability across projects

[nicobao]

Hey!

At CIDgravity we rely on lotus as submodule to build our go backend because lotus ships in one module that must build filecoin-ffi locally:

go.mod

require (
	github.com/filecoin-project/lotus v1.16.0
)

replace github.com/filecoin-project/lotus => ./extern/lotus

and in individual files of our packages, we import those lotus packages:

some_file.go

import (
	"github.com/filecoin-project/lotus/api"
	"github.com/filecoin-project/lotus/chain/actors/builtin" 
	"github.com/filecoin-project/lotus/chain/types"
	"github.com/filecoin-project/lotus/lib/sigs"
)

Besides importing lots of code that we don't actually use, it adds a lot of complexity in our build process because lotus module requires filecoin-ffi build (rust) which is slow, platform-dependent, requiring manual make.

I used https://github.com/KyleBanks/depth to inspect the dependency tree. And I discovered that, according to my research, the packages we actually use from lotus module (see above) aren't dependent on filecoin-ffi! We would not need this slow make step, we could theoretically just import pure go modules.

I started working on publishing separate modules for lotus/api, lotus/chain/types, lotus/chain/actors and lotus/lib/sigs. This will require separate modules for at least:

• lotus/chain/actors
• lotus/blockstore
• lotus/node/repo/imports
• lotus/storage/pipeline/sealiface

The goal is that I would be able to import only the modules we need instead of the whole lotus:
go.mod

require (
	"github.com/filecoin-project/lotus/api" v1.1.0
	"github.com/filecoin-project/lotus/chain/actors/builtin"  v1.2.0
	"github.com/filecoin-project/lotus/chain/types" v1.3.0
	"github.com/filecoin-project/lotus/lib/sigs" v1.1.1
)

Since go 1.18, golang supports go workspace, which would enable us to handle the creation of go modules directly in the lotus repository, instead of maintaining plenty of repositories. It would require a good chunk of refactoring. But having those modules in the same repo will help tremendously for keeping track of version compatibility, so I still think it's the best approach.

I'd like to get your feedback:

• have anyone of you faced the same issue?
• to lotus devs: which modules would be appropriate to create and what is the best approach in your opinion?

[nicobao]

lotus/chain/consensus and lotus/chain/vm obviously rely on filecoin-ffi. But lotus/chain/actors/builtin and lotus/chain/types that we use don't. It would be nice to isolate modules that use filecoin-ffi from code that doesn't.

From what I've seen, the only packages that rely on filecoin-ffi are:

• lotus/chain/vm
• lotus/chain/consensus
• lotus/storage/sealer
• lotus/storage/sealer/ffiwrapper
• lotus/lib/sigs/bls
• lotus/storage/path

For which reasons would one need to integrate with these modules?
I think you will put most of the apis to access the chain in lotus/api like you did with lotus v1.16.0 (moving stuff from lotus/miner to lotus/api). Right?

[iand]

You may find ffi-stub useful if you want to import code that depends on FFI but doesn't use it.

[aschmahmann]

ffi-stub is useful, but note that using it precludes installing binaries with go install due to golang/go#44840. Also, by being a replace directive it means it has to be included at the binary level not library so if someone wants to make libraries that wrap some lotus functionality they have to tell their consumers to use ffi-stub as well.