Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't gzip openrpc files for friendlier git wrangling #11625

Merged
merged 1 commit into from
Feb 15, 2024

Conversation

rvagg
Copy link
Member

@rvagg rvagg commented Feb 14, 2024

The .gz files are impossible to rebase/merge/etc. but the files are needed in the repo for go build because they are embeds; so we just don't gzip them. In this format they are also pretty-printed JSON, so they'll merge very nicely, but they are not compressed.

The trade-off here is that the files are embedded in the go builds, so we get larger binaries, here's the differences:

  • lotus = 238 Mb (new) vs 236 Mb (old)
  • lotus-miner = 223 Mb (new) vs 221 Mb (old)
  • lotus-worker = 158 Mb (new) vs 156 Mb (old)

(They all include all of the files, even if they don't use them, we could change that if we wanted because it's only full that's huge)

As @Stebalien was pointing out today, the reason we include these files at all in the library is so that go build theoretically works. Currently the only thing standing in the way of being able to use lotus as a library is the FFI, removing these files entirely would add another blocker to supporting that. See also (I believe) filecoin-project/filecoin-ffi#209

the .gz files are impossible to rebase/merge/etc. but the files are needed in
the repo for `go build` because they are embeds; so we just don't gzip them
@rvagg rvagg requested a review from a team as a code owner February 14, 2024 22:04
@Stebalien
Copy link
Member

I looked into making the build faster as well by avoiding goimports and... it just made a mess. I think the right way forward is to make it possible to call make -j16.

@snadrus
Copy link
Collaborator

snadrus commented Feb 15, 2024

Could we provide a redirect or proxy to these files in some static place?

If a release is dev, no. But it's rare someone is writing code off these in dev anyway, right?

So, for regular releases, we just 302 to where these files should live in some static folder that relates to our current version.

Then, it's part of the release process to copy these files to the static place that we store params.

@snadrus snadrus merged commit 080baf4 into master Feb 15, 2024
88 checks passed
@snadrus snadrus deleted the rvagg/openrpc-git-friendly branch February 15, 2024 01:51
@snadrus
Copy link
Collaborator

snadrus commented Feb 15, 2024

Accepted, but please determine if the protocol & users see these as optional.
If so: gitignore + custom build rule + docs for those that need it.

ribasushi added a commit to ribasushi/ci-abusing-lotus-fork that referenced this pull request Jun 16, 2024
ribasushi added a commit to ribasushi/ci-abusing-lotus-fork that referenced this pull request Jun 16, 2024
ribasushi added a commit to ribasushi/ci-abusing-lotus-fork that referenced this pull request Jun 16, 2024
aarshkshah1992 pushed a commit that referenced this pull request Jun 17, 2024
* chore: cleanup: remove more items forgotten after markets

* .gz somehow reappeared after #11625
rjan90 pushed a commit that referenced this pull request Jun 17, 2024
* chore: cleanup: remove more items forgotten after markets

* .gz somehow reappeared after #11625
jennijuju pushed a commit that referenced this pull request Jun 19, 2024
* fix: ci: do not use deprecated --debug goreleaser flag (#12086)

* chore: deals: remove forgotten graphsync references (#12084)

* chore: types: remove more items forgotten after markets (#12095)

* chore: cleanup: remove more items forgotten after markets

* .gz somehow reappeared after #11625

* fix: ETH RPC API: ETH Call should use the parent state root of the subsequent tipset (#11905)

* fix eth call

* tests

* changes as per review

* changes as per review

* Update node/impl/full/eth.go

Co-authored-by: Rod Vagg <rod@vagg.org>

* fix as per review

---------

Co-authored-by: Rod Vagg <rod@vagg.org>

* Update changelog to RC2

Update changelog to RC2

* Make gen / make docsgen-cli

Make gen / make docsgen-cli

* chore: api: the Net API/CLI now remains only on daemon

The only part of this repository that does lp2p is now lotus-daemon

Remove the CommonNet type, used exclusively bu the CLI stack

Adjust the rest of struct-memebership to match what went where

End result best seen in diff of `documentation/en/api-v0-methods-miner.md`

* Update changelog

Update changelog

* fix: events: sqlite db improvements (#12090)

* fix: events: sqlite db improvements

* fix unclosed multi-row query
* tune options to limit wal growth

Ref: #12089

* fix: events: use correct context for CollectEvents transaction

* fix: events: close prepared read statement

* fix: events: close initial query; handle lint failures

* Update CHANGELOG.md

---------

Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Peter Rabbitson <ribasushi@leporine.io>
jennijuju added a commit that referenced this pull request Jun 25, 2024
* release: v1.26.3 (#11908) (#11915)

* deps: update dependencies to address migration memory bloat

to address memory concerns during a heavy migration

Ref: filecoin-project/go-state-types#260
Ref: whyrusleeping/cbor-gen#96
Ref: filecoin-project/go-amt-ipld#90

* release: prep v1.26.3 patch

Prep v1.26.3 patch release:
- Update changelog, version and make gen + make docsgen-cli

* deps: update cbor-gen to tagged version

deps: update cbor-gen to tagged version

* deps: update go-state-types to tagged version

deps: update go-state-types to tagged version v0.13.2

* chore: deps: update go-state-types to v0.13.3

Fixes a panic when we have fewer than 1k proposals.

---------

Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steven Allen <steven@stebalien.com>

* build: release: v1.27.0-rc1 (#11947)

* chore: Set version as v1.27.0-rc1

Set version as v1.27.0-rc1, run make gen & make docsgen-cli

* Update changelog

Update changelog

* Update changelog

Update changelog based on feedback

* Bump pubsub-dep

Bump pubsub-dep

* Prep v1.27.0-rc2

Prep v1.27.0-rc2

* Typo fixes, and more changelog updates

Typo fixes, and more changelog updates

* chore: remove unmaintained bootstrappers (#11983)

* chore: remove unmaintained bootstrappers

chore: remove unmaintained bootstrappers

* Update mainnet.pi fixing typoed domain

fixing typo for 1475.io 'bootstarp' -> 'bootstrap'

* Update mainnet.pi

apparently the actual hostname is typoed. so bootstarp it is.

---------

Co-authored-by: smagdali <stefan@fil.org>

* chore: update go-data-transfer and go-graphsync

* add ETH addrs API to Gateway (#11979)

* fix: copy Flags field from SectorOnChainInfo

Fixes: #11962

* feat: libp2p: Lotus stream cleanup (#11993)

* set stream deadlines in Lotus

* reduce timeout

* whitelist bootstrappers

* fix tests

* Update changelog and version

Update changelog and version

* ci: deprecate circle ci in favour of github actions (#11786)

* Update changelog

Update changelog with the deprecate circle-ci

* chore: update drand (#12021)

* Update changelog / make docsgen

Update changelog / make docsgen

* chore: lint: update golangci lint config

* remove and replace some linters
* remove some exclusions
* make all exclusions more explicit matches

* chore: lint: fix lint errors with new linting config

Ref: #11967

* chore: lint: address feedback from reviews

* doc: eth: restore comment lost in linter cleanup

Ref: #11968

* chore: libp2p: update to v0.34.1 (#12027)

* update libp2p to v0.34.0

* fix libp2p err

* fix imports

* update go mod

* update go mod

* Update changelog

Update changelog

* go mod tidy

go mod tidy

* revert go version change (#12050)

* Update changelog

Update changelog

* chore: backport #12054 to release/v1.27.0 branch (#12056)

* chore: pin golanglint-ci to v1.58.2 (#12054)

Fixes: #12044

* Add backport to changelog

Add backport to changelog

---------

Co-authored-by: Rod Vagg <rod@vagg.org>

* Bump version - make gen/make docsgen

Bump version - make gen/make docsgen

* Update changelog

Update changelog

* Bump NodeBuildVersion to v1.27.1-rc1

Bump NodeBuildVersion to v1.27.1-rc1

* Add Lotus-Miner / Curio related changes

Add Lotus-Miner / Curio related changes

* Update date and upgrade warnings

Update date and upgrade warnings

* fix: ci: do not use deprecated --debug goreleaser flag (#12086)

* chore: deals: remove forgotten graphsync references (#12084)

* chore: types: remove more items forgotten after markets (#12095)

* chore: cleanup: remove more items forgotten after markets

* .gz somehow reappeared after #11625

* fix: ETH RPC API: ETH Call should use the parent state root of the subsequent tipset (#11905)

* fix eth call

* tests

* changes as per review

* changes as per review

* Update node/impl/full/eth.go

Co-authored-by: Rod Vagg <rod@vagg.org>

* fix as per review

---------

Co-authored-by: Rod Vagg <rod@vagg.org>

* Update changelog to RC2

Update changelog to RC2

* Make gen / make docsgen-cli

Make gen / make docsgen-cli

* chore: api: the Net API/CLI now remains only on daemon

The only part of this repository that does lp2p is now lotus-daemon

Remove the CommonNet type, used exclusively bu the CLI stack

Adjust the rest of struct-memebership to match what went where

End result best seen in diff of `documentation/en/api-v0-methods-miner.md`

* Update changelog

Update changelog

* fix: events: sqlite db improvements (#12090)

* fix: events: sqlite db improvements

* fix unclosed multi-row query
* tune options to limit wal growth

Ref: #12089

* fix: events: use correct context for CollectEvents transaction

* fix: events: close prepared read statement

* fix: events: close initial query; handle lint failures

* Update CHANGELOG.md

* build: release: v1.27.1-rc2 (#12101)

* fix: ci: do not use deprecated --debug goreleaser flag (#12086)

* chore: deals: remove forgotten graphsync references (#12084)

* chore: types: remove more items forgotten after markets (#12095)

* chore: cleanup: remove more items forgotten after markets

* .gz somehow reappeared after #11625

* fix: ETH RPC API: ETH Call should use the parent state root of the subsequent tipset (#11905)

* fix eth call

* tests

* changes as per review

* changes as per review

* Update node/impl/full/eth.go

Co-authored-by: Rod Vagg <rod@vagg.org>

* fix as per review

---------

Co-authored-by: Rod Vagg <rod@vagg.org>

* Update changelog to RC2

Update changelog to RC2

* Make gen / make docsgen-cli

Make gen / make docsgen-cli

* chore: api: the Net API/CLI now remains only on daemon

The only part of this repository that does lp2p is now lotus-daemon

Remove the CommonNet type, used exclusively bu the CLI stack

Adjust the rest of struct-memebership to match what went where

End result best seen in diff of `documentation/en/api-v0-methods-miner.md`

* Update changelog

Update changelog

* fix: events: sqlite db improvements (#12090)

* fix: events: sqlite db improvements

* fix unclosed multi-row query
* tune options to limit wal growth

Ref: #12089

* fix: events: use correct context for CollectEvents transaction

* fix: events: close prepared read statement

* fix: events: close initial query; handle lint failures

* Update CHANGELOG.md

---------

Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Peter Rabbitson <ribasushi@leporine.io>

* small fix in changelog

* fix: release: update goreleaser config file

Fixes: #12120

* fix go releaser and test with rc3

* Update CHANGELOG.md

* lotus v1.27.1 prep

* address review
- resolve one more conflicts
- revert 2 new line added

* doc: events: note events db migration impact

---------

Co-authored-by: Phi-rjan <orjan.roren@gmail.com>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: smagdali <stefan@fil.org>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
Co-authored-by: Piotr Galar <piotr.galar@gmail.com>
Co-authored-by: Peter Rabbitson <ribasushi@protocol.ai>
Co-authored-by: Peter Rabbitson <ribasushi@leporine.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants