Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use current process binding to limit cpus that threads can be bound to #1633

Merged
merged 1 commit into from
Nov 10, 2022
Merged

Use current process binding to limit cpus that threads can be bound to #1633

merged 1 commit into from
Nov 10, 2022

Conversation

clinta
Copy link
Contributor

@clinta clinta commented Oct 21, 2022

Core binding currently does not work if the lotus-worker process is limited to a subset of CPUs.

This PR prevents attempting to bind threads to a core that the process does not have access to.

vmx
vmx reviewed Oct 21, 2022
View reviewed changes
Copy link
Contributor

@vmx vmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this clean and minimal patch. I left comments about the tests.

I like the additional description in your PR description, feel free to add it directly to the commit message.

storage-proofs-porep/src/stacked/vanilla/cores.rs Show resolved Hide resolved
storage-proofs-porep/src/stacked/vanilla/cores.rs Show resolved Hide resolved
@vmx
Copy link
Contributor

vmx commented Oct 21, 2022

Somehow the CI didn't run. But I have hope that once you push another time, it should just work.

@clinta
Copy link
Contributor Author

clinta commented Oct 21, 2022

Somehow the CI didn't run. But I have hope that once you push another time, it should just work.

Looks like it's still happening. Not sure why, as I didn't touch the Circle CI config.

@clinta
Copy link
Contributor Author

clinta commented Oct 28, 2022

@vmx any suggestions for how to fix CI?

@vmx vmx mentioned this pull request Nov 7, 2022
Closed
@vmx
Copy link
Contributor

vmx commented Nov 7, 2022

Sorry for the delay. Thanks for the additional tests, that looks good. In regards to CI, I've contacted the CircleCI support as things seems to be triggered fine on other PRs. I created #1636 in the meanwhile, to get CI running.

@vmx
Copy link
Contributor

vmx commented Nov 10, 2022 •
edited
Loading

@vmx any suggestions for how to fix CI?

The reply I got from CircleCI support was:

Can you have the user clinta sign out of CircleCI, clear their browser cache, and then log back in again before triggering another pipeline?

@clinta
use current process binding to limit thread cores
e978001 
Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.
@clinta
Copy link
Contributor Author

clinta commented Nov 10, 2022

Fixed the clippy and rustfmt errors, and it looks like CI is working now. Thanks for the assist. I had never setup a CircleCI account before, apparently that was required.

@vmx
Copy link
Contributor

vmx commented Nov 10, 2022

I had never setup a CircleCI account before, apparently that was required.

Interesting, I also didn't know that. That's good to know for future contributors. Thanks for looking into this!

vmx
vmx approved these changes Nov 10, 2022
View reviewed changes
Copy link
Contributor

@vmx vmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and sorry for the delays due to travel and sickness.

@vmx vmx merged commit b691d7a into filecoin-project:master Nov 10, 2022
storojs72 pushed a commit that referenced this pull request Jan 11, 2023
@clinta @storojs72
fix: use current process binding to limit thread cores (#1633)
9a58f66 
Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.
storojs72 pushed a commit that referenced this pull request Jan 11, 2023
@clinta @storojs72
fix: use current process binding to limit thread cores (#1633)
e4e88bd 
Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.
storojs72 added a commit that referenced this pull request Jan 19, 2023
@storojs72 @clinta
@vmx @hanbu97 @HammerBu
Incorporate recent updates to halo2 branch (#1656)
bbf8786 
* fix: use current process binding to limit thread cores (#1633)

Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* fix: broken Links in README.md #1637 (#1649)

* feat: Introduce PoRepConfig::new_groth16() (#1635)

Instead of constructing the `PoRepConfig` directly, use a constructor
function. This simplifies the code and makes things less error-prone.

Fixes #1632.

Co-authored-by: 卜翰 <buhan970731@gmail.com>

* fix: clean up tree definitions (#1655)

There were quite a few public type definitions, that were mostly replaced by
the `SectorShape*` types. This commit cleans them up and moves them around if
appropriate.

This should make the code easier to follow and the public API surface smaller.

BREAKING CHANGE: `BinaryLCMerkleTree`, `BinaryMerkleTree`,
`BinarySubMerkleTree`, `LCMerkleTree`, `LCStore`, `MerkleStore`, `MerkleTree`,
`OctLCMerkleTree`, `OctLCSubMerkleTree`, `OctLCTopMerkleTree`, `OctMerkleTree`,
`OctSubMerkleTree`, `OctTopMerkleTree`, `QuadLCMerkleTree` and `QuadMerkleTree`
are removed from the public interface.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* chore: update Cargo.lock

* fix: there was a memmap -> memmap2 missing

* fix: make poseidon tests pass

Neptune currently is a fork of pasta_curves. That needs patching as
well, in order to get the correct names for the fields out.

* ci: Apply rustfmt and fix clippy

* Pick relevant branch of neptune

* fix: Use SHA256 hasher for binary trees

Co-authored-by: Clint Armstrong <clint@clintarmstrong.net>
Co-authored-by: Volker Mische <volker.mische@gmail.com>
Co-authored-by: hanbu97 <98807352+hanbu97@users.noreply.github.com>
Co-authored-by: 卜翰 <buhan970731@gmail.com>
storojs72 added a commit that referenced this pull request Jan 19, 2023
@storojs72 @clinta
@vmx @hanbu97 @HammerBu
Incorporate recent updates to halo2 branch (#1656)
e0fdb29 
* fix: use current process binding to limit thread cores (#1633)

Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* fix: broken Links in README.md #1637 (#1649)

* feat: Introduce PoRepConfig::new_groth16() (#1635)

Instead of constructing the `PoRepConfig` directly, use a constructor
function. This simplifies the code and makes things less error-prone.

Fixes #1632.

Co-authored-by: 卜翰 <buhan970731@gmail.com>

* fix: clean up tree definitions (#1655)

There were quite a few public type definitions, that were mostly replaced by
the `SectorShape*` types. This commit cleans them up and moves them around if
appropriate.

This should make the code easier to follow and the public API surface smaller.

BREAKING CHANGE: `BinaryLCMerkleTree`, `BinaryMerkleTree`,
`BinarySubMerkleTree`, `LCMerkleTree`, `LCStore`, `MerkleStore`, `MerkleTree`,
`OctLCMerkleTree`, `OctLCSubMerkleTree`, `OctLCTopMerkleTree`, `OctMerkleTree`,
`OctSubMerkleTree`, `OctTopMerkleTree`, `QuadLCMerkleTree` and `QuadMerkleTree`
are removed from the public interface.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* chore: update Cargo.lock

* fix: there was a memmap -> memmap2 missing

* fix: make poseidon tests pass

Neptune currently is a fork of pasta_curves. That needs patching as
well, in order to get the correct names for the fields out.

* ci: Apply rustfmt and fix clippy

* Pick relevant branch of neptune

* fix: Use SHA256 hasher for binary trees

Co-authored-by: Clint Armstrong <clint@clintarmstrong.net>
Co-authored-by: Volker Mische <volker.mische@gmail.com>
Co-authored-by: hanbu97 <98807352+hanbu97@users.noreply.github.com>
Co-authored-by: 卜翰 <buhan970731@gmail.com>
storojs72 added a commit that referenced this pull request Jan 19, 2023
@storojs72 @clinta
@vmx @hanbu97 @HammerBu
Incorporate recent updates to halo2 branch (#1656)
fef9c33 
* fix: use current process binding to limit thread cores (#1633)

Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* fix: broken Links in README.md #1637 (#1649)

* feat: Introduce PoRepConfig::new_groth16() (#1635)

Instead of constructing the `PoRepConfig` directly, use a constructor
function. This simplifies the code and makes things less error-prone.

Fixes #1632.

Co-authored-by: 卜翰 <buhan970731@gmail.com>

* fix: clean up tree definitions (#1655)

There were quite a few public type definitions, that were mostly replaced by
the `SectorShape*` types. This commit cleans them up and moves them around if
appropriate.

This should make the code easier to follow and the public API surface smaller.

BREAKING CHANGE: `BinaryLCMerkleTree`, `BinaryMerkleTree`,
`BinarySubMerkleTree`, `LCMerkleTree`, `LCStore`, `MerkleStore`, `MerkleTree`,
`OctLCMerkleTree`, `OctLCSubMerkleTree`, `OctLCTopMerkleTree`, `OctMerkleTree`,
`OctSubMerkleTree`, `OctTopMerkleTree`, `QuadLCMerkleTree` and `QuadMerkleTree`
are removed from the public interface.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* chore: update Cargo.lock

* fix: there was a memmap -> memmap2 missing

* fix: make poseidon tests pass

Neptune currently is a fork of pasta_curves. That needs patching as
well, in order to get the correct names for the fields out.

* ci: Apply rustfmt and fix clippy

* Pick relevant branch of neptune

* fix: Use SHA256 hasher for binary trees

Co-authored-by: Clint Armstrong <clint@clintarmstrong.net>
Co-authored-by: Volker Mische <volker.mische@gmail.com>
Co-authored-by: hanbu97 <98807352+hanbu97@users.noreply.github.com>
Co-authored-by: 卜翰 <buhan970731@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vmx vmx vmx approved these changes

@cryptonemo cryptonemo Awaiting requested review from cryptonemo cryptonemo is a code owner

@DrPeterVanNostrand DrPeterVanNostrand Awaiting requested review from DrPeterVanNostrand DrPeterVanNostrand is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@clinta @vmx