Incorporate recent updates to halo2 branch (#1656)

* fix: use current process binding to limit thread cores (#1633) Use the current processes bound cores to limit the possible cores that threads can be bound to. This allows core binding to work properly when the lotus-worker service is limited to certain CPUs by cgroups. * fix: update ec-gpu-gen (#1638) `ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that depend on yanked version. It's an indirect dependency of `bellperson` and `neptune`, which are upgraded here. Moving from `memmap` (which is deprecated) to `memmap2` was also needed als dependencies also switched. `chrono` updated their API, so there was also a small change needed. * fix: broken Links in README.md #1637 (#1649) * feat: Introduce PoRepConfig::new_groth16() (#1635) Instead of constructing the `PoRepConfig` directly, use a constructor function. This simplifies the code and makes things less error-prone. Fixes #1632. Co-authored-by: 卜翰 <buhan970731@gmail.com> * fix: clean up tree definitions (#1655) There were quite a few public type definitions, that were mostly replaced by the `SectorShape*` types. This commit cleans them up and moves them around if appropriate. This should make the code easier to follow and the public API surface smaller. BREAKING CHANGE: `BinaryLCMerkleTree`, `BinaryMerkleTree`, `BinarySubMerkleTree`, `LCMerkleTree`, `LCStore`, `MerkleStore`, `MerkleTree`, `OctLCMerkleTree`, `OctLCSubMerkleTree`, `OctLCTopMerkleTree`, `OctMerkleTree`, `OctSubMerkleTree`, `OctTopMerkleTree`, `QuadLCMerkleTree` and `QuadMerkleTree` are removed from the public interface. * fix: update ec-gpu-gen (#1638) `ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that depend on yanked version. It's an indirect dependency of `bellperson` and `neptune`, which are upgraded here. Moving from `memmap` (which is deprecated) to `memmap2` was also needed als dependencies also switched. `chrono` updated their API, so there was also a small change needed. * chore: update Cargo.lock * fix: there was a memmap -> memmap2 missing * fix: make poseidon tests pass Neptune currently is a fork of pasta_curves. That needs patching as well, in order to get the correct names for the fields out. * ci: Apply rustfmt and fix clippy * Pick relevant branch of neptune * fix: Use SHA256 hasher for binary trees Co-authored-by: Clint Armstrong <clint@clintarmstrong.net> Co-authored-by: Volker Mische <volker.mische@gmail.com> Co-authored-by: hanbu97 <98807352+hanbu97@users.noreply.github.com> Co-authored-by: 卜翰 <buhan970731@gmail.com>
filecoin-project · Jan 19, 2023 · e0fdb29 · e0fdb29
1 parent 5764f4b
commit e0fdb29
Show file tree

Hide file tree

Showing 45 changed files with 677 additions and 696 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -15,4 +15,7 @@ members = [
 ]
 
 [patch.crates-io]
-halo2_proofs = { git = "https://github.com/filecoin-project/halo2", branch = "serialize-vk-pk" }
+halo2_proofs = { git = "https://github.com/filecoin-project/halo2", branch = "halo2" }
+neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2" }
+pasta_curves = { git = "https://github.com/filecoin-project/pasta_curves", branch = "gpu" }
+fil_pasta_curves = { git = "https://github.com/filecoin-project/pasta_curves", branch = "gpu-fil-fork" }
diff --git a/README.md b/README.md
@@ -4,21 +4,18 @@ The **Filecoin Proving Subsystem** (or FPS) provides the storage proofs required
 
 There are currently several different crates:
 
-- [**Storage Proofs (`storage-proofs`)**](./storage-proofs)
-    A library for constructing storage proofs – including non-circuit proofs, corresponding SNARK circuits, and a method of combining them.
-
-- [**Storage Proofs Core (`storage-proofs-core`)**](./storage-proofs/core)
+- [**Storage Proofs Core (`storage-proofs-core`)**](./storage-proofs-core)
     A set of common primitives used throughout the other storage-proofs sub-crates, including crypto, merkle tree, hashing and gadget interfaces.
 
-- [**Storage Proofs PoRep (`storage-proofs-porep`)**](./storage-proofs/porep)
+- [**Storage Proofs PoRep (`storage-proofs-porep`)**](./storage-proofs-porep)
     `storage-proofs-porep` is intended to serve as a reference implementation for _**Proof-of-Replication**_ (**PoRep**), while also performing the heavy lifting for `filecoin-proofs`.
 
     Primary Components:
      -   **PoR** (**_Proof-of-Retrievability_**: Merkle inclusion proof)
      -   **DrgPoRep** (_Depth Robust Graph_ **_Proof-of-Replication_**)
      -   **StackedDrgPoRep**
 
-- [**Storage Proofs PoSt (`storage-proofs-post`)**](./storage-proofs/post)
+- [**Storage Proofs PoSt (`storage-proofs-post`)**](./storage-proofs-post)
     `storage-proofs-post` is intended to serve as a reference implementation for _**Proof-of-Space-time**_ (**PoSt**), for `filecoin-proofs`.
 
     Primary Components:
@@ -30,7 +27,7 @@ There are currently several different crates:
 
 ## Security Audits
 
-The `rust-fil-proofs` proofs code and the [Filecoin Spec](https://bafybeidxw5vxjdwsun2zc2illagf43v6w5r5w63vg455h7vjesbyqssg64.ipfs.dweb.link/algorithms/sdr/) has undergone a [proofs security audit](audits/Sigma-Prime-Protocol-Labs-Filecoin-Proofs-Security-Review-v2.1.pdf) performed by [Sigma Prime](https://sigmaprime.io/) and been deemed free of *critical* or *major* security issues.  In addition to the security review, the document provides the summary of findings, vulnerability classifications, and recommended resolutions.  All known issues have been resolved to date in both the code and the specification.
+The `rust-fil-proofs` proofs code and the [Filecoin Spec](https://spec.filecoin.io/algorithms/sdr/) has undergone a [proofs security audit](audits/Sigma-Prime-Protocol-Labs-Filecoin-Proofs-Security-Review-v2.1.pdf) performed by [Sigma Prime](https://sigmaprime.io/) and been deemed free of *critical* or *major* security issues.  In addition to the security review, the document provides the summary of findings, vulnerability classifications, and recommended resolutions.  All known issues have been resolved to date in both the code and the specification.
 
 `rust-fil-proofs` has also undergone a [SNARK proofs security audit performed by Dr. Jean-Philippe Aumasson and Antony Vennard](audits/protocolai-audit-20200728.pdf) and been deemed free of *critical* or *major* security issues.  In addition to the security analysis, the document provides the audit goals, methodology, functionality descriptions and finally observations on what could be improved.  All known issues have been resolved to date.
 

diff --git a/fil-halo2-gadgets/Cargo.toml b/fil-halo2-gadgets/Cargo.toml
@@ -10,7 +10,7 @@ repository = "https://github.com/filecoin-project/rust-fil-proofs"
 ff = "0.12.0"
 halo2_proofs = "0.2.0"
 halo2_gadgets = "0.2.0"
-neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2" }
+neptune = "~8.1.0"
 sha2 = "0.9.2"
 
 [dev-dependencies]

diff --git a/fil-halo2-gadgets/src/bin/sha256_compute_k.rs b/fil-halo2-gadgets/src/bin/sha256_compute_k.rs
@@ -131,7 +131,7 @@ fn sha256_chip_compute_k() {
                 Ok(_) => {
                     // Proof size does not change with circuit size.
                     if num_blocks == PREIMAGE_BLOCK_SIZES.start {
-                        let cost = CircuitCost::<Eq, MyCircuit>::measure(k as usize, &circ);
+                        let cost = CircuitCost::<Eq, MyCircuit>::measure(k, &circ);
                         let proof_size: usize = cost.marginal_proof_size().into();
                         println!("\tproof_size = {} bytes", proof_size);
                     }
@@ -219,7 +219,7 @@ fn compress_chip_compute_k() {
                 Ok(_) => {
                     // Proof size does not change with circuit size.
                     if num_blocks == PREIMAGE_BLOCK_SIZES.start {
-                        let cost = CircuitCost::<Eq, MyCircuit>::measure(k as usize, &circ);
+                        let cost = CircuitCost::<Eq, MyCircuit>::measure(k, &circ);
                         let proof_size: usize = cost.marginal_proof_size().into();
                         println!("\tproof_size = {} bytes", proof_size);
                     }

diff --git a/fil-proofs-param/Cargo.toml b/fil-proofs-param/Cargo.toml
@@ -32,7 +32,7 @@ structopt = "0.3.12"
 humansize = "1.1.0"
 indicatif = "0.16.2"
 dialoguer = "0.10.0"
-blstrs = "0.5.0"
+blstrs = "0.6.1"
 typenum = "1.11.2"
 generic-array = "0.14.4"
 halo2_proofs = "0.2.0"

diff --git a/fil-proofs-param/src/bin/paramcache.rs b/fil-proofs-param/src/bin/paramcache.rs
@@ -6,7 +6,7 @@ use blstrs::Scalar as Fr;
 use dialoguer::{theme::ColorfulTheme, MultiSelect};
 use filecoin_proofs::{
     constants::{
-        DefaultPieceHasher, POREP_PARTITIONS, PUBLISHED_SECTOR_SIZES, WINDOW_POST_CHALLENGE_COUNT,
+        DefaultPieceHasher, PUBLISHED_SECTOR_SIZES, WINDOW_POST_CHALLENGE_COUNT,
         WINDOW_POST_SECTOR_COUNT, WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
     },
     parameters::{public_params, window_post_public_params, winning_post_public_params},
@@ -226,18 +226,7 @@ fn generate_params_porep(sector_size: u64, api_version: ApiVersion) {
         sector_size,
         Fr,
         cache_porep_params,
-        PoRepConfig {
-            sector_size: SectorSize(sector_size),
-            partitions: PoRepProofPartitions(
-                *POREP_PARTITIONS
-                    .read()
-                    .expect("POREP_PARTITIONS poisoned")
-                    .get(&sector_size)
-                    .expect("unknown sector size") as usize,
-            ),
-            porep_id: [0; 32],
-            api_version,
-        }
+        PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
     );
 }
 
@@ -246,18 +235,7 @@ fn generate_params_empty_sector_update(sector_size: u64, api_version: ApiVersion
         sector_size,
         Fr,
         cache_empty_sector_update_params,
-        PoRepConfig {
-            sector_size: SectorSize(sector_size),
-            partitions: PoRepProofPartitions(
-                *POREP_PARTITIONS
-                    .read()
-                    .expect("POREP_PARTITIONS poisoned")
-                    .get(&sector_size)
-                    .expect("unknown sector size") as usize,
-            ),
-            porep_id: [0; 32],
-            api_version,
-        }
+        PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
     );
 }
 

diff --git a/fil-proofs-tooling/Cargo.toml b/fil-proofs-tooling/Cargo.toml
@@ -21,8 +21,8 @@ serde_json = "1.0"
 regex = "1.3.7"
 commandspec = "0.12.2"
 chrono = { version = "0.4.7", features = ["serde"] }
-memmap = "0.7.0"
-bellperson = "0.22.0"
+memmap2 = "0.5.6"
+bellperson = "0.24.0"
 rand = "0.8"
 tempfile = "3.0.8"
 cpu-time = "1.0.0"
@@ -47,7 +47,7 @@ fdlimit = "0.2.0"
 dialoguer = "0.10.0"
 structopt = "0.3.12"
 humansize = "1.1.0"
-blstrs = "0.5.0"
+blstrs = "0.6.1"
 time = "0.3.9"
 pasta_curves = "0.4.0"
 

diff --git a/fil-proofs-tooling/src/bin/benchy/prodbench.rs b/fil-proofs-tooling/src/bin/benchy/prodbench.rs
@@ -170,9 +170,8 @@ fn configure_global_config(inputs: &ProdbenchInputs) {
         .expect("POREP_PARTITIONS poisoned")
         .insert(inputs.sector_size_bytes(), inputs.porep_partitions);
     POREP_MINIMUM_CHALLENGES
-        .write()
-        .expect("POREP_MINIMUM_CHALLENGES poisoned")
-        .insert(inputs.sector_size_bytes(), inputs.porep_challenges);
+        .get_mut()
+        .insert(inputs.sector_size_bytes(), inputs.porep_challenges as usize);
 }
 
 pub fn run<F>(

diff --git a/fil-proofs-tooling/src/bin/circuitinfo/main.rs b/fil-proofs-tooling/src/bin/circuitinfo/main.rs
@@ -159,12 +159,7 @@ fn porep_info(sector_size: u64, api_version: ApiVersion) -> (CircuitInfo, usize)
         sector_size,
         Fr,
         get_porep_info,
-        PoRepConfig {
-            sector_size: SectorSize(sector_size),
-            partitions,
-            porep_id: [0; 32],
-            api_version,
-        }
+        PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
     );
     (info, partitions.into())
 }

diff --git a/fil-proofs-tooling/src/bin/update_tree_r_cache/main.rs b/fil-proofs-tooling/src/bin/update_tree_r_cache/main.rs
@@ -13,17 +13,16 @@ use filecoin_proofs::{
     SectorShapeSub2, SectorShapeSub8, SectorShapeTop2, OCT_ARITY,
 };
 use generic_array::typenum::Unsigned;
-use memmap::MmapOptions;
+use memmap2::MmapOptions;
 use merkletree::{
     merkle::get_merkle_tree_len,
-    store::{ExternalReader, ReplicaConfig, Store, StoreConfig},
+    store::{ExternalReader, LevelCacheStore, ReplicaConfig, Store, StoreConfig},
 };
 use pasta_curves::{Fp, Fq};
 use storage_proofs_core::{
     cache_key::CacheKey,
     merkle::{
-        create_lc_tree, get_base_tree_count, split_config_and_replica, LCStore, LCTree,
-        MerkleTreeTrait,
+        create_lc_tree, get_base_tree_count, split_config_and_replica, LCTree, MerkleTreeTrait,
     },
     util::{default_rows_to_discard, NODE_SIZE},
 };
@@ -75,7 +74,7 @@ where
     let base_tree_len = get_merkle_tree_len(base_tree_leafs, OCT_ARITY)?;
     let tree_r_last_root = if is_sector_shape_base(sector_size) {
         ensure!(configs.len() == 1, "Invalid tree-shape specified");
-        let store = LCStore::<DefaultTreeDomain<F>>::new_from_disk_with_reader(
+        let store = LevelCacheStore::new_from_disk_with_reader(
             base_tree_len,
             OCT_ARITY,
             &configs[0],
@@ -255,7 +254,7 @@ where
     // First, read the roots from the cached trees on disk
     let mut cached_base_tree_roots: Vec<DefaultTreeDomain<F>> = Vec::with_capacity(tree_count);
     for (i, config) in configs.iter().enumerate().take(tree_count) {
-        let store = LCStore::new_from_disk_with_reader(
+        let store = LevelCacheStore::new_from_disk_with_reader(
             base_tree_len,
             OCT_ARITY,
             config,

diff --git a/fil-proofs-tooling/src/metadata.rs b/fil-proofs-tooling/src/metadata.rs
@@ -40,6 +40,8 @@ impl GitMetadata {
         let repo = Repository::discover(&repo_path)?;
         let head = repo.head()?;
         let commit = head.peel_to_commit()?;
+
+        // Unwrap is OK as the given seconds won't be out of range.
         let date = Utc.timestamp_opt(commit.time().seconds(), 0).unwrap();
 
         Ok(GitMetadata {

diff --git a/fil-proofs-tooling/src/shared.rs b/fil-proofs-tooling/src/shared.rs
@@ -7,10 +7,10 @@ use filecoin_hashers::Hasher;
 use filecoin_proofs::{
     add_piece, fauxrep_aux, seal_pre_commit_phase1, seal_pre_commit_phase2,
     validate_cache_for_precommit_phase2, DefaultPieceHasher, DefaultTreeHasher, MerkleTreeTrait,
-    PaddedBytesAmount, PieceInfo, PoRepConfig, PoRepProofPartitions, PoStConfig, PoStType,
-    PrivateReplicaInfo, PublicReplicaInfo, SealPreCommitOutput, SealPreCommitPhase1Output,
-    SectorSize, UnpaddedBytesAmount, POREP_PARTITIONS, WINDOW_POST_CHALLENGE_COUNT,
-    WINDOW_POST_SECTOR_COUNT, WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
+    PaddedBytesAmount, PieceInfo, PoRepConfig, PoStConfig, PoStType, PrivateReplicaInfo,
+    PublicReplicaInfo, SealPreCommitOutput, SealPreCommitPhase1Output, SectorSize,
+    UnpaddedBytesAmount, WINDOW_POST_CHALLENGE_COUNT, WINDOW_POST_SECTOR_COUNT,
+    WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
 };
 use generic_array::typenum::Unsigned;
 use log::info;
@@ -137,18 +137,7 @@ where
     let sector_size_unpadded_bytes_ammount =
         UnpaddedBytesAmount::from(PaddedBytesAmount::from(sector_size));
 
-    let porep_config = PoRepConfig {
-        sector_size,
-        partitions: PoRepProofPartitions(
-            *POREP_PARTITIONS
-                .read()
-                .expect("poisoned read access")
-                .get(&u64::from(sector_size))
-                .expect("unknown sector size") as usize,
-        ),
-        porep_id,
-        api_version,
-    };
+    let porep_config = PoRepConfig::new_groth16(u64::from(sector_size), porep_id, api_version);
 
     let mut out: Vec<(SectorId, PreCommitReplicaOutput<Tree>)> = Default::default();
     let mut sector_ids = Vec::new();
@@ -343,18 +332,7 @@ pub fn get_porep_config<F: PrimeField>(sector_size: u64, api_version: ApiVersion
     let arbitrary_porep_id = [99; 32];
 
     if util::is_groth16_field::<F>() {
-        PoRepConfig {
-            sector_size: sector_size.into(),
-            partitions: PoRepProofPartitions(
-                *POREP_PARTITIONS
-                    .read()
-                    .expect("POREP_PARTITONS poisoned")
-                    .get(&(sector_size))
-                    .expect("unknown sector size") as usize,
-            ),
-            porep_id: arbitrary_porep_id,
-            api_version,
-        }
+        PoRepConfig::new_groth16(sector_size, arbitrary_porep_id, api_version)
     } else {
         PoRepConfig::new_halo2(sector_size.into(), arbitrary_porep_id, api_version)
     }

diff --git a/filecoin-hashers/Cargo.toml b/filecoin-hashers/Cargo.toml
@@ -9,17 +9,16 @@ repository = "https://github.com/filecoin-project/rust-fil-proofs"
 readme = "README.md"
 
 [dependencies]
-bellperson = "0.22.0"
-blstrs = "0.5.0"
+bellperson = "0.24.0"
+blstrs = "0.6.1"
 generic-array = "0.14.4"
 merkletree = "0.22.0"
-ec-gpu = "0.1.0"
+ec-gpu = "0.2.0"
 ff = "0.12.0"
 anyhow = "1.0.34"
 serde = "1.0.117"
 rand = "0.8.0"
-# neptune = { version = "~7.0.0", optional = true, features = ["bls", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
-neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2", optional = true, features = ["bls", "pasta", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
+neptune = { version = "~8.1.0", optional = true, features = ["bls", "pasta", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
 lazy_static = { version = "1.4.0", optional = true }
 blake2s_simd = { version = "1.0.0", optional = true }
 sha2 = { version = "0.10.2", optional = true }
@@ -31,8 +30,8 @@ fil-halo2-gadgets = { path = "../fil-halo2-gadgets" }
 [features]
 default = ["opencl", "blake2s", "poseidon", "sha256"]
 
-cuda = ["bellperson/cuda", "neptune/cuda"]
-opencl = ["bellperson/opencl", "neptune/opencl"]
+cuda = ["bellperson/cuda", "halo2_proofs/cuda", "neptune/cuda"]
+opencl = ["bellperson/opencl", "halo2_proofs/opencl", "neptune/opencl"]
 
 # available hashers
 blake2s = ["blake2s_simd"]

diff --git a/filecoin-hashers/src/types.rs b/filecoin-hashers/src/types.rs
@@ -10,7 +10,6 @@ use bellperson::{
     ConstraintSystem, SynthesisError,
 };
 use blstrs::Scalar as Fr;
-use ec_gpu::GpuField;
 use ff::{Field, PrimeField};
 use fil_halo2_gadgets::ColumnCount;
 use halo2_proofs::{
@@ -44,7 +43,7 @@ pub trait Domain:
     + Element
     + StdHash
 {
-    type Field: PrimeField + GpuField;
+    type Field: PrimeField;
 
     #[allow(clippy::wrong_self_convention)]
     fn into_bytes(&self) -> Vec<u8> {
@@ -99,7 +98,11 @@ pub trait HashFunction<T: Domain>: Clone + Debug + Send + Sync + LightAlgorithm<
 }
 
 pub trait Hasher: Clone + Debug + Eq + Default + Send + Sync {
-    type Field: PrimeField + GpuField;
+    #[cfg(not(any(feature = "cuda", feature = "opencl")))]
+    type Field: PrimeField;
+    #[cfg(any(feature = "cuda", feature = "opencl"))]
+    type Field: PrimeField + ec_gpu::GpuName;
+
     type Domain: Domain<Field = Self::Field> + LightHashable<Self::Function> + AsRef<Self::Domain>;
     type Function: HashFunction<Self::Domain>;
 

diff --git a/filecoin-proofs/Cargo.toml b/filecoin-proofs/Cargo.toml
@@ -16,12 +16,12 @@ storage-proofs-update = { path = "../storage-proofs-update", version = "~12.0.0"
 filecoin-hashers = { version = "~7.0.0", path = "../filecoin-hashers", default-features = false, features = ["poseidon", "sha256"] }
 rand = "0.8"
 lazy_static = "1.2"
-memmap = "0.7"
+memmap2 = "0.5.6"
 serde = { version = "1.0", features = ["rc", "derive"] }
 serde_json = "1.0"
 ff = "0.12.0"
 blake2b_simd = "1.0.0"
-bellperson = "0.22.0"
+bellperson = "0.24.0"
 log = "0.4.7"
 rayon = "1.1.0"
 hex = "0.4.0"
@@ -34,7 +34,7 @@ gperftools = { version = "0.2", optional = true }
 generic-array = "0.14.4"
 fr32 = { path = "../fr32", version = "~5.0.0", default-features = false }
 once_cell = "1.8.0"
-blstrs = "0.5.0"
+blstrs = "0.6.1"
 halo2_proofs = "0.2.0"
 
 [dev-dependencies]