services: add new service for fetching blocks from NeoFS

Close #3496

Co-authored-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Signed-off-by: Ekaterina Pavlova <ekt@morphbits.io>

config/protocol.testnet.yml

@@ -100,3 +100,15 @@ ApplicationConfiguration:
     Enabled: false
     Addresses:
       - ":2113"
+  NeoFSBlockFetcher:
+    Enabled: false
+    Addresses:
+      - st1.storage.fs.neo.org:8080
+    Timeout: 10m
+    DownloaderWorkersCount: 500
+    OIDBatchSize: 8000
+    BQueueSize: 32000 # must be larger than OIDBatchSize; highly recommended to be 2*OIDBatchSize or 3*OIDBatchSize
+    SkipIndexFilesSearch: false
+    ContainerID: "EPGuD26wYgQJbmDdVBoYoNZiMKHwFMJT3A5WqPjdUHxH"
+    BlockAttribute: "block"
+    IndexFileAttribute: "oid"

docs/neofs-blockstorage.md

@@ -0,0 +1,106 @@
+# NeoFS BlockStorage
+This service was proposed as a part of [ Using NeoFS to store blocks and snapshots #3463](https://github.com/neo-project/neo/issues/3463). 
+It is an alternative to the P2P synchronization protocol by using NeoFS to
+store and retrieve blockchain blocks and snapshots. This improves 
+synchronization efficiency, reduces local storage, and streamlines node 
+operations.
+
+## Components and functionality
+
+### Storage schema
+A single container is used to store blocks and index files. Each block
+is stored as a separate object with a unique OID and a set of attributes:
+- block index
+- block primary index
+- size.
+
+Index files contain OIDs of ordered blocks. Each index file is an object
+containing a constant number of OIDs in
+binary form, where every number represent the order of the index file.
+
+Attributes have the following structure: `AttributeName:IntValue`.
+In the container are stored blocks with attributes `block:1`,
+`block:2`, etc. And index files with attributes `oid:1`, `oid:2`, etc.
+
+### NeoFS BlockFetcher
+The NeoFS BlockFetcher service is designed as an alternative to P2P
+synchronisation protocol. It allows to fetch blocks, primarily used at
+the start of a node's lifecycle, providing an alternative to downloading
+blocks through the P2P NeoGo network. By default, the service is disabled.
+
+The NeoFS BlockFetcher service has two modes of operation:
+- Index File Search: Search for index files, which contain OIDs of blocks,
+  and fetch blocks from NeoFS by provided OIDs.
+- Direct Block Search: Search and fetch blocks directly from NeoFS container.
+
+### BlockFetcher operation flow
+1. **OID Fetching**:
+    Depending on the mode, the service either:
+   - Searches for index files by attribute and reads blocks OIDs from it.
+   - Searches batches of blocks directly by attributes.
+
+   Once the OIDs are retrieved, they are immediately redirected to the 
+   block downloading routines for further processing. The channel that 
+   is used to redirect block OIDs to downloading routines is buffered 
+   to provide smooth OIDs delivery without delays. Size of this channel 
+   can be configured via `OIDBatchSize` parameter.
+2. **Parallel Block Downloading**:
+   The number of downloading routines can be configured via 
+   `DownloaderWorkersCount` parameter. It's up to the user to find the 
+   balance between the downloading speed and blocks persist speed for every 
+   node that uses NeoFS BlockFetcher. Downloaded blocks are placed into a 
+   buffer with a size of `BQueueSize` parameter for further processing.
+3. **Block Insertion**:
+   Downloaded blocks are inserted into the blockchain using the same logic
+   as in the P2P synchronisation protocol. The bqueue is used to store 
+   downloaded blocks before they are inserted into the blockchain. The 
+   size of the bqueue can be configured via the `BQueueSize` parameter 
+   and should be larger than the `OIDBatchSize` parameter to avoid blocking
+   the downloading routines.
+
+Once all blocks available in the NeoFS container are processed, the service
+shuts down automatically.
+
+### NeoFS BlockFetcher Configuration
+`NeoFSBlockFetcher` configuration section contains settings for NeoFS 
+BlockFetcher module and has the following structure:
+```
+  NeoFSBlockFetcher:
+    Enabled: true
+    UnlockWallet:
+      Path: "./wallet.json"
+      Password: "pass"
+    Addresses:
+      - st1.storage.fs.neo.org:8080
+    Timeout: 10m
+    DownloaderWorkersCount: 500
+    OIDBatchSize: 8000
+    BQueueSize: 16000
+    SkipIndexFilesSearch: false
+    ContainerID: "EPGuD26wYgQJbmDdVBoYoNZiMKHwFMJT3A5WqPjdUHxH"
+    BlockAttribute: "block"
+    IndexFileAttribute: "oid"
+```
+where:
+- `Enabled` enables NeoFS BlockFetcher module.
+- `UnlockWallet` contains wallet settings to retrieve account to sign requests to 
+  NeoFS. Without this setting, the module will use randomly generated private key.
+  For configuration details see [Unlock Wallet Configuration](https://github.com/nspcc-dev/neo-go/blob/master/docs/node-configuration.md#unlock-wallet-configuration)
+- `Addresses` is a list of NeoFS storage nodes addresses.
+- `Timeout` is a timeout for a single request to NeoFS storage node.
+- `ContainerID` is a container ID to fetch blocks from.
+- `BlockAttribute` is an attribute name of NeoFS object that contains block
+  data.
+- `IndexFileAttribute` is an attribute name of NeoFS object that contains OIDs of
+  blocks objects.
+- `DownloaderWorkersCount` is a number of workers that download blocks from
+  NeoFS.
+- `OIDBatchSize` is the number of blocks to search per a single request to NeoFS
+  in case of disabled index files search. Also, for both modes of BlockFetcher 
+  operation this setting manages the buffer size of OIDs and blocks transferring channels.
+- `BQueueSize` is a size of the block queue used to manage consecutive blocks 
+  addition to the chain. It must be larger than
+  `OIDBatchSize` and highly recommended to be 2*`OIDBatchSize` or 3*`OIDBatchSize`.
+- `SkipIndexFilesSearch` is a flag that allows skipping index files search
+  in NeoFS storage nodes and search for blocks directly. It is set to `false`
+  by default.

docs/node-configuration.md

@@ -21,6 +21,7 @@ node-related settings described in the table below.
 | GarbageCollectionPeriod | `uint32` | 10000 | Controls MPT garbage collection interval (in blocks) for configurations with `RemoveUntraceableBlocks` enabled and `KeepOnlyLatestState` disabled. In this mode the node stores a number of MPT trees (corresponding to `MaxTraceableBlocks` and `StateSyncInterval`), but the DB needs to be clean from old entries from time to time. Doing it too often will cause too much processing overhead, doing it too rarely will leave more useless data in the DB. |
 | KeepOnlyLatestState | `bool` | `false` | Specifies if MPT should only store the latest state (or a set of latest states, see `P2PStateExchangeExtensions` section in the ProtocolConfiguration for details). If true, DB size will be smaller, but older roots won't be accessible. This value should remain the same for the same database. |  |
 | LogPath | `string` | "", so only console logging | File path where to store node logs. |
+| NeoFSBlockFetcher | [NeoFSBlockFetcher Configuration](https://github.com/nspcc-dev/neo-go/blob/master/docs/neofs-blockstorage.md#NeoFSBlockFetcher-Configuration) | | NeoFSBlockFetcher module configuration. See the [NeoFSBlockFetcher Configuration](https://github.com/nspcc-dev/neo-go/blob/master/docs/neofs-blockstorage.md#NeoFSBlockFetcher-Configuration) section for details. |
 | Oracle | [Oracle Configuration](#Oracle-Configuration) | | Oracle module configuration. See the [Oracle Configuration](#Oracle-Configuration) section for details. |
 | P2P | [P2P Configuration](#P2P-Configuration) | | Configuration values for P2P network interaction. See the [P2P Configuration](#P2P-Configuration) section for details. |
 | P2PNotary | [P2P Notary Configuration](#P2P-Notary-Configuration) | | P2P Notary module configuration. See the [P2P Notary Configuration](#P2P-Notary-Configuration) section for details. |

pkg/config/application_config.go

@@ -23,12 +23,13 @@ type ApplicationConfiguration struct {
 	Pprof      BasicService `yaml:"Pprof"`
 	Prometheus BasicService `yaml:"Prometheus"`
 
-	Relay     bool                `yaml:"Relay"`
-	Consensus Consensus           `yaml:"Consensus"`
-	RPC       RPC                 `yaml:"RPC"`
-	Oracle    OracleConfiguration `yaml:"Oracle"`
-	P2PNotary P2PNotary           `yaml:"P2PNotary"`
-	StateRoot StateRoot           `yaml:"StateRoot"`
+	Relay             bool                `yaml:"Relay"`
+	Consensus         Consensus           `yaml:"Consensus"`
+	RPC               RPC                 `yaml:"RPC"`
+	Oracle            OracleConfiguration `yaml:"Oracle"`
+	P2PNotary         P2PNotary           `yaml:"P2PNotary"`
+	StateRoot         StateRoot           `yaml:"StateRoot"`
+	NeoFSBlockFetcher NeoFSBlockFetcher   `yaml:"NeoFSBlockFetcher"`
 }
 
 // EqualsButServices returns true when the o is the same as a except for services
@@ -141,3 +142,13 @@ func (a *ApplicationConfiguration) GetAddresses() ([]AnnounceableAddress, error)
 	}
 	return addrs, nil
 }
+
+// Validate checks ApplicationConfiguration for internal consistency and returns
+// an error if any invalid settings are found. This ensures that the application
+// configuration is valid and safe to use for further operations.
+func (a *ApplicationConfiguration) Validate() error {
+	if err := a.NeoFSBlockFetcher.Validate(); err != nil {
+		return fmt.Errorf("failed to validate NeoFSBlockFetcher section: %w", err)
+	}
+	return nil
+}

pkg/config/blockfetcher_config.go

@@ -0,0 +1,47 @@
+package config
+
+import (
+	"errors"
+	"fmt"
+	"time"
+
+	cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
+)
+
+// NeoFSBlockFetcher represents the configuration for the NeoFS BlockFetcher service.
+type NeoFSBlockFetcher struct {
+	InternalService        `yaml:",inline"`
+	Timeout                time.Duration `yaml:"Timeout"`
+	ContainerID            string        `yaml:"ContainerID"`
+	Addresses              []string      `yaml:"Addresses"`
+	OIDBatchSize           int           `yaml:"OIDBatchSize"`
+	BlockAttribute         string        `yaml:"BlockAttribute"`
+	IndexFileAttribute     string        `yaml:"IndexFileAttribute"`
+	DownloaderWorkersCount int           `yaml:"DownloaderWorkersCount"`
+	BQueueSize             int           `yaml:"BQueueSize"`
+	SkipIndexFilesSearch   bool          `yaml:"SkipIndexFilesSearch"`
+}
+
+// Validate checks NeoFSBlockFetcher for internal consistency and ensures
+// that all required fields are properly set. It returns an error if the
+// configuration is invalid or if the ContainerID cannot be properly decoded.
+func (cfg *NeoFSBlockFetcher) Validate() error {
+	if !cfg.Enabled {
+		return nil
+	}
+	if cfg.ContainerID == "" {
+		return errors.New("container ID is not set")
+	}
+	var containerID cid.ID
+	err := containerID.DecodeString(cfg.ContainerID)
+	if err != nil {
+		return fmt.Errorf("invalid container ID: %w", err)
+	}
+	if cfg.BQueueSize < cfg.OIDBatchSize {
+		return fmt.Errorf("BQueueSize (%d) is lower than OIDBatchSize (%d)", cfg.BQueueSize, cfg.OIDBatchSize)
+	}
+	if len(cfg.Addresses) == 0 {
+		return errors.New("addresses are not set")
+	}
+	return nil
+}

pkg/config/config.go

@@ -116,6 +116,10 @@ func LoadFile(configPath string, relativePath ...string) (Config, error) {
 	if err != nil {
 		return Config{}, err
 	}
+	err = config.ApplicationConfiguration.Validate()
+	if err != nil {
+		return Config{}, err
+	}
 
 	return config, nil
 }