Safety check for full pruning #5550

deffrian · 2023-04-10T04:31:50Z

Fixes Closes Resolves #
#5495

Adds available memory check before execution full pruning
Size estimations ready only for mainnet and sepolia. Will add other networks in few weeks.

Changes

Check itself
A bit refactoring

Types of changes

What types of changes does your code introduce?

Bugfix (a non-breaking change that fixes an issue)
New feature (a non-breaking change that adds functionality)
Breaking change (a change that causes existing functionality not to work as expected)
Optimization
Refactoring
Documentation update
Build-related changes
Other: Description

Testing

Requires testing

Yes
No

If yes, did you write tests?

Yes
No

…or-full-pruning # Conflicts: # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPrunerTests.cs

src/Nethermind/Nethermind.Blockchain/FullPruning/PruningStatus.cs

src/Nethermind/Nethermind.Blockchain/FullPruning/FullPruner.cs

# Conflicts: # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPrunerTests.cs # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPruningDiskTest.cs # src/Nethermind/Nethermind.Blockchain/FullPruning/FullPruner.cs # src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs

# Conflicts: # src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs

kamilchodola · 2023-05-08T13:12:52Z

@deffrian @MarekM25
Thinking out loud now - is there any chance to put some kind of logs which will inform user that space is running out and full pruning or resync is advised? Somewhere around 250-300GB for mainnet (and respectively less for other chains). This may be configurable by Pruning options and fully disabled if somebody doesn't want to have this (or threshold can be adjusted).
This also may be pushed to health module as well since node which is close to this threshold is already put under risk.

src/Nethermind/Nethermind.Db/IPruningConfig.cs

benaadams · 2023-05-08T13:14:57Z

is there any chance to put some kind of logs which will inform user that space is running out and full pruning or resync is advised?

Warn level?

Co-authored-by: Ben Adams <thundercat@illyriad.co.uk>

LukaszRozmej · 2023-05-16T12:44:09Z

src/Nethermind/Nethermind.Blockchain/FullPruning/FullPruner.cs

@@ -158,6 +170,28 @@ private void SetCurrentPruning(IPruningContext pruningContext)

        private bool CanStartNewPruning() => _fullPruningDb.CanStartPruning;

+        private const double ChainSizeThresholdFactor = 1.3;


Would make this a long and make it in %, set it to 130, then when using it first multiply by it, then divide by 100. You keep everything integer and avoid floats.

LukaszRozmej · 2023-05-16T12:44:40Z

src/Nethermind/Nethermind.Blockchain/FullPruning/FullPruner.cs

+            long? currentChainSize = _chainEstimations.PruningSize;
+            if (currentChainSize is null)
+            {
+                if (_logger.IsWarn) _logger.Warn("Full Pruning: Chain size estimation is unavailable.");


Is this user problem?

LukaszRozmej · 2023-05-16T12:45:49Z

src/Nethermind/Nethermind.Blockchain/FullPruning/PruningStatus.cs

+    /// <summary>
+    /// Pruning failed because of low disk space
+    /// </summary>
+    NotEnoughDiskSpace,


very minor - thinking if we shoulnd't keep this enum in some order, so this would go after Disabled

LukaszRozmej · 2023-05-16T12:46:43Z

src/Nethermind/Nethermind.Db/IPruningConfig.cs

@@ -63,5 +63,8 @@ public interface IPruningConfig : IConfig
                                  "'AlwaysShutdown': shuts Nethermind down once the prune completes, whether it succeeded or failed.",
            DefaultValue = "None")]
        FullPruningCompletionBehavior FullPruningCompletionBehavior { get; set; }
+
+        [ConfigItem(Description = "Enables available disk space check.", DefaultValue = "true", HiddenFromDocs = true)]


We want this to be hidden from docs?

LukaszRozmej · 2023-05-16T12:47:44Z

src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs

@@ -314,7 +315,10 @@ private Task InitBlockchain()
                        api.PruningTrigger.Add(pruningTrigger);
                    }

-                    FullPruner pruner = new(fullPruningDb, api.PruningTrigger, pruningConfig, api.BlockTree!, stateReader, api.ProcessExit!, api.LogManager);
+                    IDriveInfo drive = api.FileSystem.GetDriveInfos(fullPruningDb.GetPath(initConfig.BaseDbPath))[0];


StateDb Path should be used, this can be mapped to different drive.

LukaszRozmej · 2023-05-16T12:49:40Z

src/Nethermind/Nethermind.Blockchain/KnownChainSizes.cs

+                BlockchainIds.Goerli => new ChainEstimations(
+                    new LinearExtrapolation(8490.MB(), 15.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.Rinkeby => new ChainEstimations(
+                    new LinearExtrapolation(34700.MB(), 20.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.Ropsten => new ChainEstimations(
+                    new LinearExtrapolation(35900.MB(), 25.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.Mainnet => new ChainEstimations(
+                    new LinearExtrapolation(90000.MB(), 70.MB(), new DateTime(2022, 04, 7)),
+                    new LinearExtrapolation(163.GB(), new DateTime(2023, 4, 11, 19, 49, 0),
+                        177653453177, new DateTime(2023, 04, 29, 00, 50, 0))),
+                BlockchainIds.Gnosis => new ChainEstimations(
+                    new LinearExtrapolation(18000.MB(), 48.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.EnergyWeb => new ChainEstimations(
+                    new LinearExtrapolation(15300.MB(), 15.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.Volta => new ChainEstimations(
+                    new LinearExtrapolation(17500.MB(), 10.MB(), new DateTime(2021, 11, 7))),
+                BlockchainIds.PoaCore => new ChainEstimations(
+                    new LinearExtrapolation(13900.MB(), 4.MB(), new DateTime(2021, 12, 7))),
+                BlockchainIds.Sepolia => new ChainEstimations(null,
+                    new LinearExtrapolation(2457.MB(), new DateTime(2023, 04, 9, 21, 11, 0),
+                        3699505976, new DateTime(2023, 04, 28, 20, 18, 0))),
+                _ => UnknownChain.Instance
+            };


@kamilchodola can you update chain sizes post newest tests?

We have problems with chain sizes, they are very unstable. I'm working on it right now

I guess we should do that a bit in similar way as we had with Pivots.
Tests will store data about chain size and then action will fetch them and apply in code.
Would be worth to read it from json file rather than hardcoding

LukaszRozmej · 2023-05-16T12:50:35Z

is there any chance to put some kind of logs which will inform user that space is running out and full pruning or resync is advised?

Warn level?

This could be checked and written in healthchecks

deffrian · 2023-05-17T15:05:50Z

@kamilchodola I will create new pr with progress logger and any other logging you want

# Conflicts: # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPrunerTests.cs

This reverts commit ca09576.

This reverts commit ca09576

deffrian added 2 commits April 5, 2023 22:32

Available space check

9bedec0

Add import

bff7b79

deffrian linked an issue Apr 10, 2023 that may be closed by this pull request

Safety check for full pruning #5495

Closed

deffrian added 3 commits April 10, 2023 13:38

Merge remote-tracking branch 'origin/master' into 5495-safety-check-f…

0844cc1

…or-full-pruning # Conflicts: # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPrunerTests.cs

Add tests & refactor

a2a3c60

Refactoring

730542f

MarekM25 reviewed Apr 11, 2023

View reviewed changes

src/Nethermind/Nethermind.Blockchain/FullPruning/PruningStatus.cs Outdated Show resolved Hide resolved

MarekM25 reviewed Apr 11, 2023

View reviewed changes

src/Nethermind/Nethermind.Blockchain/FullPruning/FullPruner.cs Outdated Show resolved Hide resolved

deffrian added 7 commits April 24, 2023 12:58

Refactor & increase threshold

6f6d1f1

Fix build

5ba8f00

Fix build

f21fd71

Update required space

584c580

Add test

4ee2619

Merge branch 'master' into 5495-safety-check-for-full-pruning

3111f28

# Conflicts: # src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs

deffrian marked this pull request as ready for review May 2, 2023 07:48

deffrian changed the title ~~[Draft] Safety check for full pruning~~ Safety check for full pruning May 2, 2023

benaadams approved these changes May 8, 2023

View reviewed changes

src/Nethermind/Nethermind.Db/IPruningConfig.cs Outdated Show resolved Hide resolved

deffrian and others added 2 commits May 9, 2023 01:16

Fix typo

6820dc3

Co-authored-by: Ben Adams <thundercat@illyriad.co.uk>

Merge branch 'master' into 5495-safety-check-for-full-pruning

00408e9

LukaszRozmej approved these changes May 16, 2023

View reviewed changes

Nikita Mescheryakov added 4 commits May 18, 2023 00:14

Merge branch 'master' into 5495-safety-check-for-full-pruning

6cb6252

# Conflicts: # src/Nethermind/Nethermind.Blockchain.Test/FullPruning/FullPrunerTests.cs

Adjustments

b97321a

Merge branch 'master' into 5495-safety-check-for-full-pruning

64d9f99

Update chain sizes

2bdef94

LukaszRozmej approved these changes Jun 12, 2023

View reviewed changes

deffrian merged commit ca09576 into master Jun 12, 2023

deffrian deleted the 5495-safety-check-for-full-pruning branch June 12, 2023 15:24

marcindsobczak added a commit that referenced this pull request Jun 14, 2023

Revert "Safety check for full pruning (#5550)"

0626087

This reverts commit ca09576.

marcindsobczak added a commit that referenced this pull request Jun 19, 2023

Revert "Safety check for full pruning (#5550)"

b9345c1

This reverts commit ca09576

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Safety check for full pruning #5550

Safety check for full pruning #5550

deffrian commented Apr 10, 2023 •

edited

Loading

kamilchodola commented May 8, 2023

benaadams commented May 8, 2023

LukaszRozmej May 16, 2023

LukaszRozmej May 16, 2023

LukaszRozmej May 16, 2023

LukaszRozmej May 16, 2023

LukaszRozmej May 16, 2023

LukaszRozmej May 16, 2023

deffrian May 17, 2023

kamilchodola May 18, 2023

LukaszRozmej commented May 16, 2023

deffrian commented May 17, 2023

		@@ -158,6 +170,28 @@ private void SetCurrentPruning(IPruningContext pruningContext)

		private bool CanStartNewPruning() => _fullPruningDb.CanStartPruning;

		private const double ChainSizeThresholdFactor = 1.3;

Safety check for full pruning #5550

Safety check for full pruning #5550

Conversation

deffrian commented Apr 10, 2023 • edited Loading

Changes

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

kamilchodola commented May 8, 2023

benaadams commented May 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LukaszRozmej commented May 16, 2023

deffrian commented May 17, 2023

deffrian commented Apr 10, 2023 •

edited

Loading