What Does Disk Usage Go Up Continually, Even More So in Pruned Mode?

[zicklag]

I ran some experiments with the substrate node template where I set the block time to 10ms so that blocks would be produce incredibly fast, and I was surprised to find that the database growth, in the scenario where all the generated blocks are empty except for the timestamp, didn't seem to be helping.

For example, after 5000 blocks in archive mode, the database directory was 20M. After the same amount of blocks with the pruning set to retain only 5 blocks, the database directory was actually larger at 55M!

What's going on here? That seems pretty strange. :)

Ideally, for my use-case I would like to have an upper-bound on how large the database usage will get. I don't need a permanent block history all the way to genesis in my case. I would like the ability for users to opt-in to having the whole chain without having to require it.

[cheme]

Indeed blocks are not pruned, it is a deliberate choice to advocate users to keep chain history on their full nodes (same thing with warp sync, history will be download in background).
Pruning is only allowed in substrate current code on the trie state (because it can be rebuild from block history).
Not sure about grandpa attestation (consensus history uses space too).
Not sure if it can be of any help, but light client could also be an alternative, these days efforts are focused on smoldot.

[arkpar]

Pruning setting only affects the state available. I.e. with pruning 100 you can query the state of the last 100 finalized blocks, but no further. The nodes still stores full history of block headers, bodies and signatures.

You can also try using experimental --keep-blocks option, that limits stored block body history, but still keeps the headers. Note that if all the nodes on your network don't have the block history, your only option to sync a new node with the network is to use fast or warp sync.

[zicklag]

Awesome, thank you! I'll try that out.

Is there any big disadvantage to needing to use fast or warp sync for adding new nodes?

[zicklag]

I tested out using --keep-blocks 5 --pruning 5 and the behavior now is that the disk usage climes steadily while the node is running, reaching roughtly 120MB after 20,000 blocks, but if I stop the node and restart it, it seems to do some sort of cleanup operation or something, because then the disk usage drops down to about 5MB, and then starts climbing again.

What is happening when I restart the node and it cleans up all that extra space? Is there a way to force it to do that periodically, without having to restart the node?

[zicklag]

Ah, I think I figured it out. It's got to be some sort of behavior driven by RocksDB, not by Substrate directly.

Testing out the ParityDB has the exact opposite behavior where it seems to do a large filesystem sync when the node exits and the disk usage grows dramatically when I shut down the node.

I'll have to look into RocksDB's configuration options to see if I can get it to clean out the extra disk space periodically. ( or maybe it already does and I haven't waited long enough for it )

[arkpar]

Is there any big disadvantage to needing to use fast or warp sync for adding new nodes?

It offers less security, although still enough for any practical use.

The effects you observe regarding database compaction should not matter for any real world blockchain application. Substrate is optimized for data size over 1Gb. On that scale temporary database overhead is insignificant.

Even with --keep-blocks and aggressive database settings you won't be able to achieve const size database. It keeps accumulating block headers and signatures. To verify that the block is valid the node at least needs to verify the full history of validator set transitions that lead to current state. Substrate does not currently support any kind of checkpoint mechanism, that would allow to get around that.

If you are developing for the constrained embedded environment take a look at smoldot as @cheme suggested.

[zicklag]

although still enough for any practical use.

Awesome, thanks!

The effects you observe regarding database compaction should not matter for any real world blockchain application. Substrate is optimized for data size over 1Gb. On that scale temporary database overhead is insignificant.

OK, cool. I found out that the strange behavior was from the WAL files created by RocksDB, and there is an option to set a maximum on the WAL file size so that should fix even the behavior that I noticed I think.

Even with --keep-blocks and aggressive database settings you won't be able to achieve const size database. It keeps accumulating block headers and signatures. To verify that the block is valid the node at least needs to verify the full history of validator set transitions that lead to current state.

That makes good sense. I'll have to measure the size of the blocks and signatures to see if will be enough space to cause me any concern. It might be that the headers are small enough for me to be fine with how much they might build up over time.

I'm not exactly in a resource constrained environment, but I want the service I'm writing to sit very much in the background on a server that will primarily be meant to run other applications, so the lower I can get it's resource usage, the better.

If you are developing for the constrained embedded environment take a look at smoldot as @cheme suggested.

Ah! I had seen Smoldot before, but I had accidentally overlooked the fact that it could also be used as a full-blown substrate node, not just a light client. That definitely looks like it could be useful for my scenario.

Thanks a lot for your help! It's been very useful!