Create Benchmarking Setup for Identity Pallet #4695

[shawntabrizi]

This PR introduces a pipeline to benchmark Substrate FRAME Pallets.

Changes In this PR

1. This PR introduces a bench database which is useful specifically for benchmarking as it gives the runtime access to control the database through host functions listed below.

2. This PR introduces a new set of host functions under the name benchmarking. These host functions allow you access to:

   • Get the current system time in nano-seconds.
   • Reset the trie db to genesis state.
   • Commit any pending storage changes to the trie db, flushing the db cache as well.

3. This PR also introduces a new Runtime API, Benchmark_dispatch_benchmark, which allows you to call into the runtime to execute benchmarking tests.

4. This Runtime API is easily accessible through a new CLI sub-command substrate benchmark which allows the user to execute benchmarks from their Substrate node executable.

5. A set of benchmarks which use this pipeline have been added for the Identity Pallet and all of its extrinsics.

How to Use

The Substrate node CLI now exposes a benchmark sub-command which can be used to run the Pallet benchmarks.

For example:

substrate benchmark --chain dev --execution=wasm --wasm-execution=compiled --pallet pallet-identity --extrinsic add_registrar --steps 10 --repeat 100 > add_registrar.csv

As shown above, the output of this command will be the benchmark results in a CSV format.

CLI Params

The benchmark sub-command shares the global node configuration parameters. Of note, when testing the Substrate runtime, it is important to set the following:

• --execution=wasm: Ensures you are running the benchmark completely in the Wasm environment.
• --wasm-execution=compiled: Ensures that you are using the compiled Wasm versus the much slower interpreted Wasm.
• --chain <your chain spec>: Needed to specify the genesis state of your benchmarking setup.

Custom parameters include:

• --pallet/-p: The pallet you want to test (i.e. pallet-identity or identity.
• --extrinsic/-e: The extrinsic you want to test (i.e. set_identity).
• --steps/-s: The maximum number of sample points to take between component ranges (see below) [Default: 1].
• --repeat/-r: The number of times to repeat every benchmark [Default: 1].

If you put an invalid --pallet or --extrinsic, the CLI will return an error and no benchmarks will be run.

Output

The output will look something like:

Pallet: "pallet-identity", Extrinsic: "request_judgement", Steps: 10, Repeat: 100
R,X,time
1,50,610000
1,50,523000
1,50,531000
1,50,529000
...

Other than the first line, which is metadata about what has been run, the rest of the output is in CSV format. time is always measured in nanoseconds.

Creating Benchmarks for a Pallet

Using the patterns defined in this PR, you should be able to create benchmarks for any pallet.

1. Create a rust module benchmark.rs in your runtime pallet.
2. Implement the Benchmarking trait for Module<T>

/// The pallet benchmarking trait.
pub trait Benchmarking<T> {
	/// Run the benchmarks for this pallet.
	///
	/// Parameters
	/// - `extrinsic`: The name of extrinsic function you want to benchmark encoded as bytes.
	/// - `steps`: The number of sample points you want to take across the range of parameters.
	/// - `repeat`: The number of times you want to repeat a benchmark.
	fn run_benchmark(extrinsic: Vec<u8>, steps: u32, repeat: u32) -> Result<Vec<T>, &'static str>;
}

3. Create a struct representing each extrinsic, and implement BenchmarkingSetup for that struct:

/// The required setup for creating a benchmark.
pub trait BenchmarkingSetup<T, Call, RawOrigin> {
	/// Return the components and their ranges which should be tested in this benchmark.
	fn components(&self) -> Vec<(BenchmarkParameter, u32, u32)>;

	/// Set up the storage, and prepare a call and caller to test in a single run of the benchmark.
	fn instance(&self, components: &[(BenchmarkParameter, u32)]) -> Result<(Call, RawOrigin), &'static str>;
}

The specific implementation of these functions is left to the user, but the implementation done in this PR is meant to be reusable and general for any benchmarks.

When running an actual benchmark, you should be sure to excute in a way similar to this:

// Set up the externalities environment for the setup we want to benchmark.
let (call, caller) = <SelectedBenchmark as BenchmarkingSetup<T, crate::Call<T>, RawOrigin<T::AccountId>>>::instance(&selected_benchmark, &c)?;
// Commit the externalities to the database, flushing the DB cache.
// This will enable worst case scenario for reading from the database.
sp_io::benchmarking::commit_db();
// Run the benchmark.
let start = sp_io::benchmarking::current_time();
call.dispatch(caller.into())?;
let finish = sp_io::benchmarking::current_time();
let elapsed = finish - start;
results.push((c.clone(), elapsed));
// Wipe the DB back to the genesis state.
sp_io::benchmarking::wipe_db();

NOTE: You may need to warm up the DB cache by calling commit_db and wipe_db once before starting your benchmarks.

4. Pipe the run_benchmarks call to map to your respective benchmark. This is a little nasty for now, but should be fixed with Create macros to simplify benchmarking. #4861

Exposing Your Pallet Benchmarks

After you have written the benchmarks for your pallet, you need to expose them through your Substrate node. You can do this via a runtime api in your main node file:

sp_api::decl_runtime_apis! {
	pub trait Benchmark
	{
		fn dispatch_benchmark(module: Vec<u8>, extrinsic: Vec<u8>, steps: u32, repeat: u32) -> Option<Vec<BenchmarkResults>>;
	}
} 

You will need to implement that API for your runtime:

impl crate::Benchmark<Block> for Runtime {
	fn dispatch_benchmark(module: Vec<u8>, extrinsic: Vec<u8>, steps: u32, repeat: u32) -> Option<Vec<BenchmarkResults>> {
		match module.as_slice() {
			b"pallet-identity" | b"identity" => Identity::run_benchmark(extrinsic, steps, repeat).ok(),
			_ => return None,
		}
	}
}

You can simply map the appropriate strings to your Module's run_benchmark function.

Benchmarking Philosophy

You can read the philosophy and process we are using for benchmarking and weighing pallets here: https://hackmd.io/UD0HojfARqyUMC9Jxs5-RA

[gavofyork]

Logic seems reasonable (sans macros). Formatting needs some love.

fn foo(self) -> Self
{
    bar()

is wrong. The rule is that two neighbouring lines at the same indent level should be syntactically interchangeable, at least in terms of their initial token. Also, indent levels should never increase or decrease by more than one per line.

As such it's either:

fn foo(self) -> Self {
    bar()

Or:

fn foo(self)
    -> Self
{
    bar()

Or:

fn foo(
    self
) -> Self {
    bar()

Or (though I've never needed to use this myself):

fn foo(
    self
)
    -> Self
{
    bar()

[shawntabrizi]

Should I hide the entire benchmarking pipeline behind a feature flag?

This means by default we should:

• Not expose a runtime API
• Not have CLI
• Not have benchmarking code in our runtime wasm

Is it possible?

[gavofyork]

i think so; we don't really want benchmarking code making it on to the chain.

[gavofyork]

double ident level