Allow Optionally Reducing The Instruction Sighash Length Or Matched Bytes

[bonedaddy]

Overview

Solana transaction sizes are fairly restrictive right now, with sufficiently complex instructions, it can be quite trivial to reach the maximum transaction size. Part of this is that the anchor instruction sighash takes up 8 bytes of space, as opposed to the traditional approach with vanilla programs often only needing 1 byte of space to indicate which instruction is being used.

While not exceedingly large, I'm aware of at least ~3 instructions in various Tulip programs that were larger than the transaction size limitation, which required some refactoring and breaking up the instruction into two different ones, which would've been possible to avoid if the instruction sighash length was reduced by 50% (from 8 bytes -> 4 bytes).

Suggested Workaround

There are four workarounds I can think of

1. use the fallback function + vanilla program approach whereby instructions are generally indicated with an enum using a u8 representation, thus only consuming 1 byte. This is non-ideal as it kind of invalidates on of the biggest selling points of anchor (imo)
2. Optional feature flag that allows for a "shortened" instruction sighash (ie: reducing the generated sighash from 8 bytes to 4 bytes).
3. Optional feature flag that allows for matching incoming instructions against a partial sighash (ie: only comparing against the first 4 bytes of the sighash instead of the 8)
4. Macro that allows specifying how many bytes to include for an instruction sighash. For example you could tag a function with a macro like #[sighash(bytes = N)] which only match against the first 4 bytes of the isntruction sighash.

#[sighash(bytes = 4)]
pub fn wao(ctx: .....) -> Result<()> {

}

When the sighash is generated by running sha256(global:wao), only the 4 bytes of the generated hash will be used as the instruction sighash.

My preferred option is 4

Note On Instruction Collision

Because the sighash is really a truncated sha256 hash, care needs to be taken that developers are not exposed to a realistic chance of a hash collision in the instruction sighash. The current 8 byte sighash pretty much guarantees this will never happen, however once the sighash length/matched bytes gets dropped by more than 50% the odds of a collision start increasing rather significantly.

Matching against a 4 byte sighash still maintains pretty high guarantees that a collision will not happen as per the following image taken from here.

[acheroncrypto]

Custom discriminators (#3097) solved this problem.