Allow transaction hash mining to prioritize transactions and protect agains spam attacks

[ThomasLobker]

Problem

Allowing completely free transactions has many great advantages, but it also poses a risk to the ecosystem. A bad actor may attack the network by anonymously broadcasting malicious transactions. Even requiring a drop of GAS is not sufficient to prevent such an attack, because based on the current value of GAS it will cost an attacker only about a dollar per week to keep flooding the mempool and preventing regular users from using the network.

Suggestion

It should be fairly easy to allow wallets to mine for transaction hashes that meet a certain difficulty. Then the seed nodes can prioritize the mempool based on the matched difficulty of the transaction hashes. If finding a difficult hash takes one or two seconds, then it will already be significantly more difficult for a bad actor to flood the mempool with malicious transactions.

How it works

NEO transactions have a remark field that allows the user to add any random string. Randomizing this value will result in a new transaction hash. We can keep randomizing this value until we find a transaction hash that meets our requirements.

For example, if the user wants a difficulty level 8 for his transaction, then the wallet will find a random string that makes the reversed transaction hash in bits end with at least 8 zeroes. Effectively making the transaction hash in hex start with double zero. The user can increase or decrease this difficulty level based on his computing power and how much priority he needs.

The node receiving the transaction in broadcast should sort all transactions in the mempool first by difficulty level, thus by how many zeroes are in the end of the reversed transaction hash in bits, then by attached network fee.

Example

Hash: 0004f2bc101398797cb21b43782b93e1ab6bc781264281fa3659208958420744
Starts with: 00 04
Reversed: 04 00
Bits: 0000 0100 0000 0000
Difficulty used to mine the hash: 10 (ends with 10 zeroes)

With JavaScript it should take no more than a couple of seconds to find such a hash. I made a small proof of concept to create transactions and mine for a hash with a certain difficulty. The goal is to keep usage on NEO free, but also add some protection against flooding.

Hoping to see some opinions

[shargon]

I like it :)
Maybe we can drop TX that don't meet this difficulty when we have high loads

[ThomasLobker]

I made a false assumption that you can only calculate the transaction hash after you signed the transaction, but this is not true. So we would need to consider ASIC hashing as well, or do something more clever that you will need to include one of the last block hashes to prove that you didn't premine the transactions.

Thanks @decentralisedkev for noticing 👍

[i359]

Maybe we can also consider sorting by coin age.

[igormcoelho]

No, please @ThomasLobker, let's not transform Neo in a useless proof-of-work. It's exactly the energy waste of Ethereum and Bitcoin that brought me to Neo.
But I have to admit that, all efforts I've pursued so far was to guarantee fee tx for big business (such as the Staking neo-project/proposals#66). Your idea will allow even small business to have free tx. So, if you want to co-author the Smart Transaction Identification System (neo-project/proposals#67), please take a look and propose a three letter code for this "proof-of-work" on Neo... so at least it won't be a random remark field, we can try to put this together inside an organized NEP. @decentralizedkev is right, best approach is to include the hash of one previous XXX block, and after a given number of blocks it's automatically expired.

[shargon]

Is not a "proof of work" @igormcoelho , is a delay for prevent the spam. You don't win nothing with this work, it is a "proof of bruteforce" :P

[ThomasLobker]

Other ideas to prevent spam attacks:
#358
#357

[erikzhang]

If a average user takes 1 second to calculate the transaction hash, the attacker can perform a 1000tps attack with only 1000 devices. But if you increase the difficulty by 1000 times, you can stop the attack, but the average user will take 1000 seconds to send a transaction.

So maybe the difficulty should change with the change of TPS?

Or do not set the difficulty, sort the transactions in the memory pool by hash, the smaller the hash value, the higher the priority?

[ThomasLobker]

@igormcoelho Yes, technically it's proof of work, but not in the sense of energy waste. Since you are not putting machines at work and not getting network strenght out of hashing power.

I believe that if we combine a couple of ideas, we can have a really robust network that is resistant to spam attacks, free to use and doesn't need complex staking mechanisms.

1. Order transactions in mempool by difficulty (require work per transaction)
2. Drop transactions older than the last 100 blocks from the mempool
3. Require a reference to a recent block hash (within last 100 blocks) to proof the transaction is still current

@erikzhang I would propose to only use the difficulty for sorting. So it won't actually prevent spam from being relayed on the network, it will just give these transactions very low priority. So regular users are not bothered by it. This will remove the incentive for a spammer to keep attacking the network.

Maybe individual seed nodes can configure a minimum difficulty. And we can add RPC calls to ask a seed node for the minimum and average difficulty levels. So in the interface of the wallet, a user can have a nice slider to increase priority if necessary.

[erikzhang]

Maybe individual seed nodes can configure a minimum difficulty. And we can add RPC calls to ask a seed node for the minimum and average difficulty levels. So in the interface of the wallet, a user can have a nice slider to increase priority if necessary.

I guess you are talking about the consensus nodes. But for security, the consensus nodes will not announce their IP addresses, nor will it open the RPC port.

[ThomasLobker]

I guess you are talking about the consensus nodes. But for security, the consensus nodes will not announce their IP addresses, nor will it open the RPC port.

I'm not talking about about the consensus nodes.

What I meant is that neo-cli could have a local setting to configure the minimum required difficulty. And the API would be extended with methods like getminimumdifficulty and getdifficultystats so that wallets can determine the best difficulty to apply for new transactions.

Would it be better to have a network wide minimum difficulty?

[dicarlo2]

Sounds like a similar concept to how Nano prevents spam attacks, I like it!

[jsolman]

In this proposal, you are suggesting to sort first by hash with the most leading zeroes. I definitely disagree with using POW to sort transactions over fees. Fees pay for the network, while POW still ends up costing money in the form of the cost of energy, but that money doesn’t go to those securing the network to pay their operating costs. Also, if we sorted by PoW first a person with the best ASIC could just flood the network and make it impossible for anyone else to get a transaction with high enough difficulty hash to even make it on the network.

Aside from that doing proof of work on devices with low powered processors would become infeasible, and that is a use case Neo should support. Clients should not be required to do high amounts of proof of work to get a transaction confirmed.

Now if we wanted to make a few different pools within the mempool sorted by different criteria I could support this being one way to get a transaction onto the network without a fee, but only if it did not affect the separate pool that is sorted first by fee per byte.

[igormcoelho]

@jsolman In fact, at first I saw as a PoW too, but now I'm convinced it's not PoW, it's just a forced-delay anti-spam measure (like captcha or slower hashings). So I vote positive for it. Now we have alert p2p message that can be used to inform difficulty directly from CN.

[jsolman]

@igormcoelho regardless of whether you see it as PoW it is by definition a proof of work to order transactions that is being proposed. Calling it a forced-delay anti-spam measure doesn’t take away that the mechanism for achieving it from this proposal is a proof of work.

@igormcoelho wouldn’t you be concerned about someone with high compute resources DDOSing the network with spam? If there were some equation where the score for ordering took into account both fee and the PoW, it could mitigate the issue, but in that case someone with huge compute power could cause everyone without it to pay high fees to get a transaction accepted on the network.

It’s not like captcha; captcha is meant to be a proof of work that a person can do that a computer cannot.

[RavenXce]

@jsolman I think this proposal is simply meant to sort free transactions, and to continue allowing them instead of requiring an arbitrary fee for all txns. Paid transactions could still get priority based on how much the user is willing to pay per byte? Using txn mining as another mechanism to order txns, and also drop those that don't meet the criteria (min difficulty or min fee/byte) during high loads seem like a good compromise.

[jsolman]

If it is only applied to sort free transactions, then I don’t have any major objection, but there should be a specific timeline when consensus nodes will begin enforcing the ordering.

[ThomasLobker]

The idea is meant as one layer of anti-spam, there will still be a need for other layers as well. I'm not sure if ASIC's can be used in such a scenario, this needs to be investigated.

I think this proposal is simply meant to sort free transactions, and to continue allowing them instead of requiring an arbitrary fee for all txns.

This exactly 👍

[canesin]

@ThomasLobker this is a great proposal. I personally like the idea of having several small features to control spam instead of a silver bullet. Also think the inclusion of latest seem blockhash in the TX, so that nodes can drop old transactions when network is saturated is a very good idea.
sorting fees + drop old + mining threshold .. seems we are getting there.

[belane]

I would suggest that all transactions should meet this proof but not order the pool according to this parameter.
In this way it helps against transaction spam but does not produce a PoW war to prioritize transactions.
As @dicarlo2 said, Nano is using it to combat spam, something that a takes less than a second for a smartphone.

[jsolman]

@belane Hmmm, How much will this really help preventing spam though if a smartphone can do the PoW in 1 second? Anyone that really wants to spam the network could certainly throw together enough computing power to fill max free transactions allowed in the block with spam if it is that low amount of compute required. Is the goal of this just to make it a little harder to spam free transactions? It seems to me more like free transactions should potentially become a PoW war if we are going to require PoW for free transactions.

If there is a resonably sized fee in the transaction why would it even need a PoW spam prevention measure? Shouldn’t a low power device be able to include a fee to avoid the PoW?

[belane]

Being a small PoW you can use enough computing power to continue spamming, but it is a small protection that at least requires an effort to the attacker.
Currently there is no protection, as @canesin says is the sum of all that can help us keep free TX.

We have to remember that @ThomasLobker proposes a timestamp (or block number) in the PoW so, generating transactions for a spam attack would have a limited time.

Ordering the free transactions by the PoW, in my opinion, would be an error because it would generate a PoW war.

[jsolman]

Hmm. If only supporting 20 free transactions per block, then it doesn’t matter if an attacker has to generate the PoW since the last block; Doing 20 proof of work computations that take 1 second on a cell phone can easily be done by a powerful machine long before the block time expires. Won’t this still make it easy to spam the network with >100 free transactions per block and effectively deny service for those attempting free transactions.

I understand the idea that some protection is better than none, but we need to ensure the amount of protection it provides is worth it before accepting a particular proposal.

[canesin]

@belane what is the problem of a PoW competition for free transactions ?

This proposal doesn't remove the sorting on fees per byte so just affect sorting of free transactions.
One could pay the cost in two ways: attaching a fee or mining a free transaction. Can't we carefully select a mining algorithm that is hard for ASICs and GPUs ?

Agree with you @jsolman and have the general feeling that proposals are moving too fast in NEO, with too short descriptions and specification discussions. Free transactions are super hard.

[ThomasLobker]

To be clear, this topic was not meant as a proposal, but only as a discussion. When all uncertainties are cleared up, we can move forward to an official proposal.

[shargon]

I think that this is one of the best improves for mantain the free fee in a network. Is not the unique, but is a needed feature for increase the viability of our awesome free network :)

For me, PoW should not be used for ordering (is not necessary i think), but we should be allowed to filter or remove TX from the pool according to his difficulty. And this difficulty should be defined acording the number of TX in the pool

Again, good job @ThomasLobker

[jsolman]

Since transactions are ordered by hash after ordering by fee, this is already effectively done by #500. Anyone could change anything about the transaction (such as an attribute with a random number) to get a different hash and fight for the lowest hash. It won't start a hash war though really because, just pay .00000001 gas more and you will be higher priority. If we really want we could create a 3rd category for transactions with 0 fees and allow a few (maybe 10 per block) to make it in each block or something; right now anything less than .001 is treated as low priority (we currently allow 20 per block) and there isn't a separate category for free transactions.

[jsolman]

I guess it is debatable whether we should allow a few completely 0 transaction fees into every block as a third category, and whether this issue should be kept open until that were implemented.

[jsolman]

With the MemPool having better performance it probably isn't necessary to require PoW to protect from spam attack, unless requiring it across the board.

The title of this issue implies two things:

• Allow transaction hash mining to prioritize transactions
  • This is already achieved by Improve Large MemoryPool Performance - Sort + intelligent TX reverification #500 since it orders by hash
• and protect agains spam attacks
  • if a PoW were required to stop spamming with too many transactions it wouldn't really stop the potential for DDOS even if the PoW wasn't met someone could keep sending the transactions and make the node have to verify them to check that the PoW was met.

So with those things considered I think this issue could be closed by #500 - with the only caveat being whether we should allow a few completely free transactions into every block as a third category -- however, I'm against that as we don't really want to promote a hash war to get free transactions into the network. People can do it now, but they should be always beaten by adding fee. Maybe we can build replace w/ fee functionality like Bitcoin has if someone wants to start with a low fee and raise fee if needed to get their transaction to verify sooner.

[igormcoelho]

I fully agree that if txs are sorted on mempool, PoW behavior occurs naturally. So, PR #500 contemplates this proposed feature. Again, congratulations to Thomas Lobker for indicating this.