feat: check cids in denylist before providing (#215)

- cap the max number of cids we'll accept in a single message. 
- We're seeing 20k spikes in our `bitswap-pending-entries` metrics per
node, so I'm putting in a hard cap of 500 wanted cids per message that
we'll process. The caller can ask again if they need more. This also
means i can put a sensible cap on how many cids the denylist service
should handle in a batch.
- check batches of cids against our denylist api.
- cache entries that are on the denylist; they are rarely removed.
- use cache to avoid asking about items we already know, and as a
fallback if denylist service cannot be reached.
- add `bitswap-denied` counter metric to see how many CIDs we skip due
to being on the denylist


see: batch endpoint for denylist api –
storacha/reads#166
see: set DENYLIST_URL in env -
elastic-ipfs/bitswap-peer-deployment#99

License: MIT

---------

Signed-off-by: Oli Evans <oli@protocol.ai>

.env.sample

@@ -12,3 +12,4 @@ DYNAMO_LINK_TABLE_V1=
 CACHE_BLOCK_INFO=true
 CACHE_BLOCK_SIZE=128
 CONCURRENCY=4
+DENYLIST_URL=https://denylist.dag.haus

README.md

@@ -57,6 +57,7 @@ _Variables in bold are required._
 | NODE_DEBUG            |               | If it contains `aws-ipfs`, debug mode is enabled.                        |
 | LOG_LEVEL            | `info` | Logging level. |
 | LOG_PRETTY            | `false` | Enable pretty logging. |
+| DENYLIST_URL | `https://denylist.dag.haus` | URL for cid checking api. |
 
 Also check [AWS specifics configuration](https://github.com/elastic-ipfs/elastic-ipfs/blob/main/aws.md).
 

metrics.yml

@@ -60,12 +60,14 @@ metrics:
       description: BitSwap Total Entries served
     bitswap-block-error:
       description: Block error (on parsing)
-
+    bitswap-denied:
+      description: Count of CIDs found on denylist
+
   process:
     elu: 
       name: bitswap-elu
       description: Bitswap Event Loop Utilization
       interval: 500
 
-version: 0.3.0
-buildDate: "20230426.1252"
+version: 0.4.0
+buildDate: "20230706.1402"

package.json

@@ -28,6 +28,8 @@
     "libp2p": "0.42.2",
     "lru-cache": "7.14.1",
     "mnemonist": "0.39.5",
+    "multiformats": "^10.0.3",
+    "p-retry": "^5.1.2",
     "pino": "8.8.0",
     "piscina": "3.2.0",
     "protobufjs": "7.2.0",

src/config.js

@@ -74,7 +74,9 @@ export function makeConfig () {
     s3MaxRetries: process.env.S3_MAX_RETRIES ? parseInt(process.env.S3_MAX_RETRIES) : 3,
     s3RetryDelay: process.env.S3_RETRY_DELAY ? parseInt(process.env.S3_RETRY_DELAY) : 100, // ms
 
-    allowReadinessTweak: process.env.ALLOW_READINESS_TWEAK === 'true'
+    allowReadinessTweak: process.env.ALLOW_READINESS_TWEAK === 'true',
+
+    denylistUrl: process.env.DENYLIST_URL ? new URL(process.env.DENYLIST_URL) : undefined
   }
 }