Implement engine_getBlobsV1

[michaelsproul]

Implement the new API for fetching blobs from the mempool to improve propagation:

• Define engine_getBlobsV1 ethereum/execution-apis#559

[Rjected]

just noting that requesting by commitment or versioned hash will require refactoring our on disk blob store to work with those types

[michaelsproul]

Thanks for the feedback @Rjected

Maybe the on-disk blob store could keep a mapping from versioned hash to transaction hash(s)? It could build the mapping at startup and keep it in memory? Or it could write the mapping to disk and read/write from it on-demand. My intuition says the in-memory mapping would be easier because there aren't too many blobs, but you'd be a better judge of whether that's feasible.

I'm guessing the DiskFileBlobStore here is what gets used in prod?

reth/crates/transaction-pool/src/blobstore/disk.rs

Lines 252 to 271 in de6332b

	/// Retrieves the blob data for the given transaction hash.
	#[inline]
	fn read_one(&self, tx: B256) -> Result<Option<BlobTransactionSidecar>, BlobStoreError> {
	let path = self.blob_disk_file(tx);
	let data = {
	let _lock = self.file_lock.read();
	match fs::read(&path) {
	Ok(data) => data,
	Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(None),
	Err(e) => {
	return Err(BlobStoreError::Other(Box::new(DiskFileBlobStoreError::ReadFile(
	tx, path, e,
	))))
	}
	}
	};
	BlobTransactionSidecar::decode(&mut data.as_slice())
	.map(Some)
	.map_err(BlobStoreError::DecodeError)
	}

[Rjected]

Thanks for the feedback @Rjected

Maybe the on-disk blob store could keep a mapping from versioned hash to transaction hash(s)? It could build the mapping at startup and keep it in memory? Or it could write the mapping to disk and read/write from it on-demand. My intuition says the in-memory mapping would be easier because there aren't too many blobs, but you'd be a better judge of whether that's feasible.

I'm guessing the DiskFileBlobStore here is what gets used in prod?

reth/crates/transaction-pool/src/blobstore/disk.rs

Lines 252 to 271 in de6332b

	/// Retrieves the blob data for the given transaction hash.
	#[inline]
	fn read_one(&self, tx: B256) -> Result<Option<BlobTransactionSidecar>, BlobStoreError> {
	let path = self.blob_disk_file(tx);
	let data = {
	let _lock = self.file_lock.read();
	match fs::read(&path) {
	Ok(data) => data,
	Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(None),
	Err(e) => {
	return Err(BlobStoreError::Other(Box::new(DiskFileBlobStoreError::ReadFile(
	tx, path, e,
	))))
	}
	}
	};
	BlobTransactionSidecar::decode(&mut data.as_slice())
	.map(Some)
	.map_err(BlobStoreError::DecodeError)
	}

Yeah, the DiskFileBlobStore is used in practice. Here is where the blob is primarily stored in the blob store:
https://github.com/paradigmxyz/reth/blob/6e146e1140e874b88c304fdc3c5ac0e046e66263/crates/transaction-pool/src/pool/mod.rs#L1188-L1191
https://github.com/paradigmxyz/reth/blob/6e146e1140e874b88c304fdc3c5ac0e046e66263/crates/transaction-pool/src/pool/mod.rs#L788-L795

And here is where it is retrieved:
https://github.com/paradigmxyz/reth/blob/6e146e1140e874b88c304fdc3c5ac0e046e66263/crates/transaction-pool/src/pool/mod.rs#L299-L308

Deletion gets slightly more tricky, since we use BlobStoreUpdates::Finalized to let the blob store know it should remove a set of blob files.

Replacement also complicates things, we would need to surface the versioned hash there:
https://github.com/paradigmxyz/reth/blob/6e146e1140e874b88c304fdc3c5ac0e046e66263/crates/transaction-pool/src/pool/mod.rs#L460-L463

So yeah, seems like a lot of things to refactor if we wanted to change the key to versioned hash entirely, although it might make sense. The in memory mapping does seem like the quickest way to implement this

[mattsse]

another approach that could work for the request, is
fetch all 4844 transactions from the pool, this shouldn't be too expensive because Vec<Arc and then simply try finding the matching tx and finally fetch the blob by txhash

[michaelsproul]

@mattsse I've updated the prototype to follow the latest spec by doing something similar with the in-memory blob cache (just iterate the full cache). I think the main inefficiency is probably recalculating the versioned hash, but that could be optimised out

[michaelsproul]

I need to revert the fmt changes to this file

[Rjected]

@michaelsproul do you mind opening this against the main reth repo? It will be easier for the team to track

[michaelsproul]

Good idea. Closing in favour of:

• Implement engine_getBlobsV1 paradigmxyz/reth#9723