unstable backend incompatible with smoldot v0.18: chainHead API doesn't work

[niklasad1]

subxt's current UnstableBackend uses chainHead_v1_* methods, but smoldot 0.16 (which Subxt is using) is expecting chainHead_unstable_* methods, so the lightclient feature in Subxt isn't currently working.

Example to re-produce this:

#![allow(missing_docs)]
use std::collections::BTreeMap;
use futures::StreamExt;
use subxt::{
    backend::unstable::UnstableBackend, client::OnlineClient, lightclient::LightClient,
    PolkadotConfig,
};

// Generate an interface that we can use from the node's metadata.
#[subxt::subxt(runtime_metadata_path = "../artifacts/polkadot_metadata_full.scale")]
pub mod polkadot {}

const POLKADOT_SPEC: &str = include_str!("../../artifacts/demo_chain_specs/polkadot.json");

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The lightclient logs are informative:
    tracing_subscriber::fmt::init();

    let (_, polkadot_rpc) = LightClient::relay_chain(POLKADOT_SPEC)?;

    let (unstable_backend, mut driver) = UnstableBackend::builder().build(polkadot_rpc);

    tokio::spawn(async move {
        while let Some(val) = driver.next().await {
            if let Err(e) = val {
                // Something went wrong driving unstable backend.
                eprintln!("Error driving unstable backend: {e}");
                break;
            }
        }
    });

    // Create Subxt clients from this unstable backend (ie using new RPCs).
    let api = OnlineClient::<PolkadotConfig>::from_backend(unstable_backend.into()).await?;

    // Create Subxt clients from these Smoldot backed RPC clients.
    // let api = OnlineClient::<PolkadotConfig>::from_rpc_client(polkadot_rpc).await?;

    let address = polkadot::storage().bounties().bounties_iter();

    let mut iter = api.storage().at_latest().await?.iter(address).await?;

    let mut data = BTreeMap::new();
    while let Some(Ok(storage)) = iter.next().await {
        let id = get_bounty_id_from_storage_key(storage.key_bytes);
        data.entry(id).and_modify(|n| *n += 1).or_insert(1);
        assert_eq!(data.len() as u32, data.iter().map(|(_, v)| v).sum::<u32>());
    }

    println!("{:?}", data);

    Ok(())
}

pub fn get_bounty_id_from_storage_key(key: Vec<u8>) -> u32 {
    let s = &key[key.len() - 4..];
    let v: [u8; 4] = s.try_into().expect("slice with incorrect length");
    u32::from_le_bytes(v)
}

Logs

2024-09-02T07:50:33.196501Z DEBUG json-rpc-polkadot: JSON-RPC => {"jsonrpc":"2.0","id":"1", "method":"chainHead_v1_follow","params":[true]}
2024-09-02T07:50:33.196553Z DEBUG json-rpc-polkadot: JSON-RPC <= {"jsonrpc":"2.0","id":"1","error":{"code":-32601,"message":"The method does not exist / is not available."}}
Error driving unstable backend: Rpc error: RPC error: RPC Error: 2024-09-02T07:50:33.196639Z DEBUG runtime-polkadot: Worker <= FailedDownload(blocks=[0xfe68…869a], error=StorageQuery(StorageQueryError { errors: [] }))
{"code":-32601,"message":"The method does not exist / is not available."}.

Solutions

1. Probably subxt could be smarter and call rpc_methods to identify which APIs are exposed by the node that subxt connected to. So, the only option is to downgrade subxt to an older client before chainHead was stabilized.
2. Stabilize the unstable backend and update smoldot to v0.18

/cc @lexnv @jsdw

[jsdw]

subxt can't connect to smoldot nodes v0.18 because the method chainHead_v1_follow is named chainHead_unstable_follow in smoldot v0.16 (which subxt is using) which ends up in "unknown method error."

This text is confusing to me :D I think youy're saying: Subxt's current UnstableBackend uses chainHead_v1_* methods, but smoldot 0.16 (which Subxt is using) is expecting chainHead_unstable_* methods, so the lightclient feature in Subxt isn't currently working.

I guess the solution to this would be to either:

1. Update Smoldot to a version (0.18 I guess) which exposes the chainHead_v1 RPC methods (no need to stabilise anything for this to happen though right?). Problem: Warp sync regression for smoldot v0.18 smol-dot/smoldot#1896 (ie slow warp sync).
2. Make the Unstable backend smarter so that it can check and use chainHead_unstable_* methods if that's what the thing we connect to with it is using. Problem: A bunch of added complexity to support osmehting that is inherently unstable. While this might work for Smoldot v0.16, it may then have issues connecting to nodes running different unstable backends or whatever.

I'm leaning to updating to Smoldot v0.18 at this point. it's still marked unstable anyway, and if there are sync issues then this will highlight them and we can report to Smoldot and try to get them fixed. The main issue for me is that the light client tests might no longer be any good if syncing is broken, or are these fine because it's a brand new dev node?

[niklasad1]

This text is confusing to me :D I think youy're saying: Subxt's current UnstableBackend uses chainHead_v1_* methods, but smoldot 0.16 (which Subxt is using) is expecting chainHead_unstable_* methods, so the lightclient feature in Subxt isn't currently working.

Yes, that is correct

[jsdw]

To add something I forgot after our discussion:

The LegacyBackend, which is the default in Subxt, does work with the light client feature, so there is no urgent need to upgrade right now (and run into the slow sync issues).

[niklasad1]

Fixed by #1400