Iot Config service use Helius for on-chain metadata #428

jeffgrunewald · 2023-04-04T16:29:01Z

Switches the iot-config service (and the rest of oracles) to use the helius database for retrieving on-chain gateway metadata instead of the blockchain-node follower client service.

This also adds additional optimizations for signature verification of sent and received messages with the iot-config service, primarily enforcing messages include the binary of the signing pubkey for O(1) verification of the signature verification instead of O(n) when checking the signature against all potentially valid signing keys; instead of attempting to verifying the signature of a message against all potential keys we validate against the key as self-reported in the message and then simply check for membership of that key in the list of potential authorized signers. This also introduces additional message signing of outgoing messages sent by the config service itself

Depends on:

config services signature/signers and metadata message types proto#308

jeffgrunewald · 2023-04-05T13:37:45Z

file_store/src/traits/msg_verify.rs

-impl_msg_verify!(SessionKeyFilterListReqV1, signature);
-impl_msg_verify!(SessionKeyFilterStreamReqV1, signature);
-impl_msg_verify!(SessionKeyFilterUpdateReqV1, signature);
+impl_msg_verify!(iot_config::OrgCreateHeliumReqV1, signature);


i've started pathing these message types here since they will have very similar names between other services (like the mobile config service)

iot_config/src/client/mod.rs

andymck

Initial first pass, will need to take a second run

iot_config/src/gateway_info.rs

iot_config/src/client/settings.rs

iot_config/src/admin_service.rs

jeffgrunewald · 2023-04-06T19:34:53Z

This is ready to review it's in "draft" status to prevent merging before updating the dependencies once they get updated

andymck

this lgtm, my eyes hurt from all the scrolling but other than that lg...
Once merged i may tweak a few things on the iot verifier side

andymck · 2023-04-07T11:57:10Z

iot_config/src/admin.rs

 }

 impl AuthCache {
    pub async fn new(
        settings: &Settings,
        db: impl sqlx::PgExecutor<'_> + Copy,
-    ) -> Result<Self, AdminAuthError> {
+    ) -> anyhow::Result<(watch::Sender<CacheKeys>, Self)> {


nit: why not import result in line 2? I notice you import in some modules but not others

this is a stylistic choice that i'm up for debate over; i don't feel overly strongly about it, but given the std::result::Result is part of the typical prelude unless you override it with use anyhow::Result then it's easier at a glance, particularly in a large file that what you're returning is not the std Result type

iot_config/src/admin_service.rs

andymck · 2023-04-07T12:15:06Z

iot_config/src/client/mod.rs

+            signature: vec![],
+        };
+        request.signature = self.signing_key.sign(&request.encode_to_vec())?;
+        tracing::debug!(pubkey = address.to_string(), "fetching gateway info");


nit: not sure what value this log statement provide

this is probably a better place for a metric on grpc requests then a log statement, good point

iot_config/src/gateway_info.rs

andymck · 2023-04-07T12:20:56Z

iot_config/src/gateway_info.rs

+    use helium_crypto::PublicKeyBinary;
+    use sqlx::{PgExecutor, Row};
+
+    pub struct IotMetadata {


slightly confusing having both GatewayMetadata and IotMetadata structs

i had hoped scoping the IotMetadata struct inside the db submodule would help with that, but i couldn't think of a better way to do unless i skipped the FromRow impl altogether; since the GatewayInfo requires the hextree map to look up a Region from the asserted location and i can't change the from_row function signature to pass that in i needed the intermediary data structure

I understand andymck's point, but I can't think of a better way either and I think scoping IotMetadata to the db submodule does help.

iot_config/src/gateway_service.rs

bbalser · 2023-04-08T14:06:20Z

iot_config/src/route.rs

+                    signer: signer.clone(),
+                    signature: vec![],
+                };
+                futures::future::ready(signing_key.sign(&update_res.encode_to_vec()))


Nice! I need to remember this.

iot_config/src/route_service.rs

bbalser

Couple Nits, but otherwise nice job!!!

…mbinators

Co-authored-by: andymck <andrewb.mckenzie@gmail.com>

iot_config/src/admin.rs

iot_config/src/gateway_info.rs

iot_config/src/gateway_service.rs

Co-authored-by: Matthew Plant <matty@nova-labs.com>

jeffgrunewald requested review from bbalser, andymck and maplant April 5, 2023 06:36

jeffgrunewald commented Apr 5, 2023

View reviewed changes

jeffgrunewald force-pushed the jg/iot-config-helius branch 2 times, most recently from 9a82be5 to 472d8e2 Compare April 5, 2023 15:09

bbalser reviewed Apr 5, 2023

View reviewed changes

iot_config/src/client/mod.rs Outdated Show resolved Hide resolved

andymck reviewed Apr 6, 2023

View reviewed changes

iot_config/src/gateway_info.rs Show resolved Hide resolved

iot_config/src/client/settings.rs Outdated Show resolved Hide resolved

iot_config/src/admin_service.rs Show resolved Hide resolved

jeffgrunewald requested review from andymck and bbalser April 6, 2023 19:34

jeffgrunewald marked this pull request as draft April 6, 2023 19:34

andymck mentioned this pull request Apr 7, 2023

handle gateway info metadata struct #424

Closed

andymck approved these changes Apr 7, 2023

View reviewed changes

jeffgrunewald force-pushed the jg/iot-config-helius branch from c453021 to 86df017 Compare April 7, 2023 19:28

bbalser reviewed Apr 8, 2023

View reviewed changes

iot_config/src/gateway_service.rs Show resolved Hide resolved

bbalser reviewed Apr 8, 2023

View reviewed changes

iot_config/src/route_service.rs Outdated Show resolved Hide resolved

bbalser approved these changes Apr 8, 2023

View reviewed changes

jeffgrunewald marked this pull request as ready for review April 8, 2023 15:42

jeffgrunewald and others added 10 commits April 10, 2023 15:24

iot config helius integration

632c8af

iot config service to use helius db

47e3f68

Update gateway_service::stream_all_gateways_info to not use stream co…

0a9a187

…mbinators

Adding batching back into gateway_service::stream_all_gateways_info

0f0fc54

order region query, cleanup some result/option matches

e6143a0

coupling iot config client under service crate

8c6b326

add region params resolver to iot config client

0af4857

swap iot config integrated client for standalone

47ece4e

iot config gateway service uses helium db pool

85ee53e

Update iot_config/src/client/settings.rs

a432500

Co-authored-by: andymck <andrewb.mckenzie@gmail.com>

jeffgrunewald added 4 commits April 10, 2023 15:27

specify admin-only api auth better where needed

de56959

implement cleanup nits

328c143

restore proto dep to master branch

8ef0ebd

final format and clippy check after rebase

7d7edfe

jeffgrunewald force-pushed the jg/iot-config-helius branch from 86df017 to 7d7edfe Compare April 10, 2023 19:33

maplant reviewed Apr 10, 2023

View reviewed changes

iot_config/src/admin.rs Outdated Show resolved Hide resolved

maplant reviewed Apr 10, 2023

View reviewed changes

iot_config/src/gateway_info.rs Outdated Show resolved Hide resolved

maplant reviewed Apr 10, 2023

View reviewed changes

iot_config/src/gateway_service.rs Outdated Show resolved Hide resolved

jeffgrunewald force-pushed the jg/iot-config-helius branch from 71fda35 to c8cde12 Compare April 10, 2023 20:01

convert "or's" to "or_else's"

b0c265b

Co-authored-by: Matthew Plant <matty@nova-labs.com>

jeffgrunewald force-pushed the jg/iot-config-helius branch from c8cde12 to b0c265b Compare April 10, 2023 20:18

jeffgrunewald merged commit 3cddbcb into main Apr 10, 2023

jeffgrunewald deleted the jg/iot-config-helius branch April 10, 2023 20:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iot Config service use Helius for on-chain metadata #428

Iot Config service use Helius for on-chain metadata #428

jeffgrunewald commented Apr 4, 2023 •

edited

Loading

jeffgrunewald Apr 5, 2023

andymck left a comment

jeffgrunewald commented Apr 6, 2023

andymck left a comment

andymck Apr 7, 2023

jeffgrunewald Apr 7, 2023

andymck Apr 7, 2023

jeffgrunewald Apr 7, 2023

andymck Apr 7, 2023

jeffgrunewald Apr 7, 2023

bbalser Apr 8, 2023

bbalser Apr 8, 2023

bbalser left a comment

Iot Config service use Helius for on-chain metadata #428

Iot Config service use Helius for on-chain metadata #428

Conversation

jeffgrunewald commented Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

andymck left a comment

Choose a reason for hiding this comment

jeffgrunewald commented Apr 6, 2023

andymck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbalser left a comment

Choose a reason for hiding this comment

jeffgrunewald commented Apr 4, 2023 •

edited

Loading