Index xcm tasks #28

v9n · 2024-01-29T22:15:40Z

I think the idea is to save disk space, we skip index event from parachainSystem' && extrinsicMethod === 'setValidationData

Unfortunately the XCM task event is through that extrinsic, so we have been missing the XCM tasks in our data.

v9n · 2024-01-29T22:19:22Z

project.yaml

@@ -18,7 +18,8 @@ schema:
 network:
  endpoint: 'wss://rpc.turing-staging.oak.tech'
  chainId: '0xd54f0988402deb4548538626ce37e4a318441ea0529ca369400ebec4e04dfe4b'
-  #dictionary: "https://api.subquery.network/sq/subquery/kusama-dictionary"
+
+  dictionary: "https://api.subquery.network/sq/subquery/kusama-dictionary"


We're using subql 2.0 but the project has been update and the default discover url is no longer valid subquery/templates@e28f826 so we explicitly set it here.

chrisli30

Looks no problem to me.

socket-security · 2024-01-30T08:49:09Z

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@polkadot/api-contract@10.11.2	Transitive: environment, filesystem, network	`+60`	73.4 MB	polkadotjs
npm/@polkadot/api@10.11.2	Transitive: environment, filesystem, network	`+59`	73.3 MB	polkadotjs
npm/@subql/cli@4.2.5	Transitive: environment, eval, filesystem, network, shell, unsafe	`+1018`	408 MB	onfinality-admin
npm/@subql/node@3.5.1	Transitive: environment, eval, filesystem, network, shell, unsafe	`+500`	190 MB	onfinality-admin
npm/@subql/testing@2.1.0	None	`+2`	236 kB	onfinality-admin
npm/@subql/types@3.3.0	Transitive: environment, filesystem, network	`+61`	73.6 MB	onfinality-admin
npm/@subql/x-sequelize@6.32.0-0.0.4	filesystem Transitive: environment	`+21`	15.8 MB	jay_ji
npm/lru-cache@10.2.0	None	`0`	458 kB	isaacs
npm/typescript@4.9.5	None	`0`	66.8 MB	typescript-bot

🚮 Removed packages: npm/@polkadot/api-contract@10.7.1, npm/@polkadot/api@10.7.1, npm/@subql/cli@2.1.0, npm/@subql/node@2.1.3, npm/@subql/testing@2.0.1, npm/@subql/types@2.1.2, npm/typescript@4.8.4

View full report↗︎

socket-security · 2024-01-30T08:49:12Z

🚨 Potential security issues detected. Learn more about Socket for GitHub ↗︎

To accept the risk, merge this PR and you will not be notified again.

Alert	Package	Note	Source
Install scripts	npm/protobufjs@6.11.4	Install script: postinstall Source: `node scripts/postinstall`	`package-lock.json` `package.json`
Native code	npm/bufferutil@4.0.8	Location: Package overview	`mixer/package-lock.json` `mixer/package.json` `package-lock.json` `package.json`
Install scripts	npm/@substrate/connect@0.7.33	Install script: postinstall Source: `node build-js-specs.mjs`	`package-lock.json` `package.json`
Install scripts	npm/@substrate/connect@0.7.35	Install script: postinstall Source: `node build-js-specs.mjs`	`package-lock.json` `package.json`
Install scripts	npm/@cosmology/protobufjs@6.11.6	Install script: postinstall Source: `node scripts/postinstall`	`package-lock.json` `package.json`
Install scripts	npm/protobufjs@7.2.6	Install script: postinstall Source: `node scripts/postinstall`	`package-lock.json` `package.json`

View full report↗︎

Next steps

What is an install script?

Install scripts are run when the package is installed. The majority of malware in npm is hidden in install scripts.

Packages should not be running non-essential scripts during install and there are often solutions to problems people solve with install scripts that can be run at publish time instead.

What's wrong with native code?

Contains native code which could be a vector to obscure malicious code, and generally decrease the likelihood of reproducible or reliable installs.

Ensure that native code bindings are expected. Consumers may consider pure JS and functionally similar alternatives to avoid the challenges and risks associated with native code bindings.

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/foo@1.0.0 or ignore all packages with @SocketSecurity ignore-all

@SocketSecurity ignore npm/protobufjs@6.11.4
@SocketSecurity ignore npm/bufferutil@4.0.8
@SocketSecurity ignore npm/@substrate/connect@0.7.33
@SocketSecurity ignore npm/@substrate/connect@0.7.35
@SocketSecurity ignore npm/@cosmology/protobufjs@6.11.6
@SocketSecurity ignore npm/protobufjs@7.2.6

v9n · 2024-01-30T18:29:17Z

I looked at the alert of Socket Security.

These are expected and normal. there is no known vulnerabiliry in that package but the way that package works:

we had package that has native code (C binding)
package define post install script

These aren't direct dependencies we pull it but our indirect dependencys of subql or substrate.

many node js package use those post install script(it's a risk) but it's popular in node somehow.

v9n · 2024-01-30T18:29:45Z

docker-compose.yml

@@ -2,7 +2,7 @@ version: "3.9"

 services:
  subquery-node:
-    image: onfinality/subql-node:v2.5.1
+    image: onfinality/subql-node:v3.5.0


@chrisli30 have to upgrade to get the re-indexing work

v9n · 2024-01-30T18:30:22Z

src/mappings/common/index.ts

-  record.idx = idx
-  record.module = event.section
-  record.method = event.method
+  const record = new Event(


new subql codegen change how we initialize these objects

v9n · 2024-01-30T18:48:17Z

@chrisli30 Deployment plan

Up till now, many of our chances can be migrated with purely SQL data manipulation. But this is the first deployment where we were missing data, so we will need to backfill and reindex data from the archive rpc node.

After observation the behavior of turing staging, this is our plan:

We will deploy this code to production: so new data will immediately become effect but we don't have old data yet. so our app can continue to serve traffic
We will run a second node manually call canary node:

 -f=/app
      - --db-schema=canary-turing
       - --log-level=${LOG_LEVEL:-info}
      - -workers=4

Once we have that data, we will

stop all indexer
rename turing-> turing-old`
rename canary-turing -> turing

Restart the indexer which now point to the new schema with the right history data.

chrisli30 · 2024-01-30T19:04:50Z

@chrisli30 Deployment plan

Up till now, many of our chances can be migrated with purely SQL data manipulation. But this is the first deployment where we were missing data, so we will need to backfill and reindex data from the archive rpc node.

After observation the behavior of turing staging, this is our plan:

We will deploy this code to production: so new data will immediately become effect but we don't have old data yet. so our app can continue to serve traffic

We will run a second node manually call canary node:
 -f=/app
      - --db-schema=canary-turing
       - --log-level=${LOG_LEVEL:-info}
      - -workers=4
Once we have that data, we will

stop all indexer

rename turing-> turing-old`

rename canary-turing -> turing

Restart the indexer which now point to the new schema with the right history data.

Thank you Vinh. Yes, that looks good. 👌

chrisli30 · 2024-01-30T19:07:08Z

many node js package use those post install script(it's a risk) but it's popular in node somehow.

Got it. Okay, let’s skip those alerts.

v9n added 2 commits January 29, 2024 14:00

index task schedule through xcm

c3a9e6d

update due to subql dictinaroy change

a95a58c

v9n commented Jan 29, 2024

View reviewed changes

add document to reindex

c175997

v9n temporarily deployed to staging January 29, 2024 22:27 — with GitHub Actions Inactive

v9n added 2 commits January 29, 2024 14:36

set up pause

62297d5

typo

d438926

v9n temporarily deployed to staging January 29, 2024 22:42 — with GitHub Actions Inactive

v9n temporarily deployed to staging January 30, 2024 00:00 — with GitHub Actions Inactive

chrisli30 approved these changes Jan 30, 2024

View reviewed changes

upgrade to 3.x to make reindex work

242d1ee

v9n added 3 commits January 30, 2024 01:27

wip

5ad51ca

.

74275b5

deploy workflow

aa0b5d3

v9n had a problem deploying to staging January 30, 2024 09:35 — with GitHub Actions Failure

bump

9efc0e5

v9n temporarily deployed to staging January 30, 2024 09:49 — with GitHub Actions Inactive

v9n temporarily deployed to staging January 30, 2024 09:59 — with GitHub Actions Inactive

resume

7f2bff7

v9n temporarily deployed to staging January 30, 2024 10:35 — with GitHub Actions Inactive

v9n commented Jan 30, 2024

View reviewed changes

v9n had a problem deploying to prod January 30, 2024 20:48 — with GitHub Actions Failure

remove debug comment

92094b6

v9n added 2 commits January 30, 2024 12:50

rename config file

f1c06a8

correct prod config rpc

70529c9

v9n temporarily deployed to prod January 30, 2024 20:51 — with GitHub Actions Inactive

v9n added 3 commits January 30, 2024 13:00

add standby canary

c644a80

add canary node

4206684

disable healtcheck on canary node

29e1548

v9n temporarily deployed to staging January 30, 2024 21:02 — with GitHub Actions Inactive

v9n temporarily deployed to staging January 30, 2024 21:23 — with GitHub Actions Inactive

v9n deployed to staging January 30, 2024 23:24 — with GitHub Actions Active

v9n temporarily deployed to prod January 30, 2024 23:25 — with GitHub Actions Inactive

v9n deployed to prod February 2, 2024 02:14 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index xcm tasks #28

Index xcm tasks #28

v9n commented Jan 29, 2024 •

edited

Loading

v9n Jan 29, 2024

chrisli30 left a comment

socket-security bot commented Jan 30, 2024 •

edited

Loading

socket-security bot commented Jan 30, 2024 •

edited

Loading

v9n commented Jan 30, 2024

v9n Jan 30, 2024

v9n Jan 30, 2024

v9n commented Jan 30, 2024

chrisli30 commented Jan 30, 2024

chrisli30 commented Jan 30, 2024

Index xcm tasks #28

Are you sure you want to change the base?

Index xcm tasks #28

Conversation

v9n commented Jan 29, 2024 • edited Loading

v9n Jan 29, 2024

Choose a reason for hiding this comment

chrisli30 left a comment

Choose a reason for hiding this comment

socket-security bot commented Jan 30, 2024 • edited Loading

socket-security bot commented Jan 30, 2024 • edited Loading

Next steps

v9n commented Jan 30, 2024

v9n Jan 30, 2024

Choose a reason for hiding this comment

v9n Jan 30, 2024

Choose a reason for hiding this comment

v9n commented Jan 30, 2024

chrisli30 commented Jan 30, 2024

chrisli30 commented Jan 30, 2024

v9n commented Jan 29, 2024 •

edited

Loading

socket-security bot commented Jan 30, 2024 •

edited

Loading

socket-security bot commented Jan 30, 2024 •

edited

Loading