light-client .scale metadata update (#285)

README.md

@@ -1,5 +1,11 @@
 # GGX Chain node set-up
 
+**Detailed production setup can be found in** [Documentation Repository](https://github.com/ggxchain/documentation/blob/main/src/sydney-testnet/validator-guides/how-to-setup-a-validator-node.md) **or** [Official Documentation Portal](https://docs.ggxchain.io/)
+
+For instant non-production deployment with docker-compose please follow steps in [Docker Compose Documentation](https://github.com/ggxchain/ggxnode/tree/main/docker-compose)
+
+**WARNING:** *Information below this point is dedicated for developers only and can lag behind the resources mentioned above or be to far in the future.*
+
 ## 1. Clone repo
 
 To get started download this repository and navigate to `ggxnode` folder, e.g.:
@@ -86,9 +92,7 @@ docker run -d -it --restart=unless-stopped --ulimit nofile=100000:100000 \
     --prometheus-external \
     --validator \
     --chain sydney \
-    --base-path=/data-sydney \
-    --bootnodes /dns/sun.sydney.ggxchain.io/tcp/30333/p2p/12D3KooWGmopnFNtQb2bo1irpjPLJUnmt9K4opTSHTMhYYobB8pC \
-    --telemetry-url "wss://telemetry.sydney.ggxchain.io/submit 0"
+    --base-path=/data-sydney
 ```
 
 #### Brooklyn - development network:
@@ -117,13 +121,11 @@ docker run \
     ggxchain-node:brooklyn \
     --base-path=/data-brooklyn \
     --chain brooklyn \
-    --bootnodes /ip4/3.74.168.122/tcp/30333/p2p/12D3KooWCUvCEgrEqNHgMJjRmq2dYJmLX5jfcmMSte5SSwtsAsao \
-    --telemetry-url "ws://test.telemetry.brooklyn.ggxchain.io/submit 0" \
     --light-client-relay-config-path eth-relay.toml \
     --light-client-init-pallet-config-path eth-init.toml
 ```
 
-You can use the following optional flags:
+Following optional flags are availabe:
 
 | Flags                             | Description |
 |-----------------------------------|---|

docker-compose/README.md

@@ -0,0 +1,128 @@
+#### Run node with docker-compose
+
+Instantly deployment using prebuild images
+
+**Requirements:**
+* [Docker](https://docs.docker.com/engine/install/)
+* [Docker Compose](https://docs.docker.com/compose/)
+* sudo apt-get install git
+
+_Please prioritize the security of your deployment. Security is beyond the scope of this guidance here, ensure your firewall is properly configured. Be aware that Docker may bypass iptables rules. Make sure you fully understand the implications of the setup._
+
+**What we about to do:**
+1) Change the node name in the config `nano <chain>.env`
+2) Activate config `cp <chain>.env .evn`
+3) Create aprorpriate data folder structure `./run-upgrade.sh`
+4) Start node `./run-upgrade.sh`
+
+**Assume we are on Linux. `ggx_user` will be userd here as for example.**
+
+```bash
+# create dedicated non-sudo user:
+sudo adduser --gecos GECOS --disabled-password --disabled-login --shell /bin/bash ggx_user
+```
+```bash
+# add user in to docker group:
+sudo usermod -aG docker ggx_user
+```
+```bash
+# get user shell
+su - ggx_user
+```
+```bash
+# clone ggxnode repository in to user home
+cd ~ && git clone https://github.com/ggxchain/ggxnode.git
+```
+```bash
+# Copy required folder in to user ${HOME}
+cd ~ && cp -r ggxnode/docker-compose/ .
+```
+* Full repository can be removed now
+```bash
+rm -rf ${HOME}/ggxnode
+```
+**Depends on the network we about to join, edit environment files:**
+
+-- for `sydney` edit `sydney.env`
+-- for `brooklyn` edit `brooklyn.env`
+
+**All we are looking here is this part:**
+```
+# -------------------- #
+NODE_NAME=my_node_name
+# -------------------- #
+```
+
+**Guidelines for Naming Variables:**
+
+* _Descriptive Names: use names that clearly indicate the variable's purpose, like `oliver_node`, `dev_node_one`, or `a3mc_little_rpc`._
+
+* _No White Spaces or Special Characters: avoid using spaces or special characters (e.g., @, #, $, %). Use underscores `_` to separate words._
+
+* _Consistent Usage: the variable name will also serve as a data storage folder name and telemetry moniker for consistency across systems._
+
+**Adjust ports if requires or levae them by default**
+```bash
+P2P_PORT=5000
+RPC_PORT=5001
+PROMETHEUS_PORT=5002
+```
+If you plan to make validator out of this node keep `RPC_METHODS=unsafe`, this will allow us to generate session keys using open RPC _( change later )_, otherwise set to `RPC_METHODS=safe`.
+
+Default pruning is `256` change if required or set to `archive`
+
+```bash
+STATE_PRUNING=256
+BLOCKS_PRUNING=256
+```
+
+`SYNC_MODE=warp` require 3 nodes to be online _( which is not always a brooklyn case )_ however on Sydney `warp` will be your best choice.
+
+**Activate network environment**
+
+**Sydney:**
+```bash
+# make sydney config active
+cp sydney.env .env
+```
+
+**Brooklyn:**
+```bash
+# make brooklyn config active
+cp brooklyn.env .env
+```
+
+**For security reasons, the internal container user ID and group ID are both set to 55500. This requires creating a special data folder owned by user 55500 and accessible by group 55500.**
+
+Script `run-upgrade.sh` will guide you how to create this folder, it will require `sudo` permission.
+
+```bash
+# make script executable
+chmod +x run-upgrade.sh
+```
+```bash
+# get data folder creation guide
+./run-upgrade.sh
+```
+Copy suggested code, `exit` non-sudo user shell and execute command. Then comeback to our user `su - ggx_user`.
+
+At this point we should have data folder set
+```
+.
+└──data
+    └── <node_name>
+```
+
+And data folder permissions configured accordingly
+
+```bash
+# check permissions
+ls -lah ${HOME}/data
+```
+**Start the node !**
+```bash
+# Start node
+cd ${HOME}/docker-compose && ./run-upgrade.sh
+```
+
+Log should populate console, press `Ctrl+C` to exit.

docker-compose/run-upgrade.sh

@@ -6,7 +6,7 @@ source .env
 if [ ! -d "$DATA_PATH" ] || [ "$(stat -c %u "$DATA_PATH")" != "55500" ] || [ "$(stat -c %g "$DATA_PATH")" != "55500" ]; then
     echo " Mount point $DATA_PATH does not exist or is not owned by 55500:55500."
     echo " Create the mount point with the following command:"
-    echo " sudo mkdir -p $DATA_PATH && sudo chown -R 55500:55500 $DATA_PATH"
+    echo " sudo -u $USER mkdir -p $DATA_PATH && sudo chown -R 55500:55500 $DATA_PATH"
     exit 1
 fi
 

docker-compose/sydney.env

@@ -11,7 +11,6 @@ STATE_PRUNING=256
 BLOCKS_PRUNING=256
 SYNC_MODE=warp
 LOG_LEVEL=info
-# For Sydney ==> data-sydney
 PATH_INTERNAL=/data-sydney
 IMAGE_NAME=ggxdocker/ggxnode-sydney-img:latest
 # Key can be created before execution, otherwise new will be created automatically

fetch_metadata.sh

@@ -1,64 +1,94 @@
 #!/bin/bash -e
 
-# This script runs a both brooklyn and sydney nodes (in --dev), then fetches metadata.
-# The metadata is saved to metadata_ggx_brooklyn.scale and metadata_ggx_sydney.scale respectively.
+# ==============================================================================
+# This script fetches blockchain metadata from specified network endpoints using
+# JSON-RPC. The script can be run with predefined network names ('sydney' or
+# 'brooklyn') or with a 'custom' RPC endpoint specified by the user.
+#
+# Usage:
+#   ./fetch_metadata.sh <network> [<custom RPC>]
+#
+# Parameters:
+#   <network>     The network name from which to fetch metadata. Supported values:
+#                 - sydney   : Use the Sydney network RPC endpoint.
+#                 - brooklyn : Use the Brooklyn network RPC endpoint.
+#                 - custom   : Use a custom RPC endpoint. Must provide the RPC URL
+#                              as the second argument.
+#   [<custom RPC>]: (Optional) The RPC endpoint URL when using the 'custom' network.
+#                   This argument is required when <network> is set to 'custom'.
+#
+# Example:
+#   ./fetch_metadata.sh sydney
+#   ./fetch_metadata.sh brooklyn
+#   ./fetch_metadata.sh custom "http://127.0.0.1:5100"
+#
+# Output:
+#   The script will generate a file named "eth_light_client_<network>.scale"
+#   containing the metadata retrieved from the specified RPC endpoint.
+#
+# Prerequisites:
+#   - jq : Command-line JSON processor
+#   - xxd: Hex dump tool
+#   - curl: Command-line tool for transferring data with URLs
+#
+# Notes:
+#   - Ensure the required tools (jq, xxd, curl) are installed and available in
+#     the system's PATH.
+#   - For the 'custom' network option, a valid RPC URL must be provided as the
+#     second argument.
+# ==============================================================================
 
-cd $(dirname $0)
-
-if [ "$#" -ne 2 ]; then
-    echo "Usage: $0 <brooklyn or sydney> <path/to/ggx-node>"
-    exit 1
-fi
+# Define default RPC endpoints
+SYDNEY_RPC="https://gate.ggxchain.net/sydney-archive"
+BROOKLYN_RPC="https://brooklyn-archive.ggxchain.net/dev-brooklyn"
 
+# Get network and custom RPC (if provided)
 NETWORK="$1"
-NODE="$2"
-
-if [ ! -f "$NODE" ]; then
-    echo "Error: ggx-node binary not found at $NODE"
-    exit 1
-fi
+CUSTOM_RPC="$2"
 
 function get_metadata() {
-    attempts=0
-    max_attempts=30
-    while sleep 1; do
-        (( attempts++ )) || true
-
-        curl -sX POST -H "Content-Type: application/json" --data \
-        '{"jsonrpc":"2.0","method":"state_getMetadata", "id": 1}' \
-        localhost:9944 | jq .result | cut -d '"' -f 2 | xxd -r -p > $1
 
-        # Check if file is empty
-        if [ ! -s "$1" ]; then
-            echo "Fetched metadata $1 is empty... retrying (attempt $attempts/$max_attempts)"
-            if [ $attempts -ge $max_attempts ]; then
-                echo "Failed to fetch metadata $1"
-                exit 1
-            fi
+    local rpc_url="$1"
+    local output_file="$2"
+    local json_rpc_request='{"jsonrpc":"2.0","method":"state_getMetadata", "id": 1}'
 
-            continue
-        fi
-
-        echo "SUCCESS"
-        echo
-        return
-    done
+    curl -sX POST -H "Content-Type: application/json" \
+        --data "${json_rpc_request}" "${rpc_url}" | \
+        jq -r .result | xxd -r -p > "${output_file}"
 }
 
-# starts a process then kills it after scale file is fetched
-start_and_kill() {
-    # Start the command in the background
-    $NODE --dev --tmp --rpc-external 1>/dev/null 2>/dev/null &
-
-    # Get its PID
-    local pid=$!
+# Display usage if no arguments are provided
+if [[ -z "${NETWORK}" ]]; then
+    echo "Usage: fetch_metadata.sh <network> [<custom RPC>]"
+    echo "Networks:"
+    echo "  sydney     - Use Sydney network RPC"
+    echo "  brooklyn   - Use Brooklyn network RPC"
+    echo "  custom     - Use a custom RPC endpoint (provide RPC URL as second argument)"
+    echo "Example:"
+    echo "  fetch_metadata.sh sydney"
+    echo "  fetch_metadata.sh custom \"http://127.0.0.1:5100\""
+    exit 1
+fi
 
-    sleep 5  # wait for node to start
-    get_metadata node/tests/data/scale/eth_light_client_$NETWORK.scale
+# Determine the appropriate RPC URL based on the network name
+if [[ "${NETWORK}" == "sydney" ]]; then
+    RPC="${SYDNEY_RPC}"
+elif [[ "${NETWORK}" == "brooklyn" ]]; then
+    RPC="${BROOKLYN_RPC}"
+elif [[ "${NETWORK}" == "custom" ]]; then
+    if [[ -n "${CUSTOM_RPC}" ]]; then
+        RPC="${CUSTOM_RPC}"
+    else
+        echo "Error: For custom network, please provide the RPC endpoint."
+        exit 1
+    fi
+else
+    echo "Error: Unknown network '${NETWORK}'. Supported networks: sydney, brooklyn, custom."
+    exit 1
+fi
 
-    # Then kill it
-    kill -9 $pid
-}
+# Set output filename based on network
+OUTPUT_FILE="eth_light_client_${NETWORK}.scale"
 
-echo "Starting $NETWORK"
-start_and_kill
+# Fetch metadata
+get_metadata "${RPC}" "${OUTPUT_FILE}"