Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2445 distribution #2030

Closed
61 of 74 tasks
jpraynaud opened this issue Oct 21, 2024 · 2 comments
Closed
61 of 74 tasks

Release 2445 distribution #2030

jpraynaud opened this issue Oct 21, 2024 · 2 comments
Assignees
Labels

Comments

@jpraynaud
Copy link
Member

jpraynaud commented Oct 21, 2024

Why

In order to deploy new features, improvements and bug fixes, we need to release a new distribution.

What

Create and release a new distribution 2445

Important

Distribution +1:

Note

  • 🔥 recompute certificate chain
    • in pre-release-preview
    • in release-preprod
    • in release-mainnet
  • Activate Cardano stake distribution certification:
    • in release-mainnet

How

Phase 1: Prepare distribution

  • Run a backward compatibility test in the CI
  • Upgrade repository dependencies (with runbook)
  • Check that all minor versions of all crates have been bumped for breaking change (if needed)
  • Check if versions of repository linked crates are correct for publication to crates.io(mithril-common, mithril-client)
  • Check if Cardano minimum supported versions in networks.json must be updated following the runbook
  • Merge a PR for rotating the CHANGELOG.md (set the distribution version)
  • Create a draft PR for rotating the documentation
  • Warmup Cardano node 10.1.2 on pre-release-preview
  • Warmup Cardano node 10.1.2 on release-preprod
  • Warmup Cardano node 10.1.2 on release-mainnet

Phase 2: Rollout pre-release distribution

  • Update the configuration of pre-release-preview:
    • Upgrade the Cardano node version to 10.1.2 in pre-release-preview
  • Tag and deploy a 2445.0-pre pre-release distribution on the pre-release-preview network
  • Fill highlights, distributions compatibility and release notes sections in the pre-release page on GitHub
  • Switch Cardano database with warmed-up node in pre-release-preview
  • Make sure pre-release-preview is working as expected
  • Make sure libraries version next have been published on npm registry
  • Run an all green client multi-platform test in the CI
  • Run an all green aggregator stress test in the CI
  • Run a Cardano transaction prover benchmark:
    • Store results in the issue
    • Make sure that there is no performance drop since last release
  • Announce the pre-release on discord channel
  • Announce any breaking change in:
    • #cardano-scaling slack channel for pre-release
    • #mithril-lace slack channel for pre-release
    • #blockfrost-scaling slack channel for pre-release
  • Wait for at least 3 signers running the new version (could be more if era change is tested) over at least 2 epochs

Phase 3: Rollout release distribution

  • Tag and deploy a 2445.0 release distribution on the pre-release-preview network
  • Fill highlights, distributions compatibility and release notes sections in the release page on GitHub
  • Make sure pre-release-preview is working as expected
  • Update networks compatibility table in release notes
  • Update the configuration of release-preprod:
    • Upgrade the Cardano node version to 10.1.2 in release-preprod
  • Update the configuration of release-mainnet:
    • Upgrade the Cardano node version to 10.1.2 in release-mainnet
  • Release the 2445.0 distribution
  • Switch Cardano database with warmed-up node in release-mainnet
  • Make sure release-mainnet is working as expected
  • Switch Cardano database with warmed-up node in release-preprod
  • Make sure release-preprod is working as expected
  • Make sure libraries have been published on crates.io
  • Make sure libraries version latest have been published on npm registry
  • Merge the PR for rotating the documentation
  • Merge a PR for rotating the CHANGELOG.md (updating the release date of the distribution)
  • Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
  • Announce any breaking/new feature change in:
    • #cardano-scaling slack channel for release
    • #mithril-lace slack channel for release
    • #blockfrost-scaling slack channel for release
  • Dev blog post:
    • Dev blog post about aggregator Prometheus endpoint and Grafana dashboard

Phase 4: Activate features

  • Cardano stake distribution:
    • When signer release >= 0.2.182 (2437.1) is run by > 95% of the stake (within 2 weeks, or abort),
    • Activate the certification of Cardano stake distribution in release-mainnet
    • Restart the release-mainnet deployment in the CI
    • Make sure release-mainnet is working as expected
    • Announce the release on discord channel
    • Dev blog post: Activation of Cardano stake distribution in release-mainnet
@jpraynaud
Copy link
Member Author

Cardano node warmup

Select the Mithril network

pre-release-preview

export NETWORK=preview
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/pre-release-preview/genesis.vkey)

release-preprod

export NETWORK=preprod
export AGGREGATOR_ENDPOINT=https://aggregator.release-preprod.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-preprod/genesis.vkey)

release-mainnet

export NETWORK=mainnet
export AGGREGATOR_ENDPOINT=https://aggregator.release-mainnet.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-mainnet/genesis.vkey)

Phase 0: Verify enough disk space is available in data disk

At least 50% of available space on data disk

df -h

Phase 1: Warmup a temporary Cardano node from a Mithril snapshota

Create temp dir

rm -rf /home/curry/data/temp
mkdir -p /home/curry/data/temp
cd /home/curry/data/temp

Download Mithril client

wget https://github.com/input-output-hk/mithril/releases/download/2442.0/mithril-2442.0-linux-x64.tar.gz
tar xzf mithril-2442.0-linux-x64.tar.gz mithril-client
chmod u+x mithril-client
./mithril-client --version

Download latest snapshot

./mithril-client cardano-db download latest

Restore the Cardano node

docker run -d --name cardano-node-warmup -v cardano-node-ipc:/ipc -v cardano-node-data:/data --mount type=bind,source="/home/curry/data/temp/db",target=/data/db/ -e NETWORK=${NETWORK} ghcr.io/intersectmbo/cardano-node:10.1.2
docker logs -f --tail 100 cardano-node-warmup

Phase 2: Switch Cardano database from the main node with the warmed-up one

Copy Cardano database to aggregator

docker stop cardano-node-warmup
docker stop cardano-node-aggregator
mv /home/curry/data/$NETWORK/mithril-aggregator/cardano/db /home/curry/data/$NETWORK/mithril-aggregator/cardano/db.bak
mv /home/curry/data/temp/db /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
sudo chown -R curry /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
sudo chgrp -R curry /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
docker start cardano-node-aggregator
docker logs -f --tail 100 cardano-node-aggregator

Cleanup directory (when node is back up)

docker rm cardano-node-warmup
rm -rf /home/curry/data/$NETWORK/mithril-aggregator/cardano/db.bak
cd /home/curry/
rm -rf /home/curry/data/temp

@jpraynaud
Copy link
Member Author

Cardano transactions prover benchmark

Note

run on GCP with e2-highmem-2 machine with 2 vCPUs and 15GB of RAM
not during Cardano database snapshotting

  • Benchmark results:
total_requests concurrency transactions/request requests/s
500 50 1 53.93
500 50 10 39.81
500 50 20 25.17
500 50 30 29.80
500 50 40 23.59
500 50 50 21.11

vs previous results

total_requests concurrency transactions/request requests/s
500 50 1 56.11
500 50 10 41.44
500 50 20 33.54
500 50 30 30.91
500 50 40 27.23
500 50 50 24.67
  • Command run:
# Aggregator endpoint
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator

# Transactions file to prove
export TRANSACTIONS_FILE=transactions-preview.txt

# Transactions proved per request range definition
export TRANSACTIONS_PER_REQUEST_MIN=0
export TRANSACTIONS_PER_REQUEST_MAX=50
export TRANSACTIONS_PER_REQUEST_STEP=10

# Apache benchmark total request sent per benchmark
export AB_TOTAL_REQUESTS=500

# Apache benchmark concurrency level range definition
export AB_CONCURRENCY_MIN=50
export AB_CONCURRENCY_MAX=50
export AB_CONCURRENCY_STEP=50

# Run the benchmark:
./benchmark-aggregator-prover.sh
  • Command result:
MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK

>> Aggregator endpoint: https://aggregator.pre-release-preview.api.mithril.network/aggregator
>> Aggregator route: /proof/cardano-transaction
>> Transactions file: transactions-preview.txt
>> Transactions available: [100]
>> Transactions per request range: [0 10 20 30 40 50]
>> AB concurrency range: [50]
>> AB total requests per run: [500]
>> AB total runs: 6
>> Output file: benchmark.csv

>> [#1/6] Running stress test with 500 requests with 1 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (53.93 requests/s)

>> [#2/6] Running stress test with 500 requests with 10 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (39.81 requests/s)

>> [#3/6] Running stress test with 500 requests with 20 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (25.17 requests/s)

>> [#4/6] Running stress test with 500 requests with 30 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (29.80 requests/s)

>> [#5/6] Running stress test with 500 requests with 40 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (23.59 requests/s)

>> [#6/6] Running stress test with 500 requests with 50 transactions per request and 50 concurrency
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
>>>> Success (21.11 requests/s)

>> Benchmark completed:

total_requests,concurrency,transactions/request,requests/s
500,50,1,53.93
500,50,10,39.81
500,50,20,25.17
500,50,30,29.80
500,50,40,23.59
500,50,50,21.11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant