Osmosis Full Node Validator Setup
Not ensuring the following caused my validator and sentry to fail during mainnet 1. This was generally an unknown issue at the time but I was the unlucky one to run into it. Thanks to "Simo | AC" the issue was correctly identified. I recommend Intel Cascade Lake (2019) generation of processor or newer.
I am using Google Cloud N2 Compute Instance.
Check Instruction Set:
cat /proc/cpuinfo | grep flags
Should have rdseed cpu instruction set listed
Outbound:
Any
Inbound (Sentry):
SSH (22 or custom port) from your personal IP only
P2P (26656) from Anywhere
RPC (26657) from Anywhere
Inbound (Validator):
SSH (22 or custom port) from your personal IP only
P2P (26656) from Sentry01
P2P (26656) from Sentry02
P2P (26656) from Sentry03
RPC (26657) from <MONITORING_NODE_IP>
Note that .profile/.bash_profile should be called when you logon to a Linux session. .bashrc will be called from .profile/.bash_profile This include when your start Akash as a service.
Insert the following lines at the bottom of your .bashrc file:
To open/create the file:
nano ~/.bashrc
ulimit -n 65530
# Update environment variables to include go
cat <<'EOF' >>$HOME/.profile
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin
# Setup Cosmovisor
export DAEMON_NAME=osmosisd
export DAEMON_HOME=/home/<LOGGED ON USER>/.osmosisd
EOF
source $HOME/.profile
To save the file:
CTRL+o and ENTER
CTRL+x
The following is optional but will give your a colorful terminal window. Note for my Google Cloud instance, this was already there so just add the above lines to the bottom of the .bashrc file:
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
# don't put duplicate lines in the history. See bash(1) for more options
# ... or force ignoredups and ignorespace
HISTCONTROL=ignoredups:ignorespace
# append to the history file, don't overwrite it
shopt -s histappend
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000
# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize
# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"
# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "$debian_chroot" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
xterm-color) color_prompt=yes;;
esac
# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
force_color_prompt=yes
if [ -n "$force_color_prompt" ]; then
if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
# We have color support; assume it's compliant with Ecma-48
# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
# a case would tend to support setf rather than setaf.)
color_prompt=yes
else
color_prompt=
fi
fi
if [ "$color_prompt" = yes ]; then
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt
# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
;;
*)
;;
esac
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'
# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
#if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
# . /etc/bash_completion
#fi
Now lets configure our .profile/.bash_profile file. Good news, for my Google Could and Ubuntu 20.04 installation the file was already configured to call .bashrc. If not, just create the file and paste the following into it. You should create it in your $HOME directory:
To open/create the file
nano ~/.profile
# ~/.profile: executed by Bourne-compatible login shells.
if [ "$BASH" ]; then
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
fi
mesg n 2> /dev/null || true
To save the file:
CTRL+o and ENTER
CTRL+x
Install Go
cd ~/
# First remove any existing old Go installation
sudo rm -rf /usr/local/go
# Install the latest version of Go using this helpful script
curl https://raw.githubusercontent.com/canha/golang-tools-install-script/master/goinstall.sh | bash
The Go installation may have added the following to you .bashrc. To check this:
nano ~/.bashrc
If you it is there, skip then next section and only add the Cosmovisor entries. Otherwise add both of the following:
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin
# Setup Cosmovisor
export DAEMON_NAME=osmosisd
export DAEMON_HOME=/home/<LOGGED ON USER>/.osmosisd
go version
Get the Omosis Code
git clone https://github.com/osmosis-labs/osmosis
cd osmosis
git checkout v3.1.0
You can now build Osmosis node software. Running the following command will install the executable osmosisd (Osmosis node daemon) to your GOPATH.
make install
Verify that everything is OK. If you get something like the following, you've successfully installed Osmosis on your system.
osmosisd version --long
This should output the following:
name: osmosis
server_name: osmosisd
version: 3.1.0
commit: 13916d1e10bca718b6ea7f4b84715710bc319e6d
build_tags: netgo,ledger
go: go version go1.16.6 linux/amd64
Osmosis reintroduces the client-side config that was removed in earlier Stargate versions of the Cosmos SDK.
If you haven't done so already, please save the mainnet chain-id to your client.toml. This will make it so you do not have to manually pass in the chain-id flag for every CLI command.
osmosisd config chain-id osmosis-1
osmosisd init --chain-id=osmosis-1 <YOUR_MONIKER>
The above command will create the necessary configurations under ~/.osmosisd with the below structure:
├── config
│ ├── app.toml
│ ├── config.toml
│ ├── genesis.json
│ ├── node_key.json
│ └── priv_validator_key.json
└── data
└── priv_validator_state.json
cd $HOME/.osmosisd/config/
curl https://github.com/osmosis-labs/networks/raw/main/osmosis-1/genesis.json > $HOME/.osmosisd/config/genesis.json
I recommend you use cosmovisor to run both Validator and Sentry nodes. This will make low-downtime upgrades more smoother, as validators(and sentries) don't have to manually upgrade binaries during the upgrade, and instead can preinstall new binaries, and cosmovisor will automatically update them based on on-chain SoftwareUpgrade proposals.
cd $HOME
git clone https://github.com/cosmos/cosmos-sdk
cd cosmos-sdk
git checkout v0.42.5
make cosmovisor
cp cosmovisor/cosmovisor $GOPATH/bin/cosmovisor
cd $HOME
mkdir -p ~/.osmosisd
mkdir -p ~/.osmosisd/cosmovisor
mkdir -p ~/.osmosisd/cosmovisor/genesis
mkdir -p ~/.osmosisd/cosmovisor/genesis/bin
mkdir -p ~/.osmosisd/cosmovisor/upgrades
Cosmovisor requires some ENVIRONMENT VARIABLES be set in order to function properly. We added these above in our .bashrc file which is being called from .profile
cp $GOPATH/bin/osmosisd ~/.osmosisd/cosmovisor/genesis/bin
Configure some extra client settings:
Now you have the option to install a Sentry or Validator node. Review the following important note regarding the config.toml configuration and edit it appropriately based on which you are installing. One portion is for both Validator and Sentry:
Install your Sentry/Validator on an enterprise class server. I am using Google Cloud for my Validator and one of my 3 Sentries.
This tells the node whether to gossip via "peer-exchange reactor" your validator's node address:
For a Validator:
pex = false # This will keep your validator private
max_open_connections = 3 # These connections are for 3rd parties such as Exchanges and Wallet providers. These connections will be blocked by the firewall. We only connect to our sentries.
For a Sentry:
pex = true # We want to share these to make the network resilient and connections available
max_open_connections = 20 # These connections are for 3rd parties such as Exchanges and Wallet providers
private_peer_ids = <TENDERMINT_NODE_ID_OF_YOUR_VALIDATOR> # Prevents your Validator node address from being gossiped by the Sentry
laddr = "tcp://0.0.0.0:26667" #Note we have to change the default 26657 to 26667 for NGINX rate control (DDoS protection) config coming later.
persistent_peers = <PEERS_PROVIDED_BY_AKASH_AND_YOUR_VALIDATORS_AND_SENTRIES> # You will open connections to the servers in this list
For Both:
unconditional_peer_ids = "<YOUR_VALIDATORS_AND_SENTRIES>" # NOTE: These are the **Tendermint Node IDs and not IP address** - Always accept connections
prometheus = true # Allow monitoring metrics via Prometheus
prometheus_listen_addr = ":26660" # Uncomment this to set the listening port for prometheus
Do not change the following setting in your app.toml:
minimum-gas-prices = ""
Configues NGINX Rate Control for DDoS protection only on our Sentries and not the Validator because the Validator only talks to our Sentries
Install nginx web server:
sudo apt-get install nginx
Backup the original nginx.conf:
sudo cp //etc/nginx/sites-available/default /etc/nginx/sites-available/default_original
Open the nginx configuration file:
sudo nano /etc/nginx/sites-available/default
Replace the contents of /etc/nginx/sites-available/default with the following:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
# Rate Limiting - Protection from DDoS
limit_req_zone $binary_remote_addr zone=req_zone:10m rate=5r/s;
upstream rpc_nodes {
least_conn;
server 127.0.0.1:26667;
}
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 26657;
location / {
limit_req zone=req_zone burst=20 nodelay;
proxy_pass http://rpc_nodes;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host:26657;
}
}
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
Save the ngnix.conf file:
CTRL+o and ENTER
CTRL+x
Confirm there are no errors in the file
sudo nginx -t
Restart nginx:
sudo systemctl restart nginx
This concludes the Sentry configuration.
We are going for most secure so we are using a Ledger Nano S or X is fine.
Note: When using the Ledger to create your keys and validator, you will run all the commands from your personal PC. Once created you will copy your priv_validator_key.json and node_key.json to your validator server in the cloud. YOU NEVER START THE VALIDATOR ON YOUR PERSONAL PC!!!! This will keep you from double signing!!!!
Install your Ledger as instructed by the Ledger team to your PC. Install the Cosmos app to your Ledger device using Ledger Live. If you are on Linux, just install the Ledger there (not in a VM). Trying to use it connected to an Oracle VirtualBox VM would not work BUT with Osmosis it worked fine. I just connect the Ledger to the VM and osmosisd recognized it with issue.
Important: For the Ledger key there will be no mnemonic or keyring passphrase. This is all handled on the Ledger. You will need to have the Ledger connected and opened to the Cosmos App to execute transactions from your personal PC.
Lets create our first key.
osmosisd keys add <GIVE_YOUR_KEY_A_NAME> --ledger
For non-ledger key
osmosisd keys add <GIVE_YOUR_KEY_A_NAME>
osmosisd tx staking create-validator \
--amount=1000000uosmo \
--pubkey=$(osmosisd tendermint show-validator) \
--moniker="<GIVE IT A NAME>" \
--chain-id=osmosis-1 \
--commission-rate="0.05" \
--commission-max-rate="0.20" \
--commission-max-change-rate="0.05" \
--min-self-delegation="1" \
--details="<ENTER DETAILS ABOUT YOUR VALIDATOR>" \
--security-contact="<EMAIL ADDRESS - THIS IS OPTIONAL>" \
--website="<WEBSITE - THIS IS OPTIONAL>" \
--identity=<KEYBASE ID - THIS IS OPTIONAL> \
--from=<YOUR KEY NAME>
The above should have been successful. If not, reach out to me on Telegram, @SkyNet_Validators and or in the Osmosis Discord channel, #validators
osmosisd tx staking edit-validator --identity=<KEYBASE_ID> --details "<YOUR_PERSONALIZED_MESSAGE>" --security-contact "<EMAIL_ADDRESS" --website "<WEBSITE>" --fees 5000uakt --ledger --indent --moniker <GIVE_YOUR_VALIDATOR_A_NAME> --from <KEY_NAME_YOU_GAVE_ABOVE>
If you want an image for your validator, create an account at https://keybase.io and add a profile picture. Then add it to the above command '--identity='. It will take a little bit of time to show in the explorer.
Our preferred method is to use a service to start the node
Fetch osmosisd location path and use it for 'WorkingDirectory' & 'ExecStart' in the next step
which osmosisd
The above command will output the following:
/usr/bin/osmosisd
Make akash a system service
sudo nano /lib/systemd/system/osmosisd.service
Paste the following into the above osmosisd.service files:
[Unit]
Description=Osmosis Daemon
After=network-online.target
[Service]
User=$USER
ExecStart=$(which cosmovisor) start
Restart=always
RestartSec=3
LimitNOFILE=65530
Environment="DAEMON_HOME=$HOME/.osmosisd"
Environment="DAEMON_NAME=osmosisd"
Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=false"
Environment="DAEMON_RESTART_AFTER_UPGRADE=true"
Environment="DAEMON_LOG_BUFFER_SIZE=512"
[Install]
WantedBy=multi-user.target
Enable and start the system service using:
sudo systemctl enable osmosisd.service
sudo systemctl start osmosisd.service
You can verify osmosis started properly with:
sudo systemctl status osmosisd
With the above configuration you now have a first class validator setup that adheres to validator best practices, Ledger Nano secured keys, and NGINIX Rate Control DDoS protection. Here is SkyNet | Validators network diagram here: