Celestia Validator Node Deployment on Mocha-4 Testnet

Introduction

Celestia is a modular blockchain designed to enhance scalability and decentralization by separating consensus and data availability layers. Running a Celestia validator node on the Mocha-4 testnet can be a complex process, but automation through Ansible simplifies the deployment, management, and monitoring of the node. This blog provides a step-by-step guide to automate the entire process, from setting up the infrastructure to monitoring performance.

Why Automate Celestia Validator Node Deployment?

Manually deploying a validator node involves a series of repetitive and error-prone steps. Automation offers the following benefits:

Consistency: Ensures that every deployment follows the same configurations.
Efficiency: Reduces manual intervention, saving time and effort.
Scalability: Makes it easier to deploy multiple nodes.
Security: Securely stores sensitive credentials and configuration files.

Prerequisites

Before diving into automation, you need to set up the required infrastructure components:

1. AWS EC2 Instance Setup

Login to AWS Console: Navigate to the AWS EC2 Console and click on "Launch Instance."
Choose AMI: Select Ubuntu 24.04 LTS.
Instance Type: Choose t2.medium (2 vCPUs and 4GB RAM).
Storage: Add a 50GB volume for blockchain data.
Security Group Configuration: Open necessary ports:
- TCP 26656 (P2P Communication)
- TCP 26657 (RPC Communication)
- TCP 26658 (State Sync)
- TCP 8080 (Prometheus)
- TCP 3000 (Grafana)
Launch Instance: Review and download the private key for SSH access.

2. SSH into the Instance

Use the following command to connect to your instance:

ssh -i path/to/your-key.pem ubuntu@your-ec2-public-ip

Once logged in, switch to the root user:

sudo -i

After switching to the root user, you can continue with the playbook execution or manually perform any required configurations.

System Preparation

Before proceeding with the Celestia validator node installation, it's important to update and upgrade the system to ensure all the packages are up to date.

Run the following command to update the package list:

sudo apt-get update -y

After the update is completed, upgrade all the installed packages to their latest versions:

sudo apt-get upgrade -y

These commands will ensure that your system is running the latest available software and security patches. Once the upgrade process is complete, your system will be ready for the Celestia validator node installation.

Clone the Celestia Validator Deployment Repository

To begin the setup, we need to clone the repository that contains the Ansible playbook and related scripts for automating the Celestia validator node deployment.

Use the following command to clone the repository to your local machine:

git clone https://github.com/nishankkoul/celestia-validator-deployment.git

Once the repository is cloned, navigate to the directory where the repository has been downloaded:

cd celestia-validator-deployment

This repository contains all the necessary configurations, playbooks, and scripts required to deploy and manage the Celestia validator node. Proceed with the next steps to configure and deploy the node.

Install Ansible

Now, we need to install Ansible, which will be used to automate the deployment and configuration of the Celestia validator node.

Install Ansible using the following command:

sudo apt install ansible -y

After installation, verify that Ansible is installed successfully by checking its version:

ansible --version

This will ensure that Ansible is set up correctly and ready for use in automating the deployment process. You can now proceed to run the Ansible playbook for the Celestia validator node.

Configure Variables

Next, we'll navigate to the folder where the configuration variables are stored and edit the main.yml file to customize the deployment for your environment.

Change into the celestia_validator/vars directory:

cd celestia_validator/vars

Open the main.yml file in the vim editor:

vim main.yml

Edit the variables in this file as needed. You can adjust settings such as the Celestia validator's name, network configuration, and other parameters based on your setup.

# vars/main.yml

celestia_repo: "https://github.com/celestiaorg/celestia-app.git"
celestia_version: "v3.2.0"  #Change the version accordingly
celestia_chain_id: "mocha-4"
celestia_wallet_name: "my-wallet"  #Set the Celestia Wallet Name
celestia_networks_repo: "https://github.com/celestiaorg/networks"
celestia_genesis_file: "/root/networks/mocha-4/genesis.json"
celestia_snap_url: "https://s3.imperator.co/testnets-snapshots/celestia/celestia_chain_4141707.tar.zst"
celestia_service_name: "celestia-appd"
celestia_service_exec: "/usr/local/bin/celestia-appd start"
celestia_config_path: "/root/.celestia-app/config"
celestia_priv_validator_file: "/root/.celestia-app/data/priv_validator_state.json"
celestia_snap_name: "celestia_chain_4141707.tar.zst"

Make sure to refer https://www.imperator.co/services/chain-services/testnets/celestia for the latest version of the snapshot and set the variables 'celestia_snap_url' and 'celestia_snap_name' accordingly. You can refer to some other authentic snapshots as well.

After editing the file, save your changes and exit the editor:

Press Esc to exit insert mode.
Type :wq to write the changes and quit the editor.

This will ensure that the necessary variables are set up before executing the Ansible playbook to deploy the Celestia validator node.

Execute the Ansible Playbook

Once the variables are configured, we are ready to execute the Ansible playbook to deploy the Celestia validator node.

Run the following command to execute the playbook:

ansible-playbook deploy_celestia.yml -vvv

The -vvv flag provides detailed output, which is useful for debugging and monitoring the playbook execution.

The playbook will run through various tasks, such as installing necessary dependencies, configuring the node, applying the snapshot, generating the wallet, and more.
Wait for the playbook to complete. The output will show each task as it is executed, along with any relevant success or failure messages.

Wallet Address Details

After successfully executing the Ansible playbook, the wallet address details are securely stored in the following location: /etc/celestia/wallet_mnemonic.txt

This file contains the mnemonic phrase generated for the wallet associated with the Celestia validator node. The wallet address is essential for interacting with the validator on the testnet.

Important Security Note:

→ The file is stored with strict access control (0600 permissions), meaning only the root user has access to it. This ensures the security of sensitive information like the mnemonic phrase, which is required to restore or manage the wallet in the future.

→ The task for storing this wallet mnemonic is set with no_log: true to prevent it from being logged during playbook execution.

Encrypt the Wallet Mnemonic File

To securely store the wallet mnemonic, we will encrypt the file using ansible-vault to prevent unauthorized access. This ensures that the sensitive information (such as the seed phrase) remains protected.

Run the following command to encrypt the wallet_mnemonic.txt file:

ansible-vault encrypt /etc/celestia/wallet_mnemonic.txt

You will be prompted to enter a password. This password will be used to decrypt the file later, so make sure to store it securely.

Decrypting the File (If Needed)

To decrypt the file and view its contents, use the following command:

ansible-vault decrypt /etc/celestia/wallet_mnemonic.txt.vault

Check the Status of the Celestia Validator Service

After setting up the validator node, it's important to verify that the celestia-appd service is running properly.

Check the Service Status:

To check the status of the Celestia application service, run the following command:

systemctl status celestia-appd

Check Daemon Logs in Real-Time

To ensure that the Celestia validator node is operating correctly, you can check the logs of the celestia-appd service in real-time. This helps you monitor its activities and troubleshoot any potential issues as they occur.

View Logs in Real-Time

To view the real-time logs of the Celestia daemon, run the following command:

sudo journalctl -u celestia-appd.service -f

Check if Your Node is in Sync Before Going Forward

To verify if your Celestia validator node is successfully synced with the testnet, you can check the synchronization status by comparing the block height of your node with the latest block height on the explorer. Once they match, it means your node is fully synced.

Check Node Sync Status

To check the current sync status of your node, run the following command:

curl -s localhost:26657/status | jq .result | jq .sync_info

NOTE: DO NOT PROCEED FORWARD TILL THE NODE IS IN SYNC COMPLETELY! ALSO, MAKE SURE TO KEEP MONITORING THE CPU USAGE USING THE 'top' COMMAND AND ADJUST THE INSTANCE TYPE ACCORDINGLY

You can stay aware of the latest block height of the network here: https://mocha.celenium.io/

Claim Test Tokens from the Celestia Faucet

To participate in the Celestia testnet, you need test tokens for your validator. You can claim these test tokens from the official Celestia Mocha-4 testnet faucet.

How to Get Test Tokens

Join the Celestia Discord: To access the faucet, you must first join the Celestia Discord server. You can do so by visiting the following link: Celestia Discord
Request Test Tokens: Once you're in the Discord server, navigate to the mocha-faucet channel. In this channel, you can request test tokens for your wallet.
Use the Command to Request Tokens: In the mocha-faucet channel, use the following command to request test tokens:
```
 $request <celestia-wallet-address>
```

Check Wallet Balance

Once you have successfully claimed test tokens and your wallet is set up, you can check the balance of your Celestia wallet using the following command:

celestia-appd q bank balances <your-wallet-address>

Replace with the actual wallet address you created earlier.

This command queries the Celestia blockchain and displays the balance of your specified wallet address. It will show the amount of tokens currently held in your wallet, which is crucial before proceeding with the validator registration process.

Execute the `validator_wallet.sh` Script and Lock the File Using Ansible Vault

Follow the steps below to edit the validator_wallet.sh script, make it executable, execute it, and lock the file containing the valoper_address using Ansible Vault. Firstly, 'cd' into the scripts directory.

Step 1: Edit the `validator_wallet.sh` Script

Open the script 'validator_wallet.sh' using the vim/nano editor and then replace the with the actual name of your celestia wallet.

#!/bin/bash

export CELESTIA_WALLET=<WALLET-NAME>

if [ -z "$CELESTIA_WALLET" ]; then
  echo "Error: CELESTIA_WALLET environment variable is not set."
  echo "Please set it using 'export CELESTIA_WALLET=<your_wallet_name>' and rerun this script."
  exit 1
fi

# Fetch the validator address and store it in a variable
CELESTIA_VALOPER=$(celestia-appd keys show $CELESTIA_WALLET --bech val -a)

# Write the actual validator address to a file in /etc/celestia/
echo $CELESTIA_VALOPER > /etc/celestia/valoper_address.txt

# Add the CELESTIA_VALOPER environment variable to .bash_profile
echo "Exporting CELESTIA_VALOPER to .bash_profile..."
echo "export CELESTIA_VALOPER=${CELESTIA_VALOPER}" >> $HOME/.bash_profile

# Reload .bash_profile to make the variable available in the current session
source $HOME/.bash_profile

# Confirm the export
echo "CELESTIA_VALOPER environment variable has been set."

Save the script and exit the editor.

Step 2: Make the Script Executable

Change the permissions of the script to make it executable:

chmod +x validator_wallet.sh

Step 3: Execute the Script

Run the script to create the validator wallet and generate the valoper_address:

./validator_wallet.sh

The valoper_address.txt file will be created at /etc/celestia/valoper_address.txt.

Step 4: Encrypt the File with Ansible Vault

To secure the valoper_address.txt file, we will encrypt it using Ansible Vault.

ansible-vault encrypt /etc/celestia/valoper_address.txt

You will be prompted to enter a password for encryption. After entering the password, the file will be encrypted, and the content will be securely stored.

Exposing Addresses within the Environment

To expose the Celestia-related variables CELESTIA_ADDR, CELESTIA_VALOPER, and CELESTIA_WALLET within the environment, follow the steps below.

Step 1: Expose `CELESTIA_WALLET`

Export the CELESTIA_WALLET variable:

export CELESTIA_WALLET=<WALLET-NAME>
echo $CELESTIA_WALLET
echo 'export CELESTIA_WALLET='${CELESTIA_WALLET} >> $HOME/.bash_profile
source $HOME/.bash_profile

Make sure to replace it with your wallet name here.

Step 2: Expose CELESTIA_ADDR

Use the following commands to fetch the address and export it as CELESTIA_ADDR:

CELESTIA_ADDR=$(celestia-appd keys show $CELESTIA_WALLET -a)
echo $CELESTIA_ADDR
echo 'export CELESTIA_ADDR='${CELESTIA_ADDR} >> $HOME/.bash_profile
source $HOME/.bash_profile

Step 3: Expose CELESTIA_VALOPER

Use the following commands to fetch the validator operator address and export it as CELESTIA_VALOPER:

CELESTIA_VALOPER=$(celestia-appd keys show $CELESTIA_WALLET --bech val -a)
echo $CELESTIA_VALOPER
echo 'export CELESTIA_VALOPER='${CELESTIA_VALOPER} >> $HOME/.bash_profile
source $HOME/.bash_profile

Step 4: Secure Sensitive Information

Ensure that the .bash_profile is accessible only to the root user or the relevant system administrator to prevent unauthorized access to the environment variables.

chmod 600 $HOME/.bash_profile

By following these steps, you have successfully exposed the Celestia-related addresses and wallet information as environment variables, making them readily available for use in your scripts or commands.

Execute the `create_validator.sh` File to Create a Validator Node

The create_validator.sh script is designed to create a validator node and securely store the generated information. Follow these steps to execute the script:

Make the Script Executable and Execute it

Before executing the script, ensure it has the appropriate permissions to run:

cd celestia-validator-deployment/celestia_validator/scripts
chmod +x create_validator.sh
./create_validator.sh

Make sure to replace the MONIKER with whatever alias you want to set your validator node with.

MONIKER="my-celestia-wallet"

celestia-appd tx staking create-validator \
    --amount=1000000utia \
    --pubkey=$(celestia-appd tendermint show-validator) \
    --moniker=$MONIKER \
    --chain-id=mocha-4 \
    --commission-rate=0.1 \
    --commission-max-rate=0.2 \
    --commission-max-change-rate=0.01 \
    --min-self-delegation=1000000 \
    --from=$CELESTIA_ADDR \
    --keyring-backend=test \
    --fees=21000utia \
    --gas=220000

With the validator node successfully created, you are ready to move forward with additional configurations or interactions as needed.

Delegate to a Validator

Delegating tokens to a validator is a critical step in supporting the Celestia network. Follow these steps to delegate tokens:

Step 1: Execute the Delegation Command

Run the following command to delegate 1,000,000 utia tokens to your validator:

celestia-appd tx staking delegate $CELESTIA_VALOPER 1000000utia --chain-id mocha-4 --fees=420utia --from $CELESTIA-ADDR

The command bash celestia-appd tx staking delegate $CELESTIA_VALOPER 1000000utia --chain-id mocha --fees=420utia --from $CELESTIA-ADDR delegates 1,000,000 utia tokens from the wallet address stored in $CELESTIA-ADDR to the validator identified by $CELESTIA_VALOPER on the mocha testnet. It also deducts a transaction fee of 420 utia from the wallet. This action helps secure the network and allows the delegator to earn staking rewards over time. Ensure sufficient balance in the wallet for both the delegation and transaction fee.

Step 2: Verify Delegation

To ensure the delegation was successful, use the following command:

celestia-appd q staking validator $CELESTIA_VALOPER

The command celestia-appd q staking validator $CELESTIA_VALOPER retrieves detailed information about the validator associated with the $CELESTIA_VALOPER address. It provides details such as the validator's commission rates, public key, delegator shares, description (moniker, website, etc.), status (e.g., bonded or unbonded), total tokens staked, minimum self-delegation requirement, and other metadata. This command is useful for monitoring and verifying the validator's configuration and operational status within the Celestia network.

You have now successfully delegated tokens to your validator, contributing to the Celestia network.

To verify that your validator is a part of the network, visit mintscan.io/celestia-testnet

If the node gets more delegations, it will change to Active state.

Monitoring using Prometheus and Grafana

Metrics are a powerful tool for monitoring the health and performance of a system. Celestia provides support for metrics to make sure, as an operator, your system continues to remain up and running. Metrics can also provide critical insight into how Celestia is used and how it can be improved.

Setup

Celestia uses Prometheus to publish metrics. It can be enabled through the $HOME/.celestia-app/config/config.toml file.

#######################################################
###       Instrumentation Configuration Options     ###
#######################################################
[instrumentation]

# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = true

# Address to listen for Prometheus collector(s) connections
prometheus_listen_addr = ":26660"

# Maximum number of simultaneous connections.
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
max_open_connections = 3

# Instrumentation namespace
namespace = "celestia"

Restart the Service

Next, you need to restart the service using the command:

systemctl restart celestia-appd

curl localhost:26660/metrics

You will now be able to see the metrics being scraped.

Visualization

Now your nodes are publishing metrics, we need something to scrape it and a visualizer to create a dashboard. Commonly, Prometheus is paired with Grafana.

First, you will need to install Prometheus and for that execute the below command:

sudo apt install prometheus -y

Next, create a config file $HOME/.celestia-app/config/prometheus.yml and fill out some basic settings as follows:

global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating
  # with external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: "codelab-monitor"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries
  # scraped from this config.
  - job_name: "prometheus"

    # Override the global default and scrape targets from this job every
    # 5 seconds.
    scrape_interval: 5s

    static_configs:
      # Point to the same endpoint that Celestia is publishing on
      - targets: ["localhost:26660"]

Note, that Prometheus by default runs its server on :9090. If you are running this on the same machine as your consensus node, it will collide with gRPC which runs on the same port. To avoid this, either switch off gRPC (if it's not needed), change the gRPC port in app.toml, or run Prometheus on a different port e.g. --web.listen-address="0.0.0.0:8080"

To run the prometheus server:

prometheus --config.file=/root/.celestia-app/config/prometheus.yml --web.listen-address="0.0.0.0:8080" &

This will begin the server on http://{EC2-IP-ADDRESS}:8080

A prometheus server can scrape metrics from multiple nodes at once; a good way of bringing together information from many machines to one place.

To visualize the information, you can use Grafana. For Grafana Installation, exeucte the below commands:

sudo apt-get update -y
sudo apt-get upgrade -y 

sudo apt-get install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_11.4.0_amd64.deb
sudo dpkg -i grafana-enterprise_11.4.0_amd64.deb

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

This will begin the server on http://{EC2-IP-ADDRESS>}:3000. If you open the URL on your browser you will see the Grafana login page. Use admin for both the user and password to log in.

You will need to link the Prometheus server as a data source. To do that go to "Configuration" in the sidebar and then "Data Sources". Add a new data source specifying the URL of the Prometheus instance (http://:9090). Click "Save & test" to confirm.

Dashboard

Next, we will be setting up the dashboard for these metrics: CPU Usage, Disk Usage, Free Memory, Peer Connections, Block Height, and Sync Status.

Authentication

Now, we will be creating a user in Grafana for authentication purposes. Go to Home -> Administration -> Users and Access -> Users.

Click on New User.

In this way, we can create a new user. Also, we can give admin access to the user after the user is created.

Conclusion

Automating the deployment of a Celestia validator node using Ansible significantly simplifies the setup process, ensures consistency, and enhances security. With Prometheus and Grafana, you can monitor your node's health and performance efficiently.

By following this guide, you can confidently participate in the Celestia Mocha-4 testnet, secure your validator, and contribute to the network's decentralization.