🦏
ETH Home Staking Collection
DVT Home Staking Curriculum
DVT Home Staking Curriculum
  • The DVT Home Staking Curriculum
  • Curriculum breakdown & timeline
  • Understanding ETH validators
    • Introduction to ETH Validators
    • Roles & Responsibilities of a node operator
    • Rewards and penalties
    • Importance of client diversity
    • Distributed Validator Technologies (DVTs)
    • Economics of using DVTs (WIP)
      • Diva Staking (WIP)
      • Obol (WIP)
      • SSV (WIP)
    • Bonded Validators
    • Economics of bonded validators (WIP)
  • Hardware & systems setup
    • Setup Overview
    • Hardware & system requirements
    • Procuring your hardware
    • Assemble your hardware
    • Practicing for free on Cloud VMs
      • Google Cloud
      • Alibaba Cloud
  • Linux OS, Networking, & Security
    • Install and prepare the OS
    • Networking & network security
    • Device level security setup
    • Verifying checksums
  • Installing & configuring your EL+CL clients
    • Set up and configure execution layer client
      • Nethermind
      • Besu
      • Geth
      • Erigon
      • Reth
    • Set up and configure consensus layer client
      • Teku BN
      • Nimbus BN
      • Lodestar BN
      • Lighthouse BN
      • Prysm BN
  • Keystore generation & MEV-Boost
    • Validator key generation
    • Set up and configure MEV-boost
  • Native Solo Staking Setup
    • Validator client setup
      • Teku VC
      • Nimbus VC
      • Lodestar VC
      • Lighthouse VC
      • Prysm VC
    • Depositing 32 ETH into your validator
    • Exiting your validator
  • Monitoring, Maintenance, and Updates
    • Set up monitoring suite
      • Installing & configuring Prometheus
      • Installing & configuring Node Exporter
      • Installing & configuring Grafana
      • Beaconcha.in App API
      • Client Uptime Check
    • Maintenance & Updates
      • Nethermind
      • Besu
      • Teku
      • Nimbus
      • Lodestar
      • Updating the monitoring suite
      • Preparing for Pectra
  • DVT Setup
    • Diva Staking
      • Diva Staking client setup
        • Default - All-in-one setup
        • Advanced - with standalone Lodestar VC
      • Registering your Diva node
      • Updating your Diva client
      • Monitoring your Diva Node
    • Obol
      • Techne Bronze Speedrun (Launchpad)
      • Obol + Bonded Validators (Techne Silver)
        • Obol + Lido CSM
    • SSV
      • SSV + Lido CSM (WIP)
      • SSV Operator
      • SSV Staker
  • Bonded Validators Setup
    • Lido CSM
      • Generating CSM keystores
      • Set Fee Recipient Address
        • Method 1: Configure on validator keys
        • Method 2: Configure on separate validator client
        • Verifying Fee Recipient Registered on MEV Relays
      • Upload/Remove/View validator keys
      • Rewards & bonds
      • Exiting CSM validators
        • "Lazy" exits (TESTNET ONLY)
        • Proper Exits
      • Role/Address management
      • Monitoring
      • Automations
        • CSM with ETHPillar
        • CSM with ETH Docker
        • CSM with Dappnode
    • Puffer
      • Non-Enclave: 2 ETH
    • Ether.fi
      • Receive distributed validator keyshares
    • Stader (WIP)
    • Rocketpool (WIP)
  • Liquid Staking Vaults
    • Stakewise V3
  • Mainnet
    • Mainnet Deployment
    • Heroglpyhs (WIP)
  • Best practices
    • Slashing prevention
    • Maximising uptime and performance
    • Optimising security
    • Managing your withdrawal wallet
  • Tips
    • Advanced networking
    • Downloading files from your node
  • Useful resources
    • General resources
    • Holesky Faucets
  • Automation/tools
    • ETHPillar
    • ETH Docker
    • Automated power on/off
      • Wake-on-LAN (WoL)
      • Network UPS Tools (NUT)
    • Validator Healthcheck Alerts
  • Solo Stakers Guild
    • Lido CSM+SSV+Obol (Testnet)
Powered by GitBook
On this page
  • Monitoring On-chain metrics
  • Operator UI
  • Divascan
  • Beaconcha.in & Rated.Network
  • Monitoring device performance with Grafana
  • Monitoring client performance with system logs
  • Expected outputs
  • Common errors
  • Diva Client
  • Lodestar Client
  1. DVT Setup
  2. Diva Staking

Monitoring your Diva Node

PreviousUpdating your Diva clientNextObol

Last updated 1 year ago

There will be 3 levels of metrics you as the Diva operator will be monitoring on.

  1. On-chain metrics: How effective you are as an operator in maximising uptime and performance for the validator keys assigned to you - e.g. attestation effectiveness, active/inactive status, rewards & penalties

  2. Device-level metrics: How does the health of your hardware look like? - e.g. internal temperature, CPU/RAM/storage usage, read/write speeds, network latency

  3. Client-level metrics: Are there any warnings or errors due to misconfigurations or anomalies? - e.g. database corruption, out-of-memory errors, no auto-restart, undiscoverable endpoints, potential slashing violations

Monitoring On-chain metrics

Operator UI

You will be able to see all the validator key shares you have been assigned by accessing your Operator UI via a browser.

Recap on how to access the Operator UI if needed below.

Divascan

The tabs you will be interested in as a Diva node operator are "Operators" and "Nodes". Each operator can operate multiple nodes and each node can in turn be assigned multiple validator key shares.

Clicking into your operator ID will bring you to a page showing all of the nodes that have been registered to your operator ID, along with how many validator key shares have been assigned to each node.

Clicking into a Node ID here will show you a summary of all validator key shares assigned to that node, along with their corresponding public keys on the beacon chain.

It is worth noting that this method merely provides a very high level overview.

Negative performance does not immediately mean that there is something wrong with your node because each validator key has a fault tolerance rate of 5/16 - e.g. if more than 11 nodes are offline, the validator key assigned will be offline even if your node is online.

The reverse is true as well. Just because your assigned validator keys are chugging along, it does not mean that your nodes are performing up to standards.

The Diva Staking team will monitor the performance of each operator in a trust-less manner via the use of Zero-Knowledge Proof oracles and you will be penalised or ejected from the network for low performance.

For this reason, we will need other methods of monitoring your performance.

Beaconcha.in & Rated.Network

This method is used to monitor the performance of your assigned validator keys individually.

Although it may not be practical to do this for large amounts of key shares, it is still an extremely useful tool for monitoring alerts and troubleshooting.

Beaconcha.in

2) You will then be able to see the status, the summary performance, and individual attestations of the validator you want to inspect. However, the industry defers to another explorer for the "Effectiveness" metric.

Rated.Network

Monitoring device performance with Grafana

The Diva service comes with a pre-configured docker container running Grafana, a highly extensible system/device-level monitoring dashboard.

Here's how you can access this dashboard:

1) First, make sure the Grafana docker container is running and exposing port 3000.

docker ps -a

Expected output:

2) Open port 3000 on your device-level firewall

sudo ufw allow 3000

3) Identify the internal IP address of your node device.

ip a

Expected output: Look for an IP address that is similar to the format - 192.168.xx.xx. That will be the internal IP address.

4) Choose one of the following methods to access the Grafana dashboard depending on your network access.

Local (e.g. at home):

Connect your working laptop to the same router as your node device and enter http://<internal_IP_address>:3000 of the node device (e.g. 192.168.1.45) in your browser.

Remote (e.g. out of home):

Create an SSH tunnel into your node device. Note: Your SSH port needs to be open/forwarded from your modem/router into your node device. Check out the port forwarding section below if needed.

ssh -L 3000:<internal_IP_address:3000 <user>@<external_ip_address> -p <SSH_port> -i <SSH_key> -v 

Once you are logged in to your node device, you can now enter http://127.0.0.1:3000 into the browser on your working laptop.

Make sure you enterhttp and not https here

Monitoring client performance with system logs

1) Check that all docker containers are active and running

docker ps -a

Expected output:

2) Check and follow logs of individual containers

docker logs <CONTAINER_ID> -f --tail 5

This prints out the last 5 lines of the monitoring logs and new lines as they appear. You can change the number of lines as needed. Check the output for warnings or errors and troubleshoot them against the list of common errors in the next section.

Expected outputs

Main containers:

Note: suggestedFeeRecipeint is set to the "zero" address and burnt during this testnet phase but will be set to a designated pooling address on the mainnet.

*Before being assigned any validator key shares from Diva Staking

Monitoring/Telemetry containers:

Common errors

Diva Client

Lodestar Client

1) Unable to connect to Diva container endpoint.

Dec-16 15:34:07.791[]                 info: Lodestar network=goerli, version=v1.12.1/85e44ef, commit=85e44efe73d1ad5228486c73ea9b1fcf3f036193
Dec-16 15:34:07.793[]                 info: Connecting to LevelDB database path=/var/lib/lodestar/validators/validator-db
 ✖ FetchError: Request to http://diva:9000/api/v1/eth2/publicKeys failed, reason: connect ECONNREFUSED 172.27.0.4:9000
    at wrappedFetch (file:///usr/app/packages/api/src/utils/client/fetch.ts:10:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at externalSignerGetKeys (file:///usr/app/packages/validator/src/util/externalSignerClient.ts:111:15)
    at getRemoteSigners (file:///usr/app/packages/cli/src/cmds/validator/signers/index.ts:170:54)
    at Object.validatorHandler [as handler] (file:///usr/app/packages/cli/src/cmds/validator/handler.ts:91:19)

Fixes:

  • Make sure your Diva client is active by running docker ps -a

Divascan () is an on-chain explorer similar to Etherscan that provides metrics on all operators, nodes, activated validator keys, and stakers of Diva Staking. This is a community-contributed project and is not an official product of Diva Staking.

Under these tabs, you will be able to look for your operator ID - which is the wallet address you used when registering your node during the step.

1) Go to and search for the validator pubkey you want to inspect. The validator pubkey can be found using Divascan (see above section)

Rated.Network provides a more wholistic measure of "Effectiveness" (more details ) and you use it in the same way as Beaconcha.in - by searching for the validator pubkey or ID that you want to inspect.

If none of the scenarios matches your issue, you can raise them in the for targeted support.

1) Warning that says "cannot start p2p network without identity". This means you have not

Ensure that you have

Registering your Diva node
https://divascan.io/
Registering your Diva node
https://holesky.beaconcha.in/
here
Advanced networking
Diva Staking discord channel
registered your Diva node via the Operator UI
registered your Diva node via the Operator UI
Example of a pre-configured Diva dashboard on Grafana
Lodestar VC