Monitoring your Diva Node
Last updated
Last updated
There will be 3 levels of metrics you as the Diva operator will be monitoring on.
On-chain metrics: How effective you are as an operator in maximising uptime and performance for the validator keys assigned to you - e.g. attestation effectiveness, active/inactive status, rewards & penalties
Device-level metrics: How does the health of your hardware look like? - e.g. internal temperature, CPU/RAM/storage usage, read/write speeds, network latency
Client-level metrics: Are there any warnings or errors due to misconfigurations or anomalies? - e.g. database corruption, out-of-memory errors, no auto-restart, undiscoverable endpoints, potential slashing violations
You will be able to see all the validator key shares you have been assigned by accessing your Operator UI via a browser.
Recap on how to access the Operator UI if needed below.
Divascan (https://divascan.io/) is an on-chain explorer similar to Etherscan that provides metrics on all operators, nodes, activated validator keys, and stakers of Diva Staking. This is a community-contributed project and is not an official product of Diva Staking.
The tabs you will be interested in as a Diva node operator are "Operators" and "Nodes". Each operator can operate multiple nodes and each node can in turn be assigned multiple validator key shares.
Under these tabs, you will be able to look for your operator ID - which is the wallet address you used when registering your node during the Registering your Diva node step.
Clicking into your operator ID will bring you to a page showing all of the nodes that have been registered to your operator ID, along with how many validator key shares have been assigned to each node.
Clicking into a Node ID here will show you a summary of all validator key shares assigned to that node, along with their corresponding public keys on the beacon chain.
It is worth noting that this method merely provides a very high level overview.
Negative performance does not immediately mean that there is something wrong with your node because each validator key has a fault tolerance rate of 5/16 - e.g. if more than 11 nodes are offline, the validator key assigned will be offline even if your node is online.
The reverse is true as well. Just because your assigned validator keys are chugging along, it does not mean that your nodes are performing up to standards.
The Diva Staking team will monitor the performance of each operator in a trust-less manner via the use of Zero-Knowledge Proof oracles and you will be penalised or ejected from the network for low performance.
For this reason, we will need other methods of monitoring your performance.
This method is used to monitor the performance of your assigned validator keys individually.
Although it may not be practical to do this for large amounts of key shares, it is still an extremely useful tool for monitoring alerts and troubleshooting.
1) Go to https://holesky.beaconcha.in/ and search for the validator pubkey you want to inspect. The validator pubkey can be found using Divascan (see above section)
2) You will then be able to see the status, the summary performance, and individual attestations of the validator you want to inspect. However, the industry defers to another explorer for the "Effectiveness" metric.
Rated.Network provides a more wholistic measure of "Effectiveness" (more details here) and you use it in the same way as Beaconcha.in - by searching for the validator pubkey or ID that you want to inspect.
The Diva service comes with a pre-configured docker container running Grafana, a highly extensible system/device-level monitoring dashboard.
Here's how you can access this dashboard:
1) First, make sure the Grafana docker container is running and exposing port 3000.
Expected output:
2) Open port 3000 on your device-level firewall
3) Identify the internal IP address of your node device.
Expected output: Look for an IP address that is similar to the format - 192.168.xx.xx. That will be the internal IP address.
4) Choose one of the following methods to access the Grafana dashboard depending on your network access.
Local (e.g. at home):
Connect your working laptop to the same router as your node device and enter http://<internal_IP_address>:3000 of the node device (e.g. 192.168.1.45) in your browser.
Remote (e.g. out of home):
Create an SSH tunnel into your node device. Note: Your SSH port needs to be open/forwarded from your modem/router into your node device. Check out the port forwarding section below if needed.
Once you are logged in to your node device, you can now enter http://127.0.0.1:3000 into the browser on your working laptop.
Make sure you enterhttp and not https here
1) Check that all docker containers are active and running
Expected output:
2) Check and follow logs of individual containers
This prints out the last 5 lines of the monitoring logs and new lines as they appear. You can change the number of lines as needed. Check the output for warnings or errors and troubleshoot them against the list of common errors in the next section.
If none of the scenarios matches your issue, you can raise them in the Diva Staking discord channel for targeted support.
Main containers:
Monitoring/Telemetry containers:
1) Warning that says "cannot start p2p network without identity". This means you have not registered your Diva node via the Operator UI
1) Unable to connect to Diva container endpoint.
Make sure your Diva client is active by running docker ps -a
Ensure that you have registered your Diva node via the Operator UI
