Backup of Network UPS Tools (NUT)

Network UPS Tools (NUT)

My method involves some custom shell scripts to work around the limitations of my UPS model and/or my current understanding of NUT. Though I try my best to implement security best practices here, please use this with caution as you will need to run custom scripts on your validator node.

You will need a UPS with a USB port for this setup.

Plug in your device (node) to the UPS power socket and connect them via a USB cable.

Install the Network UPS Tools package on your device.

sudo apt update
sudo apt install nut nut-client nut-server

Inspect the NUT folder.

sudo ls -l /etc/nut

You should see the following configuration files which we will be customising in this guide.

nut.conf ups.conf  upsd.conf  upsd.users  upsmon.conf  upssched.conf

Identify key information about your UPS.

sudo nut-scanner

Example output:

Scanning USB bus.
No start IP, skipping SNMP
Scanning XML/HTTP bus.
No start IP, skipping NUT bus (old connect method)
Scanning IPMI bus.
[nutdev1]
	driver = "blazer_usb"
	port = "auto"
	vendorid = "0665"
	productid = "5161"
	product = "USB to Serial"
	vendor = "INNO TECH"
	bus = "003"

Back up the exiting ups.conf file as a copy and edit the main file.

sudo cp /etc/nut/ups.conf /etc/nut/ups.conf.example
sudo nano /etc/nut/ups.conf

Replace the file contents by matching the output from the nut-scanner output above. Use CTRL+T and then CTRL+V to clear all file contents.

Example:

[nutdev1]
    driver = blazer_usb
    port = auto
    desc = "USE PRODUCT OR VENDOR HERE"
    vendorid = 0665
    productid = 5161

CTRL+O, ENTER, CTRL+X to save and exit.

Back up the exiting upsd.conf file as a copy and edit the main file.

sudo cp /etc/nut/upsd.conf /etc/nut/upsd.conf.example
sudo nano /etc/nut/upsd.conf

Replace the file contents with the following. Use CTRL+T and then CTRL+V to clear all file contents.

LISTEN 0.0.0.0 3493

CTRL+O, ENTER, CTRL+X to save and exit.

Back up the exiting nut.conf file as a copy and edit the main file.

sudo cp /etc/nut/nut.conf /etc/nut/nut.conf.example
sudo nano /etc/nut/nut.conf

Replace the file contents with the following. Use CTRL+T and then CTRL+V to clear all file contents.

MODE=netserver

CTRL+O, ENTER, CTRL+X to save and exit.

Back up the exiting upsd.users file as a copy and edit the main file.

sudo cp /etc/nut/upsd.users /etc/nut/upsd.users.example
sudo nano /etc/nut/upsd.users

Replace the file contents with the following. Use CTRL+T and then CTRL+V to clear all file contents.

[monuser]
  password = secret
  upsmon master
#"upsmon", "secret", and "master" need to match the contents of the upsd.users
# to be set below

CTRL+O, ENTER, CTRL+X to save and exit.

Back up the exiting upsmon.conf file as a copy and edit the main file.

sudo cp /etc/nut/upsmon.conf /etc/nut/upsmon.conf.example
sudo nano /etc/nut/upsmon.conf

Add the following as new lines to the bottom of the existing file content.

MONITOR nutdev1@localhost 1 upsmon secret master
#"upsmon", "secret", and "master" need to match the contents of the upsd.users set above
NOTIFYFLAG ONBATT SYSLOG+EXEC+WALL
NOTIFYCMD /etc/nut/onbatt.sh # we will create this shell script later

And edit the SHUTDOWNCMD line to the following.

SHUTDOWNCMD "/sbin/shutdown -h now && /sbin/upsdrvctl shutdown"

CTRL+O, ENTER, CTRL+X to save and exit.

Create the onbatt.sh shell script which tells your device to shut down gracefully when it detects that power supply to your UPS has been cut off (e.g., due to a power outage).

Note: I am using a simple script to handle the automated shutdown sequence because I can't get my NUT to work as intended. Let me know if anyone managed to get yours working.

sudo nano /etc/nut/onbatt.sh

Paste the following content.

#!/bin/bash
# Log the start of the script
logger "NUT: Checking UPS status for ONBATT event"

# Fetch the UPS status
status=$(upsc prolink@localhost ups.status)

# Check if the UPS is on battery
if [[ "$status" == "OB" ]]; then
    logger "NUT: UPS on battery power detected. Waiting 60 seconds before shutting down."
    
    # Wait for 60 seconds
    sleep 60
    
    # Log the shutdown initiation
    logger "NUT: Initiating shutdown after 60-second delay."
    sudo /sbin/shutdown -h now "UPS on battery power"
else
    logger "NUT: UPS status is not ONBATT (status: $status). No shutdown triggered."
fi

CTRL+O, ENTER, CTRL+X to save and exit.

Make this shell script executable.

sudo chmod +x /etc/nut/onbatt.sh

Allow the nut user to run only the /sbin/shutdown to power down your device without needing the sudo (superuser) password.

sudo visduo

Add the following as a new line in the file.

nut ALL=(ALL) NOPASSWD: /sbin/shutdown

Restart the NUT services.

sudo service nut-server restart
sudo service nut-client restart
sudo systemctl restart nut-monitor
sudo upsdrvctl stop
sudo upsdrvctl start

Inspect the nut-server and nut-monitor for errors.

sudo journalctl -fu nut-server  -o cat | ccze -A
sudo journalctl -fu nut-monitor -o cat | ccze -A

Example output:

#nut-server
Starting Network UPS Tools - power devices information server... 
fopen /run/nut/upsd.pid: No such file or directory 
listening on 0.0.0.0 port 3493 
listening on 0.0.0.0 port 3493 
Connected to UPS [prolink]: blazer_usb-prolink 
Connected to UPS [prolink]: blazer_usb-prolink 
Startup successful 
Started Network UPS Tools - power devices information server. 
User [email protected] logged into UPS [prolink]

#nut-monitor
Starting Network UPS Tools - power device monitor and shutdown controller... 
fopen /run/nut/upsmon.pid: No such file or directory 
UPS: prolink@localhost (master) (power value 1) 
Using power down flag file /etc/killpower 
Startup successful 
nut-monitor.service: Can't open PID file /run/nut/upsmon.pid (yet?) after start: Operation not permitted 
nut-monitor.service: Supervising process 1206 which is not our child. We'll most likely not notice when it exits. 
Started Network UPS Tools - power device monitor and shutdown controller. 
nut-monitor.service: Supervising process 1206 which is not our child. We'll most likely not notice when it exits. 
nut-monitor.service: Supervising process 1206 which is not our child. We'll most likely not notice when it exits. 

CTRL+C to exit logging view.

Enable auto-power-on in BIOS

Auto-power-on may not be enabled on your devices by default. If so, you will need to plug in a monitor and keyboard into your device to enable it.

1. Restart your device and press F2 repeatedly during boot to enter BIOS Setup.

2. Select Advanced, then select the Power menu.

3. Expand the Secondary Power Settings sub-menu and set After Power Failure to Power On.

4. Press F10 to save and exit the BIOS Setup.

Summary of setup

• Your device (e.g., validator node) will now gracefully shut down when your UPS is cut off from it's power supply and turn itself back on when power is restored.

• If it doesn't (like mine), then you will need to configure a separate Wake-on-LAN (WOL) server to handle automatic power-on signals. Fortunately, this can be done using a $50 Raspberry Pi.

Configure Wake-on-LAN (WOL)

This configuration is applied to any device that needs to be remotely powered on automatically after recovering from a power failure. i.e., the Wake-on-LAN clients

Install wakeonlan and ethtool on your device.

sudo apt install wakeonlan
sudo apt install ethtool

Identify the ethernet interface of your device.

ip a

It will be the one that has the 192.168.xx.xx IP address assigned. For example:

Check your existing Wake-on-LAN status. Replace enp2s0with the actual ethernet interface of your device.

sudo ethtool enp2s0 | grep "Wake-on"

If you see the following, you can proceed to the next sub-section. Else, continue following along.

	Supports Wake-on: pumbg
	Wake-on: g

If you see Wake-on: d or any other letter here, it means wake-on-lan is disabled or not optimally configured so you need to change this letter to g.

sudo ethtool -s enp2s0 wol g

Next, make this configuration persistent even after rebooting your system.

sudo nano /etc/netplan/01-network-manager-all.yaml

Add the following lines to your file.

  ethernets:
    <ETHERNET_INTERFACE>:
      wakeonlan: true

Example of how your file should look like.

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      wakeonlan: true

CTRL+O, ENTER, CTRL+X to save and exit.

Apply the new configuration.

sudo netplan apply

Enable Wake-on-LAN in BIOS

Wake-on-LAN may not be enabled on your devices by default. If so, you will need to plug in a monitor and keyboard into your device to enable it.

1. Restart your device and press F2 repeatedly during boot to enter BIOS Setup.

2. Select Advanced, then select the Power menu.

3. Expand the Secondary Power Settings sub-menu and set Wake-on-LAN from S4/S5 to: Power On - Normal Boot.

4. Press F10 to save and exit the BIOS Setup.

Setup an automated Wake-on-LAN server

You will need to use a Raspberry Pi or a similar low-powered device without a standby power mode for this setup. i.e., no on/off button, turns on once connected to a power source.

This device will serve as the Wake-on-LAN server that sends "power on" signals to all your other devices in the same network after recovering from a power failure.

Create a wake-on-LAN script on your Raspberry Pi that covers all your other devices.

sudo nano /usr/local/bin/wake_devices

Paste the following content:

Add more devices as needed as new lines in the format ["Name"]="MAC Address" within the declare -A devices=(...) segment.

#!/bin/bash
# Wake all devices by sending magic packets using wakeonlan
# Logs are saved to a designated folder, including device names

# Define the folder to store logs
LOG_FOLDER="$HOME/wol_logs"
LOG_FILE="$LOG_FOLDER/wol_$(date '+%Y-%m-%d_%H-%M-%S').log"

# Create the log folder if it doesn't exist
mkdir -p "$LOG_FOLDER"

# Define a list of devices with their names and MAC addresses
declare -A devices=(
    ["testnode"]="aa:bb:cc:dd:ee:ff" # replace with your actual device name and MAC address 
    # Add more devices as needed as new lines here in the format ["Name"]="MAC"
)

# Start logging
echo "WOL Script started at $(date)" > "$LOG_FILE"
echo "Log file: $LOG_FILE" >> "$LOG_FILE"

# Send a WOL packet to each device
for name in "${!devices[@]}"; do
    mac="${devices[$name]}"
    echo "Sending WOL packet to $name ($mac)..." | tee -a "$LOG_FILE"
    wakeonlan "$mac" >> "$LOG_FILE" 2>&1
    if [ $? -eq 0 ]; then
        echo "Successfully sent WOL packet to $name ($mac)" | tee -a "$LOG_FILE"
    else
        echo "Failed to send WOL packet to $name ($mac)" | tee -a "$LOG_FILE"
    fi
done

# End logging
echo "WOL Script finished at $(date)" >> "$LOG_FILE"
echo "Logs saved to $LOG_FILE"

CTRL+O, ENTER, CTRL+X to save and exit.

Make this script executable.

sudo chmod +x /usr/local/bin/wake_devices

We want this script to run automatically whenever our WOL server restarts after a power failure. Create a new systemd service file to run the script at startup.

sudo nano /etc/systemd/system/wake_devices.service

Add the following content:

[Unit]
Description=Wake-on-LAN script to wake all devices
After=network.target

[Service]
ExecStartPre=/bin/sleep 300
ExecStart=/usr/local/bin/wake_devices
Restart=on-failure
User=raspberrypi 
#use your actual system user above

[Install]
WantedBy=multi-user.target

We want the WOL script to run only after all your other devices have completely turned off in the event of a instant recovery following a power failure, which will cause this script to fail its purpose. Hence the deliberate 300 second delay imposed in this service file via /bin/sleep 300

Load and run the WOL service.

sudo systemctl daemon-reload
sudo systemctl enable wake_devices.service
sudo systemctl start wake_devices.service
sudo systemctl status wake_devices.service
sudo journalctl -u wake_devices

Use CTRL+C to exit the monitoring/logging view.

Manual Wake-on-LAN via Telegram Bot

This is useful as a backup to the automated WOL setup above in case you need to manually "wake up" your devices remote after recovery from a power failure.

Key Features:

1. Does not require opening ports to each of your devices

2. Conveniently "wakes up" all your devices via Telegram. i.e., without needing to download new apps

3. Run on your Wake-on-LAN server. e.g., Raspberry Pi that runs 24/7

4. Requires the "Setup an automated Wake-on-LAN server" sub-section above to be completed

Create a new Telegram bot by following the steps below.

1. Open Telegram and Message the BotFather:

   • Search for "BotFather" in Telegram and start a conversation.

2. Create a New Bot:

   • Send /newbot to the BotFather.

   • Follow the instructions in the BotFather chat to name your bot and get its API token.

3. Save the API Token:

   • Example token: 123456789:ABCDEFGHIJKLMNOPQRSTUVWXYZ.

4. Add the Bot to Your Private Group:

   • Invite the bot to your Telegram group.

5. Get Your Chat ID:

   • Use the bot to retrieve the chat ID:

     • Send a message in the Telegram group with your bot

     • Navigate to https://api.telegram.org/bot<YourBOTToken>/getUpdates on your browser while replacing <YourBOTToken> with your actual Telegram bot API token

Install dependencies on your WOL server.

sudo apt install python3-pip
sudo apt install python3-dotenv
sudo apt install python3-python-telegram-bot

Create a new folder to store the bot files.

sudo mkdir -p /usr/local/bin/TG_WOL_BOT

Create the .env file to store private and sensitive information such as your Telegram bot token and chat ID.

sudo nano /usr/local/bin/TG_WOL_BOT/.env

Add your Bot API Token and Chat ID as variables into .env

#example
BOT_TOKEN=123456789:ABCDEFGHIJKLMNOPQRSTUVWXYZ
ALLOWED_CHAT_ID=-123456789

CTRL+O, ENTER, CTRL+X to save and exit.

Secure the .env file so that only your current user can access the file.

sudo chmod 600 /usr/local/bin/TG_WOL_BOT/.env
sudo chown -R $USER:$USER /usr/local/bin/TG_WOL_BOT

Create the Telegram bot script.

sudo nano /usr/local/bin/TG_WOL_BOT/WOL_bot.py

Paste the following content:

from telegram.ext import Application, CommandHandler
import subprocess
import os
from dotenv import load_dotenv

# Load environment variables
load_dotenv("/usr/local/bin/TG_WOL_BOT/.env")

BOT_TOKEN = os.getenv("BOT_TOKEN")
ALLOWED_CHAT_ID = int(os.getenv("ALLOWED_CHAT_ID"))

async def start(update, context):
    """Send a welcome message."""
    chat_id = update.effective_chat.id
    if chat_id != ALLOWED_CHAT_ID:
        await update.message.reply_text("Unauthorized!")
        return
    await update.message.reply_text("Hi! I'm your Wake-on-LAN bot.")

async def wol_command(update, context):
    """Run the Wake-on-LAN script."""
    chat_id = update.effective_chat.id
    if chat_id != ALLOWED_CHAT_ID:
        await update.message.reply_text("Unauthorized!")
        return
    try:
        await update.message.reply_text("Running Wake-on-LAN script...")
        # Run the WoL script
        result = subprocess.run(
            ["/usr/local/bin/wake_devices"],
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True
        )
        if result.returncode == 0:
            await update.message.reply_text("Wake-on-LAN script executed successfully!")
        else:
            await update.message.reply_text(f"Script failed with error:\n{result.stderr}")
    except Exception as e:
        await update.message.reply_text(f"An error occurred: {str(e)}")

def main():
    """Start the bot."""
    if not BOT_TOKEN:
        print("Error: BOT_TOKEN environment variable not set.")
        return
    if not ALLOWED_CHAT_ID:
        print("Error: ALLOWED_CHAT_ID environment variable not set.")
        return

    # Create the application instance
    application = Application.builder().token(BOT_TOKEN).build()

    # Add command handlers
    application.add_handler(CommandHandler("start", start))
    application.add_handler(CommandHandler("wol", wol_command))

    # Start the bot
    application.run_polling()

if __name__ == "__main__":
    main()

CTRL+O, ENTER, CTRL+X to save and exit.

How to use the Telegram Bot: Type /start and then /wol in the chat group created with your bot.

• /start: Greets the user and checks if they are authorized.

• /wol: Executes the wake_devices script

The Authorization Check ensures only messages from the allowed Chat ID trigger actions.

Make this python file executable.

sudo chmod +x /usr/local/bin/TG_WOL_BOT/WOL_bot.py
sudo chown -R $USER:$USER /usr/local/bin/TG_WOL_BOT

Create a systemd service file for the bot:

sudo nano /etc/systemd/system/WOL_bot.service

Add the following content:

[Unit]
Description=Telegram Bot for Wake-on-LAN
After=network.target

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/TG_WOL_BOT/WOL_bot.py
Restart=always
User=raspberrypi
#use your actual system user above
WorkingDirectory=/usr/local/bin/TG_WOL_BOT

[Install]
WantedBy=multi-user.target

CTRL+O, ENTER, CTRL+X to save and exit.

Load and run the WOL bot service.

sudo systemctl daemon-reload
sudo systemctl enable WOL_bot.service
sudo systemctl start WOL_bot.service
sudo systemctl status WOL_bot.service
sudo journalctl -fu WOL_bot.service

Use CTRL+C to exit the monitoring/logging view.