A fail-safe for changing network config on headless servers

Lenovo Tiny PC

I’ve been setting up a Lenvo tiny system, which came with an M.2 wifi card. I’m going to probably replace it with another Intel GB / 2.5 GB adapter at some point, but right now I don’t really have a reason to. It is always a little scary when you make major changes to your network configuration on a headless system. This box isn’t physically far away, but it is indeed tucked away in another part of the house and I have no desire to bring a monitor over to it.

I realized, the wifi could serve a purpose in case I bork the bridge config somehow, or something unexpected breaks my configuration. Because I used wifi for the Debian netinstall, ifupdown already connects to wifi on boot, so there’s my failsafe. However, I don’t want or need that to stay up after the wired network is up. So I came up with this:

@reboot sleep 60 && ping -c 3 -I br0 10.0.0.1 >/dev/null 2>&1 && ifdown wlp2s0

Put this in root’s crontab, 60 seconds after cron starts it will try to ping something on the LAN, trying three times. If it can ping successfully, we bring down our backup interface. Beautiful.

This would make even more sense with a cheap USB network adapter just to have a failsafe if you’re experimenting and don’t want to lock yourself out of a system without a monitor (or even a physical serial port, for that matter).

Fail-safe, for what? – long version, for those so-inclined

Well, I wanted to be able to easily network some VMs on this thing, so I set up bridged networking. For those who don’t know, this is how Proxmox lets your VMs basically all share a NIC and each get a DHCP lease from your main network, no double-NAT nonsense, and no need to configure static routes either. I briefly did try ProxMox on this thing, and not to talk down of it but there is a reason I don’t use it… has way too much going on out of the box that I’ll never use. It is a great product, and it definitely has its place but I like the minimal-manual style of setting things up. If I wanted to take advantage of the cluster / high availability features or ZFS snapshots then it is a great way to save you a ton of hassle manually setting up some pretty complicated stuff… But I’m not doing anything that fancy. Proxmox IMO is overkill if you just need to run a few VMs, jails or containers. It is convenient as a “poor man’s KVM” though.

XScreenSaver Install Script for Debian MATE Desktop

XScreenSaver

MAKING IT BE (a little) LESS PAIN IN THE ASS

If you’re using Gnome/MATE screensaver (or locker), I’d recommend you don’t. Weather it is because you value X locking security, or just admit it they’re cool screensavers, toss that in the bin and install the real deal.. XScreenSaver.

I have written a script which takes care of all of the following:
1. On Debian, XScreenSaver needs you to manually specify 4 packages to get the program, “extra” savers, gl savers, and “gl extra” savers. This will get you the complete collection, as the author distributes it.
2. Remove MATE Screensaver
3. Apt pin / pref so MATE Screensaver isn’t reinstalled next time you run apt, because MATE Desktop thinks it needs it
4. Autostarts XScreenSaver
5. makes a soft link so you can still lock from the “System” menu up top, and click “Lock Screen”. Now XS will do it.

This will save me HOURS over time.

Get it here: http://lostgeek.net/files/xscreensaver-mate.sh

The PiFrame — Pi Zero 2 LCD Weather Clock


The
          PiFrame

   Raspberry Pi Zero 2 WH — $18
I2C 20×4 LCD Display — $5
Shadowbox Frame — $7

Doing a geeky project for under $30?? Priceless…

Ah, the Raspberry Pi. That $35 single board computer everyone was scalping for 3x what they were worth during the chip shortages. Well, I used to own several of them… and unfortunately no longer do. I will say, for the MSRP price they aren’t a bad option. The whole ecosystem is quite attractive for many reasons, and the brand receives praise left and right for it. I will indeed say, they’re basically swiss army knives for a hacker. A whole miniature linux system, with a quad core 64 bit CPU and often 1 – 4 GB of RAM. IMO the 8 GB is a waste of money, of course, I tend to like lean configurations so perhaps I just feel that way because I’d never use 4 GB on a Pi let alone 8. AND, if I did need 8 GB or more, I’d use a DDR4 mini PC, not a Pi!

Anywho, in the spirit of what the Pi is all about, I wanted something cheap to hack on. I have a Pi 5, but it pulls full time duty as a server. And, what can I say? It works so well for this, and the small size and lower power requirements are part of that attraction for me. Now, PCIe gigabit ethernet, and PCIe NVME storage are a pretty strong motivation for my willingness to keep the Pi 5 4 GB I’ve got employed as a server. Without those, I’d use a thin client or old laptop in a heartbeat. Oh yeah, the spirit of the Pi, that’s where I started blabbing right?

So the Pi Zero, it’s like an original 2012 Pi, but with optional Wifi. You loose onboard ethernet (but it was USB anyway on the early models, and you do have a USB port to add a NIC…) but you get a very small package still boasting full 40 pin GPIO. They refreshed the Pi Zero in late 2021 with the Pi Zero 2. If you want WiFi and BT, you want the Zero 2 W. Want pre-soldered GPIO pins too? Get the WH.

** NOW a little PSA here, I bought a Pi Zero 2 WH on Amazon… so that came /w a soldered GPIO pin header. Quite handy, even has color coded spots at the base of each pin so you know what is GPIO, 5v, Ground, etc… Except, mine was put on upside down. Took me forever to figure this out, and I would have been pretty pissed if I needed to RMA it because some shoddy reseller is doing these headders themselves to save 30 cents and mislabeling the pins. I don’t care now that I know, but being largely for the education market this is a bit discouraging to see. If I were in the same situation as a young kid, the Pi may very well have gone in the bin.

You can get a pack of two 20 character / column x 4 row LCD screens, with pre-soldered i2c “backpack” for about ten bucks. And, you can get it in green, red, blue, whatever you want. I went with the OG, green LCD.

Let there
        be light!

So… what does it do? Well, it’s an excuse to have another Linux box in your fleet, I mean, what more do you want?? But since you asked, it does anything you tell it to. Right now, mine spends five seconds showing me the date, time, and my web server uptime. Then it shows me local weather for another five seconds. There’s more in the pipe though, and trying out new code is incredibly easy.

LCD
        Display LCD Display

What makes this clock… tick?? Python.

#!/usr/bin/env python

import drivers
from time import sleep, strftime
import argparse
import requests
import subprocess

def get_uptime():
    try:
        # Run the 'uptime -p' command and capture the output
        #result = subprocess.run(['uptime', '-p'], capture_output=True, text=True, check=True)
        result = subprocess.run(['cat', '/tmp/uptime'], capture_output=True, text=True, check=True)
        uptime_str = result.stdout.strip()  # E.g., "up 1 day, 1 hour, 45 minutes"
        
##        # Use awk to format it as "up 1d 1h 45m"
##        formatted_uptime = subprocess.run(
##           ['awk', '{print "WWW up ", $2 " weeks", $4 "d", $6 "h"}'], input=uptime_str, text=True, capture_output=True
##        ).stdout.strip()

## The above works, when you've had < 7 days up... then we need the following... (and yes, I could have made this MUCH more elegant)

        # Use awk to format and convert weeks into days, then calculate total days
        formatted_uptime = subprocess.run(
            ['awk', '{week_days=($2*7); total_days=week_days+$4; print "HTTPD.lan up", total_days "d", $6 "h"}'], 
            input=uptime_str, text=True, capture_output=True
        ).stdout.strip()
        return formatted_uptime

    except subprocess.CalledProcessError as e:
        print(f"Error getting uptime: {e}")
        return "Uptime not available"

# Load the driver
lcd = drivers.Lcd()

# Weather API settings
API_KEY = "000000000000000000000" ## The API keys are free, just sign up. Painless or I wouldn't have bothered.
ZIP_CODE = "00000" ## Your Zip code here!
COUNTRY_CODE = "US"
WEATHER_URL = f"http://api.openweathermap.org/data/2.5/weather?zip={ZIP_CODE},{COUNTRY_CODE}&appid={API_KEY}&units=imperial"

# Function to fetch weather data
def get_weather():
    try:
        response = requests.get(WEATHER_URL)
        data = response.json()
        if data and data["cod"] == 200:
            temp = round(data["main"]["temp"])
            humidity = data["main"]["humidity"]
            wind_speed = round(data["wind"]["speed"])
            wind_dir = data["wind"].get("deg", "N/A")
            return temp, humidity, wind_speed, wind_dir
    except Exception as e:
        print("Error fetching weather:", e)
    return None, None, None, None

# Parse command-line arguments
parser = argparse.ArgumentParser(description="LCD Display Script")
parser.add_argument("--wc", action="store_true", help="Only display weather and clock pages in rotation")
args = parser.parse_args()

try:
    while True:
        # Date/Time page
        lcd.lcd_clear()
        lcd.lcd_display_string(strftime("Today is %A,"), 1)
        lcd.lcd_display_string(strftime("     %B %d"), 2)

        # Display uptime on the 4th row
        uptime = get_uptime()  # Call the function and store the uptime
        lcd.lcd_display_string(f"{uptime}", 4)

        # Continuously update the time (third row)
        for _ in range(10):  # Display for ~10 seconds
            lcd.lcd_display_string(strftime("     %I:%M:%S %p"), 3)
            sleep(1)

        # Weather page
        if args.wc:  # Include weather in both modes (if --wc is passed)
            temp, humidity, wind_speed, wind_dir = get_weather()
            if temp is not None:
                lcd.lcd_clear()
                lcd.lcd_display_string("    Boscawen, NH    ", 1)
                lcd.lcd_display_string(f"    Temp: {temp}F   ", 2)
                lcd.lcd_display_string(f"   {humidity}% Humidity", 3)
                lcd.lcd_display_string(f"  Wind: {wind_speed}mph", 4)
                sleep(5)

except KeyboardInterrupt:
    print(" ~ Clearing ~ ")
    lcd.lcd_clear()

Now, I’m not really much of a programmer. Nope. But, ugly or not there it is. I suggest you do what I did, and start here: The Raspberry Pi Guy has a page with sample code and some other helpful stuff on Github. Using the 16×2 code on a 20×4 is as easy as changing 16 to 20 and 2 to 4. Well, gotta add lines 3 and 4 below 1 and 2. But not rocket surgery.

I recommend using the overlay FS and read only /boot partition if you do something like this to avoid accidental SD card filesystem corruption from unsafe shutdowns. I actually added a systemd service so that on target of reboot, halt or shutdown a shell script will kill the python process, then launch another which blanks the screen and replaces the text with “IT IS NOW SAFE TO TURN OFF YOUR COMPUTER” — if you know, you know. About 1 second after that hits the LCD, the Pi powers off and the Act LED goes dark. The LCD will stay lit, and retain the last thing printed on it as long as power is connected.

Now, the BEST thing to do for your filesystem / SD card is to power off via SSH before unplugging any Pi. However, to power my “clock” up, all I do is plug it in. If you put in your crontab a line starting with @reboot, you’ll be able to easily start scripts at boot. I did this as root, because I think you need to be root to use the GPIO. Probably a way around this, but this runs nothing other than the display stuff at the moment.

Cron on the Pi Zero 2 W. aka PiFrame:
@reboot /root/lcd/bens3.py –wc
@reboot curl -s https://ben.lostgeek.net/uptime.txt -o /tmp/uptime
0 * * * * curl -s https://ben.lostgeek.net/uptime.txt -o /tmp/uptime

What this does is at boot, we pull uptime from a text file on my webserver and we start up the python program with the –wc arg, “weather clock”. This applies to the code above, so I left it as is. Only one more part is needed.

Cron on the server:
0 * * * * uptime -p > /var/www/html/ben/uptime.txt

This puts an up to date uptime file in my web directory once an hour. And the keen observers among us probably noticed that the Zero also will refresh this information at the top of each hour too. Easy peasy.

Raspberry Pi 5 PCIe Gen 3 Follow-Up

Good News,

Perhaps you’ve read my last article related to the Raspberry Pi 5, where I explore what kind of NVMe SSD experience one can get for a mere $25 or less spent on Amazon. And, the TLDR is that for under $25 I was able to purchase a brand new PCI-E M.2 HAT for the Pi 5, as well as a name brand Gen 3 128 GB SSD. The SK Hynix SSD was not brand new, it was pulled from something but SMART showed that it had less than 300GB written to it, so IMO absolutely worth buying.

So in that write up, I discuss the PCI link speed situation… basically, the Pi is only certified for Gen 2… but Gen 3 offers nearly double the speed! Immediately, I was conflicted. Even at Gen 2 speed, this already beats using SD card storage by a MILE. Way more reliable, MUCH faster, and much larger. There is no down side, other than costing a small bit more. Since this runs my server, I could have happily stuck with Gen 2 and still gotten more than my money’s worth in terms of bang-for-buck computing goodness out of the upgrade. However, many folks on the net reported no issues whatsoever running at 3.0 link speed, so I figured I’d try it, run several benchmarks, use it for several days. If I saw even one error, minor or otherwise, Gen 3 would be officially off the table.

Now, I’d be willing to bet that what I did next was not entirely necessary. But basically, there are some files you can read back from a system directory in linux which will tell you if you’re having NVME errors. Non-volatile memory advanced error reporting, or NVMAER for short. I go into more detail in the original article.

I run many scripts via cron on this machine. The ease of scripting as a sysadmin’s tool is a large part of what makes *nix such an excellent platform for power users. So… I wrote up a script, which runs once per hour. It checks for any of three types of NVMAER errors; when it does this, it appends a time stamped line to a log file stating wether or not it found errors. If It does find any, it will say which kind of errors they were, and I’ll know WHEN they happened.

Now, I’m sure what I am doing here is entirely redundant. I’d be very surprised if the system logs don’t already store all this and more, but it doesn’t tax this system to run a small script once per hour, and it gave me more confidence that if anything wonky did happen at Gen 3 speed that I wouldn’t possibly miss it! The last thing I wanted was to have minor errors which weren’t obvious, then down the road I end up with corrupt files.

I figured I’d share that script, because it came out pretty clean. And, it is a really good example of how one can go about building the tool, which solves a given problem. The multiple error types which are possibly made it just interesting enough, in my opinion. AND, even if this may be entirely redundant, just the ease of use made it worth it for me. So maybe you’ll find it of use yourself if you’re also self-validating PCIE 3 speeds on your Pi 5 SSD.

#!/bin/bash

LOGFILE="/var/log/cronlog/nvmaer.log"   # Log file. You'll want to change this! The directory must already exist.

# Define color codes
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Current date
DATE=$(date)
LOG_DATE=$(date +"%m/%d/%Y %H:%M")
echo "CHECK DATE: $DATE"

# Initialize variables
exit_code=0
log_entry=""

# Function to check errors
check_errors() {
    local error_type="$1"
    local file_path="$2"
    
    echo "Checking for $error_type errors..."
    
    # Capture the output of the command
    errors=$(awk '$2 != 0' "$file_path")
    
    # Check if errors is empty
    if [ -z "$errors" ]; then
        # No errors found
        echo -e "${GREEN}PASSED${NC}"
        log_entry+=" $error_type PASSED"
    else
        # Errors detected
        echo -e "${YELLOW}ERRORS DETECTED!${NC}"
        echo "$errors"
        log_entry+=" $error_type ERRORS DETECTED!"
        log_entry+=$'\n'"$errors"
        # Set the exit code to indicate problems
        exit_code=1
    fi
    echo "- - - - - - - - - - - - - - - - - - - -"
}

# Check for FATAL errors
check_errors "FATAL" "/sys/devices/platform/axi/1000110000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/aer_dev_fatal"

# Check for NONFATAL errors
check_errors "NONFATAL" "/sys/devices/platform/axi/1000110000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/aer_dev_nonfatal"

# Check for CORRECTABLE errors
check_errors "CORRECTABLE" "/sys/devices/platform/axi/1000110000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/aer_dev_correctable"

# Log results
if [ $exit_code -eq 0 ]; then
    # No errors detected
    echo -e "CHECK $LOG_DATE -- NO ERRORS, ${GREEN}[CHECK PASSED]${NC}" >> "$LOGFILE"
else
    # Errors detected
    echo "$log_entry" >> "$LOGFILE"
fi

# Exit with the appropriate exit code
exit $exit_code

Original article: https://lostgeek.net/pi5nvme

© 2025 LostGeek.NET - All Rights Reserved. Powered by ClassicPress, NGINX, Debian GNU/Linux.