You are here

Home » Forums » General discussion

Local Software Server Issues

5 posts / 0 new
Log in or register to post comments
Last post
Mon, 09/01/2025 - 22:38
#1
W1LMS
Local Software Server Issues
Hello, 

Following the directions on Creating a Local AREDN® Software Server, I set one up on a Linux machine connected to of our nodes a week ago, and wrote a script for my team and I to keep them updated in each region. However, I quickly noticed that even though nightly builds were being provided, the software server was not showing them. Eventually, I discovered that the afs/misc/collect.py script that updates the JSON targets was also removing the new versions from afs/www/config.js that were just synced over. As a result, we are permanently stuck with offering whichever versions that were originally downloaded on the original sync. You can, however, manually navigate the tree to download the file you want, but I am certain the python script was not intended to make changes to the config.js. As a work around, I make a backup copy of the config.js before running the python script and restore it after the script completes. This has fixed (or band-aid) the issue for now. Deleting the entire root directory and starting fresh also works, wink.

Thanks for you help!
Tony
Top
Tue, 09/09/2025 - 13:54
W1LMS
Local Software Server Issues
I did not see an area on github that was for the software server, so I have only posted the issue here. Is there an area where to post issues that the AREDN team monitors/responds?

## Description
The `collect.py` script overwrites the `versions` object in `config.js` during firmware synchronization, breaking custom configurations for local firmware servers.

## Steps to Reproduce
1. Set up local firmware server using rsync to mirror AREDN downloads
2. Customize `config.js` with local `image_url` 
3. Run `collect.py` to update firmware database
4. Observe that `versions` object is rebuilt, losing new versions

## Expected Behavior
`collect.py` should preserve existing `config.js` content, and push the `image_url` setting to the JSON files.

## Actual Behavior
`collect.py` pushes the `image_url` through out all the models JSON files, but also breaks the `versions` object, overwriting the just added new versions. I will spend more time troubleshooting as soon as I have time, but guessing it is searching and modifying *.JS* files instead of *.JSON files. Not sure

## Environment
- AREDN firmware version: 3.25.8.0+

## Workaround
Currently using a post-sync script to restore custom `image_url` after `collect.py` runs by backing up and then restoring the `config.js`

 
Top
Wed, 09/10/2025 - 06:36
nc8q
nc8q's picture
My AREDN software server update script
Hi, Tony:

My ( self modified from AREDN published ) script seems to work well.
The 'stable' and 'latest nightly' appear in the dropdown box when updating firmware on a node.
It runs on a Raspberry Pi using a spinning HD for '/'.
Salt to taste.

73, Chuck

#!/bin/bash
date > 0
echo "With Spectrum cable this took 7 minites 33 seconds."
echo "With AltaFiber this takes 3 minutes 03 seconds."
echo "With babel_only, now 5 minutes 53 seconds."
echo "2025-07-14: Addded bandwidth limit, 2 MB/sec, time now 18 minutes, 37 seconds."
cd /var/www/html/
nice -n 19 /usr/bin/rsync -z --bwlimit=2m -rv --delete --size-only downloads.arednmesh.org::aredn_firmware /var/www/html/arednSoftware/
nice -n 19 wget -O /var/www/html/arednSoftware/afs/www/config.js http://downloads.arednmesh.org/afs/www/config.js
nice -n 19 cat /var/www/html/arednSoftware/afs/www/config.js
cd ~
cat 0
date
 
Top
Thu, 09/11/2025 - 02:30
W1LMS
My AREDN software server update script
That's very interesting.

I can clearly see collect.py modifying config.js. The new nightly versions that are listed in the config.js after the sync are removed when running collect.py. I am able to reproduce this 100% of the time. So I started making a backup of the config.js before running collect.py, then restoring on completion, a band-aid for now. Config.js appears in the list of JS* files collect.py modifies, last file in the list, though I cannot figure out why it needs to even touch the file. All commands are ran from the root sync folder. 

That script I created works around the issue, but it seems very odd that you have not had the same issue. I have tested it on 4 of our fanless PC's and am able to reproduced it. Perhaps I should send a screen recording.
Top
Fri, 09/19/2025 - 22:39
W4JJN
Another option...
Below is my script.

If you dig deeper, you will find that the overview.json image url's are not getting updated either. You can tell because the nodes will still pull from the main server and not the mirror.  That doesn't help me because some of my nodes are not able to reach the internet without turning on Mesh to WAN.

This one does the usual rsync, config.js replacement and backup, runs collect.py, restores the config.js as usual. Then, it overwrites all the image urls in the json files, reverts the hardcoded http to https forced upgrades back to http, and then writes a basic index.html page for the root directory.

#!/bin/bash
# Script to synchronize a local AREDN firmware mirror with the official repository.
# It uses rsync for synchronization, runs the collect.py script to update the index files,
# and corrects the URL within config.js and all overview.json files.

# --- Enable robust error handling ---
set -euo pipefail

# --- Configuration (to be updated by users) ---
readonly DEST="/var/www/html/"
readonly LOG_FILE="/var/log/sync_aredn.log"
readonly NEW_URL="http://w4jjn-x86-arednmirror.local.mesh/"
readonly CALLSIGN="W4JJN"
readonly WWW_USER="www-data"
readonly WWW_GROUP="www-data"

# --- Configuration (should be static unless AREDN makes changes) ---
readonly SOURCE="downloads.arednmesh.org::aredn_firmware"
readonly CONFIG_FILE="${DEST}afs/www/config.js"
readonly INDEX_FILE="${DEST}afs/www/index.js"
readonly OLD_URL="http://downloads.arednmesh.org/"
readonly DATA_PATH="${DEST}afs/www/data/"
readonly CONFIG_BACKUP="${DEST}afs/www/configbackup"

# --- Functions ---

# Function to write log messages.
log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

# Function for cleanup actions to ensure correct state even on failure.
cleanup() {
  log "Performing cleanup actions..."

  # Restore config.js from backup if it exists.
  if [[ -f "$CONFIG_BACKUP" ]]; then
    log "Restoring config.js from backup."
    mv "$CONFIG_BACKUP" "$CONFIG_FILE" >> "$LOG_FILE" 2>&1
  fi
  
  # Ensure file permissions are correct for the web server user.
  log "Fixing file and directory permissions."
  chown -R "${WWW_USER}":"${WWW_GROUP}" "${DEST}" >> "$LOG_FILE" 2>&1
  log "Permissions fixed."
  log "Synchronization and configuration completed."
  log "--------------------------------------------------"
}

# Main function to perform the synchronization.
main() {
  log "Starting script..."

  # --- rsync: Synchronize the mirror with the source. ---
  log "Starting rsync synchronization from $SOURCE to $DEST."
  /usr/bin/rsync -r --delete --size-only "$SOURCE" "$DEST" >> "$LOG_FILE" 2>&1
  log "rsync completed successfully."

  # --- Correct config.js before collect.py runs. ---
  log "Patching config.js with mirror URL."
  sed -i "s#$OLD_URL#$NEW_URL#g" "$CONFIG_FILE" >> "$LOG_FILE" 2>&1

  # --- collect.py workaround: Backup and restore config.js. ---
  log "Backing up patched config.js before running collect.py."
  cp -a "$CONFIG_FILE" "$CONFIG_BACKUP" >> "$LOG_FILE" 2>&1
  log "Running collect.py to generate index files."
  cd "${DEST}"
  chmod +x "${DEST}afs/misc/collect.py"
  "${DEST}afs/misc/collect.py" "${DEST}" "${DEST}afs/www/" >> "$LOG_FILE" 2>&1
  cd - > /dev/null # Return to the original directory.

  # --- Fix hardcoded URLs in index files. ---
  log "Fixing hardcoded links in index files."
  sed -i 's/https:/http:/g' "$INDEX_FILE" >> "$LOG_FILE" 2>&1

  # --- Force-fix overview.json files. ---
  log "Force-replacing URLs in all overview.json files."
  find "$DATA_PATH" -type f -name "overview.json" -exec sed -i "s|$OLD_URL|$NEW_URL|g" {} + >> "$LOG_FILE" 2>&1

  # --- Generate a custom index.html for the mirror. ---
  log "Generating custom index.html."
  cat > "${DEST}index.html" << EOF
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>${CALLSIGN} AREDN Software Mirror</title><style type="text/css" media="screen">body{background:#e7e7e7;font-family:Verdana,sans-serif;font-size:11pt;}#page{background:#fff;margin:50px;border:2px solid #c0c0c0;padding:10px;}#header{background:#4b6983;border:2px solid #7590ae;text-align:center;padding:10px;color:#fff;}#header h1{color:#fff;}#body{padding:10px;}span.tt{font-family:monospace;}span.bold{font-weight:700;}a:link{text-decoration:none;font-weight:700;color:#C00;background:#ffc;}a:visited{text-decoration:none;font-weight:700;color:#999;background:#ffc;}a:active{text-decoration:none;font-weight:700;color:#F00;background:#FC0;}a:hover{text-decoration:none;color:#C00;background:#FC0;}</style></head><body><div id="page"><div id="header"><h1>${CALLSIGN} AREDN Software Mirror</h1></div><div id="body"><h2>This server is a mirror of the AREDN official software server. It uses a script to perform regular synchronization to the official software repository so that mesh devices can update even without WAN access.</h2><ul><li>The sync runs every morning at 0100 ET and usually takes about 10 minutes to fully sync and reconfigure. Suggest running your updates either at midnight or after 0200ET</li><li>To manually search and download software, please use the search box at <a href="${NEW_URL}afs/www/">${NEW_URL}afs/www/</a></li><li>To use:<br>1. Go to your node config page and click on the current firmware name. This will open a pop-up called "Firmware".<br>2. Click on the "Advanced Options" sub-header near the bottom.<br>3. Copy and paste the URL <b>${NEW_URL}</b> into the "Firmware URL" text box. 4. Click on Done, and then Commit the changes.<br>5. Re-open the Firmware pop-up by clicking on the current firmware version.<br>6. You should now be able to use the refresh button to the right of the "Download Firmware" selector to refresh the available list and choose a firmware to install</li></ul></div></div></body></html>
EOF
}

# --- Script Execution ---
# Set up a trap to call the cleanup function upon script exit.
trap cleanup EXIT

# Call the main function to run the script logic.
main

Top

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer