You are here

OTA update robustness

7 posts / 0 new
Last post
k7dmk
OTA update robustness
I'm  curious how the ota update mechanism is actually implemented. Is there any error checking like a checksum performed on the uploaded firmware before it's flashed?  We have several remote nodes that are a long drive away, but are tough to replace if they get bricked. Does an error in the process fail gracefully, or catastrophically with a bad FW load?  Anyone have any success/fail tails to tell?

We are running 3.16.1.1 on most of the nodes to be updated and typically have ETX values of less than 3.

Thanks, Dave K7DMK


 
K5DLQ
K5DLQ's picture
Hi Dave,
Hi Dave,
A few things:
  • we are using the native (plus a few extra steps) sysupgrade process from OpenWRT.
  • The latest nightly builds do contain a check for the filename to ensure that it's appropriate for the device. (3.16.1.x and 3.18.9.0 do not have these checks)
  • You should reboot immediately before doing an upgrade.
  • Most of the time, when upgrades have failed on me, they fail because of lack of memory (thus, the prior reboot recommendation).  In these cases, the node will stay at the same version after the reboot. (soft fail)
  • I would recommend uninstalling any packages that you may have manually installed on the node first.  (ie.  MeshChat)
  • Next, I HIGHLY RECOMMEND that you UPGRADE to v3.18.9.0 (or at least v3.16.1.2) IMMEDIATELY due to security vulnerability fixes. 
Here's the release notes for v3.18.9.0:   https://www.arednmesh.org/content/aredn-release-notes-v31890
Here's the changelog of features/bugfixes since v3.18.9.0 that are in the nightly builds:  http://downloads.arednmesh.org/snapshots/trunk/CHANGELOG.md
 
K6AH
K6AH's picture
...shut down any tunnels and reboot the nodes
Procedurally, shut down any tunnels and reboot the nodes before performing the upgrade and you won;'t have any troubles.  You aren't unique in having remote nodes... all of us have them to one degree or another and have been doing firmware upgrades like this routinely.

Andre, K6AH
 
k7dmk
OTA Upgrade
Darryl, and Andre: thanks for the explanation, tips and reassuring experience concerning the ota fw upgrade. I'm embolden to proceed now.   I don't mind driving, but I'm trying to avoid adding to my small Ubiquiti paperweight collection.

Thanks again, Dave

 
AE6XE
AE6XE's picture
Barring a hardware failure, I
Barring a hardware failure, I've not meet a ubiquiti I couldn't get re-flashed.   But I have traveled to tower sites once or twice due to cockpit error :) .
k1ky
k1ky's picture
OTA Upgrade
Ditto all of the above.  I have around 98% success rate with OTA upgrades on over 400+ and only one (1) that had to be brought down off the tower and swapped out due to my loading the "wrong" firmware - hence the new firmware validation feature.  The procedures outlined above are a must, especially the remove modules and reboot before upgrading.  If you are on 3.16.1.1, suggest "stepping up" slowly at least to a production version, either 3.16.1.2 or 3.18 before moving on to any of the nightlies.  I had a pair of nodes high up on a hardened site tower that had been up for 386 days since last upgrade.  Talk about a pucker factor on those upgrades!  Thankfully they went without a hitch!
k1ky
k1ky's picture
OTA Upgrade II
And don't forget to check for STATIC WAN on the older Firmware loads 3.16.1.1 and probably Pre 3.18 and switch them back to DHCP on the WAN before performing upgrades.

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer