You are here

Packet storm mitigation patch

32 posts / 0 new
Last post
w6bi
w6bi's picture
Packet storm mitigation patch
Nightly build 571 hit the streets this morning. It contains the packet storm mitigation patch.
 
If you're in a medium or large network, or a highly meshed network, and your node passes traffic through itself (i.e., not a end-point), seriously consider using this build.

Orv W6BI
 
AE6XE
AE6XE's picture
All tower, tunnel nodes, or
All tower, tunnel nodes, or other prominent hub sites, should install the nightly build.   Darryl K5DLQ installed on the global tunnel server today.  Once the new image is installed, it is encouraged to tunnel connect to other global sites and ramp up the node count.   We do not known how far OSLR will scale up, and worth finding this limit.   A node that has the latest nightly build installed, will no longer forward bogus OLSR messages, and contribute to an OLSR storm event.

You may not always want to be joined to a global AREDN network, to ensure your emcomm support of 'local' incidences does not depend on the internet.  However, participating for a period of time to help find the upper limit, would be helpful.  

Tim KN6PLV, just rolled up the sleeves, dug in, figured out the root cause, and simply submitted the code to fix the OLSR storm issue.  This issue has been blocking the AREDN mesh network from scaling up.  Many thanks to Tim for fixing a major pain many have experienced and spent a lot of time investigating.

Joe AE6XE 
w6bi
w6bi's picture
Testing date?
SFWEM and SoCal will link together.  I'm going to suggest October 30th as the test date.  Seems appropriate. :-)
Maybe other groups can link up that same day.

Orv W6BI
KV3T
KV3T's picture
I have 8 nodes (Chicago) I
I have 8 nodes (Chicago) I can add to the party.  The 30th is as good a day as any for me.  I will need a tunnel log in to participate.
KM6IAU
KM6IAU's picture
Yucca Valley nodes

All my nodes are now on 571.  That includes:

There are a small handful of stations in Yucca Valley with equipment pointed at Paxton Hill.  They are running 3.20.3 or later.  None of them are tunnel servers or clients.

We're happy to join the party. laugh

w6bi
w6bi's picture
Patch progress
The SoCal mesh mapper, which detect > 350 nodes, reports that as of today 17.4% of nodes have been updated to 571.   In our fairly linear network, that probably is enough to prevent storms (we'll see).

In a highly-meshed network which provides a storm multiple paths to traverse, a higher percentage of updated nodes may be necessary to squelch a storm.
KD1HA
KD1HA's picture
Nightly build 571 and hAC Lite Tunnel
 I'm unable to reinstall the tunnel on the hAC Lite. Trying to be ready for the storm test! Had to drop back to the full release.  

Thanks 
Denis
K6CCC
K6CCC's picture
What error indications?  I
What error indications?  I installed tunnel on two hAPs with nightly 571 without issue.
 
KD1HA
KD1HA's picture
Firmware installed fine and
Firmware installed fine and just tried to load the "Click to Install "Tunnel software needs to be installed" and it showed a failure I don't remember the return. Yes I have the internet and I have done this many, many times before without an issue. Not sure why this happened. Before I try to reload the firmware again and shutdown my users I just thought I would check and see if anyone is having the same issue with the hap Lite. Maybe just a glitch?

Thanks,
Denis
 
k1ky
k1ky's picture
Fimware Nightly Tunnel installs
Looks like the latest Nightly is now 591 - only the latest nightly and Production versions point to the libraries for Tunnel modules.  I assume that Nightly 591 also includes the "Storm Patch" - so suggest updating to 591 and then try to install the Tunnel module.
k1ky
k1ky's picture
Upgrade procedure suggestion

Another helpful tip when upgrading tunnel server/client units: You don't need to "disable" your Client and Server connections before upgrade.  Simply "remove" the VTUN package, reboot, upgrade firmware, ** Install any other optional packages such as the 2 iPerf modules, reboot**, reinstall the VTUN module.  Once that is complete, all Tunnel Client and Server connections are restored.  Saves a lot of unnecessary clicking.

nc8q
nc8q's picture
How sweet it is.
Tom:
Thanks.

remove vtun
upgrade
refresh / download vtun
reboot

voila!

My hAP and loco-m-xw are running 599 via this procedure. :-)
OBTW, it seems the hAP needs 4 minutes, .vs. 3 minutes, after an upgrade prior to reloading the web page.

Chuck
 
KD1HA
KD1HA's picture
Thank you Tom and Chuck. 

I just tried the tunnel software for 599 and the same problem failed to load package. It does load the tunnel package after reloading the 3.20.3.0

Denis

K6CCC
K6CCC's picture
Strange, I updated two hAPs
Strange, I updated two hAPs to Nightly 599 an hour or two ago and both worked properly - including tunnel software installation.
 
KD1HA
KD1HA's picture
It is strange. Any idea?

It is strange. Any idea? Were you running 3.20.4.0 prier to upgrading?

K6CCC
K6CCC's picture
No.  Nightly 571
No.  Nightly 571
KD1HA
KD1HA's picture
I guess what I should be
I guess what I should be asking is did you start the Nightly Builds from 3.20.4.0?
K6CCC
K6CCC's picture
I assume so.
I assume so.  I pretty much put every production build and nightly into all 13 nodes.  So, yes, I would assume at some point, I have production 3.20.4.0
 
KD1HA
KD1HA's picture
Ok I will upgrade to 3.20.4.0
Ok I will upgrade to 3.20.4.0 then up to the latest 599 and see how that goes. Thank you.

Denis
k1ky
k1ky's picture
Latest "Production" is 3.21.4.0

Just for clarification - 3.21.4.0 is the latest "Production" version.  It contains all the latest security patches that have been released.  The "nightly's" may include new OLSR and OPENWRT releases along with other goodies that are being continually added before the next production version is released.
Last time I looked, the latest nightly is version 599 

2021-10-26 10:12 6.4M  
K5DLQ
K5DLQ's picture
Yes, nightly 599 dropped on
Yes, nightly 599 dropped on 10/26
KV3T
KV3T's picture
I would like to participate
I would like to participate this weekend with my 8 Chicago nodes. Orv emailed me asking for the id of the node to connect (which is KV3T-2G-M2) and i sent it back last week but he must not have seen my response yet. If someone can give me a tunnel connection I'm happy to participate. Not looking for a long term tunnel, just for this exercise. I'm good on qrz.
w6bi
w6bi's picture
Email...
Casey, I don't find any email from you.   Ping me again and we'll get you set up.
73
Orv W6BI
KV3T
KV3T's picture
Well then I'm glad I posted
Well then I'm glad I posted here.  I just sent the message again.  It is to the gmail address that came up when I hit reply to the message you sent me via the AREDN.  It matches your QRZ email address.  It is coming from my email address which can be found on QRZ.  I'm not sure if these forums are indexed by google so I'll not put email addresses inline.  I host my own email server on my own domain, so check your spam box.

If for some reason that doesn't come through again, the node is KV3T-2G-M2, and you can send me the credentials however you reached out last week.  That made its way to me somehow.  It had the subject "Re: [Amateur Radio Emergency Data Network]" so I assume it was via a pm feature on the forums here, but I could find such a feature.

thanks!!!
k1ky
k1ky's picture
Now it's Nightly 607 released 10/29
In the middle of upgrading my nodes and I noticed that Nightly 607 has been released.  Sure wish that could have waited until after Saturday! I won't need to upgrade my 599 nodes but will continue new upgrades with 607 so I can load IperfSpeed and VTUN.
KV3T
KV3T's picture
Orv, i have no idea why my
Orv, i have no idea why my emails aren't making it to you, but yours are making it to me. My eight nodes are up and connected. Thank you.
w6bi
w6bi's picture
Tunnel up
I saw that - enjoy!

73
Orv W6BI
KV3T
KV3T's picture
Oh boy.  There is definitely
Oh boy.  There is definitely something going on.  When connected to the tunnel, I am unable to reach all of my nodes, and none of my nodes that I can reach would load the mesh status page.  I disconnected from the tunnel, and connectivity was restored, but I cannot reach the mesh page on any given node until it is rebooted.  This is observed on a Rocket M5, Mikrotik LDF, LHG-XL-AP, an AR150, and a few others.  I'll pull the support data, let me know if someone wants it.  Some nodes I cannot get to until I disconnect the tunnel, but I can access and get the data before I reboot them.  When I'm done, I'll go ahead and reconnect so others can continue the test.
AE6XE
AE6XE's picture
Be sure to wait for mesh
Be sure to wait for mesh status, up to a minute to respond. Older devices will take longer needing more horse power to process the data, now much larger. This will impact performance too, and unrelated to network performance.
K6CCC
K6CCC's picture
What firmware version -
What firmware version - particularly the node running the tunnel?
KV3T
KV3T's picture
They were all up to date.  I
They were all up to date.  I jumped on the mattermost server with some others and we were able to determine that I had two devices that didn't have enough memory.  Only 32 MB.  And those devices were between me and the tunnel server.  So the tunnel and the larger network were ok, but my connection to it was not.  Based on my observations, with that number of clients (just over 1000) the device needs about 40 MB of RAM just to maintain itself and its connection to the network.  The devices less than that were randomly disconnecting, and would completely fail if you tried to load the mesh status page.  One of them was sitting at a load of over 4, just at idle, trying to keep up with all of the nodes.
w6bi
w6bi's picture
Observations
Casey, those observations are fairly close to what we saw.  Thanks for participating.  Report to follow!

73
Orv W6BI

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer