Packet storm mitigation patch

32 posts / 0 new

Sat, 10/09/2021 - 20:29

w6bi

Nightly build 571 hit the streets this morning. It contains the packet storm mitigation patch.

If you're in a medium or large network, or a highly meshed network, and your node passes traffic through itself (i.e., not a end-point), seriously consider using this build.

Orv W6BI

Thu, 10/14/2021 - 21:56

AE6XE

All tower, tunnel nodes, or

All tower, tunnel nodes, or other prominent hub sites, should install the nightly build. Darryl K5DLQ installed on the global tunnel server today. Once the new image is installed, it is encouraged to tunnel connect to other global sites and ramp up the node count. We do not known how far OSLR will scale up, and worth finding this limit. A node that has the latest nightly build installed, will no longer forward bogus OLSR messages, and contribute to an OLSR storm event.

You may not always want to be joined to a global AREDN network, to ensure your emcomm support of 'local' incidences does not depend on the internet. However, participating for a period of time to help find the upper limit, would be helpful.

Tim KN6PLV, just rolled up the sleeves, dug in, figured out the root cause, and simply submitted the code to fix the OLSR storm issue. This issue has been blocking the AREDN mesh network from scaling up. Many thanks to Tim for fixing a major pain many have experienced and spent a lot of time investigating.

Joe AE6XE

Sat, 10/16/2021 - 21:17

w6bi

Testing date?

SFWEM and SoCal will link together. I'm going to suggest October 30th as the test date. Seems appropriate. :-)
Maybe other groups can link up that same day.

Orv W6BI

Mon, 10/18/2021 - 13:38

KV3T

I have 8 nodes (Chicago) I

I have 8 nodes (Chicago) I can add to the party. The 30th is as good a day as any for me. I will need a tunnel log in to participate.

Thu, 10/21/2021 - 04:01

KM6IAU

Yucca Valley nodes

All my nodes are now on 571. That includes:

KM6IAU-Onaga-Node2: (Mikrotik LHG XL HP5) link to Snow Peak, internet gateway
KM6IAU-Onaga-Node1: (Mikrotik LHG XL HP5) link to Paxton Hill in Yucca Valley, tunnel server, internet gateway
KM6IAU-PaxtonHill-Node0: (Ubiquiti RM5) RF-disabled atop Paxton Hill
KM6IAU-PaxtonHill-Node1: (Ubiquiti PBE-M5-300) North-facing atop Paxton Hill
KM6IAU-PaxtonHill-Node2: (Ubiquiti PBE-M5-300) West-facing PBE300 atop Paxton Hill

There are a small handful of stations in Yucca Valley with equipment pointed at Paxton Hill. They are running 3.20.3 or later. None of them are tunnel servers or clients.

We're happy to join the party.

Thu, 10/21/2021 - 13:21

w6bi

Patch progress

The SoCal mesh mapper, which detect > 350 nodes, reports that as of today 17.4% of nodes have been updated to 571. In our fairly linear network, that probably is enough to prevent storms (we'll see).

In a highly-meshed network which provides a storm multiple paths to traverse, a higher percentage of updated nodes may be necessary to squelch a storm.

Mon, 10/25/2021 - 17:55

KD1HA

Nightly build 571 and hAC Lite Tunnel

I'm unable to reinstall the tunnel on the hAC Lite. Trying to be ready for the storm test! Had to drop back to the full release.

Thanks
Denis

Mon, 10/25/2021 - 18:29

(Reply to #7) #8

K6CCC

What error indications? I

What error indications? I installed tunnel on two hAPs with nightly 571 without issue.

Mon, 10/25/2021 - 20:13

KD1HA

Firmware installed fine and

Firmware installed fine and just tried to load the "Click to Install "Tunnel software needs to be installed" and it showed a failure I don't remember the return. Yes I have the internet and I have done this many, many times before without an issue. Not sure why this happened. Before I try to reload the firmware again and shutdown my users I just thought I would check and see if anyone is having the same issue with the hap Lite. Maybe just a glitch?

Thanks,
Denis

Mon, 10/25/2021 - 20:59

#10

k1ky

Fimware Nightly Tunnel installs

Looks like the latest Nightly is now 591 - only the latest nightly and Production versions point to the libraries for Tunnel modules. I assume that Nightly 591 also includes the "Storm Patch" - so suggest updating to 591 and then try to install the Tunnel module.

Mon, 10/25/2021 - 21:32

#11

k1ky

Upgrade procedure suggestion

Another helpful tip when upgrading tunnel server/client units: You don't need to "disable" your Client and Server connections before upgrade. Simply "remove" the VTUN package, reboot, upgrade firmware, ** Install any other optional packages such as the 2 iPerf modules, reboot**, reinstall the VTUN module. Once that is complete, all Tunnel Client and Server connections are restored. Saves a lot of unnecessary clicking.

Tue, 10/26/2021 - 15:10

(Reply to #11) #12

nc8q

How sweet it is.

Tom:
Thanks.

remove vtun
upgrade
refresh / download vtun
reboot

voila!

My hAP and loco-m-xw are running 599 via this procedure. :-)
OBTW, it seems the hAP needs 4 minutes, .vs. 3 minutes, after an upgrade prior to reloading the web page.

Chuck

Tue, 10/26/2021 - 16:09

#13

KD1HA

Thank you Tom and Chuck.

I just tried the tunnel software for 599 and the same problem failed to load package. It does load the tunnel package after reloading the 3.20.3.0

Denis

Tue, 10/26/2021 - 16:43

(Reply to #13) #14

K6CCC

Strange, I updated two hAPs

Strange, I updated two hAPs to Nightly 599 an hour or two ago and both worked properly - including tunnel software installation.

Tue, 10/26/2021 - 16:47

(Reply to #14) #15

KD1HA

It is strange. Any idea?

It is strange. Any idea? Were you running 3.20.4.0 prier to upgrading?

Tue, 10/26/2021 - 16:58

(Reply to #15) #16

K6CCC

No. Nightly 571

Tue, 10/26/2021 - 17:01

(Reply to #16) #17

KD1HA

I guess what I should be

I guess what I should be asking is did you start the Nightly Builds from 3.20.4.0?

Tue, 10/26/2021 - 17:09

#18

K6CCC

I assume so.

I assume so. I pretty much put every production build and nightly into all 13 nodes. So, yes, I would assume at some point, I have production 3.20.4.0

Tue, 10/26/2021 - 20:13

#19

KD1HA

Ok I will upgrade to 3.20.4.0

Ok I will upgrade to 3.20.4.0 then up to the latest 599 and see how that goes. Thank you.

Denis

Tue, 10/26/2021 - 20:23

#20

k1ky

Latest "Production" is 3.21.4.0

Just for clarification - 3.21.4.0 is the latest "Production" version. It contains all the latest security patches that have been released. The "nightly's" may include new OLSR and OPENWRT releases along with other goodies that are being continually added before the next production version is released.
Last time I looked, the latest nightly is version 599

2021-10-26 10:12

6.4M

Wed, 10/27/2021 - 10:37

#21

K5DLQ

Yes, nightly 599 dropped on

Yes, nightly 599 dropped on 10/26

Thu, 10/28/2021 - 09:17

#22

KV3T

I would like to participate

I would like to participate this weekend with my 8 Chicago nodes. Orv emailed me asking for the id of the node to connect (which is KV3T-2G-M2) and i sent it back last week but he must not have seen my response yet. If someone can give me a tunnel connection I'm happy to participate. Not looking for a long term tunnel, just for this exercise. I'm good on qrz.

Thu, 10/28/2021 - 11:34

(Reply to #22) #23

w6bi

Email...

Casey, I don't find any email from you. Ping me again and we'll get you set up.
73
Orv W6BI

Thu, 10/28/2021 - 17:43

#24

KV3T

Well then I'm glad I posted

Well then I'm glad I posted here. I just sent the message again. It is to the gmail address that came up when I hit reply to the message you sent me via the AREDN. It matches your QRZ email address. It is coming from my email address which can be found on QRZ. I'm not sure if these forums are indexed by google so I'll not put email addresses inline. I host my own email server on my own domain, so check your spam box.

If for some reason that doesn't come through again, the node is KV3T-2G-M2, and you can send me the credentials however you reached out last week. That made its way to me somehow. It had the subject "Re: [Amateur Radio Emergency Data Network]" so I assume it was via a pm feature on the forums here, but I could find such a feature.

thanks!!!

Fri, 10/29/2021 - 12:30

#25

k1ky

Now it's Nightly 607 released 10/29

In the middle of upgrading my nodes and I noticed that Nightly 607 has been released. Sure wish that could have waited until after Saturday! I won't need to upgrade my 599 nodes but will continue new upgrades with 607 so I can load IperfSpeed and VTUN.

Sat, 10/30/2021 - 11:44

(Reply to #25) #26

KV3T

Orv, i have no idea why my

Orv, i have no idea why my emails aren't making it to you, but yours are making it to me. My eight nodes are up and connected. Thank you.

Sat, 10/30/2021 - 12:45

(Reply to #26) #27

w6bi

Tunnel up

I saw that - enjoy!

73
Orv W6BI

Sat, 10/30/2021 - 14:19

#28

KV3T

Oh boy. There is definitely

Oh boy. There is definitely something going on. When connected to the tunnel, I am unable to reach all of my nodes, and none of my nodes that I can reach would load the mesh status page. I disconnected from the tunnel, and connectivity was restored, but I cannot reach the mesh page on any given node until it is rebooted. This is observed on a Rocket M5, Mikrotik LDF, LHG-XL-AP, an AR150, and a few others. I'll pull the support data, let me know if someone wants it. Some nodes I cannot get to until I disconnect the tunnel, but I can access and get the data before I reboot them. When I'm done, I'll go ahead and reconnect so others can continue the test.

Sat, 10/30/2021 - 15:28

(Reply to #28) #29

AE6XE

Be sure to wait for mesh

Be sure to wait for mesh status, up to a minute to respond. Older devices will take longer needing more horse power to process the data, now much larger. This will impact performance too, and unrelated to network performance.

Sat, 10/30/2021 - 14:47

#30

K6CCC

What firmware version -

What firmware version - particularly the node running the tunnel?

Sat, 10/30/2021 - 17:57

#31

KV3T

They were all up to date. I

They were all up to date. I jumped on the mattermost server with some others and we were able to determine that I had two devices that didn't have enough memory. Only 32 MB. And those devices were between me and the tunnel server. So the tunnel and the larger network were ok, but my connection to it was not. Based on my observations, with that number of clients (just over 1000) the device needs about 40 MB of RAM just to maintain itself and its connection to the network. The devices less than that were randomly disconnecting, and would completely fail if you tried to load the mesh status page. One of them was sitting at a load of over 4, just at idle, trying to keep up with all of the nodes.

Sat, 10/30/2021 - 19:33

(Reply to #31) #32

w6bi

Observations

Casey, those observations are fairly close to what we saw. Thanks for participating. Report to follow!

73
Orv W6BI

Amateur Radio Emergency Data Network

Menu SF

You are here

Packet storm mitigation patch

Amateur Radio Emergency Data Network

Search form

Menu SF

You are here

Packet storm mitigation patch