You are here

Degradation In 3 GHz performance with 5G roll out?

22 posts / 0 new
Last post
NM7B
Degradation In 3 GHz performance with 5G roll out?
The 3 GHz portion of our mesh in Hillsboro, OR seems to have some spotty degradation where I used to see 100% LQ and NLQ.  It seems to be assymmetric, like one node is having trouble hearing other nodes, but the peer nodes are receiving at 100% link quality, so I'm not convinced it just the annual tree growth.   It seemed to coincide with the deployment of 5G.  I have not dismissed that the node has degraded, and will swap in a spare identical node at some point.

Has anyone else observed this on their 3 GHz network?

73's,
Collier
NM7B
 
KE6GYD
KE6GYD's picture
Re: 3 GHz degradation
Have not see it here in Irvine, CA.  We have three 3GHz links about 4 to 7 miles in distance.  Our signals are 100% LQ and NLQ with 30+ SNR and a low of 39 Mbx on one link and 50+ Mbs on the other two.  All our links use Rocket M3's with Rocket dishes pointing to each other.  No sector antennas.73
Bob
KE6GYD

 
K5DLQ
K5DLQ's picture
what channels are you using?
what channels are you using?
KE6GYD
KE6GYD's picture
We are using channels 3400,
We are using channels 3400, 3410, 3430, and 3440
nc8q
nc8q's picture
Degradation In 3 GHz performance with 5G roll out?

Hi, Collier:

What channels/frequencies are you using?

73, Chuck

 

K5DLQ
K5DLQ's picture
Make sure that you are not
Make sure that you are not going out of band into 3450.   Depending on your channel width, you might be.
NM7B
Using the lowest channel
We did migrate out of the reallocated frequencies and decided to move to 3380 MHz as the return loss with the antennas being used is the highest on that end of the band, so we're well clear of the lost frequencies.  That's what doesn't seem to make sense: I would expect the new 5G service to be far enough away to not impact impact reception at this one node.   We'll see: I'm going to swap in a different Rocket/sector when we reconfigure this site.  

A separate path that is much shorter at about 3/4 mile also has seen degradation, but it's symmetrical.  That link is at 3400 MHz, and may be impacted by season tree growth in the path. 

Thank you, and 73's,
Collier
 
nc8q
nc8q's picture
Using LQ/NLQ as a metric

Hi, Collier:

I am thinking that LQ/NLQ is not a good metric to use to quantify 'degradation'.
SNR might be a better tool. Did you save SNR  values over time?
I am unclear what 'spotty' degradation is as compared to other types of degradation.
Since AREDN firmware automaticly seeks highest throughput, perhaps measureing the chosen MCS might be a good tool.
I am wondering if your 3 GHz links are point-to-point, point-to-multi-point, or (mesh) multipoint-to-multipoint.
Are the degraded nodes co-located on cellular towers?
Where might the 5G interference be from?
I have used a point-to-point link and tested on all the channels.
I don't know why, but some channels were better than others.
Have you tried other channels?

I hope this helps,
Chuck

NM7B
update
Hi, Chuck,

The SNR values haven't changed more than a few dB, with the most challenging path around 14 dB.  What I have noticed is that iperf runs have degraded too.  Instead of multiple samples with a small variation in throughput values, I'm getting 3 to 5 test samples that are 0 byte transferred.  I've also noticed some degradation in the VoIP telephony service.   

The main node that I monitor is in the middle of a square mile campus with the nearest cellsite about 3/4 mile away.  I don't know if that site supports 3 GHz, but there is an airport only a few blocks away, so they may not be using it yet. 

The system is currently configured as point to multipoint, with a wide variety of distances from a node on an adjacent building to a dish about 3.5 miles away.  

Trying different channels is a good idea.  I will give that a try after logging the current system readings.  I also will be scheduling some antenna work and will swap in a different but identical 3 GHz node to see if that changes anything.  

Thank you again for brainstorming my observations.
AB7PA
Have you tried using LQM

Have you tried using the new Link Quality Manager feature of the latest firmware?  If you want to use LQM you should probably load the latest Nightly Build on both ends of a link and it should self adjust to get the best throughput.  Do you have access to a spectrum analyzer?  Depending on the 5G implementation in your area, they definitely may be overlapping with your frequencies.  The FCC is sunsetting Amateur Radio use in the entire band, so this may be an opportunity for you to consider swapping equipment to use a different band.

NM7B
LQM enabled
I did update all of the nodes I admin to 3.22.6, and have enabled LQM.  I'll let it run for a while and see what it reports. 

I am in the process of phasing out our 3 GHz nodes as I saw the FCC's cautionary note that amateurs will have to vacate the remaining 3 GHz privileges probably on short notice.  The county EOCs that we connect to are currently on 3 GHz as well, so I need to maintain 3 GHz for that connection path at least in the near future.  The county EM has requested funding to add five 5 GHz AREDN nodes to some of their key facilities in the area.

Thank you,
Collier
 
NM7B
Can't QSY one of my nodes
I attempted to move my 3 GHz nodes up from 3380 MHz to 3410 MHz, but got stuck by this: https://www.arednmesh.org/content/rocket-m5hp-xw-hardware

 
nc8q
nc8q's picture
attempted to move my 3 GHz nodes
I have not seen this.
Nanostation M3 (XM x4).
Stables 2018 through 3.22.6.0.
Nightlies through 1449.
[HTTP_USER_AGENT] => Mozilla/5.0 (X11; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0


 
Image Attachments: 
NM7B
Rocket only
I have a mix of 3 GHz NanoStations and Rockets.  The NanoStations QSYed no issues.  It's the one Rocket that coughed up the Bad Gateway issue.  I even tried power-cycling the Rocket twice but that didn't resolve the issue.   
AB7PA
Post a support data dump

Can you post a support data dump of the offfending node right after it exhibits the "Bad Gateway" message?
You can always get a support dump using the URL directly:  http://your-nodename:8080/cgi-bin/supporttool

NM7B
Support dump attached
Per instructions, I reproduce the issue by selecting a new channel, then clicking on "Save Changes".  When the "Bad Gateway" message appeared, I immediately manually changed to the ./supporttool URL and saved the attached dump.

As a point of reference, I tried the same channel change and save on the other Rocket M3 I have, and it did not exhibit this "Bad Gateway" issue.  That Rocket is also running 3.22.6

Thank you for delving into this!
Collier
NM7B
 
AB7PA
Here is the error

Here is the error, and I'll pass it along to the coders.  THX.

Sat Jun 25 19:26:08 2022 daemon.err uhttpd[976]: ping: sendto: Network unreachable
Sat Jun 25 19:26:08 2022 daemon.err uhttpd[976]: /usr/bin/lua: /www/cgi-bin/setup:1271: attempt to concatenate global 'wifi2_channel' (a nil value)
Sat Jun 25 19:26:08 2022 daemon.err uhttpd[976]: stack traceback:
Sat Jun 25 19:26:08 2022 daemon.err uhttpd[976]: /www/cgi-bin/setup:1271: in main chunk
Sat Jun 25 19:26:08 2022 daemon.err uhttpd[976]: [C]: ?
This seems to indicate that a ping to/from your node is unable to connect to that IP subnet.
NM7B
Thank you!
Thank you for the quick parsing of the dump.  I'm kind of baffled why one Rocket M3 threw such a fit, while the other one worked perfectly.  I'll have to see if there's any hardware differences.  I purchased two in the first round, then the third at a later date.

73's,
Collier
NM7B
 
AB7PA
Check tomorrow's Nightly Build firmware for fix

An update was just merged and should appear in tomorrow's Nightly Build firmware that will hopefully fix your "Bad Gateway" message.  Let us know how it goes when you have a chance to install that NB firmware.  THX

NM7B
Missing Optional Settings
Thank you for the heads-up on the nightly build.  I'll give it a try.

Another anomalous behavior that might or might not be related: For the Rocket M3 that is throwing the Bad Gateway message, all of the Optional settings, specifically everything below the upper webpage screen where the channel, power output, DHCP parameters are set, everything below there is missing.  I'm not able to enter any location information because all of those entry fields are completely missing.  I checked the other Rocket M3 that doesn't throw the Bad Gateway message, and the location information setting fields are visible and available to be entered/modified.   I tried re-flashing 3.22.6 to see if that would alleviate the issue, but the missing fields persisted after the reflash.  

I was preparing the spare Rocket M3/sector antenna to be swapped in to see if that makes a difference for my asymmetric path performance, and was attempting to copy the location data when I realized that I couldn't find the Optional Settings field.  This antenna work will also involve adding a 5 GHz node which is the long-term replacement for 3 GHz.  Antenna work at this site is very involved, so physical access to the node is extremely limited.

Thank you,
Collier
NM7B
 
kj6dzb
kj6dzb's picture
Has anyone used airview on
Has anyone used airview on your local spectrum. I don't use the AREDN fw at this site. I know I have a very crowded rf inviorment! https://youtu.be/YsdjY-4IfGo https://youtu.be/RRMajsGiVt4 I'm a little limited by NSM3 its got a narrower usable bandwidth compared to the airfiber.
nc8q
nc8q's picture
local spectrum available in Nightly Build
Spectral view available in Nightly Build.
 
Image Attachments: 

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer