You are here

Timing Variability and VoIP quality

14 posts / 0 new
Last post
w8erd
Timing Variability and VoIP quality
We have trouble with VOIP voice quality.  Sometimes it is perfect for a few seconds and then is very choppy, making it generally unusable.
Tracert show significant timing variability among the nodes. There is essentially no other traffic flowing on the nodes.  What could be causing
this?  See these tracert:
 
Here are 4 samplings. Only the first one showed a long delay from your node to your phone.
 
To your phone:
 
Tracing route to W8ERD-Telephone.local.mesh [10.176.63.206]
over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  localnode.local.mesh [10.82.37.145]
  2   241 ms    11 ms    12 ms  N8DCA-LewisCenter.local.mesh [10.34.42.155]
  3    59 ms    86 ms     7 ms  N8DCA-Ostrander.local.mesh [10.36.23.218]
  4   151 ms     7 ms    15 ms  W8ERD-Tower.local.mesh [10.150.7.249]
  5   854 ms   285 ms   185 ms  W8ERD-Telephone.local.mesh [10.176.63.206]
 
Number 2:
Tracing route to W8ERD-Telephone.local.mesh [10.176.63.206]
over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  localnode.local.mesh [10.82.37.145]
  2     2 ms    19 ms    39 ms  N8DCA-LewisCenter.local.mesh [10.34.42.155]
  3    35 ms   569 ms    46 ms  N8DCA-Ostrander.local.mesh [10.36.23.218]
  4    26 ms   185 ms   168 ms  W8ERD-Tower.local.mesh [10.150.7.249]
  5    41 ms    10 ms    15 ms  W8ERD-Telephone.local.mesh [10.176.63.206]
 
Number 3:
Tracing route to W8ERD-Telephone.local.mesh [10.176.63.206]
over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  localnode.local.mesh [10.82.37.145]
  2     7 ms   131 ms     3 ms  N8DCA-LewisCenter.local.mesh [10.34.42.155]
  3    14 ms     7 ms    20 ms  N8DCA-Ostrander.local.mesh [10.36.23.218]
  4     7 ms     7 ms    92 ms  W8ERD-Tower.local.mesh [10.150.7.249]
  5    80 ms    70 ms   158 ms  W8ERD-Telephone.local.mesh [10.176.63.206]
 
Number 4:
Tracing route to W8ERD-Telephone.local.mesh [10.176.63.206]
over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  localnode.local.mesh [10.82.37.145]
  2    21 ms     7 ms     4 ms  N8DCA-LewisCenter.local.mesh [10.34.42.155]
  3     9 ms    15 ms     4 ms  N8DCA-Ostrander.local.mesh [10.36.23.218]
  4    47 ms    74 ms    24 ms  W8ERD-Tower.local.mesh [10.150.7.249]
  5     7 ms     7 ms    16 ms  W8ERD-Telephone.local.mesh [10.176.63.206]

 
K5DLQ
K5DLQ's picture
Are you using the G729 codec
Are you using the G729 codec on your phone and PBX?  I've found it very efficient and the sound quality is good.
w8erd
G729
We will try that soon.  Thanks for the suggestion.

Bob  W8ERD
w8erd
G729
We tried that today, and it is different, but no better. Still unusable.  Whereas before we had a mixture of clear voice and no voice, with G729 we have
continuous mush, with occasional words understandable.

We have a Grandstteam IP phone at one end and a Grandstream Handytone at the other. Both set to use G729 as the first choice.

Bob W8ERD
KE2N
KE2N's picture
buffer

No one has mentioned this (yet) but one thing you MUST do when using wireless for VOIP is to have a generous jitter buffer in all the phones.  Look at your ping times and see what time corresponds to about 90% of the pings and then set the jitter buffer to cover that (ideally this will be less than 300 ms - a longer latency than this will interfere with your ability to carry on a conversation). 
 

w8erd
jitter buffer
Thanks! We will try that. The default setting on the Grandstream phone is "adaptive jitter buffer length" and "300ms".  
Just for an experiment, we will change that to "fixed" and "800ms" as that is then max allowed.

Bob W8ERD
w8erd
jitter buffer
We changed it to a fixed length of 800 ms.  Then we could understand a few words now and then.   Still unusable unfortunatey.

Bob W8ERD
N2MH
N2MH's picture
Network Topology

You didn't mention your network topology. Looking at the AREDN Map, it appears that all 3 of these nodes are on the same channel, -2 in the 2 GHz band, as well as most of the other nodes in your area.

I suspect that if you go to each of these three nodes (LewisCenter, Ostrander, Tower) and look at each of their neighbors you will find that none of them see the other two, or get good link qualities from each. I also suspect that each of these three nodes see many other nodes, only several of which may be in common to everybody.

I further suspect that you have a good case of "Hidden Terminal Syndrome".

https://en.wikipedia.org/wiki/Hidden_node_problem

Keep in mind that unlike tcp traffic, eg, Web, voice packets are udp packets and do not get acknowledged. Also, you may not see any "real" traffic but the channel is constantly being occupied by each node transmitting his SSID and OLSR routing information. Depending on propagation and whoever may have their node on or off the air at any given time, the QRM to your voice call from these broadcasts may vary over time.

Bottom line, I think your voice packets are getting stepped on by other nodes out there that do not hear them. Since the voice packets are udp in nature, they will get dropped and lost forever. Thus, the voice call will experience drop outs.

In addition to the solutions provided in the WiKi, you and your neighbors should consider setting up a true backbone network with various local networks isolated from each other both by distance, channel, and band.

BTW, a utility that shows this effect more clearly than traceroute is "MTR" (My Trace Route). This utility is available for both linux and Windows. I believe that in Windows it is called "WinMTR".

73, Mark


 

w8erd
VoIP variability
Thanks Mark!

Some more info and questions.
All the nodes with N8DCA-XXX callsign are on high towers provided by EMA (also used for their own  communications).
All other nodes are generally at home locations. Everything is at 2.4 ghz, 5 mhz BW, ch -2.
Most, probably all of the N8DCA nodes see all the others, although not always with good quality.
Would it help if I sent the mesh status info from the nodes in question?
Would it help if the "distance" parameters were changed?

I looked at the Hidden Node reference you gave, but it does not seem we could do any of the things suggested to improve things.
Increase power - not possible, but would be great.
Omni antennas - All are.
Obstacles - no
move nodes - not possible.

My thoughts for the ideal situation would be to link all the n8dca towers to a central one at 5 ghz, with directional antennas. Then have each tower 
serve only its local users at 2.4 GHz.  But we have no money to do much of anything. We still have 3 more EMA towers where we could install nodes, but
have no money to do so.  

Bob W8ERD
nc8q
nc8q's picture
Everything is at 2.4 ghz, 5 mhz BW, ch -2.
Hi, Bob:

 It seems to me that your response to Mark's note narrowly focused on his link to 'Hidden Transmitter Syndrome'.
While that reference is an excellent explanation of what 'HTS' is, its solutions do not mirror the tools we have available in 'Part 97'.
Let's review and extrapolate on Mark's comments.

Mark: "it appears that all 3 of these nodes are on the same channel, -2 in the 2 GHz band, as well as most of the other nodes in your area."

This is not a good network policy. Part 97 and supported Ubiquiti devices gives of four (4) bands and multiple channels with a few that are unshared.
It appears that you chose only one (1) channel for everything.

 Mark described the Hidden Transmitter Syndrome and shared a link.

 I looked at the AREDN MAP (http://usercontent.arednmesh.org/K/5/K5DLQ/livemap2.html) and it appears that your area
has at least 24 nodes on the same channel. This, pretty much, guarantees a hidden transmitter issue.

Bob: "Most, probably all of the N8DCA nodes see all the others, although not always with good quality."

 This, pretty much, describes a hidden transmitter prone scenario.
IMHO, nodes should see (100% LQ) the node(s) they should see - and not see all others.

 It was not mentioned, but there may be another issue at play here: 'Exposed Nodes'.
https://en.wikipedia.org/wiki/Exposed_node_problem

Bob: "All the nodes with N8DCA-XXX callsign are on high towers provided by EMA"

 These might be exposed nodes.

Bob: "All other nodes are generally at home locations. Everything is at 2.4 ghz, 5 mhz BW, ch -2."

These might be hidden transmitters.

Mark: "you and your neighbors should consider setting up a true backbone network with various local networks isolated from each other both by distance,
channel, and band."

  I emphasize 'band'.

  It seems to me that the primary core feature of AREDN mesh firmware is its automatic adaptive routing.
i.e. We need not configure 'layer 3' routing.
A secondary core feature is the 'DtD' ethernet link capability.
However, expecting this firmware to compensate for insufficient linking (LQs<100%) and
(hidden transmitters and exposed nodes) 'layer 1' insufficient RF channel selection might be too much.

 I hope this helps, Chuck
 
w8erd
VoIP and network topology
If we were to start over from scratch there could be better node arrangements, but it grew over the years and is what it is and there is little money to make big changes.

Could we do an experiment to find a better configuration?  For example, could we temporarily "turn off" some of the main nodes in the network?
The smaller ones probably make little difference. I can control from my node all the main nodes.  How could they best be "turned off"?  Obviously this must be a reversible situation so I can turn them back on again.  One possibility that comes to mind is to turn off all the nodes except those that are being used for the test.

Then In the longer term, if the network is needed for some special event or emergency, we could reconfigure it with this knowledge to make it more reliable.

Bob W8ERD
KD1HA
KD1HA's picture
Turn off or...
Maybe just change to -1 on the nodes needed for the test?

Denis KD1HA
w8erd
VoIP Tests
That is an excellent idea. (I had thought of the same thing myself :)

Bob W8ERD
w8erd
VoIP Quality
We have conducted extensive experimentation since I last posted.
1. We put the 4 nodes involved on ch -1, as was suggested here. That did not work, as there seems to be an unknown signal on ch -1 causing interference to one
of the end nodes. Using WiFi Scan we can see it, but we are not sure of what we seeing, because the MAC addresses listed do not correspond to anything we know, and seem inconsistent. At one point we seemed to be getting interference from another strong signal on ch -2, but the Mac address made no sense.
2. We went back to ch -2, and changed the SSID of the 4 nodes. Now the VOIP call goes thru, but is very mushy and distorted.  But by accident we discovered this is due to audio overload somewhere. If we hold the telephones far away from our faces, the audio becomes clear.  This may have been true earlier also before we tried the G.729 compression and buffer size changes.

So we just have kind of a mess and don't know where to go from here.

Bob W8ERD

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer