After hours of debugging, I got this 20MHz single-polarity link
moving up to 55Mbit. A high resolution spectrum analysis on both
sides did indeed show a better frequency to use for the link.
Spectrum analysis captures here:
https://imgur.com/a/4H7GB
There was also an IP conflict between two modems @ Capitol Park.
The conflicting IP has been removed from CapitolPark-S3.
The QueenAnne modem used for the link is not one used anywhere else
on the network. It's a very low-end modem, and as a result it was
having CPU/RAM starvation issues when running our regular diagnostic
tools, which lead to out-of-memory conditions and kernel crashes. A
different test methodology had to be used to verify the link speed
(testing through the modem, instead of to the modem).
The modem that links QueenAnne with the Westin building (on the
QueenAnne side) had a mis-configured OSPF router-id. This was
fixed.
I'm still seeing weird routing decisions being made by OSPF. These
are triggered by our point-to-point route entries (
44.24.242.0/24
space). More research needs to be done here, and perhaps a re-write
of how we define point-to-point interface addresses. Any OSPF
experts in the house?
I also discovered R1.QueenAnne was still vulnerable to hacking due
to a mis-configuration of its control software. It missed the
updates that were sent out to the whole network. This has been
fixed now. R1.QueenAnne also didn't have the diagnostic
bandwidth-server setup correctly. This was fixed.
With the CapitolPark-QueenAnne link performing well now:
[eo@CapitolPark-QueenAnne] /system resource> /tool
bandwidth-test 44.24.241.81 direction=both
status: running
duration: 57s
tx-current: 28.0Mbps
tx-10-second-average: 28.4Mbps
tx-total-average: 27.5Mbps
rx-current: 27.6Mbps
rx-10-second-average: 28.0Mbps
rx-total-average: 27.0Mbps
lost-packets: 288
random-data: no
direction: both
tx-size: 1500
rx-size: 1500
its OSPF config has been reset to a normal preference level, so
that packets no longer try to avoid that link as they are routed
through the network. This link can be sped up by upgrading to a
dual-polarity modem @ CapitolPark.
While testing if the OSPF hop cost was being calculated correctly in
the Beacon-Haystack-QueenAnne RF link (they both connect to the same
dish @ Haystack), I discovered a mis-config on the Haystack.Beacon
modem (bad LAN IP binding) which was preventing it from bringing up
OSPF on its LAN interface. This was fixed and that modem should act
like an actual router now, moving traffic.
During the same Beacon testing, I was reminded that our Baldi-Beacon
RF link
sucks.
It was optimized for speed to Tukwila, which is now gone, so a trip
needs to be scheduled to Baldi to rotate the dish a few degrees
north and get that link going strong.
I normally wouldn't send this kind of verbose email to psdr@, but I
hope it's illuminating as to the type and extent of work required to
keep this network running well.
--Bart