MONITORING SYSTEMS PROBLEM
The number one problem we find when we onboard a new network is not the routing. It is the monitoring.
Most conversations about network monitoring best practices start with tool selection. Ours starts somewhere else, because the tool is almost never the problem. At first glance every setup looks fine. Someone spun up LibreNMS or Observium in an afternoon, pointed it at the gear, and the graphs filled in. Hundreds of metrics, neat dashboards, green everywhere. And everyone moved on, because it felt done. Monitoring in place, box checked.
Then customers start dropping for 30 seconds at a time, every few minutes, and every dashboard is still green. Nobody got paged.
That is not monitoring failing. That is monitoring working exactly as designed. The design is the problem.
What a 5-Minute Average Erases
Most setups poll every 5 minutes on defaults. SNMP reads the counters, divides by 300 seconds, draws a point. A 30 second burst at line rate inside that window averages out to a 10 percent utilization bump on the graph. Meanwhile buffers overflow, the PFE chokes, TCP sessions reset, and the customer experiences a full outage. The graph shows a blip nobody would look at twice.
Run the numbers on a 10G link. Thirty seconds at line rate is 300 gigabits of traffic. Spread across a 300 second polling window, that renders as 1 Gbps average, a calm 10 percent. The customer saw a dead connection. The graph saw a Tuesday.
That same averaging hides microbursts, brief congestion, and short reconvergence events every single day. The problems that churn customers are usually the short ones, and short is exactly what a 5 minute average erases.
Network Monitoring Best Practices Start With Polling Design
Here is the part most people skip. Monitoring is not the graphs. It is how you use the data.
It makes no sense to poll the SFP serial number every 5 minutes, and it makes every sense to poll the critical backbone links far more often than the default. Same tool, completely different value, depending on whether someone tuned it. The practical rule we apply: classify what each metric is for. Inventory data can be read daily. Environmental data every few minutes. Utilization and error counters on customer-affecting links deserve the shortest interval the platform and the device CPU can sustain, and on modern gear, streaming telemetry over gNMI beats SNMP polling entirely for exactly this class of metric.
Polling design is also a load question. Shortening every interval on every OID is how you turn the SNMP monitoring system itself into a source of device CPU problems. Tuning means spending the polling budget where the customer impact lives, not everywhere equally.
Alerting Is Ongoing Work, Not Setup
And then there is alerting, which is where almost everyone stops. Out-of-the-box thresholds and default timers are a starting point, not a configuration. Good alerting is ongoing work: tuning what fires, what stays quiet, what escalates, so a problem gets caught before the customer feels it.
That is a core part of what a 24/7 managed NOC actually does, and it never finishes. Every incident that reached a customer before it reached an alert is a tuning task. Every page that woke an engineer for nothing is also a tuning task. The alert set should look different every quarter, because the network does.
There is no ready-made NMS that does it for you, because every network is custom. The topology is custom, the traffic is custom, the failure modes are custom. The monitoring has to be too.
The First Question We Ask
So when we onboard a network, green dashboards tell us almost nothing about network visibility. The first question we ask is always the same: what can this monitoring not see?
The answers repeat across networks. Sub-interval events, as above. Control plane state that nobody graphed. Optical levels polled but never thresholded. Backup paths that are monitored for being up, never for being able to carry the load. It is the same list we work through in the first week of every engagement, and it is why the first deliverable is never a new dashboard. It is a list of blind spots.
Monitoring tells you the network is up. It does not tell you the network is fine. Those are two different claims, and the gap between them is where customers quietly churn.
