blog articles

PROCESS BEATS HERO CULTURE IN NETWORK OPERATIONS

NOC best practices: documented process and runbook automation over hero engineer culture

PROCESS BEATS HERO CULTURE IN NETWORK OPERATIONS

 

The rockstar engineer might fix today’s outage. The harder question is what happens if they resign tomorrow.

In most network operations teams, the honest answer is uncomfortable: undocumented changes, tribal knowledge, no written policies, and a leadership team quietly hoping the next incident does not expose the gaps. That is not an operations model. That is a single point of failure with a salary.

The Bus Factor Problem

Engineers have a name for this: the bus factor, the number of people who can leave before the operation stops functioning. A NOC built around one or two brilliant individuals has a bus factor of one or two, and everything about it looks fine right up until the day it does not. The changes only they understood, the monitoring thresholds only they knew the history of, the vendor relationships that lived in their inbox: none of it survives the handover, because there was no handover, because nothing was written down.

The failure is not the engineer’s. Talented people default to solving problems, not documenting them, when the organization rewards the first and never asks for the second. The failure belongs to the operating model that made one person’s memory a production dependency.

NOC Best Practices Start With What Survives Departure

In network operations, customers are not really buying skills. Skills are the entry ticket. They are buying predictability, transparency, and proof, and none of those can depend on which individual answers the page. If a partner cannot show templates, policies, and logs, there is no partner. There is a gamble with a services agreement stapled to it.

At ITcare we made a deliberate choice years ago: build process maturity first, then layer talent on top of it. In practice, the NOC best practices that came out of that decision look like this:

Templates and runbooks that make every incident response repeatable, so the resolution quality does not depend on who is on shift. Every fault the senior engineers solve becomes a runbook the next shift can execute, which is the same feedback loop we described in our incident escalation process: heroes get paged forever, rotations backed by runbooks do not.

Policies enforced under ISO 27001, not personal preference. Change control, access management, and incident handling follow the certified process regardless of seniority, because “I always do it this way” is not an auditable control.

Runbook automation that removes the hero dependency at its root. When the diagnostic capture, the standard remediations, and the escalation packaging run as code, the 3am response is identical whether it is the veteran or the engineer in their fifth month executing it.

Proof of work. Every step documented, logged, and auditable, because screenshots of green dashboards are not evidence. Timestamps in the ticketing system are.

Process Is What Lets Talent Compound

Here is the part the hero-culture debate usually gets wrong: this is not process instead of talent. Our engineers carry JNCIE, JNCIP, and CCNA-to-CKS certifications, and we write about their growth constantly, because deep expertise is exactly what designs a good runbook in the first place. Process is what captures that expertise so it compounds instead of evaporating. The senior engineer who documents a novel fault has just taught every future shift. The one who fixes it silently has taught no one, and will fix it again at 3am next quarter.

Process maturity does not slow the operation down. It is why our 24/7 NOC resolves roughly 90 percent of incidents within the first 10 minutes: the known faults are runbooks, the automation handles the capture, and the humans spend their time on the genuinely novel. A high bus factor and fast resolution are the same property viewed from two angles.

So the next time someone pitches you senior engineers on demand, ask the harder question: where is the process those engineers work inside? Ask to see a runbook, a change policy, an escalation rule. The talent gets the incident fixed tonight. The process is what guarantees the same fix on the night they no longer work there.