The Physical Thing That Breaks First

Ask a room full of network engineers what the smallest physical component is they've watched bring down an entire service, and you will get a genuine argument. Not a shrug. An argument.

The reaction

Not a shrug. An argument.

That's what happened on LinkedIn a fortnight ago, after a post about a submarine cable fault off Samoa picked up 21 comments from people who actually do this for a living: submarine cable project managers, principal fibre engineers, ROV operations specialists, offshore ops supervisors. People who correct you on repeater weight to the kilogram, because it matters to them.

The debate that followed said more than the original post did.

The finding

144 kilometres off Samoa.

Here's what happened. A submerged optical amplifier, a repeater, developed a fault 144 kilometres off Samoa. Niue and the Cook Islands lost proper internet because of it. The Manatua Consortium's own release ruled out anchor damage, trawling, and sabotage. Whatever went wrong, went wrong inside the system itself.

The first read on that, including my own, was: a component doing what components eventually do. Wear, not damage.

One of the commenters, a principal fibre engineer, pushed back on that specifically. A shunt fault, he pointed out, means the cable's insulation has been compromised and it's earthing into the sea. That's not wear. That's damage. Something has hit it. Which raises a harder question than the first one: if it's not external, what does that leave? Manufacturing defect, material fatigue at depth, a termination failing under load?

Nobody in that thread had a confirmed answer. That's the point. A fully-costed, professionally-run international system, and the actual cause of its failure was still genuinely unclear a week after the fault was confirmed.

Beyond submarine cable

Why this isn't really a submarine cable story

Here's where it gets useful for anyone who isn't in undersea telecoms.

The redundancy conversation in most networks happens at the routing layer. Failover paths, diagram after diagram, all showing how traffic moves if one path drops. What that conversation almost never covers is what happens when the physical thing carrying the signal is the thing that fails, and the fix requires someone to physically go and find it.

For Manatua, that means a repair vessel, a queue of other faults competing for the same limited repair capacity, and a timeline nobody can commit to because "how long is a piece of string" was the literal answer one of the engineers gave when asked.

For a building in Sydney, it means something much smaller, but structurally identical.

Same shape, smaller scale

The domestic mirror

This is where the thread stopped being about Samoa.

Boris Siljanoski, Owner Operator/Technician at Antenna Industries, replied with this:

"1 faulty F connector when slightly disturbed brought whole MATV system down ever more annoyingly it would only fault depending what orientation the feed cable was hanging on, doing other service work disturbed it and troubleshooting it then made it work again before you got to it."

That's not a coincidence of scale. That's the same failure mode as a shunt fault on a seabed cable, just playing out in a comms cupboard instead of the Pacific. My reply to Boris was the actual thesis of this article:

"That's the same physical layer stubbornness, just a different scale, Boris. Intermittent, orientation-dependent, self-clearing before you can pin it down. Whether it's an F connector or a subsea power core, an unstable physical connection behaves the same way and hides from you the same way while you're chasing it."

Garry M., whose profile reads "Connecting operational realities," added a story from a lightning strike in Malongo Camp, Cabinda, that melted an entire equipment gateway:

"There was a spare in Luanda, Angola which id left for this eventuality. Id packed in Angola 6 months before. Was going through the redundancy process, two days before id left. A colleague messaged me on Bebo from Cabinda 2 days before my leaving date. The site was down, and had been for 2 weeks."

He'd had the foresight to leave a spare, six months earlier, for exactly this scenario. It still nearly wasn't there when it mattered, because a separate internal process had already started moving that spare somewhere else. The plan existed. It just didn't live inside the diagram anyone was tracking. As I said in reply: that's the same shape as the Manatua fault, the failure point isn't dramatic, it's a component doing what components eventually do, and the plan for it lives outside the neat diagram everyone points to.

Even the correction mid-thread made the point stronger. Peter Jamieson, Principal Engineer in Core Engineering Fibre, pushed back on my first read of the Manatua fault:

"You say it's a 'shunt fault', so that means it should/could still be operational. The insulation of the cable has been compromised and it's earthing into the sea. So it's not a component failure, the repeater hasn't failed, the cable has been damaged by something that has exposed the power core. So something has hit it."

I'll take that correction on the page too, since it's true and it sharpens the point rather than undercutting it: even the causes that don't involve a ship dragging an anchor across the seabed are still physical-world failure modes that no network diagram accounts for.

Bringing it home

Bringing it back to your building

Most enterprise networks have never had their physical layer traced end to end since the day it was fitted out. Ports get relabelled informally. Patch leads get moved during other work and never get put back exactly right. A PoE injector or an RF splitter sits behind a screen nobody's looked behind in years.

None of that shows up on a routing diagram. All of it is exactly the kind of thing that decides how long an outage actually lasts, because you can't fail over around a physical layer you can't locate.

The teams that handle this well aren't the ones with the most detailed failover documentation. They're the ones who can also tell you, specifically, what physical thing is most likely to go first, and where it actually is.

That's a different kind of audit to the one most businesses run. It's not asking "is our network redundant." It's asking "if the thing that fails is physical, do we already know what and where."

The physical layer risk cluster.

This article sits alongside two others documenting the same blind spot from different angles:

→

Network documentation audits: when the comms room knowledge walks out.

What a physical layer audit finds when the person who knew the comms room has already left, and the record never caught up.

→

Route diversity: when both paths are "diverse." And share the same trench.

Redundancy on a diagram and redundancy in the ground are two different claims. One fibre cut proved it.

Common questions

Physical layer risk, explained.

What is "physical layer risk" and why does redundancy at the network layer not cover it?

Physical layer risk is the chance that the actual cabling, connectors, or hardware carrying a signal fails, rather than the routing path around it. Network-layer redundancy assumes an alternate path exists and works. It doesn't account for a single physical component, a connector, a splitter, a patch lead, being the point of failure the alternate path was never built to bypass.

How can one small component take down an entire service?

Because most infrastructure isn't built with true component-level redundancy at every point. A single faulty connector or damaged cable section can sit at a point in the network where there's no alternate physical path, meaning its failure isn't routed around, it's simply a failure.

Why does it matter whether a fault is intermittent versus permanent?

Intermittent, orientation-dependent, or self-clearing faults are harder to diagnose than a permanent break because they can disappear the moment someone starts investigating. This is exactly the kind of fault that undocumented, untested infrastructure hides for years, since nobody's actively watching for it until it causes a visible outage.

Does having a spare part or a backup plan solve this?

It helps, but it isn't the whole answer. A spare only protects you if it's still where you think it is, and if the process for using it hasn't quietly moved on without anyone updating the plan. Physical layer risk needs an accurate, current record, not just a spare sitting in a cupboard.

How do I find out what could fail first in my own building?

The only reliable way is tracing the physical infrastructure as it actually exists today, not as it was designed or last documented. That means testing cabling and hardware directly, not relying on labels or old diagrams that may no longer reflect reality.