Redundancy Isn’t Always Redundant

Recently, a cloud provider we work with experienced a two-hour connectivity failure to one of their regional locations. The root cause? A set of “redundant” transport links that turned out not to be redundant at all. Although sold to the provider as diverse paths between locations, all circuits ultimately converged on a single line card in the transport provider’s switch.

At CST, we design our hosted environments to tolerate the loss of individual links, sites, or components, and those safeguards operated exactly as intended and none of our customers were affected by this incident. Still, events like this are valuable reminders of the hidden fragilities that can exist behind the scenes.

For those of us who have spent long enough in infrastructure, stories like this are familiar. It’s the classic scenario: two fibres in different ducts that magically join up in the last 50 metres; “diverse” power feeds that share the same breaker upstream; or cross-site links that, unbeknown to the customer, terminate on the same piece of hardware.

From the outside, redundancy is a simple promise: If A fails, B continues to work.

In reality, unless that redundancy is carefully engineered, verified, and continuously audited, it may only exist on paper.

Assumptions vs. Reality

The provider in this incident had ordered fully redundant links. Their intention was sound, and, importantly, they believed they had achieved redundancy. But unless you explicitly ask a transport provider where each fibre is routed, what equipment each side terminates on, and how protection switching is implemented, the word “redundant” can mean many things.

Just as backups are only useful if you can restore them, a redundant link is only redundant if you’ve:

• validated the physical and logical diversity
• confirmed it terminates on different optical equipment
• checked that the upstream provider has not introduced a hidden single point of failure.

Why This Matters for Our Customers

Our role as a hosting and high-availability specialist is to design solutions that continue to work - even when the unexpected happens. Outages like this are reminders of a simple truth: you cannot rely solely on what a provider claims. You must verify it. That means:

• Asking “awkward” questions during provisioning - Where do the circuits run? Which chassis, which modules, which protection schemes?
• Documenting the answers - Because what is true at deployment time may not be true six months later.
• Auditing regularly - Providers change routing, replace cards, consolidate infrastructure, or perform maintenance that unintentionally introduces new single points of failure. Drift happens, and unless someone checks, you won’t know until it breaks.

Designing for the Real World

No provider, no matter how large, is immune to incidents. Equipment fails, people make mistakes, and complexity breeds fragility. The key is not to pretend otherwise, but to architect around that reality within the constraints of budget and business risk:

• Multi-region or multi-availability zone deployments
• Cross-provider or cross-cloud failover
• True path diversity validated at the fibre and equipment level
• In-application resilience such as queueing, retries, health checks, and autoscaling

Every layer of additional resilience reduces the blast radius of unexpected failures - and allows businesses to keep operating while someone else scrambles to replace a faulty line card.

Final Thoughts

Outages like the one described are not signs of incompetence - they are signs of the inherent complexity of modern networking. Redundancy is not a checkbox; it is a discipline. If it isn’t tested, validated, and revisited over time, then it may not exist when you need it most.

As always, our commitment is to design hosting architectures that expect things to fail - and keep your services available when they do.