Network Fabric, Switching and Carrier Connectivity
Follow packets from the cabinet to the meet-me room and find the places where physical reality limits a logical design.
Learning outcomes
- Trace a packet from a server port to the carrier boundary
- Compare access-layer placement and its cabling consequences
- Separate logical redundancy from physical independence
OVERSUBSCRIPTION
RATIO = TOTAL ACCESS PORT CAPACITY ÷ UPLINK CAPACITY
A design choice, not a defect. What matters is whether the ratio matches the traffic the workloads actually produce.
The access layer is where physical reality arrives
A server port connects to an access switch, and where that switch lives determines what runs through the room. Placed in the cabinet, the copper is short and stays inside it, and what leaves is a small number of fibre uplinks. Placed at the end or middle of a row, the switching is shared across cabinets and every server connection becomes a horizontal run to reach it.
Neither is better in general. Per-cabinet switching strands ports in partly filled cabinets and adds devices to manage; shared switching pools those ports and pays for it in dense copper trunks that occupy pathway space, add weight and can obstruct airflow where they pass. The cable decision and the thermal decision are the same decision, made in different drawings.
Leaf-spine changed what “the fabric” means
Traditional three-tier designs assumed most traffic left the building, so access fed aggregation and aggregation fed a core. When applications became distributed, most traffic started moving between servers instead, and a topology optimised for north-south flow made east-west traffic take long, uneven paths.
A leaf-spine fabric answers that by connecting every leaf to every spine, so any two leaves are the same number of hops apart. Capacity is added by adding spines rather than by making individual devices larger. The trade is a great deal more cabling between leaf and spine layers, and a design whose behaviour depends on the uplinks being present and balanced rather than on one large device being healthy.
Two fabrics are only two if they stay apart
Independent A and B fabrics are the network version of A/B power feeds, and they fail the same way. Two switches in one cabinet, fed from one power strip, uplinked through one cable route, are not two paths — they are one path with duplicated equipment in it.
Real independence has to survive the whole trace: separate power feeds, separate physical routes, separate uplink paths, and ideally separate management and change windows so one mistaken configuration cannot be applied to both at once. The failure that takes out both fabrics is rarely a switch; it is a shared route, a shared power source or a shared change.
The carrier boundary is where the building stops
Connectivity leaves the building through entrance facilities, and diversity there is physical rather than contractual. Two carriers whose fibre enters through the same duct, the same building entry or the same street are exposed to the same excavator. Genuine diversity means separate physical routes, separate entry points, and evidence rather than assurance.
The meet-me room is where carriers and tenants cross-connect, which makes it simultaneously the most useful and the most concentrated point in the connectivity design. It deserves the same treatment as any other shared dependency: know what depends on it, know what happens if it is unavailable, and know whether there is any path that does not traverse it.
Latency is a budget, and distance spends it
Every element in a path costs time: propagation over distance, serialisation, queuing during congestion, and processing at each hop. Distance is the one that cannot be optimised away — it is set by physics and by where the facilities are, which is why latency-sensitive placement is a real-estate decision as much as a network one.
The rest is design and operations. Fewer hops, sensible oversubscription, and congestion that is measured rather than assumed will get a fabric most of the way. A predictable path is usually worth more than a marginally shorter one, because a fabric whose behaviour changes with load is very hard to reason about during an incident.
How it appears in Data Center Fan
Data Center Fan models connectivity as capacity and incident surface — fibre cuts, attacks and congestion affect contracts and uptime. It abstracts topology heavily and does not simulate switching, routing or a real fabric.
Common misconception
“Top-of-rack means the switch is at the top of the rack.” It is a topology name, not a mounting position; the switch is frequently installed in the middle of the cabinet so the runs to servers above and below are shortest.
Knowledge check
Two independent fabrics feed a cabinet, but both uplinks share one cable tray. What is true?
- The design is fully redundant
- A physical event in that tray can remove both paths
- Independence is guaranteed by the logical separation
Logical independence does not survive a shared physical route. Two paths sharing a tray, a riser or a room share exposure to anything that happens there.
Frequently asked questions
What is a leaf-spine fabric?
A two-layer topology where every leaf switch connects to every spine switch, so any two leaves are the same number of hops apart. It suits east-west traffic between servers and scales by adding spines rather than by making individual devices larger.
What is the difference between three-tier and leaf-spine?
Three-tier designs stack access, aggregation and core, which suits traffic leaving the building. Leaf-spine connects every leaf to every spine, which gives predictable hop counts for traffic moving between servers. The shift followed applications becoming distributed.
What is a meet-me room?
The space in a facility where carriers and tenants cross-connect to each other. It is the point where the building’s connectivity concentrates, which makes it both extremely useful and a shared dependency worth understanding.
What is carrier diversity?
Having connectivity that does not share a physical fate. Two carriers are only diverse if their fibre follows separate routes and enters the building at separate points; contracts with two providers whose cables share a duct are not diversity.
What is oversubscription in a data center network?
The ratio between the total capacity of access ports and the uplink capacity available to them. It is a deliberate design choice rather than a fault — the question is whether the chosen ratio matches the traffic the workloads actually generate.
Why does cabling affect cooling?
Because dense copper bundles occupy pathway and under-floor space, add weight, and can obstruct the air the floor is meant to deliver. A switch-placement decision that adds horizontal cabling is also a decision about airflow in the spaces that cabling passes through.