Why is Multi-Site preferred over a single stretched VXLAN fabric?

Because the BGW essentially acts as a 'filter' for the data plane, translating local VNIs to remote VNIs and stripping local encapsulation. This prevents a broadcast loop or a Spanning Tree failure in one site from affecting the CP/DP of the second site.

Can I use older 9300-series (non-EX/FX) switches as Border Gateways?

Cloud Scale ASICs like the FX2 or GX are required because they support the VXLAN-to-VXLAN 'double lookup' required by a BGW. Older ASICs cannot perform the necessary encapsulation and decapsulation in a single pass at line rate.

Is vPC required for Border Gateway High Availability?

Anycast BGW is significantly better for scalability. It allows multiple BGWs to share a virtual IP, enabling L3 ECMP for inter-site traffic and avoiding the proprietary limitations and failure-domain risks of a vPC Peer-Link.

What are the specific requirements for the Inter-Site Network (ISN)?

The ISN only needs to provide IP reachability and support an MTU of at least 1550 bytes. It does not need to be VXLAN-aware or EVPN-aware; it simply routes the encapsulated packets between BGW VTEPs.

What does the 'rewrite-evpn-rt-asn' command actually do?

The 'rewrite-evpn-rt-asn' command enables a BGW to re-originate EVPN routes with its own ASN, which is crucial for multi-AS designs where Route Targets must be mapped correctly between independent BGP domains.

Should I use Ingress Replication or Multicast for BUM traffic across sites?

For most enterprises, Ingress Replication (IR) is the preferred method. It simplifies the ISN configuration by using the BGW to replicate BUM traffic as unicast to remote sites, removing the need for PIM or an RP in the core.

Mastering Cisco Nexus 9000 VXLAN EVPN Multi-Site: 2026 Design Guide

The era of the "stretched" Layer 2 domain via OTV or VPLS is dead, and if you are still relying on back-to-back vPC for multi-pod connectivity, your blast radius is a ticking time bomb. In 2026, the only defensible architecture for interconnecting distributed high-performance data centers is the Nexus 9000 VXLAN EVPN Multi-Site model, leveraging hardware-based Border Gateway (BGW) functions to localize failure domains while maintaining seamless workload mobility.

The Fallacy of the Stretched Fabric

For years, network architects chased the "holy grail" of a single, unified fabric across multiple physical sites. This approach, while theoretically simplifying vMotion and IP consistency, created a fragile "fate-sharing" environment. A single BUM (Broadcast, Unknown Unicast, Multicast) storm in Site A could, and often did, bring down Site B and C simultaneously. Cisco’s NX-OS implementation of VXLAN EVPN Multi-Site solves this by decoupling the internal site protocols from the Inter-Site Network (ISN).

In a standard Multi-Site design, each site operates its own independent BGP EVPN control plane. The Border Gateways (BGWs) act as the termination point for intra-site VXLAN tunnels and the origination point for inter-site tunnels. Unlike traditional stretching, the BGW rewrites the Next-Hop and modifies the Route Targets, ensuring that MAC/IP learning is isolated. If a loop occurs in Site A, the Spanning Tree or storm control mechanisms within that site catch it before it propagates across the high-latency ISN link.

Hardware Selection: Cloud Scale or Bust

Don't try to run Multi-Site on first-generation Nexus 9300 series. You need the Cloud Scale ASICs (EX, FX, FX2, FX3, or the newer GX/GX2 series) to support the required VXLAN-to-VXLAN termination and the necessary TCAM entries for L3VNI and L2VNI mapping. For the Border Gateway role specifically, the Nexus 9364C-GX or the 9336C-FX2 are our gold standards for 100G/400G density.

The 9300-GX series is particularly critical in 2026 as we move toward 400G backbones. They provide the deep buffers necessary to handle micro-bursts that occur when transitioning from high-bandwidth internal spines to potentially lower-bandwidth ISN circuits. Avoid the 9200 series for BGW roles; the feature parity simply isn't there for complex EVPN Multi-Site deployments.

Architecture of the Border Gateway (BGW)

The Border Gateway is the most critical component in this design. You have two primary deployment models: Distributed BGW (Spines as BGWs) or Dedicated BGW (Leafs as BGWs). At TechLeague, we almost exclusively recommend the Dedicated BGW model (Border Leafs) for large-scale enterprise environments. While using Spines as BGWs saves a few RU of rack space, it forces your spines to maintain a full BGP routing table for the external world, increasing CPU pressure and complicating your management boundary.

High Availability: Anycast BGW vs. vPC BGW

Old-school engineers still try to put their BGWs in a vPC pair. This is a mistake. VXLAN EVPN Multi-Site supports the Anycast BGW (virtual IP) model. In this setup, multiple BGWs share the same anycast IP address for the VXLAN tunnel endpoint (VTEP). Using Anycast BGW allows for up to 6 or 8-way ECMP paths across the ISN, providing much higher resiliency and throughput than a two-node vPC cluster.


evpn multisite border-gateway 100
  delay-restore time 30
  provision-model anycast-gw
  multisite-id 1

By using the anycast model, you eliminate the need for a vPC Peer-Link between gateways, which has traditionally been a point of failure in convergence-sensitive environments. If one BGW fails, the BGP convergence on the ISN simply reroutes traffic to the other anycast members without requiring complex LACP negotiation.

The Inter-Site Network (ISN): Stop Overcomplicating It

The ISN is often the most misunderstood part of the design. It does not need to speak EVPN. In fact, it shouldn't. The ISN is purely a transport medium that provides IP reachability between BGW VTEPs. It must support MTU 1550+ (to accommodate the 50-byte VXLAN header) and run an IGP like OSPF or IS-IS.

The BGWs establish Multi-Hop MP-BGP EVPN peerings with each other over the ISN. We strongly advocate for using BGP Peer Groups to manage these connections, especially when scaling beyond three sites. Here is a typical BGW configuration snippet for ISN peering:


router bgp 65001
  neighbor 10.255.255.10 remote-as 65002
    description Peer_to_Site_B_BGW
    update-source loopback1
    ebgp-multihop 5
    address-family l2vpn evpn
      send-community extended
      rewrite-evpn-rt-asn

The rewrite-evpn-rt-asn command is vital. It allows the BGW to translate Route Targets between sites, ensuring that labels are correctly interpreted as traffic moves through the ISN.

L3 Multicast and BUM Traffic Handling

Handling BUM traffic across sites without melting your WAN is the differentiator between a junior admin and a senior engineer. You have two choices: Ingress Replication (IR) or Local-Bias Multicast. For 2026 designs, Ingress Replication with Multi-Site Optimizations is the standard unless you have very specific high-bandwidth multicast requirements. IR creates individual unicast copies of BUM packets for each remote site, but the BGW is smart enough to send only one copy to the remote site, which the remote BGW then replicates locally to its leafs. This "Head-end Replication" significantly reduces the bandwidth burden on your ISN.

If you must use Multicast in the core, ensure your ISN supports PIM-BSR or PIM-RP. However, the complexity of managing an RP across multiple administrative domains usually outweighs the efficiency gains for 90% of enterprises.

Security and Policy: Micro-segmentation with Group-Based Policy (GBP)

Standard VXLAN EVPN only carries the VNI and L2/L3 info. If you want to maintain security posture across sites, you must implement VXLAN-GPO (Group-Based Policy). This allows you to carry SGTs (Scalable Group Tags) in the VXLAN header. When a user in Site A (SGT 10) talks to a server in Site B, the BGW preserves that tag. This allows your firewalls or Nexus leafs at Site B to enforce policy based on the SGT rather than an IP address that might have changed.

As we detailed in our guide on Nexus ACI vs. Standalone EVPN, the manual orchestration of SGTs in NX-OS is the primary "pain point" that often drives customers toward ACI. However, if you are a CLI-first shop, sticking with NX-OS Multi-Site gives you better visibility into the control plane and avoids the "black box" nature of APIC.

Verification and Operational Reality

Troubleshooting Multi-Site requires a mastery of the show nve multisite commands. You aren't just checking adjacency; you're checking whether the BGW is actually advertising the correct site-id. A common failure point is mismatched Site-IDs within the same fabric, which causes the BGW to shut down the Inter-Site VIP to prevent loops.


# Check BGW status
show nve multisite border-gateway
# Check remote site reachability
show bgp l2vpn evpn summary
# Verify VNI mapping across sites
show nve vni

Expect your ISN latency to play a role in BGP Convergence. Tune your keepalive and holdtime timers, but don't go sub-second unless you have a dedicated dark fiber link. On a standard 10ms-20ms WAN transit, a 3-second keepalive is the "sweet spot" to prevent flapping during minor jitter events.

Conclusion

Cisco Nexus 9000 VXLAN EVPN Multi-Site is the only way to scale modern data centers without inheriting the risks of a giant, flat Layer 2 domain. By using Anycast BGWs and Cloud Scale ASICs, you build a resilient, multi-homed environment that supports the high-speed demands of 2026. If your current design relies on stretched vPCs or complex OTV overlays, it's time to re-platform. To get an expert review of your fabric design or to bring in a Tier-3 engineering team for your deployment, visit our techleague.io pricing page to see how we can accelerate your infrastructure modernization.