Why do we need the overlay?

Traditional network devices are hardware-centric and forward packets based on the destination IP address. Furthermore, each network node makes a separate independent decision on how, when, and to whom to forward each packet. This creates the following inefficiencies and drawbacks:

  • Network segmentation and network slicing is not possible - duplicate IP address ranges cannot traverse a single IP network natively;
  • Scaling is hard. Equal-cost multipathing (ECMP) over multiple types of WAN transports at scale is practically impossible. 
  • Design changes require hardware interactions
  • Virtualization and Abstraction is not possible
  • Multicast does not natively traverse public transport such as the Internet.
  • And many more;
Overlay Tunnels
Figure 1. Overlay Tunnels

One of the main business propositions of Cisco SD-WAN is that it can use any given IP transport in an efficient, secure, and flexible manner. In order to do that, the solution abstracts the packet forwarding away from the network and application logic. This is done by building IPsec tunnels between the routers' WAN attachment points. The traffic that is going through the tunnels is encapsulated with a new IP header where the source/destination addresses are replaced with the T1/T2 addresses. This way, the intermediate network between T1 and T2 does not need to know anything about the original traffic. Most network engineers are very familiar with the tunneling techniques that exist. However, here are some examples of overlay tunnels - IPsec tunnels, Virtual Extensible LAN (VXLAN), Generic Encapsulation (NVGRE), Stateless Transport Tunning (SST), Network Virtualization Overlays 3 (NVO3), etc. Cisco's SD-WAN solution uses DTLS/TLS tunnels in the overlay.  

Cisco SD-WAN Underlay vs Overlay

Cisco SD-WAN architecture is divided into two very distinct parts: the underlay network and the overlay fabric.

The underlay network represents the hardware infrastructure - all network devices that connect to the available WAN transports and local site networks. The router interfaces that connect the WAN transport networks are always configured in VPN0 (the Transport VPN). The attachment points that connect to the transports are called TLOCs (colored with red in figure 2). TLOCs play a very important role in abstracting the underlay network away from the overlay fabric and the applications.  The main and only function of the underlay network is to provide IP reachability between TLOCs

Underlay vs Overlay
Figure 2. Underlay vs Overlay

Cisco's SD-WAN Overlay network is made of IPsec tunnels that traverse from site to site using the underlay network forming the so-called SD-WAN Fabric. Each overlay tunnel is formed between two TLOCs. The routing within the overlay is governed by the Overlay Management Protocol (OMP), a control-plane protocol very similar to BGP. The OMP protocol runs over secure DTLS or TLS connections between the WAN edge routers and the vSmart controllers. The process is very similar to the BGP operation, the vSmart controller acts as a BGP route reflector (RR), it receives, modifies, and re-advertises routes from the vEdge routers, but never participate in the data-plane (in the packet forwarding).

Cisco SD-WAN Underlay vs Overlay Routing
Figure 3. Cisco SD-WAN Underlay vs Overlay Routing

Network Segmentation

Abstracting the packet forwarding away from the network and application logic opens a world of possibilities. This allows for the use of VPNs that divide the overlay network into different network segments. Essentially, segmentation is done at the WAN edge routers, and the segmentation information is carried as a VPN label in the pakcets. However, the underlay network (Transport VPN0), that connects the WAN edge routers to the WAN transport, is completely unaware of the network segments (VPNs). Only the overlay knows about the VPNs; the underlay network follows the standard IP routing.

Different Overlay Topologies per VPN
Figure 4. Different Overlay Topologies per VPN

Key Takeaways

Let's try to summarize the difference between Cisco SD-WAN's Underlay vs Overlay in one table:

Cisco SD-WAN Underlay Overlay
Description The underlay network represents the hardware infrastructure - all network devices that connect to the available WAN transports and local site networks. The overlay network represents the IPsec/GRE tunnels that are built between the underlay TLOCs.
Function To provide IP reachability between TLOCs. To provide network segmentation, security, and flexibility.
Packet Forwarding Packets traverse over the WAN following the standard IP routing principles. Next-hop is an IP address. Packets are forwarded between overlay nodes over IPsec tunnels. Next-hop is a TLOC of a remote peer.
Packet Control Hardware oriented. Software oriented.
Packet Delivery Responsible for delivery of packets. Abstracted away from the delivery of packets.
Control-Plane Protocol Standard control-plane protocols such as OSPF, IS-IS, BGP, and static routing. Cisco's Overlay Management Protocol (OMP)
Multipathing (ECMP) Achieving Equal-cost Multi-pathing (ECMP) over multiple different types of WAN transports is associated with overhead and complexity. Very hard to achieve at scale. Support for scalable multi-path forwarding over multiple virtual IPsec/GRE tunnels.
Deployment time Deployment times are long. Design changes typically require hardware changes and manual activities. Ability to rapidly deploy new functions at scale. Design changes in the overlay are done in a centralized fashion.
Multitenancy Multitenancy could be achieved via VLANs/VRFs/NAT. Requires a custom and complex control plane to propagate the VRFs across the network. Large scale implementations are associated with configuration overhead and complexity. Natively supports Multitanency and has the ability to manage overlapping IP addresses between multiple tenants.
Scalability Less scalable due to legacy technology limitation. Designed to provide great scalability, security, and flexibility.




Sat, 05/01/2021 - 08:34

Best Comparison chart for underlay and overlay


Wed, 07/28/2021 - 12:10

I went through videos on youtube and also studied the Cisco SD-WAN book, but the differences between Underlay and Overlay, you made it very clear.

The Diagramatic or Visio is fantastic for Overlay and Underlay.

Thank You.