Based on my teaching experience, most engineers who start learning SD-WAN somehow assume that the technology just appeared out of thin air. That it was invented on its own. Well, the reality is completely different. SD-WAN is the natural evolution of the large-scale WAN design. To understand what that means, let's quickly go through the history of WAN design at a very high level.
1990s - The era of Frame Relay
In the 1990s, the most common WAN design used Frame Relay to link branches to a central site. It was the first telco technology that allowed the WAN to scale to hundreds of offices because multiple virtual circuits on the regional hub shared one access link. This was much more cost-efficient than ordering separate leased lines to every location.
The problem with Frame Relay was its complexity, because the enterprise had to handle routing on top. This involved running routing protocols over NBMA, circuit provisioning, addressing, etc. Network security was not even a thing in the WAN at that point.
Early 2000s - The era of MPLS
Then in the early 2000s, carriers started offering MPLS Layer 3 VPNs. This significantly reduced the complexity of running a large-scale WAN because the provider handled the routing in the core. You just shared routes with them via BGP or OSPF, and they transported them to all remote locations, providing IP reachability.
The main problem was the cost and the circuit delivery times.
Mid 2000s - The era of Managed Services
By the mid 2000s, managed WAN services were a common way to pass more work to a carrier. Routing stayed much the same, and the carrier could also manage branch routers, monitor the network, and handle faults. This helped small network teams run large global WANs.
The main problem was the growing dependency on the service provider. Any changes meant submitting tickets and waiting for maintenance windows.
Late 2000s - The era of Hybrid WAN
By the late 2000s, broadband Internet was already everywhere. Organizations quickly realized that they could take advantage of the cheap Internet and rely less on the expensive MPLS service.
The main problem with the hybrid WAN approach was managing routing, because you ran L3VPN with the telco on one side and another routing protocol over an IPsec tunnel on the other side. Load balancing traffic and managing failover rules was harder.
Early 2010s - The era of DMVPN starts
And that brings us to the DMVPN era.
It was already a well-established technology around that time, and organizations quickly realized it offered many advantages over the legacy large-scale WAN designs. The main advantage was that it is an overlay on top of any transport - broadband, 4G/LTE, MPLS, etc.
This advantage appealed to organizations of all sizes. Why rely on carriers every time you want to deploy a new branch or make a change? Enterprises wanted the freedom to use whatever WAN circuit they had on-site.
Organizations quickly realized that the Internet is the new WAN. It is cheap, reliable, and already everywhere. You build an overlay on top. Providers simply supply the links. This gave network teams far more freedom to choose services, add sites, and change carriers.
Somewhere around that time, a few distinguished Cisco engineers who worked on large-scale WAN designs left the company, as shown in the diagram below.
Let's go back to that period and discuss the most common DMVPN inefficiencies, so we can see how the ex-Cisco engineers who founded Viptela addressed them in their first SD-WAN solution called Viptela (later acquired by Cisco and renamed Catalyst SD-WAN).
DMVPN inefficiencies
DMVPN was a major advance, but it also had its problems. We won't cover each one, but we'll cover the major ones at a high level that drove SD-WAN development.
DMVPN issue N1- Security
This one may surprise you. The most common complaint from large-scale customers was about DMVPN security. Wait, what?
Let me explain. Notice the hub’s public mGRE tunnel interface. It has a public Internet IP address and accepts tunnel requests from remote sites all over the Internet. That's how DMVPN works. Anyone who could reach that address (basically the entire Internet) could try to establish a DMVPN tunnel with the hub. If you own the network, does that sound good to you?
Now imagine an engineer who has just been laid off and left the company with a copy of the hub router’s config. He knows the mGRE address, the NHRP key, and the IPsec pre-shared key. Who is stopping that person from powering up any Cisco router, configuring a DMVPN tunnel, and joining your WAN network, as shown in the diagram below?
What if the organization is a large-scale multinational company with thousands of employees? How do you remove that person’s access without changing the NHRP and IPsec keys across the whole network?
Customers wanted mGRE tunnels over the Internet, but they also wanted the network to allow only devices the organization owns. For example, devices with serial numbers from the organization's Smart Account.
Welcome the vBond orchestrator.
And that is why vBond was introduced. The engineers who founded Viptela recognized this DMVPN security inefficiency and introduced the vBond orchestrator.
vBond checks each router’s serial number before approving it to join the network, as shown in the diagram below.
Now let's imagine the same scenario again. The attacker's device must first authenticate with vBond before it can reach the hub routers and establish IPsec tunnels. Since its serial number is not in the organization's Smart Account, vBond doesn't allow it access to the network.
DMVPN issue N2- Routing
Now, let's go back to another DMVPN inefficiency that became obvious in large-scale WAN networks - the hub routers must handle the data plane (forwarding and IPsec encryption) and the control plane (routing) at the same time.
In almost all large-scale designs, the hub routers run EIGRP or serve as iBGP route reflectors. In practice, the same hub devices advertised routes, handled NHRP, set up IKE/IPsec, and tracked tunnels with BFD. Now imagine a network with 3000-5000 thousand of spokes. You can imagine how much RAM and CPU the hub routers needed to maintain all those routing peerings, as shown in the diagram below.
Many customers complained that the hub routers are the bottleneck of the network. They spent too many resources maintaining routing peerings and establishing IKE/IPsec, leaving little resources for data forwarding. To account for that, organizations must replace them with bigger chassis every time the network grows.
By then, it was already common in service provider networks to use separate iBGP servers as route reflectors. The advantages were obvious - server hardware is much cheaper, and you can upscale a server/virtual machine on the fly as the network grows.
Viptela engineers quickly realized that this concept can easily be translated to enterprise WAN networks. And that's how the vSmart controller was born, as shown in the diagram below.
The vSmart controller is basically an iBGP route reflector running as a separate virtual machine on commodity hardware. This allows you to upscale/downscale it with the size of the network on the fly. Now the hub routers don't need to maintain thousands of control-plane peerings with remote branches, leaving more resources for data-plane scaling.
DMVPN issue N3- Segmentation (VRFs)
Large-scale customers complained about another issue: the DMVPN network does not support VRFs. So, to have Layer 3 segmentation, you need a separate DMVPN network per VRF. Well, imagine having 50 VRFs!?
Some organizations found a workaround - they run MPLS-over-DMVPN and carry multiple VRFs through a shared overlay using MPLS and MP-BGP. Service providers were okay with that workaround, but enterprises weren't.
So, Vitelia engineers quickly realized they had to make the overlay carry multiple VRFs. With the current set of protocols, this was not possible, so they decided to change the routing concept and create three new BGP address families that advertise more information. But you cannot simply create a new BGP RFC - it takes years. That's why they forked BGP into a new protocol called OMP and created three new address families: vRoute, TLOC, and Service. That's how OMP was born.
OMP is essentially BGP in disguise. It is also a good starting point to think about it as iBGP with three new address families.
DMVPN issue N4 - Complexity
The last major challenge grew from the first three: managing a DMVPN network with thousands of branches was very complex and time-consuming. All routers need to be configured manually. All security settings need to be audited manually, etc.
Yes, the network team could use scripts and remote setup tools, but DMVPN itself did not bring all those functions together in one system. The network team had to build and maintain its own scripts and tools.
The ex-Cisco engineers quickly realized that large-scale WAN needs centralized management; otherwise, the management plane (e.g., the network team) becomes the bottleneck. That's how the vManage controller was invented, as shown in the diagram below.
The vManage controller is a virtual machine that runs on cheap commodity hardware and scales with the network. It is the central management platform that lets the network team configure, monitor, and troubleshoot the network through a single web interface.
And that is how the first version of Viptela SD-WAN came to life. Of course, this is not the full story, but the idea is to show that most SD-WAN concepts and technologies are a natural evolution of previous large-scale DMVPN designs.
Viptela was founded in 2012. Its founders had years of experience in the Cisco advanced routing and WAN design departments. Their team knew the demands of large networks and the problems customers faced when running them. Every controller and protocol in Viptela SD-WAN was invented to solve inefficiencies in the legacy DMVPN design.