This lesson will explore a Cisco SD-WAN functionality called Dynamic On-demand Tunnels. We will see when this feature could come in handy when designing the overlay topology and how we configure on-demand tunnels.
The Business Need
It is very common for organizations to deploy IPsec tunnels in a full-mesh SD-WAN overlay to provide a low-latency path to voice and video services. In most cases, this leads to many idle tunnels between remote sites that rarely or never communicate with each other. Additionally, when an organization uses less-powerful WAN edge platforms such as Cisco ISR 1100 series on spoke sites, the full-mesh topology becomes a scaling limitation because, at some point, the routers can't handle the thousands of IPsec tunnels required.
There are three solutions to this scaling problem:
- Changing the topology to hub-and-spoke: This is the easiest way to scale up the overlay fabric. However, spokes aren't able to communicate with other spokes via low-latency direct tunnels (all traffic goes through the hub).
- Manually adjusting the topology to custom partial mesh: This option would require a complex control-plane policy and won't scale efficiently. Additionally, it is hard to predict which spokes communicate with each other in many scenarios, so it is very hard to determine which sites to set up direct tunnels between.
- Using dynamic on-demand tunnels: The most efficient option to both scale up the overlay and keep having a direct low-latency path between remote sites is by using on-demand tunnels. The topology is hub-and-spoke, but spoke-to-spoke traffic triggers the WAN edge routers to set up a temporary direct IPsec tunnel between the two sites. (similarly to DMVPN Phase 3)
Figure 1 illustrates a typical use-case of the Cisco SD-WAN Dynamic On-demand Tunnels.
What are On-demand Tunnels?
On-demand Tunnels are a Cisco SD-WAN functionality that triggers the establishment of an IPsec overlay tunnel between two sites only when there is traffic between the WAN edger routers. After the flow of direct traffic stops and the pre-defined idle-timer expires, the tunnel is placed in an inactive state. In this state, the tunnel doesn't consume bandwidth or CPU and therefore does not affect the router's performance. However, the on-demand tunnel can quickly be brought back up again when direct traffic re-occurs.
In summary, dynamic spoke-to-spoke tunnels allow organizations to use less expensive, less powerful WAN edge routers and, at the same time, be able to scale up the overlay and use direct low-latency paths between remote sites.
How do On-demand Tunnels work?
We could break the process of enabling this functionality within an SD-WAN overlay into four distinct configuration steps:
The first step in setting up on-demand tunnels is to configure the control plane properly. There are two essential pre-requisites when enabling the functionality in Cisco SD-WAN:
- All spoke sites must receive all TLOCs and vroutes of all other spokes. This practically means that the initial topology (before enabling on-demand functionality) must be a full-mesh and not hub-and-spoke.
- All spoke vRoutes must have a backup path set up through tloc-action in a centralized control policy, as shown in the output below. The ultimate tloc represents the potential direct path to reach this prefix directly.
Figure 2 illustrates both pre-requisites in a simple example. Notice that the topology is full-mesh (there are tunnels between vEdge-4 and vEdge-5) and also that spokes have a backup path for the prefixes of other spokes (ultimate tloc with tloc-action backup).
The output below shows an example of a vRoute that has a backup next-hop enforced via a centralized control policy. The next-hop tloc points to the HUB. The ultimate tloc points directly to the remote spoke and represents the direct data plane path to the destination prefix.
--------------------------------------------------- omp route entries for vpn 4 route 10.5.4.0/24 --------------------------------------------------- RECEIVED FROM: peer 18.104.22.168 path-id 53 label 1010 status C,I,R loss-reason not set lost-to-peer not set lost-to-path-id not set Attributes: originator 22.214.171.124 type installed tloc 126.96.36.199, mpls, ipsec ultimate-tloc 188.8.131.52, mpls, ipsec -- backup domain-id not set overlay-id 1 site-id 5 preference not set tag not set origin-proto connected origin-metric 0 as-path not set community not set unknown-attr-len not set
We will see how to configure the control plane in the configuration portion of this lesson.
Okay, we said that the initial topology before implementing the on-demand functionality must be full-mesh (all spokes knowing all TLOCs). But we want to have a hub-and-spoke topology with dynamic spoke-to-spoke tunnels, right? Well, how's that going to happen?
The magic happens in the data plane - once enabled for on-demand, WAN edge routers stop forming tunnels to other routers that are also enabled with the on-demand functionality, as shown in figure 3 above. Hence, if all spokes are enabled with on-demand and the hubs are not, the topology becomes hub-and-spoke because spokes won't form tunnels to other spokes (all are configured with on-demand) but at the same time spokes will form tunnels with the hub (hubs are not enabled for on-demand).
Triggering the on-demand tunnel
The initial traffic between two spoke sites is routed through the static next-hop tloc in the vroute which is pointing to the hub. This forces the traffic to go through the hub.
When a vEdge router is enabled for on-demand tunnels and receives traffic from a prefix that has an ultimate-tloc-backup, it responds back via the hub and simultaneously triggers the provisioning of direct on-demand tunnels to the site sourcing the traffic (not to the individual vEdge!). This is very important - in dual-homed spoke sites, when traffic triggers the on-demand functionality, direct tunnels are formed to all WAN edge routers on the souring spoke site.
For spoke sites with multiple WAN edge routers, it is very important to configure the on-demand functionality on all routers; otherwise, the on-demand feature may behave unpredictably.
Once traffic triggers the setting up on-demand tunnels on both spoke sites, the vEdge routers start to monitor the state of the tunnels.
An on-demand tunnel has two states:
- Active - Once an on-demand tunnel is established, it is placed in an active state. This means that traffic is actively traversing the direct tunnel and BFD is up. In this state, the tunnel activity is constantly observed. When the traffic stops flowing, a pre-defined idle-timer starts (default one is ten minutes). When this timer expires, the tunnel is placed in an Inactive state.
- Inactive - When an on-demand tunnel is placed in an inactive state, it basically means that the tunnel is removed and no BFD probes are being sent for detection. Inactive tunnels do not use any bandwidth or CPU resources of the router.
On-demand Tunnels Benefits
Establishing dynamic spoke-to-spoke tunnels offers many performance advantages over the pure-vanilla hub-and-spoke topology:
- Significantly improved performance in scenarios with less-powerful routers working in a full-mesh overlay.
- Significantly improved latency between spokes.
- Reduced bandwidth usage.
- Reduced CPU and memory usage.
Configuring On-demand Tunnels
Let's now jump into the configuration portion of this lesson and see how we can enable spoke-to-spoke tunnels in our lab topology.
The initial state
For this lab example, we will use the topology shown in figure 6 below. For the purpose of this example, I have reverted the topology back to full-mesh and deleted all centralized control policies applied in previous lessons. The overlay topology is just a default full-mesh SD-WAN fabric.
To verify this, let's check that each router has overlay tunnels to all other sites (full-mesh).
vEdge-4# show bfd sessions | t SYSTEM SITE DETECT TX IP ID LOCAL COLOR COLOR STATE MULTIPLIER INTERVAL UPTIME ------------------------------------------------------------------------------- 184.108.40.206 1 mpls mpls up 7 1000 0:00:00:14 220.127.116.11 1 biz-internet biz-internet up 7 1000 0:00:00:19 18.104.22.168 1 mpls mpls up 7 1000 0:00:00:14 22.214.171.124 1 biz-internet biz-internet up 7 1000 0:00:00:19 126.96.36.199 3 mpls mpls up 7 1000 0:00:00:14 188.8.131.52 3 biz-internet biz-internet up 7 1000 0:00:00:19 184.108.40.206 5 mpls mpls up 7 1000 0:00:00:14 220.127.116.11 5 biz-internet biz-internet up 7 1000 0:00:00:18 18.104.22.168 6 mpls mpls up 7 2000 0:00:00:14 22.214.171.124 6 biz-internet biz-internet up 7 1000 0:00:00:19
Additionally, each prefix is reachable via the direct overlay path to the respective remote site. Therefore, the overlay routing is also "as it is by default".
vEdge-4# show ip routes vpn 4 | t ADDRESS PATH NEXTHOP VPN FAMILY PREFIX ID PROTOCOL IFNAME TLOC IP COLOR ENCAP -------------------------------------------------------------------------------- 4 ipv4 10.1.4.0/24 0 omp - 126.96.36.199 mpls ipsec 4 ipv4 10.1.4.0/24 1 omp - 188.8.131.52 biz-internet ipsec 4 ipv4 10.1.4.0/24 2 omp - 184.108.40.206 mpls ipsec 4 ipv4 10.1.4.0/24 3 omp - 220.127.116.11 biz-internet ipsec 4 ipv4 10.3.4.0/24 0 omp - 18.104.22.168 mpls ipsec 4 ipv4 10.3.4.0/24 1 omp - 22.214.171.124 biz-internet ipsec 4 ipv4 10.4.4.0/24 0 connected ge0/4 - - - 4 ipv4 10.5.4.0/24 0 omp - 126.96.36.199 mpls ipsec 4 ipv4 10.5.4.0/24 1 omp - 188.8.131.52 biz-internet ipsec 4 ipv4 10.6.4.0/24 0 omp - 184.108.40.206 mpls ipsec 4 ipv4 10.6.4.0/24 1 omp - 220.127.116.11 biz-internet ipsec
There is nothing fancy about the initial topology, so let's go ahead and demonstrate the Cisco SD-WAB on-demand functionality.
Step 1: Configuring the control plane
The first step in enabling the feature is to configure the control plane. We will need to provision a new centralized control policy that includes the tloc-action backup. This will, later on, ensure that spokes know that a direct path to the destination prefix exists so they can trigger the establishment of on-demand tunnels.
Full Content Access is for Registered Users Only (it's FREE)...
- Learn any CCNA, DevNet or Network Automation topic with animated explanation.
- We focus on simplicity. Networking tutorials and examples written in simple, understandable language for beginners.