The Default Overlay Fabric

As we have already discussed throughout this course - the default behavior of the Cisco SD-WAN overlay fabric is to build a full mesh of IPsec tunnels between all WAN edge routers with different site-ids. Let's visualize this with the example shown in figure 1. If we have six WAN edge routers connected to a single WAN transport as shown on the left, the default overlay outcome with no policies applied results in 15 IPsec tunnels as shown on the right side. You can easily calculate how many tunnels will be established with the full mesh formula n*(n-1)/2. In case the company has 100 branches, there will be 4950 overlay tunnels.

Cisco SD-WAN Default Overlay Fabric with one transport
Figure 1. Cisco SD-WAN Default Overlay Fabric with one transport
In a real-world example, it is more likely that each vEdge device is connected with at least two underlay transports typically one public internet circuit and one private MPLS one. In this case, there will be twice as many overlay connections as shown in figure 2. In this scenario, both WAN transports are completely independent, meaning that blue TLOCs cannot reach green TLOCs. 

Cisco SD-WAN Default Overlay Fabric with two transports
Figure 2. Cisco SD-WAN Default Overlay Fabric with two transports

At this point, it is important to understand the behavior of the WAN edge routers and what is a TLOC. A TLOC (Transport Locator) is a data plane attachment point identified by a 3 values tuple - (System IP, Color, Encap). TLOCs do not have a straight one to one analog in traditional networking, but the closest simplest explanation is that they are just tunnel endpoints. Therefore, if a vEdge device receives a TLOC (tunnel attachment point) from the vSmart controllers, it attempts to establish a tunnel from each of its own TLOCs to this new TLOC. If the tunnel is successfully built, it establishes a BFD session over the tunnel and starts keeping track of the state and the characteristics of it.

IMPORTANT  The number of established IPsec tunnels and the arbitrary network topology is controlled by the advertisements of Transport Locators (TLOCs) by the vSmart controllers. By default, all TLOCs are advertised to all vEdge routers, resulting in a full mesh overlay fabric. To create a custom network topology, you must filter which TLOCs are advertised to which specific WAN Edge devices.

The next example is not very common but illustrates well the overlay fabric creation. If we have four WAN edge routers attached to two underlay transports as shown on the left in figure 3 and there is full reachability between both clouds, in this case, IPsec tunnels are established between different color TLOCs resulting in the overlay shown on the right. Even with only four sites, there are 24 tunnels, because blue TLOCs can reach green TLOCs, and tunnels between blue and green endpoint are also built. Therefore, if the company has 100 sites, this results in 200*199/2-100 = 39700 IPsec tunnels! If there are two vEdge devices per site, the number is doubled, and so on.

Cisco SD-WAN Full MEsh Overlay Fabric
Figure 3. Cisco SD-WAN Full MEsh Overlay Fabric

You can clearly see that in some rare cases, this could be the desired topology, but in most mid-sized and enterprise networks, there is little need to have direct branch-to-branch communication. There is also a scaling limitation because WAN edge devices at the remote sites are typically not sized to handle hundred of thousands of IPsec tunnels and BFD sessions. A better more practical design approach is the use of a Hub-and-Spoke topology.

Configuring a Centralized Policy

In Cisco SD-WAN, to implement hub and spoke topology means to restrict the spoke-to-spoke overlay connections.  To do this, a centralized policy must be created and applied so that the remote sites will only receive the Transport Locators of the data center WAN Edges from the vSmart controllers. Each vEdge router attempts to establish a tunnel to all known TLOCs, so the goal is to make the vSmart advertise only the data center TLOCs toward the branches. 

For this set of lab examples, we are going to use the topology shown in figure 4. 

Cisco SD-WAN Deep-Dive - Main Lab Topology
Figure 4. Cisco SD-WAN Deep-Dive - Main Lab Topology

The end goal after applying the centralized policy is to have the data center Site-50 act as a hub and all other sites (60,70,80,90) act as spokes and establishing tunnels to the DC only and not in between them.

To understand the overlay fabric at the moment, let's check how many BFD sessions one of the spoke routers has. This is the easiest way to quickly check how many operational IPsec tunnel a WAN edge device has.

vEdge-4# show bfd sessions
                                SOURCE TLOC      REMOTE TLOC                       
SYSTEM IP      SITE ID STATE  COLOR            COLOR            SOURCE IP   DST IP   
-------------------------------------------------------------------------------------
50.50.50.50    50      up     mpls             mpls             10.70.1.1   10.50.1.1 
50.50.50.50    50      up     mpls             public-internet  10.70.1.1   50.1.1.1  
50.50.50.50    50      up     public-internet  mpls             70.1.1.1    10.50.1.1 
50.50.50.50    50      up     public-internet  public-internet  70.1.1.1    50.1.1.1  
50.50.50.51    50      up     mpls             mpls             10.70.1.1   10.50.2.1 
50.50.50.51    50      up     mpls             public-internet  10.70.1.1   50.1.2.1  
50.50.50.51    50      up     public-internet  mpls             70.1.1.1    10.50.2.1 
50.50.50.51    50      up     public-internet  public-internet  70.1.1.1    50.1.2.1  
60.60.60.60    60      up     mpls             lte              10.70.1.1   60.1.1.1  
60.60.60.60    60      up     public-internet  lte              70.1.1.1    60.1.1.1  
80.80.80.80    80      up     mpls             mpls             10.70.1.1   10.80.1.1 
80.80.80.80    80      up     mpls             public-internet  10.70.1.1   80.1.1.1  
80.80.80.80    80      up     public-internet  mpls             70.1.1.1    10.80.1.1 
80.80.80.80    80      up     public-internet  public-internet  70.1.1.1    80.1.1.1  
90.90.90.90    90      up     mpls             mpls             10.70.1.1   10.90.1.1 
90.90.90.90    90      up     public-internet  mpls             70.1.1.1    10.90.1.1

You can see based on the site-id, that vEdge-4 has overlay connections to all other spoke sites (highlighted in yellow). If we successfully configure the hub-and-spoke topology, we should only see connections to Site-50, which would be the hub site. 

The first step to constructing the Centralized Control Policy is to use the Policy Wizard in Configuration > Policies as shown below.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 1
Figure 5. Lab1 - Restricting Spoke-to-Spoke tunnels Step 1

From inside the Centralized Policy Wizard, we need to create two Lists that will be used to match the hub and spokes site-ids. Lists are very simple constructs that are used to match specific values. In Cisco SD-WAN, lists work the same way as they do in traditional networking where we use access-lists and prefix-lists to match specific routes and then specify the action in another construct called a route-map.

For this example, we create one list called Hub that matches site-id 50 and one called Spokes that matches the site-ids of all remote sites - 60,70,80,90.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 2
Figure 6. Lab1 - Restricting Spoke-to-Spoke tunnels Step 2

After clicking Next, the wizard moves to the next page called "Configure Topology and VPN Membership". There we select the Custom Control (Route and TLOC) from the Add Topology drop-down menu. You can see in the available options that there is a pre-defined Hub-and-Spoke topology, but to better understand the principles of the Centralized policies, it is better to go down this route.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 3
Lab1 - Restricting Spoke-to-Spoke tunnels Step 3

On the next page, the first thing that we have to specify is the Name and Description of the Control Policy. Composing good and scalable policy names is a skill in itself. A general rule of thumb is to use all capital letters and always include a version number. 

Next, we click Sequence Type and select TLOC to create a policy that applies to TLOCs.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 4
Lab1 - Restricting Spoke-to-Spoke tunnels Step 4

At this point, we should add a new Sequence Rule that matches the Hub sites and then specify Action: Accept. The idea is that the control policy has a default action Reject at the end similarly to Cisco IOS access-lists having explicit deny all at the end. Therefore, the idea is to explicitly match the TLOCs of the Hub site and accept them, and all other TLOCs (of the spoke sites) will be rejected by the default action. In the end, the spokes will only know the TLOCs of the hub and won't establish IPsec tunnels to other spokes.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 5
Lab1 - Restricting Spoke-to-Spoke tunnels Step 5

As we explained above, after we match the hub sites, we specify the Accept action.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 6
Lab1 - Restricting Spoke-to-Spoke tunnels Step 6

In the end, on the Configure Topology and VPN Membership page, you should see the control policy we have just created. If that is the case, we click Next and go to the Configure Traffic Rules page. For this lab, there is nothing that needs to be configured to this page, so we click Next and go to Apply Policies to Sites and VPNs page.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 7
Lab1 - Restricting Spoke-to-Spoke tunnels Step 7

The first piece of configuration required on this page is the policy name and description. Then under the Topology tab, we must specify where and in what direction the FILTER-OUT-SPOKES-TLOCS-V1 control policy will be applied. In our case, we apply it in the outbound direction to the spoke sites.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 8
Lab1 - Restricting Spoke-to-Spoke tunnels Step 8

In the end, we should see the Centralized Policy created but not activated. The last thing we must do is to activate it as shown in the screenshot below.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 9
Lab1 - Restricting Spoke-to-Spoke tunnels Step 9

Note that the vSmart won't accept the policy if it is not in a "vManaged mode".  This means that the vSmart must have a template applied from vManage. The type of the template does not matter, it could be a Device template or a Feature template, but a template has to be applied. This enables vManage to have authoritative control over the vSmarts. It is common for production environments to initially deploy the vSmart controllers with a CLI template from vManage, so the controllers are in "vManaged mode" from the get-go.

In our lab example, the vSmart has been initially deployed with a CLI template so vManage can successfully push the Centralized Policy to the controllers.

Lab1 - Restricting Spoke-to-Spoke tunnels Step 10
Lab1 - Restricting Spoke-to-Spoke tunnels Step 10

At this point, the overlay fabric must be analogous to a traditional Hub and Spoke topology. We can now check again the active BDF sessions on one of the spokes WAN edge devices. If our control policy is correct and has taken effect, we must see IPsec tunnels to the hub site only (site-id 50) and not to other spokes (sites 60,70,80,90). 

vEdge-4# show bfd sessions
                                SOURCE TLOC      REMOTE TLOC                       
SYSTEM IP      SITE ID STATE  COLOR            COLOR            SOURCE IP   DST IP   
-------------------------------------------------------------------------------------
50.50.50.50    50      up     mpls             mpls             10.70.1.1   10.50.1.1 
50.50.50.50    50      up     mpls             public-internet  10.70.1.1   50.1.1.1  
50.50.50.50    50      up     public-internet  mpls             70.1.1.1    10.50.1.1 
50.50.50.50    50      up     public-internet  public-internet  70.1.1.1    50.1.1.1  
50.50.50.51    50      up     mpls             mpls             10.70.1.1   10.50.2.1 
50.50.50.51    50      up     mpls             public-internet  10.70.1.1   50.1.2.1  
50.50.50.51    50      up     public-internet  mpls             70.1.1.1    10.50.2.1 
50.50.50.51    50      up     public-internet  public-internet  70.1.1.1    50.1.2.1  

Obviously, vEdge-4 has overlay connections only to the data center site 50.

Key Takeaways

Let's recap what we have done in this lab and highlight the key takeaways from this lesson. Use figure 15 for a reference.

We created a Centralized Policy using the vManage GUI. In this policy, we have created another Control Policy that matches the hub TLOCs and accepts them, all other TLOCs are rejected by the default action at the end - Reject. After that, we applied this policy in an outbound direction to the spoke sites. Note that the OUT direction is from the perspective of the OMP updates of the vSmart controller to the WAN edge devices. So what happened after we activated the policy. As it is shown in figure 15, vManage pushed this Centralized policy as a NETCONF transaction to the vSmart controllers. 

Cisco SD-WAN Policy Direction
Figure 5. Cisco SD-WAN Policy Direction

If we check the policy in the running configuration of the vSmart, it looks like the output shown below. Note that it has a very similar construct as the route-maps in Cisco IOS. We have lists that match something (in green). In route-maps, we typically use access-lists, prefix-lists, or as-path lists to specify values of interest.

Then we have a sequence with a match-action construct (in yellow). In Cisco IOS we have route-maps with similar match-set logic, for example, match ACL1 set local-preference 110.

And then we apply this match-action construct somewhere (in orange). In the case of Cisco SD-WAN, we have applied it to the OMP updates in the outbound direction. In the case of a typical route-map, we apply it on a BGP neighbor for example.

!
policy
 lists
  site-list HUB
   site-id 50
  !
  site-list SPOKES
   site-id 60
   site-id 70
   site-id 80
   site-id 90
  !
 !
 control-policy FILTER-OUT-SPOKES-TLOCS-V1
  sequence 1
   match tloc
    site-list HUB
   !
   action accept
   !
  !
  default-action reject
 !
!
apply-policy
 site-list SPOKES
  control-policy FILTER-OUT-SPOKES-TLOCS-V1 out
 !
!

You can see that there is nothing quite new in the way the overlay fabric is configured with Centralized Policies, only the underlying mechanisms are different. The logical construct is pretty much the same as it has always been in traditional networking. 

Comments

dougwillik

Tue, 04/27/2021 - 04:25

beautiful and very in-depth deep-dive to the inner-workings of SD-WAN policy definitions . The best i have seen so far. So glad i found this content . Great Job .!! five stars

prasad.digraskar

Thu, 06/24/2021 - 06:49

Hi Ivan, Yes TLOC advertisement is blocked by vSmart and the only edge has a BFD session with Hub. but after applying policy o lost my DC & edge vRoutes. I think I should configure Hub_And_Spoke routing. Correct me if I am wrong

In reply to by prasad.digraskar

Ivan.Ivanov

Thu, 06/24/2021 - 10:36

Hi Prasad, the vRoute advertisements hit the default reject at the end of the policy and get dropped at vSmart. This lab covers how to control the topology, in the next lab I have explained how you can enable the routing. Hope this help!

shoaib.020@gmail.com

Fri, 09/10/2021 - 15:16

Great explanation. Thank you very much

Rynqlo

Tue, 10/19/2021 - 06:54

The example is great!

ciscouser

Mon, 12/27/2021 - 02:24

Can you share the EVE lab file? or dev some labs for study

Aida

Tue, 01/04/2022 - 00:27

Much appreciated !!

vektorprime

Thu, 04/07/2022 - 15:12

Typo: math the TLOCs

vektorprime

Thu, 04/07/2022 - 15:15

and another "deeply"