This lesson will make the VPN5's topology a full-mesh while at the same time VPNs 1 through 4 a hub-and-spoke. It is a typical use case where an organization wants to have its data services transit a regional hub site. At the same time, real-time services such as voice and video communicate directly through the fastest possible path.

Per VPN Topologies

You probably have heard multiple times that “Cisco SD-WAN allows us to create arbitrary custom topologies per VPN,” as shown in figure 1 below.

This statement mistakenly leads some network engineers into thinking that the overlay IPsec tunnels are established “per VPN” and that there could be different overlay topologies for each VPN. Let me put it straight from the very beginning - there is no such thing as different per-VPN overlay topologies. The overlay fabric has nothing to do with VPNs. Controlling the topology in Cisco SD-WAN is a function of deciding which TLOCs the vSmart controller sends to which particular vEdge routers.TLOC routes are always on the transport side of the network (VPN 0) and are not associated with any service VPN. Therefore, we can't do any “per-VPN” manipulation to a TLOC and subsequently to the topology of overlay tunnels. The VPN id is just a label in the packets going through the overlay tunnels, as shown in figure 2 below.

Okay, so everything is a marketing trick then? Well, what we can do “per VPN” is to manipulate the OMP routing in such a way that different service VPNs take different paths in the network. This is possible because vRoutes are always associated with a service VPN. Therefore, we can match vroutes based on a VPN-id and then change their next hop. An example is shown in the output below:

sequence 1
 match route
  vpn 4
 action accept 
  set tloc-list HUB-TLOCS 

In reality, creating different logical per-VPN topologies is just that - manipulating the routing on a per-VPN basis.

However, the topology of IPsec/GRE tunnels is only one and sits on the transport side of the network; hence has nothing to do with the service VPNs.

Initial Lab state

Ok, so back to the lab lesson. To achieve the goal of this example, we need to make sure that VPN5 communicates directly in a full-mesh manner and VPNs 1-4 communicate through the hub. This is not possible with the current state of our lab setup because the transport side topology is hub-and-spoke from the previous lessons, and there aren't direct spoke-to-spoke tunnels at all. Therefore, no VPN can communicate directly between spokes because there aren't IPsec tunnels to go through. We can verify this on any of the spokes (for example, on vEdge-6). You can see that there are overlay tunnels to the hub’s vEdges only (site-id 1).

vEdge-6# show bfd sessions | t

                             SYSTEM   SITE                                  
SRC IP     DST IP     PROTO  IP       ID   LOCAL COLOR   COLOR         STATE
----------------------------------------------------------------------------
10.10.0.4  10.10.0.2  ipsec  1.1.1.1  1    mpls          mpls          up   
39.3.0.4   39.3.0.2   ipsec  1.1.1.1  1    biz-internet  biz-internet  up   
10.10.0.4  10.10.0.1  ipsec  1.1.1.2  1    mpls          mpls          up   
39.3.0.4   39.3.0.1   ipsec  1.1.1.2  1    biz-internet  biz-internet  up  

This is due to the control policy applied toward the spoke sites. In sequences 1 and 2, we accept only the hub’s TLOCs, and all other tloc routes get rejected by the default action at the end of the policy. 

control-policy OUTBOUND-TO-SPOKES
  sequence 1
   match tloc
    tloc-list VEDGE1-TLOCS
   !
   action accept
    set
     preference 80
  !
  sequence 2
   match tloc
    tloc-list VEDGE2-TLOCS
   !
   action accept
  !
  sequence 10
   match route
    prefix-list 10-1-1-0/24
    tloc 1.1.1.2 color mpls encap ipsec
   !
   action accept
    set
     preference 90
  !
  sequence 11
   match route
    site-list HUB
   !
   action accept
   !
  !
  sequence 21
   match route
    site-list SPOKES
   !
   action accept
    set
     service FW
  !

To ensure that VPN 5 is a full-mesh, we must change the overlay topology to full-mesh. We can do this by removing sequences 1 and 2 from the control policy and configuring a new sequence 1 that accepts all tloc routes. This change will ensure that the spokes receive all tloc routes and create a full mesh of overlay tunnels between all vEdges.