This lesson will demonstrate a route-leaking scenario that is not common in real-world deployments but is very instructive. It will show you how VPNs and LABELs work in the context of route leaking. By the end, spokes will communicate in VPN 5 via the hub, with the hub not having VPN 5 configured.

Scenario Overview

An organization has a hub-and-spoke SD-WAN network with two transport colors - biz-internet and mpls(restricted). The hub has only VPN3 configured, while the spokes have only VPN5. Figure 1 shows an example where vEdges 3-5 are spokes and vEdge-1 is the hub.

Notice that the mpls color is restricted, and only two combinations of overlay tunnels are formed: biz-internet <-> biz-internet and mpls <-> mpls.

Scenario Requirements

You are tasked to configure the network in such a way so that it satisfies the following requirements:

• Spokes must be able to communicate with other spokes via the hub.
• You are not allowed to configure VPN 5 on the hub.
• You are not allowed to configure VPN 3 on spokes.

Figure 2 illustrates the scenario requirements. Notice that vEdges 3, 4, and 5 have only one VPN - 5, and vEdge-1 has only one VPN configured - 3.

Initial Topology and Configs

We will use the topology illustrated in Figure 2 to demonstrate a solution to the scenario.

The initial configuration of each SD-WAN device can be found on the section page here. We will only use vEdges 1, 3, 4, and 5.

The only difference is that we will leave only VPN 3 on the hub and VPN 5 on the spokes.

Configuring Route Leaking

Before configuring route leaking between VPNs 3 and 5, we need to change the default full-mesh topology to hub-and-spoke.

Adjusting the topology to hub-and-spoke

We will need the following policy lists that we will invoke in the control policy that adjusts the overlay topology to hub-and-spoke.

policy
 lists
  site-list HUB
   site-id 1
  !
  site-list SPOKES
   site-id 3-5
 !
  tloc-list HUB-TLOCS
   tloc 1.1.1.1 color mpls encap ipsec
   tloc 1.1.1.1 color biz-internet encap ipsec
  !

And then, we will need to define a control policy that only sends the hub's tlocs to the spokes (seq 11), making the topology hub-and-spoke. Also, we will need to change the next-hop of spokes' vroutes to point to the hub (seq 21).

 control-policy VSMART-TO-SPOKES
  sequence 11
   match tloc
    site-list HUB
   !
   action accept
  !
  sequence 21
   match route
    site-list SPOKES
   !
   action accept
    set
     tloc-list HUB-TLOCS
  !
  sequence 31
   match route
    site-list HUB
   !
   action accept
  !
  default-action reject
!

There is nothing unusual with the control policy, so we can just go ahead and apply it in an outbound direction to spokes.

apply-policy
 site-list SPOKES
  control-policy VSMART-TO-SPOKES out
 !
!

Once the policy is applied on vSmart, we can verify the topology on any spoke. For example, if we check on vEdge-3, we will see that it has only established an overlay tunnel to the hub (vEdge-1), so the topology is already hub-and-spoke.

vEdge-3# show bfd sessions | t

SYSTEM   SITE                                     DETECT      TX                  
IP       ID    LOCAL COLOR   COLOR         STATE  MULTIPLIER  INTERVAL  UPTIME    
----------------------------------------------------------------------------------
1.1.1.1  1     mpls          mpls          up     7           1000      0:00:11:09
1.1.1.1  1     biz-internet  biz-internet  up     7           1000      0:00:11:09

However, if we check the routing on any of the spokes, we will see that spokes do not receive routes of other spokes. For example, you can see that vEdge-3 hasn't received any VPN5 routes and only knows about its directly connected network. 

vEdge-3# show omp routes vpn 5 | t

                                                  ATTRIBUTE                               
VPN    PREFIX         FROM PEER  LABEL  STATUS    TYPE       TLOC IP   COLOR         ENCAP
------------------------------------------------------------------------------------------
5      10.3.5.0/24    0.0.0.0    1012   C,Red,R   installed  3.3.3.3   mpls          ipsec
                      0.0.0.0    1012   C,Red,R   installed  3.3.3.3   biz-internet  ipsec

So the control policy that we have just applied to spokes has broken the OMP routing between spokes. Spokes don't receive routes of other spokes in VPN5. Can you understand why that happens? We will get to it a little bit later.

Now let's go ahead and configure the route leaking between VPN3 and VPN5.

Configuring Route Leaking 

In Cisco SD-WAN, we leak routes from one VPN into another using an Inbound Control Policy. We match the routes that we want to leak, and then we tell vSmart to copy them into another VPN routing table by using the "export-to" command. In our particular example, we will need two inbound control policies - one applied to the routes coming from the hub and one applied to the routes coming from the spokes. Let's go ahead and configure both policies.

policy
 control-policy HUB-TO-VSMART
  sequence 11
   match route
    vpn 3
   !
   action accept
    export-to
     vpn 5
    !
  !
  default-action accept
 !
 control-policy SPOKES-TO-VSMART
  sequence 11
   match route
    vpn 5
   !
   action accept
    export-to
     vpn 3
    !
  !
  default-action accept
 !

The policies are pretty simple and self-explanatory. Notice that the default action on both policies is "accept."

Now let's go ahead and apply the policies in an inbound direction from the perspective of vSmart.

apply-policy
 site-list HUB
  control-policy HUB-TO-VSMART in
 !
 site-list SPOKES
  control-policy SPOKES-TO-VSMART in
  control-policy VSMART-TO-SPOKES out
 !
!

Once the policies are applied to the respective site lists, we can go ahead and check the result.

Route Leaking Control-plane

One of the most essential aspects of route leaking is that route leaking between VPN routing tables only happens on vSmart! Many network engineers tend to think that some route redistribution happens on the WAN routers themselves, which is wrong. Figure 3 shows an overview of the route leaking process.

We can verify that VPN5's routes are copied into VPN3's routing table on vSmart.

vSmart# show omp routes vpn 3 | t
Code:
C   -> chosen
I   -> installed
Red -> redistributed
Rej -> rejected
L   -> looped
R   -> resolved
S   -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA  -> On-demand inactive
U   -> TLOC unresolved

                                           ATTRIBUTE                              
VPN PREFIX       FROM PEER  LABEL STATUS   TYPE       TLOC IP  COLOR         ENCAP
----------------------------------------------------------------------------------
3   10.1.3.0/24  1.1.1.1    1008  C,R      installed  1.1.1.1  mpls          ipsec
                 1.1.1.1    1008  C,R      installed  1.1.1.1  biz-internet  ipsec
3   10.3.5.0/24  3.3.3.3    1012  C,R,Ext  original   3.3.3.3  mpls          ipsec
                                           installed  3.3.3.3  mpls          ipsec
                 3.3.3.3    1012  C,R,Ext  original   3.3.3.3  biz-internet  ipsec
                                           installed  3.3.3.3  biz-internet  ipsec
3   10.4.5.0/24  4.4.4.4    1019  C,R,Ext  original   4.4.4.4  mpls          ipsec
                                           installed  4.4.4.4  mpls          ipsec
                 4.4.4.4    1019  C,R,Ext  original   4.4.4.4  biz-internet  ipsec
                                           installed  4.4.4.4  biz-internet  ipsec
3   10.5.5.0/24  5.5.5.5    1005  C,R,Ext  original   5.5.5.5  mpls          ipsec
                                           installed  5.5.5.5  mpls          ipsec
                 5.5.5.5    1005  C,R,Ext  original   5.5.5.5  biz-internet  ipsec
                                           installed  5.5.5.5  biz-internet  ipsec

VPN5's routes are seen as Extranets in the VPN3 routing table, but they still point to the VPN5 label of the next-hop router. 

On the other hand, VPN3's routes are seen in VPN5's routing table as Extranets, but they still point to the VPN3 label of the next-hop router. 

vSmart# show omp routes vpn 5 | t
Code:
C   -> chosen
I   -> installed
Red -> redistributed
Rej -> rejected
L   -> looped
R   -> resolved
S   -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA  -> On-demand inactive
U   -> TLOC unresolved

                                          ATTRIBUTE                             
VPN PREFIX       FROM PEER LABEL  STATUS  TYPE       TLOC IP  COLOR        ENCAP
--------------------------------------------------------------------------------
5   10.1.3.0/24  1.1.1.1   1008   C,R,Ext original   1.1.1.1  mpls         ipsec
                                          installed  1.1.1.1  mpls         ipsec
                 1.1.1.1   1008   C,R,Ext original   1.1.1.1  biz-internet ipsec
                                          installed  1.1.1.1  biz-internet ipsec
5   10.3.5.0/24  3.3.3.3   1012   C,R     installed  3.3.3.3  mpls         ipsec
                 3.3.3.3   1012   C,R     installed  3.3.3.3  biz-internet ipsec
5   10.4.5.0/24  4.4.4.4   1019   C,R     installed  4.4.4.4  mpls         ipsec
                 4.4.4.4   1019   C,R     installed  4.4.4.4  biz-internet ipsec
5   10.5.5.0/24  5.5.5.5   1005   C,R     installed  5.5.5.5  mpls         ipsec
                 5.5.5.5   1005   C,R     installed  5.5.5.5  biz-internet ipsec

So from a control-plane point of view, we have complete routing information in both VPNs' routing tables. Let's go ahead and check whether there is connectivity between VPN3 and VPN5 and how the data plane works.

Route Leaking Data-plane

If we ping one of the spokes from the hub, we will see that there is connectivity between the hub's VPN3 and the spokes' VPN5. Therefore the route leaking works as expected.

vEdge-1# ping vpn 3 10.3.5.1
Ping in VPN 3
PING 10.3.5.1 (10.3.5.1) 56(84) bytes of data.
64 bytes from 10.3.5.1: icmp_seq=1 ttl=64 time=48.4 ms
64 bytes from 10.3.5.1: icmp_seq=2 ttl=64 time=40.8 ms
64 bytes from 10.3.5.1: icmp_seq=3 ttl=64 time=43.3 ms
64 bytes from 10.3.5.1: icmp_seq=4 ttl=64 time=44.1 ms
64 bytes from 10.3.5.1: icmp_seq=5 ttl=64 time=39.9 ms
^C
--- 10.3.5.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 39.987/43.349/48.433/2.978 ms

Pay close attention to how the route leaking works in the data plane. Let's use the communication between vEdge-1's VPN3-10.1.3.0/.24 and vEdge-4's VPN5-10.4.5.0/24. When vEdge-1 makes an IP lookup in its VPN3 routing table for destination 10.4.5.0/24, it understands that the prefix is reachable via tlocs (4.4.4.4, biz-internet) and (4.4.4.4, mpls) using label 1019. When vEdge-4 receives the packets, label 1019 tells the router to make an IP lookup in VPN5's routing table!  

The same process happens in the opposite direction, as shown in figure 4 above. The label is what actually leaks the traffic from one local VPN into another remote VPN in the data plane.

The Problem

We have seen that the route leaking between VPN3 and VPN5 works as expected, and there is connectivity between the two segments. However, there is still no connectivity between the spokes in VPN5.

vEdge-3# ping vpn 5 10.5.5.1
Ping in VPN 5
PING 10.5.5.1 (10.5.5.1) 56(84) bytes of data.
From 127.1.0.2 icmp_seq=1 Destination Net Unreachable
From 127.1.0.2 icmp_seq=2 Destination Net Unreachable
From 127.1.0.2 icmp_seq=3 Destination Net Unreachable
From 127.1.0.2 icmp_seq=4 Destination Net Unreachable
^C
--- 10.5.5.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3103ms

And as you can see, the spokes do not receive the VPN5 routes from other spokes. For example, vEdge-3 only knows about its own directly connected prefix in VPN5 and does not know about vEdge-4 and vEdge-5's VPN5 prefixes.

vEdge-3# show omp routes vpn 5 | t

                                                  ATTRIBUTE                               
VPN    PREFIX         FROM PEER  LABEL  STATUS    TYPE       TLOC IP   COLOR         ENCAP
------------------------------------------------------------------------------------------
5      10.1.3.0/24    1.1.1.30   1008   C,I,R     installed  1.1.1.1   mpls          ipsec
                      1.1.1.30   1008   C,I,R     installed  1.1.1.1   biz-internet  ipsec
5      10.3.5.0/24    0.0.0.0    1012   C,Red,R   installed  3.3.3.3   mpls          ipsec
                      0.0.0.0    1012   C,Red,R   installed  3.3.3.3   biz-internet  ipsec

However, if we check on vSmart, we will find out that the controller has received all spoke routes.

vSmart# show omp routes vpn 5 | t

                                                ATTRIBUTE                               
VPN    PREFIX       FROM PEER  LABEL  STATUS    TYPE       TLOC IP   COLOR         ENCAP
----------------------------------------------------------------------------------------
5      10.1.3.0/24  1.1.1.1    1008   C,R,Ext   original   1.1.1.1   mpls          ipsec
                                                installed  1.1.1.1   mpls          ipsec
                    1.1.1.1    1008   C,R,Ext   original   1.1.1.1   biz-internet  ipsec
                                                installed  1.1.1.1   biz-internet  ipsec
5      10.3.5.0/24  3.3.3.3    1012   C,R       installed  3.3.3.3   mpls          ipsec
                    3.3.3.3    1012   C,R       installed  3.3.3.3   biz-internet  ipsec
5      10.4.5.0/24  4.4.4.4    1019   C,R       installed  4.4.4.4   mpls          ipsec
                    4.4.4.4    1019   C,R       installed  4.4.4.4   biz-internet  ipsec
5      10.5.5.0/24  5.5.5.5    1005   C,R       installed  5.5.5.5   mpls          ipsec
                    5.5.5.5    1005   C,R       installed  5.5.5.5   biz-internet  ipsec

Obviously, vSmart does not advertise the VPN5 routes to spokes. The output below shows that vSmart only advertises the prefix 10.1.3.0/24 (the hub's directly connected subnet) to spokes. Why does that happen?

vSmart# show omp routes vpn 5 advertised | t

VPN    PREFIX              TO PEER          
--------------------------------------------
5      10.1.3.0/24         3.3.3.3          
                           4.4.4.4          
                           5.5.5.5          

The answer is that "vSmart is smart" and doesn't advertise spoke routes to other spokes to prevent black holes in the data plane. The problem is that in the outbound control policy to spokes, we are changing the next-hops of spoke routes to point to the hub. These routes are in VPN5 and have labels pointing to VPN5. However, the hub doesn't have a label for VPN5. If vSmart advertises the spoke routes to other spokes, these routes will point to unexisting labels on the hub, and the traffic will be blackholed. 

Figure 5 illustrates what would have happened had the vSmart controller advertised the spoke routes to other spokes. The Smart controller does not advertise OMP routes if their next-hop router doesn't have a label for the respective VPN. For example, vSmart won't advertise 10.3.5.0/24 to vEdge-4 because the route points to next-hop router vEdge-1, and vEdge-1 doesn't have a label for VPN5.