The Business Need

Cisco SD-WAN's device portfolio includes WAN edge routers that support WAN connections over 3G/4G LTE. This is a great option in remote areas where Internet circuits are expensive or not available

However, 3G/4G LTE is not a service provider leased line and is not designed for communicating a large amount of data at a constant rate 24/7. In many parts of the world, an LTE SIM card comes with a data limit that only allows for a certain volume of data to be sent over the LTE line per month. After the data limit is exhausted, either the radio link speed is greatly decreased or there are additional charges for provisioning additional data.

Therefore, in many real-world deployments, where we have a remote site connected to two WAN transports, one of which is LTE, we would generally like to use the LTE radio link only in case the other transport goes down.

Why do we need a Last Resort Circuit
Figure 1. Why do we need a Last Resort Circuit

One way of offloading the traffic from the LTE link is by configuring a higher TLOC preference and higher WEIGHT to the primary WAN transport. This will make sure that in normal circumstances, most of the traffic will pass through the 'primary' tunnel. However, this is not an optimal solution, because even the IPsec tunnel to the LTE TLOC is generating constant traffic. There is a BFD session that exchanges keepalives every second (as shown in figure 1) and there are DTLS control connections via which the vEgde constantly pings the controllers (as shown in figure 2) 

These control/overlay connections will still consume a lot of data, even though application traffic does not go over this TLOC.

Control Connections over 4G/LTE
Figure 2. Control Connections over 4G/LTE

Let's verify that on vEdge-3 using the CLI. You can see that there is a BFD session that is UP and the TX interval time is 1 second. Therefore, each second there will be at least two BFD probes to this TLOC (one originated by vEdge-1 and one by vEdge-3). But what if there are multiple WAN edge routers and there are many BFD sessions? Depending on the 4G LTE plan, this may not be very efficient and consume a lot of data unnecessarily.

vEdge-3# show bfd sessions | tab

                              SRC    DST    SYSTEM    SITE  LOCAL                          DETECT      TX                                 
SRC IP     DST IP      PROTO  PORT   PORT   IP        ID    COLOR  COLOR            STATE  MULTIPLIER  INTERVAL  UPTIME      TRANSITIONS  
------------------------------------------------------------------------------------------------------------------------------------------
10.10.0.1  10.10.1.51  ipsec  12386  12366  15.1.1.1  15    mpls   mpls             up     7           1000      0:01:35:14  0            
39.3.0.2   39.3.0.1    ipsec  12366  12346  15.1.1.1  15    lte    public-internet  up     7           1000      0:02:30:41  0

We can also verify that there are control connections over this orange TLOC. Therefore the WAN edge router is constantly pinging the controllers to make sure they are reachable. This may consume additional data as well.

vEdge-3# show control connections | tab
#some columns are omitted for clarity

                         LOCAL      LOCAL                                                                                                                   
          PEER     SITE  PRIVATE    PRIVATE             PUBLIC  SYSTEM    LOCAL  REMOTE   PRIVATE    PRIVATE                                        BEHIND  
INSTANCE  TYPE     ID    IP         PORT     PUBLIC IP  PORT    IP        COLOR  COLOR    IP         PORT     STATE  UPTIME      V ORG NAME         PROXY   
------------------------------------------------------------------------------------------------------------------------------------------------------------
0         vsmart   1     10.10.0.1  12386    10.1.1.30  12346   1.1.1.30  mpls   default  10.1.1.30  12346    up     0:02:02:45  networkacademy-io  No      
0         vsmart   1     39.3.0.2   12366    10.1.1.30  12346   1.1.1.30  lte    default  10.1.1.30  12346    up     0:02:03:46  networkacademy-io  No      
0         vbond    0     10.10.0.1  12386    10.1.1.10  12346   0.0.0.0   mpls   mpls     10.1.1.10  12346    up     0:02:02:12  networkacademy-io  -       
0         vbond    0     39.3.0.2   12366    10.1.1.10  12346   0.0.0.0   lte    lte      10.1.1.10  12346    up     0:02:02:36  networkacademy-io  -       
0         vmanage  1     39.3.0.2   12366    10.1.1.20  12546   1.1.1.20  lte    default  10.1.1.20  12546    up     0:01:23:45  networkacademy-io  No

A better solution - Last Resort Circuit

A better solution to this problem would be to form an IPsec tunnel over this 4G TLOC only in case that the primary WAN transport goes down. Well, Cisco SD-WAN provides such an option in the solution. It is called Last Resort Circuit and is very straightforward and easy to set up.

Cisco SD-WAN Last Resort Circuit
Figure 3. Cisco SD-WAN Last Resort Circuit

The idea is visualized in figures 3 and 4. We would like to advertise the LTE TLOC to the vEdges but only form a tunnel when the primary IPsec tunnel goes down.

Cisco SD-WAN Last Resort Circuit Control Connections
Figure 4. Cisco SD-WAN Last Resort Circuit Control Connections

The same logic applies to the control connections as well. We would like to form a control connection and OMP peering over the LTE TLOC only in case of primary link failure.

Last Resort Circuit Configuration

 Let's first check the initial configuration of both TLOCs of vEdge-3. There is nothing out of the ordinary. 

!
vpn 0
 interface ge0/0
  ip dhcp-client
  ipv6 dhcp-client
  tunnel-interface
   encapsulation ipsec
   color lte
   allow-service all
  !
  no shutdown
 !
 interface ge0/1
  ip dhcp-client
  tunnel-interface
   encapsulation ipsec
   color mpls restrict
   allow-service all
  !
  no shutdown
 !
!

To enable the Last Resort Feature on the 4G transport, we just configure the command Last Resort Circuit under the tunnel-interface configuration as follows:

!
vpn 0
 interface ge0/0
  ip dhcp-client
  ipv6 dhcp-client
  tunnel-interface
   encapsulation ipsec
   color lte
   last-resort-circuit
   allow-service all
  !
  no shutdown
 !

Now let's verify whether there is still a tunnel via the 4G TLOC.

You can see that there is no tunnel over the LTE connection.

vEdge-3# show bfd sessions | tab

                              SRC    DST    SYSTEM    SITE  LOCAL                DETECT      TX                                 
SRC IP     DST IP      PROTO  PORT   PORT   IP        ID    COLOR  COLOR  STATE  MULTIPLIER  INTERVAL  UPTIME      TRANSITIONS  
--------------------------------------------------------------------------------------------------------------------------------
10.10.0.1  10.10.1.51  ipsec  12386  12366  15.1.1.1  15    mpls   mpls   up     7           1000      0:00:06:47  1

There aren't control connections as well.

vEdge-3# show control connections | tab
#some columns are omitted for clarity

                        LOCAL      LOCAL                                                                                                                   
          PEER     SITE PRIVATE    PRIVATE             PUBLIC  SYSTEM    LOCAL  REMOTE   PRIVATE    PRIVATE                                        BEHIND  
INSTANCE  TYPE     ID   IP         PORT     PUBLIC IP  PORT    IP        COLOR  COLOR    IP         PORT     STATE  UPTIME      V ORG NAME         PROXY   
-----------------------------------------------------------------------------------------------------------------------------------------------------------
0         vsmart   1    10.10.0.1  12386    10.1.1.30  12346   1.1.1.30  mpls   default  10.1.1.30  12346    up     0:00:02:50  networkacademy-io  No      
0         vbond    0    10.10.0.1  12386    10.1.1.10  12346   0.0.0.0   mpls   mpls     10.1.1.10  12346    up     0:00:02:51  networkacademy-io  -       
0         vmanage  1    10.10.0.1  12386    10.1.1.20  12546   1.1.1.20  mpls   default  10.1.1.20  12546    up     0:00:02:34  networkacademy-io  No

That is how simple it is to set up the Last Resort Circuit feature in Cisco SD-WAN. Now let's verify that the feature will work when the primary transport is down.

Verification

To verify that the feature is working, we are going to shut down the primary tunnel and see whether an IPsec overlay will form over the 4G.

How does the Last Resort Circuit works
Figure 5. How does the Last Resort Circuit work

We shut down the interface marked with the mpls color as it is shown in figure 5:

!
 interface ge0/1
  tunnel-interface
   encapsulation ipsec
   color mpls restrict
  !
  shutdown
 !
!

Now if we check the BFD sessions, we can see that a session over the 4G/LTE TLOC has just come up.

vEdge-3# show bfd sessions | tab

                           SRC    DST    SYSTEM    SITE  LOCAL                          DETECT      TX                                 
SRC IP    DST IP    PROTO  PORT   PORT   IP        ID    COLOR  COLOR            STATE  MULTIPLIER  INTERVAL  UPTIME      TRANSITIONS  
---------------------------------------------------------------------------------------------------------------------------------------
39.3.0.1  39.3.0.2  ipsec  12366  12366  15.1.1.1  15    lte    public-internet  up     7           1000      0:00:00:29  0

You can see that CIsco SD-WAN Last Resort Circuit is a very useful and flexible feature that can be easily deployed at remote sites that use data-constrained WAN transports.