In real-world deployments, there will always be cases where the WAN edge routers cannot be directly connected to each available WAN transport as is shown in figure 1. In the example on the left, only one WAN edge device is connected to a single transport. This significantly reduces the overall availability and creates inefficiencies in the overlay fabric. TLOC-extension feature is designed to overcome this problem by extending the WAN transports to both SD-WAN routers without requiring direct attachment to both service provider clouds.
Of course, the SD-WAN routers can connect to each WAN through front-facing switches as shown on the right but this is not a recommended approach because it adds additional costs and results in having another device to operate.
What is TLOC Extension?
TLOC extension is a feature that allows a WAN Edge router to communicate over the WAN transport connected to the adjacent WAN Edge router through a TLOC-extension interface. In the example shown in figure 2, vEdge-1 is directly connected to the Internet and also uses the TLOC extension feature to connect to the MPLS transport via vEdge-2. In the end, vEdge-1 has an overlay tunnel built over both WAN clouds even though it is directly attached only to one of the WAN transports. On the opposite side, vEdge-2 is attached to the MPLS cloud and uses the TLOC extension feature to connect to the Internet via WAN Edge 1.
The feature is set up in a per-interface manner and provides transparent connectivity from one interface (called a TLOC extension interface) to a particular WAN transport. If we take vEdge-1 for example, it is unaware that the blue tunnel to the MPLS cloud passes through another WAN edge device because vEdge-2 extends their directly connected interface transparently to the MPLS transport.
TLOC Extension Types
Cisco SD-WAN allows for multiple ways of implementing TLOC extensions on WAN edge routers. The most typical and straightforward way of extending the transports is by using directly connected interfaces between the vEdges. However, depending on the local site design, the SD-WAN routers can be connected through Layer 2 switches as is illustrated in figure 3. L2 TLOC-extensions describe extensions between vEdges that are Layer2-adjacent are situated in the same broadcast domain/the same subnet.
The SD-WAN routers can also be connected at Layer 3 via any sort of IP routing. L3 TLOC-extensions describe extensions between two vEdges that are separated by an L3 device and are situated in different IP subnets. L3 TLOC extensions are implemented using GRE tunnels.
I'd also like to point out that TLOC-extensions can be configured using physical interfaces such as Ge0/0, Eth0, etc., and also using L3 subinterfaces.
At this point, it is a good time to point out some notable limitation of this feature:
- As you might already know, TLOCs are only supported on routing interfaces. Well, the same applies to TLOC-extension interfaces as well. They are only supported on L3 routed interfaces. Switchports and SVIs cannot be configured as WAN/Overlay ports and can only be used within the service VPNs.
- LTE can not be used as a TLOC extension interface between WAN Edge routers.
- L3 TLOC-extension isn't supported on Viptela vEdge routers but only on SD-WAN devices running IOS-XE.
- The feature would not work on transport ports that are bound to the loopback tunnels.
Configuring TLOC Extensions
A TLOC extension interface is always configured in VPN 0 and has an IP address assigned. Then the WAN interface to which it is bound is specified. For example, the vEdge-1's extension interface is ge0/4 and is bound to the Internet through ge0/0 and vEdge-2's extension interface is ge0/5 and is bound to the MPLS cloud through ge0/1. Then static default routes are configured in VPN0 on each WAN Edge router, pointing to the adjacent vEdge's IP as a next-hop.
However, some IP reachability considerations must be done in order for the overlay tunnels and BFD sessions to come up with remote peers over the TLOC extension interfaces. For example, to reach the MPLS cloud, vEdge-1 must be configured with a default route pointing to Ge0/5 of vEdge-2. Another very important part is the reverse IP reachability. To make sure that packets can be routed back to the TLOC extension interface, WAN Edge 2 must advertise subnet B to the MPLS provider. This can be done using any dynamic routing protocol that the service provider is willing to run. In a typical production deployment, some sort of inbound route policy is applied to deny all incoming dynamic routes from the provider since there is a static route pointing to the MPLS cloud.
In the case of WAN edge router 2, to reach the Internet cloud, it must be configured with a default route pointing to vEdge-1’s ge0/4 IP address. In a typical production deployment, subnet-A would be from the RFC1918 address space and NAT must be utilized on WAN edge-1 towards the INET cloud in order to ensure that traffic can be routed back from the Internet to vEdge-2 over the extension interface.
Let's go through a configuration example and see how this feature is implemented.
Extending vEdge-1 to the MPLS
Let's first extend router-1 to the MPLS cloud. For this example, we are going to use the private subnet 10.51.1.0/30 between Ge0/5 interfaces of both routers. This prefix 10.51.1.0/30 must also be advertised to the MPLS provider in order to have IP reachability in the other direction. For this purpose, we are going to utilize BGP.
It is important to understand that in order to extend vEdge-1 to the MPLS cloud, the actual TLOC extension happens on the adjacent router. From the perspective of router-1, Ge0/5 is connected to the MPLS provider in the same way as it would have been with a direct attachment link. Therefore, there is nothing special in the configuration on the vEdge-1 as you can see below. The interface is configured with IP address, color, and encapsulation type and a default route is specified. That's about it.
! vpn 0 ! interface ge0/5 ip address 10.51.1.1/30 tunnel-interface encapsulation ipsec color mpls restrict allow-service all ! no shutdown ! ip route 0.0.0.0/0 10.51.1.2 ip route 0.0.0.0/0 188.8.131.52 !
The "magic" happens on the neighboring router-2. In there, we should configure two things. The first one is to tell interface Ge0/5 that it extends the MPLS cloud attached to Ge0/1. The other important thing is to advertise the subnet 10.51.1.0/30 toward the service provider using BGP.
! vpn 0 ! interface ge0/1 ip address 10.50.2.1/30 tunnel-interface encapsulation ipsec color mpls restrict allow-service all ! no shutdown ! interface ge0/5 ip address 10.51.1.2/30 tloc-extension ge0/1 no shutdown ! ip route 0.0.0.0/0 10.50.2.2 ! router bgp 64551 address-family ipv4-unicast network 10.51.1.0/30 ! neighbor 10.50.2.2 no shutdown remote-as 65051 address-family ipv4-unicast ! ! !
Once both steps are configured, we can see that vEdge-1 has a valid TLOC marked with the mpls color that has a BFD session in UP state even though the router is not directly connected to the MPLS cloud.
vEdge-1# show omp tlocs --------------------------------------------------- tloc entries for 184.108.40.206 mpls ipsec --------------------------------------------------- RECEIVED FROM: peer 0.0.0.0 status C,Red,R loss-reason not set lost-to-peer not set lost-to-path-id not set Attributes: attribute-type installed encap-key not set encap-proto 0 encap-spi 256 encap-auth sha1-hmac,ah-sha1-hmac encap-encrypt aes256 public-ip 10.51.1.1 public-port 12406 private-ip 10.51.1.1 private-port 12406 public-ip :: public-port 0 private-ip :: private-port 0 bfd-status up domain-id not set site-id 50 overlay-id not set preference 0 tag not set stale not set weight 1 version 2 gen-id 0x80000019 carrier default restrict 1 groups [ 0 ] border not set unknown-attr-len not set
Extending vEdge-2 to the Internet
Let's now extend WAN edge 2 to the Internet provider. The configuration is simpler than the MPLS one because we do not need to advertise the interconnecting subnet 192.168.51.0/30 to the Internet provider, but only to NAT it when leaving the transport interface toward the provider.
All required configuration is shown in the diagram above. Once the config is applied, we can see that vEdge-2 has a valid TLOC marked with the public-internet color. Note that in the TLOC route, we can see the private-public IP address pair.
vEdge-2# show omp tlocs --------------------------------------------------- tloc entries for 220.127.116.11 public-internet ipsec --------------------------------------------------- RECEIVED FROM: peer 0.0.0.0 status C,Red,R loss-reason not set lost-to-peer not set lost-to-path-id not set Attributes: attribute-type installed encap-key not set encap-proto 0 encap-spi 256 encap-auth sha1-hmac,ah-sha1-hmac encap-encrypt aes256 public-ip 18.104.22.168 public-port 12366 private-ip 192.168.51.2 private-port 12366 public-ip :: public-port 0 private-ip :: private-port 0 bfd-status up domain-id not set site-id 50 overlay-id not set preference 0 tag not set stale not set weight 1 version 2 gen-id 0x80000014 carrier default restrict 0 groups [ 0 ] border not set unknown-attr-len not set
If we now check whether there is a BFD session over that color, you can see that there is one is UP state sourced from the TLOC extension interface 192.168.51.2 to a public IP address 22.214.171.124.
vEdge-2# show bfd sessions SOURCE TLOC REMOTE TLOC DST PUBLIC SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP ------------------------------------------------------------------------------------- 126.96.36.199 80 up mpls mpls 10.50.2.1 10.80.1.1 188.8.131.52 80 up public-internet public-internet 192.168.51.2 184.108.40.206