A TLOC is a Transport Locator that represents an attachment point where a Cisco WAN Edge device connects to a WAN transport. A TLOC is uniquely identified by a tuple of three values - (System-IP address, Color, Encapsulation). 

  • System-IP: The System-IP is the unique identifier of the WAN edge device across the SD-WAN fabric. It is similar to the Router-ID in traditional routing protocols such as BGP. It does not need to be routable or reachable across the fabric.
  • Transport Color: The color is an abstraction used to identify different WAN transports such as MPLS, Internet, LTE, 5G, etc. In scenarios where transport types are duplicated (for example two different Internet providers) and should be treated differently from each other, the colors could be arbitrary, such as Green, Blue, Silver, Gold, etc.
  • Encapsulation Type: This value specifies the type of encapsulation this TLOC uses - IPsec or GRE. To successfully form a data plane tunnel to another TLOC, both sides must use the same encryption type.
Cisco SD-WAN Forming an IPsec tunnel - Step 1
Figure 1. Cisco SD-WAN Forming an IPsec tunnel - Step 1

Let's stop here and look at the example shown in figure 1. WAN edge router 1 has an interface Ge0/0 with IP address 10.1.2.3 that connects to the Internet. As we all know, this IP is private and not routable over the Internet, so on the provider side, this private address is translated to the public address 31.1.2.3 through one-to-one NAT. On the other side, vEdge-2 has interface Ge0/1 with a publicly routable IP address 150.2.2.2. 

Now let's say that we are tasked to manually configure an overlay tunnel between WAN edge 1 and WAN edge 2. If the devices are regular Cisco IOS routers and we are configuring a GRE tunnel with IPsec on top, we are going to specify a tunnel interface with tunnel-source IP and tunnel-destination IP on both ends taking the NAT into account. Also, we are going to specify the IPsec authentication and encryption parameters, the IPsec session keys, tunnel keys, and many additional settings depending on the situation. Now imagine that vEdge-1 and vEdge-2 must establish the same tunnel but in an automatic fashion and on-demand. Well, they will need to exchange all these parameters with each other. That is what the concept of Transport Locators (TLOCs) is created for.

A TLOC route consists of all required information needed by a remote peer in order to establish an overlay tunnel with that TLOC. This includes private and public IP addresses and ports, site-id, preference, weight, status, encapsulation info such as encryption and authentication parameters, and much more. Note that the three-tuple uniquely identifies a particular TLOC but the TLOC route itself consists of much more info. For example, the vEdge-1 connection to the Internet is uniquely identified by the TLOC (1.1.1.1, green, ipsec), but the corresponding TLOC route that vEdge-1 advertises to the SD-WAN controller consists of the IP address of interface Ge0/0, the public IP address after the NAT, all IPsec parameters and many more values required by vEdge-2 in order to establish an IPsec tunnel to vEdge-1.

Cisco SD-WAN Forming an IPsec tunnel - Step 2
Figure 2. Cisco SD-WAN Forming an IPsec tunnel - Step 2

Having all we have said about TLOCs in mind, let's now follow the process of forming an overlay tunnel between WAN edge-1 and WAN edge-2. When vEdge-1 is first connected to the Internet via interface Ge0/0, it gets an IP address/Mask/Default Gateway via DHCP and it gets the transport color and encapsulation type from the controller or from a manual configuration. It then tries to establish all control plane connections to vBond/vManage/vSmart and in the process, it detects its publicly routable NATed address. Once it is able to communicate to the control plane via this interface, it creates a unique TLOC identified by the system-IP 1.1.1.1 (not the interface IP), the WAN transport color - green, and the encapsulation type - IPSec. This TLOC is then advertised to vSmart along with many other attributes that you can see in the output below.

vEdge-1# show omp tlocs detail
---------------------------------------------------
tloc entries for 1.1.1.1
                 green
                 ipsec
---------------------------------------------------
 ADVERTISED TO:                   
peer    9.9.9.9
    Attributes:
     encap-key         not set
     encap-proto       0
     encap-spi         256
     encap-auth        sha1-hmac,ah-sha1-hmac
     encap-encrypt     aes256
     public-ip         31.1.2.3
     public-port       12346
     private-ip        10.1.2.3
     private-port      12346
     public-ip       ::
     public-port     0
     private-ip      ::
     private-port     0
     domain-id         not set
     site-id           10
     overlay-id        not set
     preference        0
     tag               not set
     stale             not set
     weight            1
     version           2
     gen-id            0x80000011
     carrier           default
     restrict          0
     groups            ( 0 )
     border              not set
     unknown-attr-len  not set
     preference        0
     tag               not set
     stale             not set
     weight            1
     version           2
    gen-id             0x80000015
     carrier           default
     restrict          0
     groups            ( 0 )
     border             not set
     unknown-attr-len  not set

Once vSmart receives this TLOC route via OMP, it redistributes it to WAN edge router 2. The same process happens on the other side where vEdge-2 advertises its WAN connection to the Internet as TLOC (2.2.2.2, green, ipsec). In the end, both WAN edge routers have all the required information in order to form an overlay tunnel between each other. 

Cisco SD-WAN Forming an IPsec tunnel - Step 1
Figure 3. Cisco SD-WAN Forming an IPsec tunnel - Step 1

Now with the TLOC routes redistributed to both WAN Edge routers, they can attempt to form an overlay tunnel using IPsec and to form a BFD session with each other.

Cisco SD-WAN Forming an IPsec tunnel - Step 4
Figure 4. Cisco SD-WAN Forming an IPsec tunnel - Step 4

Multiple TLOCs

At this point, it is very likely that you might have thought - "Ah I see, a TLOC is just a tunnel endpoint and one TLOC is equal to one tunnel". If we have a WAN edge router that has two connections to two different WAN providers, for example, two Internet circuits, it would have two TLOCs. One that represents Internet-1 and one that represents Internet-2 as it is visualized on the example shown in figure 5.  Well, this is the point where it gets a little bit more complicated.

By default, WAN Edge routers try to form an overlay tunnel to every TLOC over each available WAN transport, including TLOCs that belong to other colors if there is IP reachability between the two transport networks. In the example shown in figure 5, there is any-to-any reachability between the two clouds because they are basically two Internet providers. In this scenario, four IPsec tunnels will form T1-T3, T1-T4, T2-T3, T2-T4. There is one important exception, WAN edge devices do not form overlay tunnels to other devices in the same site ie having the same site-id.

Forming the overlay with over TLOCs
Figure 5. Forming the overlay with over TLOCs

This is where the Cisco SD-WAN overlay magic happens. If you advertise all TLOCs to all WAN edge routers, a full-mesh overlay fabric will be formed (assuming there is IP reachability between the underlay transports). However, if you want to have a custom overlay topology, you just control which particular TLOCs are advertised to which particular WAN edge routers. The logic is as follows - if vEdge-X receives TLOCs Y and Z, it will attempt to form IPsec tunnels from each own TLOC. If vEdge-X does not receive TLOCs Y and Z, it won't ever attempt to establish overlay tunnels to these TLOCs.

Overlay routing and TLOCs

Ok, but what does that mean for the overlay routing? Transport Locators are also a key part of the overlay OMP routing. Each omp route points to a TLOC next-hop and not to the particular overlay tunnel used to get there. By doing that, Cisco SD-WAN abstracts the overlay routing away from underlay routing, and WAN edge devices do not have to keep any underlay information in the overlay routing tables. For an OMP route to be considered as valid, the vEdge device must have at least one IPsec tunnel with a BFD session in UP state to the next-hop TLOC.

vEdge-1# show omp routes 
---------------------------------------------------
omp route entries for vpn 1 route 172.16.50.0/24
---------------------------------------------------
            RECEIVED FROM:                   
peer            9.9.9.9
path-id         10
label           1
status          C,I,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     originator       2.2.2.2
     type             installed
     tloc             2.2.2.2, green, ipsec
     ultimate-tloc    not set
     domain-id        not set
     overlay-id        1
     site-id          20
     preference       not set
     tag              not set
     origin-proto     connected
     origin-metric    0
     as-path          not set
     unknown-attr-len not set