[Part 9] Cisco SD-WAN - Data Plane

The Cisco Software Defined - WAN Data Plane Operation

[Part 9] Cisco SD-WAN - Data Plane

Data Plane Works

In a traditional network, the data plane is responsible for moving packets of data from one place to another. This is often done using the public internet or private wide area networks (WANs) like DMVPN, MPLS, or point-to-point connections. These technologies use overlays to encapsulate and secure the data packets.

However, as WANs grow larger, the existing technologies face challenges in scaling, especially when it comes to securing control and data planes. This requires a lot of processing power to handle tasks like key exchanges and routing updates.

To ensure security in the data plane, IPsec encryption and its tools are commonly used. However, when networks rely heavily on IPsec for security, scalability becomes a concern. The processing power required for key exchanges grows exponentially with the number of nodes in the WAN.

For example, a network with 100 nodes would need 10,000 key exchanges (n2), and each device would have to maintain 999 keys (n–1).

Read more: Cisco SD-WAN vSmart Controllers

Cisco SD-WAN also utilizes IPsec for securing the data plane, but it has made modifications to support larger deployments. It uses a centralized controller (vSmart) to distribute keys and routing information, allowing for scalability without requiring each WAN Edge device to negotiate keys with all other nodes in the network.

Furthermore, networks face scalability challenges when they need to support segmentation or different topologies per network segment. Traditional methods like MPLS L3VPN and DMVPN can be complex and require experienced network engineers to implement and troubleshoot.

In the Cisco SD-WAN solution, segmentation is implemented natively and doesn't require advanced experience to set up and support. It allows for different topologies per network segment, such as a full mesh for corporate users and a hub and spoke for specific devices.

TLOC Color

The Color attribute of the TLOC route is crucial in identifying the transport being used. Each transport should ideally have a different color assigned to it. The Color attribute allows policies to be constructed to influence how the data plane is built within the SD-WAN network.

There are 22 predefined colors available to choose from, and they also define whether the underlying transport is private or public. This distinction determines the IP address that should be used when establishing a data plane tunnel to a remote site.


By default, WAN Edge devices will attempt to build data plane tunnels to every other site using every available color.

Read more: Cisco SDWAN TLOC?

When setting up the IPsec data plane, routers in the network will automatically try to establish connections with all other routers in the fabric, creating full mesh connectivity.

For example, if cEdge-1 has the color "biz-internet", and "private1" and another router has the color "public-internet," and "private1" they will still establish an IPsec tunnel if they have IP connectivity between them.

This happens by default. If two routers have IP connectivity, regardless of their assigned "color" (a designation used in the network), they will build an IPsec tunnel between them.

One scenario where this decision becomes important is when your private WAN (MPLS) doesn't have direct IP connectivity to the internet. In such cases, you wouldn't want MPLS-connected routers to attempt to establish connections with internet-connected routers. (refer to the below figure)

Depending on the country or region, you may want full mesh tunnels between routers. However, when connecting different countries or regions (like the US and UK), it may be preferable to use point-to-point tunnels specifically between hub sites.

If there is no IP connectivity between TLOCs or if the design doesn't allow data connectivity between certain colors, there are two options available.

One option is to advertise the restrict attribute with the TLOC, which tells other devices in the network not to try to establish connectivity with the restricted color. The other option is to configure tunnel groups, which serve the same purpose.

Color Restrict

Let's focus on the "restrict" keyword first. It is an attribute within the TLOC route that can be set to either on or off for each site. In the example provided, the TLOC route uses the color "biz-internet" and has the "restrict" attribute set to 1.

This means that the device will only form tunnels with other TLOCs advertising the same color. If the restrict attribute were set to 0, the color would be unrestricted and able to form tunnels with other colors.

In the given scenario, if "restrict" is not set, each device would end up with four data plane tunnels, as IPsec tunnels would be established across all colors.

To enable "restrict", use the below command lines:

vpn 0
 interface ge0/0
   color public-internet restrict

The output below verifies the TLOC "restrict" enabled.

tloc entries for
            RECEIVED FROM:                   
status          C,Red,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
     attribute-type    installed
     encap-key         not set
     encap-proto       0
     encap-spi         258
     encap-auth        sha1-hmac,ah-sha1-hmac
     encap-encrypt     aes256
     public-port       12346
     private-port      12346
     public-ip         ::
     public-port       0
     private-ip        ::
     private-port      0
     bfd-status        up
     domain-id         not set
     site-id           10
     overlay-id        not set
     preference        0
     tag               not set
     stale             not set
     weight            1
     version           3
    gen-id             0x8000000b
     carrier           default
     restrict          1            <<<<<<<<<<--## RESTRICT enabled
     on-demand          0
     groups            [ 0 ]

Tunnel Group

Another way to control data plane connectivity is by utilizing tunnel groups. When you use tunnel groups, only tunnels that have matching tunnel groups or no tunnel group defined at all will establish data plane connectivity, regardless of their color. It is advisable to have tunnel groups defined for all sites if you decide to implement this approach.

A common scenario where tunnel groups come into play is when a data center has two physical connections to the same MPLS provider, but the branch sites have only one physical connection each. However, the network design requires establishing connectivity across both physical interfaces in the data center.

To achieve this, the tunnel group is advertised as an attribute in the TLOC route, as shown in the above illustration. The tunnel group value can range from 0 to 4294967295, allowing for flexibility in setting up the desired configurations.

Data Plane Encryption

Up until now, we've been discussing various concepts related to the control plane's role in establishing the data plane. Similar to other overlay technologies, Cisco SD-WAN achieves encryption and authentication using IPsec. However, the scale of this process is handled differently, especially concerning key exchange.

In the traditional approach, the Internet Key Exchange (IKE) protocol manages key exchange. In the first phase of IKE, two peers negotiate encryption, authentication, hashing, and other techniques to establish a secure channel for the second phase of the IPsec tunnel. The second phase of IKE sets up a tunnel for transmitting user data. During this phase, several elements are negotiated, such as the encapsulation protocol (Authentication Header or Encapsulation Security Protocol), encryption algorithm, authentication type, and tunnel lifetime.

Cisco SD-WAN supports the following methods for key exchange:

  • Authentication: Ensures that the communicating endpoints are valid and authentic, using 2048-bit keys with RSA encryption. The SD-WAN solution supports Encapsulation Security Payload (ESP) and Authentication Header (AH) for sender authentication.

  • Encryption: Utilizes the AES protocol with a 256-bit key length for data encryption.

  • Integrity: Verifies that data traffic traversed the network without tampering. This is achieved using the Galois Counter Mode (GCM) variant of AES-256, which has a built-in hashing mechanism for data integrity. Additionally, Anti-Replay Protection is enabled to safeguard against duplication attacks.

As the network grows larger, this negotiation process can become a scalability concern. Even after the initial negotiation, the devices must continue tracking the tunnel state, which consumes CPU cycles.

To address this issue, Cisco SD-WAN implements these negotiations within the control plane. The WAN Edge already has a tunnel established to the control plane with its encryption, authentication, and integrity. This infrastructure is leveraged for data plane negotiations. Each WAN Edge generates an AES-256 bit key (per transport) for encryption and integrity.

This key is then advertised to vSmart, along with the corresponding TLOCs, in an OMP update. These route advertisements are propagated throughout the network. Remote WAN Edges use this information to build IPsec tunnels between themselves.

This model of key distribution eliminates the need for individual negotiations as in IKE, reducing the burden on the system. Additionally, to enhance security, WAN Edges regenerate their keys every 24 hours, with the flexibility to adjust the rekey timer based on specific requirements. Importantly, renegotiating keys does not disrupt existing traffic, as this process occurs in parallel with the existing tunnels.

Encryption with Pairwise

This approach adds an extra layer of security by avoiding the use of the same key across all devices in the network fabric for encryption and decryption.

Pairwise encryption works by creating specific key pairs between two WAN Edges. When encrypting and decrypting data between WAN Edge 1 and WAN Edge 2, a unique key pair is generated for this specific pair of devices. Similarly, a different key pair is used for traffic between WAN Edge 1 and WAN Edge 3.

In this way, each WAN Edge has its own unique set of keys for communication with different peers.

The public key is the only piece of information exchanged over OMP (Overlay Management Protocol).

The advantage of this method is that security-conscious customers need not worry about the private key being exchanged as well. The key exchange process still occurs through the vSmart controller, and unique pairs are generated for each transport.

In summary, the pairwise encryption key model ensures greater security by creating unique key pairs for communication between WAN Edges, providing an additional layer of protection against potential vulnerabilities that could arise from using the same key across all devices in the fabric.

Get the Cisco SD-WAN Zero-to-One ebook