SD-WAN Colors, Restrict and Private vs Public Colors

You have two edges with two transports each: MPLS and Internet. Do the maths and there are four possible tunnels between them. Then you run show sdwan bfd sessions and see two. Is something broken?

Usually not. The answer is in two words in the tunnel config: the color, and whether it says restrict. This post explains what a color is, what restrict does, and why some colors count as private and others as public. Then it removes restrict in a live overlay and counts the tunnels. I built the overlay in my home lab on Cisco Catalyst SD-WAN 20.18.2.1, with C8000v routers on IOS XE 17.18.02.

This is part of the Cisco SD-WAN series. The previous part, TLOCs Explained, introduced the color as one of the three parts of a TLOC. For the old names of SD-WAN Manager (formerly vManage) and the other controllers, see the name map.

The problem: which pipes should talk to each other?

An edge doesn't know what a circuit is. It sees an IP address on an interface. It can't tell a private MPLS cloud from a broadband line unless you tell it.

That matters for two reasons:

• Reachability. An MPLS address often can't reach an Internet address. If the edge tries anyway, it builds a tunnel that never comes up.
• Addressing. Behind NAT, an Internet edge has two addresses: the one on its interface and the one the world sees. Which one should a peer aim at?

Colors answer both.

The idea: labels on the pipes

Think of a building with a blue water pipe and a red gas pipe. The colors don't change what's inside. They tell the plumber which pipes may be joined.

In SD-WAN a color is a label on a WAN interface. It names the transport: mpls, biz-internet, lte and so on. Every edge advertises its TLOCs, with colors, through OMP. Every other edge then decides which pairs to try.

The default rule is simple. An edge tries a tunnel from each of its TLOCs to each remote TLOC. Two transports on each side gives a full mesh of four.

The parts

Restrict

restrict is a flag on a color. It means: only build tunnels to remote TLOCs with the same color. Water pipe to water pipe, never water to gas.

Here is the tunnel config on edge1 in my lab:

edge1#show sdwan running-config | include color
   color mpls restrict
   color biz-internet

The flag travels with the TLOC. OMP carries it as an attribute, so the remote edge knows too:

edge1#show sdwan omp tlocs | include entries|restrict|^ +mpls|^ +biz
tloc entries for 10.0.0.2
                 mpls
     restrict          1
tloc entries for 10.0.0.2
                 biz-internet
     restrict          0

A 1 means restricted. edge2's mpls TLOC will only accept tunnels from another mpls TLOC.

Private and public colors

Colors also fall into two groups. Cisco's docs list them:

The group decides which address a peer uses. Every TLOC has a private address (on the interface) and a public address (what the Validator saw after any NAT). According to Cisco's docs, when both ends use private colors, the edges build the tunnel to each other's private addresses. When either end is public, they use the public addresses.

That makes sense. An MPLS cloud has no NAT, so the interface address works. Across the Internet, only the post-NAT address is reachable.

My lab has no NAT, so the private and public addresses are the same. You can see it in the local properties, where both columns read 10.10.1.11. I can't show the private/public choice changing on the wire here, so treat that rule as Cisco's documented behavior rather than something I tested.

The lab: removing restrict

Both edges start with color mpls restrict on Gi2 and plain color biz-internet on Gi3. Here are the BFD sessions on edge1, one per tunnel (columns trimmed):

edge1#show sdwan bfd sessions
SYSTEM IP   STATE  SOURCE COLOR   REMOTE COLOR   DST PUBLIC IP
10.0.0.2    up     mpls           mpls           10.10.2.12
10.0.0.2    up     biz-internet   biz-internet   10.20.2.12

Two tunnels out of four possible.

Step 1: remove restrict on edge1 only

There's no no restrict command on IOS XE. I had to remove the color and set it again without the keyword. color mpls on its own returned "No modifications to commit".

edge1(config)# sdwan
edge1(config-sdwan)# interface GigabitEthernet2
edge1(config-interface-GigabitEthernet2)# tunnel-interface
edge1(config-tunnel-interface)# no color
edge1(config-tunnel-interface)# color mpls
edge1(config-tunnel-interface)# commit

Forty seconds later:

edge1#show sdwan bfd sessions
SYSTEM IP   STATE  SOURCE COLOR   REMOTE COLOR   DST PUBLIC IP
10.0.0.2    up     mpls           mpls           10.10.2.12
10.0.0.2    up     mpls           biz-internet   10.20.2.12
10.0.0.2    up     biz-internet   biz-internet   10.20.2.12

Three tunnels, not four. edge1's mpls TLOC now reaches edge2's biz-internet TLOC, because neither of those is restricted. But edge2's mpls TLOC is still restricted, so edge1's biz-internet TLOC can't join it.

Restrict works on both ends. A cross-color tunnel needs both TLOCs to allow it.

Step 2: remove restrict on edge2 too

Same change on edge2:

edge1#show sdwan bfd sessions
SYSTEM IP   STATE  SOURCE COLOR   REMOTE COLOR   DST PUBLIC IP
10.0.0.2    up     mpls           mpls           10.10.2.12
10.0.0.2    up     mpls           biz-internet   10.20.2.12
10.0.0.2    up     biz-internet   mpls           10.10.2.12
10.0.0.2    up     biz-internet   biz-internet   10.20.2.12

edge1#show sdwan bfd summary
sessions-total           4
sessions-up              4
sessions-max             4

Full mesh: four tunnels. All four came up because my lab routes between the two transports. edge2 could already ping edge1's Internet address from its MPLS interface.

In a real network that usually isn't true. MPLS and Internet are separate worlds. Without restrict, the edges would still try the cross tunnels. They would sit down forever, and you'd chase BFD alarms for tunnels that could never work.

Step 3: revert

I set color mpls restrict again on both edges. The sessions went back to two:

edge1#show sdwan bfd summary
sessions-total           2
sessions-up              2
sessions-max             4
sessions-flap            2

sessions-max remembers the peak of four.

Restrict and the controllers

Restrict changes one more thing. Look at the local properties on edge1:

INTERFACE         COLOR          RESTRICT/CONTROL/STUN   VS/VM
GigabitEthernet2  mpls           yes/yes/no              1/1
GigabitEthernet3  biz-internet   no/yes/no               1/0

Both colors carry control connections to the Controller (VS = 1). Only mpls connects to SD-WAN Manager (VM = 1), since an edge keeps one Manager connection. Restrict didn't stop control traffic here. It only limits data-plane tunnels between edges.

Verify colors on your own edges

What breaks

• Cross-color tunnels stuck down. No restrict on a private color, and no path between the transports. Add restrict, or block the pairs with policy.
• "color mpls" did nothing. Setting the same color doesn't clear restrict. Use no color first, then set the color again.
• Same color on two transports. Two interfaces with the same color on one edge confuse the TLOC model. Each transport needs its own color.
• Private color across NAT. Two edges with private colors aim at private addresses. If NAT sits between them, the tunnel won't form. Cisco's docs say to use a public color on NATed links.
• Wrong group for a carrier. An Internet link colored mpls gets treated as private. Pick colors by how the link behaves, not by the circuit's name.

Key takeaways

• A color is a label on a transport. It tells edges which pipes may be joined and which address to use.
• With no restrict, edges try every pair: four tunnels for two transports on each side. My lab built all four.
• restrict limits a TLOC to same-color peers. It must be off on both ends for a cross-color tunnel. One side off gave me three tunnels, not four.
• On IOS XE, clear restrict with no color then the color again.
• Private colors use private addresses between private-color peers, and public colors use the post-NAT address. That last part comes from Cisco's docs. My lab has no NAT to show it.