Cisco Catalyst SD-WAN Components: Manager, Controller, Validator and WAN Edge

You open a new Cisco SD-WAN network and count five boxes. Three of them never forward a user packet, and one sits at each branch. When a branch won't come up, which box do you look at first?

To answer that, you need to know what each component does, which plane it works in, how the components connect, and the order a new router follows to join. Everything below is from an overlay I built in my home lab on Cisco Catalyst SD-WAN 20.18.2.1, with C8000v routers on IOS XE 17.18.02.

If SD-WAN itself is new to you, read What Is SD-WAN? first. If you learned the old names (vManage, vSmart, vBond), keep the name map open. I use the new names here and show the old ones only where the CLI still prints them.

One router used to do every job

On a classic WAN router, everything happens in one box. You type the config on it, it runs OSPF or BGP to learn routes, and it forwards the packets. That works for five sites. At 200 sites you have 200 configs, 200 routing processes, and no single place that sees the whole network.

SD-WAN splits those jobs up. Each job gets its own component, and you can scale each component on its own.

Four components, four jobs

An airport is a good picture of the split. The airline office plans the schedule, but it never flies a plane. Air traffic control tells each plane where the others are. Passport control checks who you are before you go any further. The planes carry the passengers.

In Cisco Catalyst SD-WAN:

• SD-WAN Manager is the airline office. You configure and monitor everything from it.
• SD-WAN Controller is air traffic control. It tells each router which routes and keys to use.
• SD-WAN Validator is passport control. It checks each new router and points it to the Manager and Controller.
• The WAN Edge routers are the planes. They carry your user traffic.

Cisco calls the first three the control components.

My lab has one of each control component and two WAN Edge routers.

The system IP works like a router ID. It names the device, whatever its interface addresses are.

What each component does

SD-WAN Manager

The Manager is where you work. It has the GUI and the REST API. You build configuration groups or templates on it, and it pushes them to the routers over NETCONF. It also collects statistics and alarms.

The Manager is not in the path of user traffic. You'll see that in the lab output further down: no tunnel or BFD session points at 1.1.1.1.

SD-WAN Controller

The Controller is the routing brain. Each WAN Edge router opens a session to it and runs OMP, the Overlay Management Protocol. Each router sends the Controller its local prefixes and transport details. The Controller applies policy and passes them on to the other routers. It also hands out the IPsec keys, so the routers never run IKE between themselves.

If you know a BGP route reflector, you already know the shape. Cisco's design guide uses the same comparison. Here is the Controller seeing edge1 as an OMP peer:

controller# show omp peers
PEER       TYPE   DOMAIN ID  OVERLAY ID  SITE ID  STATE  UPTIME      R/I/S
10.0.0.1   vedge  1          1           100      up     0:00:04:25  12/0/4

R/I/S means routes received, installed and sent. The Controller received 12 routes from edge1 and installed none. Like a route reflector, it passes routes on without forwarding traffic itself.

If a router loses every Controller session, it keeps forwarding with the last routes and keys it had. Cisco's design guide says this lasts for the OMP graceful restart timer, 12 hours by default.

SD-WAN Validator

The Validator is the first box a new router talks to. It has three jobs:

1. It checks the router's certificate, and its chassis and serial number against the list of allowed devices.
2. It tells the router where the Manager and Controllers are.
3. It records the public IP address and port it sees the router coming from. Cisco describes this as acting as a STUN server, which lets two branches behind NAT find each other.

In real networks the Validator needs an address that every branch can reach, often a public one. In my lab it's 172.16.30.30 on UDP 12346, and nothing sits behind NAT.

The Validator keeps the allowed-device list. Both lab routers are on it as valid:

validator# show orchestrator valid-vedges
CHASSIS NUMBER                             SERIAL NUMBER         VALIDITY  ORG
C8K-PAYG-463-xxxx-xxxx-xxxx-xxxxxxx1402B   12B44262E7BD...CDA29  valid     pinglabz-lab
C8K-PAYG-7BA-xxxx-xxxx-xxxx-xxxxxxx086BB   12B44262E7BD...CDA2A  valid     pinglabz-lab

I masked part of each chassis and serial number.

WAN Edge routers

The WAN Edge routers forward user traffic. Each one builds IPsec tunnels to the other routers over every transport it has. Inside each tunnel it runs BFD to check that the path is alive.

Each transport has a color, which is a label for the link. My routers use mpls and biz-internet. Think of the color as the name painted on the pipe. I cover colors and TLOCs in later posts.

The Manager's overview page shows the whole set: one Validator, one Controller, one Manager, and two reachable WAN Edge routers.

The planes keep user traffic away from the control components

A plane is a kind of traffic. Cisco's design guide splits Catalyst SD-WAN into four:

• The management plane carries config and statistics between you, the Manager and the routers.
• The control plane carries OMP routes, keys and policy between the Controller and the routers.
• The data plane carries user packets in IPsec tunnels, router to router.
• The orchestration plane is the Validator's work. It matters when a device joins or its NAT mapping changes.

User traffic only travels on the data plane, so it never passes through the three control components. In the lab, edge1's BFD sessions go straight to edge2, one per color:

edge1#show sdwan bfd sessions
SYSTEM IP  SITE ID  STATE  SOURCE COLOR  REMOTE COLOR  SOURCE IP   DST PUBLIC IP  DST PORT  ENCAP
10.0.0.2   200      up     mpls          mpls          10.10.1.11  10.10.2.12     12426     ipsec
10.0.0.2   200      up     biz-internet  biz-internet  10.20.1.11  10.20.2.12     12426     ipsec

I trimmed some columns to fit. None of the 1.1.1.x control component addresses appear. A host behind edge1 (192.168.10.10) reaches a host behind edge2:

h1# ping -c 5 192.168.20.10
64 bytes from 192.168.20.10: seq=0 ttl=62 time=3.005 ms
64 bytes from 192.168.20.10: seq=1 ttl=62 time=1.920 ms
64 bytes from 192.168.20.10: seq=2 ttl=62 time=1.891 ms
64 bytes from 192.168.20.10: seq=3 ttl=62 time=1.249 ms
64 bytes from 192.168.20.10: seq=4 ttl=62 time=1.656 ms
5 packets transmitted, 5 packets received, 0% packet loss

The TTL drops by two, one for edge1 and one for edge2. I'll prove with a packet capture that the Controller never touches user traffic in the planes post.

How the components connect

Each WAN Edge router opens control connections to the control components. They use DTLS by default, which is TLS over UDP. You can switch the Manager and Controller connections to TLS over TCP. Connections to the Validator are always DTLS. Here is edge1:

edge1#show sdwan control connections
PEER TYPE  PROT  SYSTEM IP  SITE ID  PEER PRIVATE IP  PEER PORT  LOCAL COLOR   STATE  UPTIME
vsmart     dtls  1.1.1.2    1000     172.16.20.20     12346      mpls          up     0:00:31:04
vsmart     dtls  1.1.1.2    1000     172.16.20.20     12346      biz-internet  up     0:00:31:04
vbond      dtls  0.0.0.0    0        172.16.30.30     12346      mpls          up     0:00:31:05
vbond      dtls  0.0.0.0    0        172.16.30.30     12346      biz-internet  up     0:00:31:05
vmanage    dtls  1.1.1.1    1000     172.16.10.10     12746      mpls          up     0:00:31:05

I trimmed some columns. The CLI still says vsmart, vbond and vmanage. Read it one peer at a time:

• edge1 has one Controller connection per color. With two transports it has two sessions, so OMP survives if one transport fails. It still runs only one OMP session, peered on the system IPs.
• edge1 has a single Manager connection, over one color. Cisco's design guide says the router keeps the Manager connection on the first transport that connects.
• The Validator connections need more explanation, below.

Why the Validator connections stay up in this lab

Cisco's design guide calls a WAN Edge's Validator connection temporary. The router uses it to find the Manager and Controllers, then tears it down.

My routers kept theirs. I checked edge1 twice, 80 seconds apart. Both Validator connections were up both times, with uptimes of 29 and 31 minutes, the same as the Controller and Manager sessions. The connection history shows them coming up at 04:37 UTC and not being torn down since.

Cisco's troubleshooting guide explains why. By default a router wants control connections to two Controllers (max-control-connections defaults to 2). My lab has one Controller. Until the router reaches the number it wants, it keeps a permanent connection to the Validator. The design guide calls this state "out of equilibrium". My edge config doesn't set max-control-connections, so the default applies:

edge1#show sdwan running-config sdwan | include max-control|tunnel-interface|interface
 interface GigabitEthernet2
  tunnel-interface
 interface GigabitEthernet3
  tunnel-interface

Cisco's fix for a one-Controller overlay is max-control-connections 1 under each tunnel interface. I left the default in place so the output above matches the lab. With two Controllers, or with that setting, you should see only the vsmart and vmanage rows.

Ports

Why does edge1 reach the Manager on 12746 and not 12346? The Manager and Controller can run up to eight cores, and each core listens on its own port. Core 0 uses UDP 12346, and each core after that adds 100: 12446, 12546, and up to 13046 for core 7. Port 12746 is core 4. Cisco's design guide notes that a router connects to only one Manager core. In my lab edge1 landed on core 4 and edge2 on core 6 (UDP 12946). The Validator has only one core and always listens on 12346. Every Manager and Controller core keeps its own connection to it. My Validator sees the Manager on eight ports, 12346 to 13046, and the Controller on two, 12346 and 12446.

The router's own ports follow a different rule. A WAN Edge starts on UDP 12346. If a connection fails, it tries 12366, 12386, 12406 and 12426 in turn, then goes back to 12346. Cisco calls this port hopping. The same port is used for the router's IPsec tunnels and BFD sessions. In my lab edge1 ended up on 12406 and edge2 on 12426, which are both on that list. A port offset shifts the whole set by the offset you configure. With TLS, the Manager and Controller listen on TCP 23456 for core 0 instead, also adding 100 for each core.

How a router joins

A new branch router follows the same order every time. Cisco's design guide describes these steps, and my lab matched them:

1. The router knows the Validator's address. Mine came from a minimal bootstrap config. In production it often comes from Plug and Play. The router also needs a certificate, and its chassis must be on the allowed list.
2. The router opens a DTLS connection to the Validator, and each side checks the other. The Validator checks the chassis and serial against its list and the organization name (pinglabz-lab here).
3. The Validator sends back the Manager and Controller addresses. It also tells the Controller and Manager that a new router is joining.
4. The router connects to the Manager and gets its config.
5. The router connects to the Controller on each transport. OMP comes up, and routes, TLOCs and keys arrive.
6. The router builds IPsec tunnels with BFD to the other routers. Now user traffic flows.

On the router, show sdwan control local-properties shows what step 2 depends on. The organization name and the root CA must match the control components:

edge1#show sdwan control local-properties
personality                       vedge
organization-name                 pinglabz-lab
root-ca-chain-status              Installed
certificate-status                Installed
certificate-validity              Valid
dns-name                          172.16.30.30
site-id                           100
system-ip                         10.0.0.1
chassis-num/unique-id             C8K-PAYG-463-xxxx-xxxx-xxxx-xxxxxxx1402b
number-vbond-peers                1
number-active-wan-interfaces      2

dns-name is the Validator's address. In my lab a lab enterprise root CA signs the router certificates, so no Cisco Smart Account is needed. Cisco also supports its own PKI and Plug and Play Connect for this. The Manager's certificate page shows both routers with installed certificates:

The connection history on edge1 shows what an early failure looks like. Before I added the routers to the allowed list, the Validator rejected edge1 with SERNTPRES (serial number not present):

PEER TYPE  PEER PRIVATE IP  LOCAL COLOR   STATE           LOCAL ERROR  REMOTE ERROR  REPEAT COUNT
vbond      172.16.30.30     mpls          challenge_resp  RXTRDWN      SERNTPRES     8
vbond      172.16.30.30     biz-internet  challenge_resp  RXTRDWN      SERNTPRES     8

Most joins that fail stop at step 2. The usual causes are a wrong organization name, a missing root CA, a chassis that isn't on the list, or a clock outside the certificate's valid dates. I cover control connection troubleshooting in a later post.

Quick reference

The Cisco SD-WAN series hub lists every post in order. The next one takes the control, data and management planes further and uses a packet capture to show the Controller never touching a user packet.