OMP Explained: BGP's Cousin in the Overlay

Your branch router has two WAN links and a LAN. In a Cisco SD-WAN overlay, it doesn't run OSPF or BGP to the other branches. So how does edge1 learn that 192.168.20.0/24 lives behind edge2, and which tunnel to use to get there?

The answer is OMP, the Overlay Management Protocol. The good news: if you know BGP route reflectors, you already know about 80% of it. This post maps OMP onto BGP, shows the three kinds of OMP routes, and then follows one new prefix through the overlay. I built the lab in my home lab on Cisco Catalyst SD-WAN 20.18.2.1, with C8000v routers on IOS XE 17.18.02.

This is part of the Cisco SD-WAN series. The previous part, Control Plane vs Data Plane vs Management Plane in SD-WAN, showed that the SD-WAN Controller (formerly vSmart) never forwards user traffic. OMP is the protocol it uses to do its actual job. If SD-WAN itself is new, read What is SD-WAN? first. The components post introduces SD-WAN Manager (formerly vManage) and the other boxes. For the other names, see the name map.

The problem: full mesh doesn't scale

Picture 100 branches that all need each other's routes. With BGP between them, you'd need 4,950 sessions. Nobody does that. In BGP you add a route reflector, and each branch peers only with the reflector.

SD-WAN takes that idea and builds it in. OMP has exactly one design: every WAN Edge router peers with the Controller, and never with another edge.

The idea: OMP is BGP with a route reflector you can't remove

Think of a newsroom. Reporters in each city file their stories with the central desk. The desk decides what runs and sends the stories out to every city. Reporters never phone each other. The desk itself never goes out to report.

That is OMP. The edges are the reporters. The Controller is the desk. Cisco's design guide describes the Controller as acting "as a route reflector", and the OMP chapter of Cisco's routing guide compares OMP to BGP directly.

The part that is not like BGP: you don't configure neighbors. OMP runs inside the DTLS control connection that each edge already has to the Controller. When the control connection comes up, so does OMP.

The parts: three kinds of OMP routes

Cisco's routing guide lists three route types that OMP carries:

1. OMP routes (vRoutes). Prefixes from the LAN side of each edge. Connected, static, OSPF or BGP routes from a service VPN. Each one points at a TLOC.
2. TLOC routes. One per transport on each edge. They say "this edge's mpls link is at 10.10.1.11, port 12406". They are what a vRoute resolves through. The next post in this series covers them in detail.
3. Service routes. Advertise services such as a firewall that other sites can steer traffic through. My lab doesn't use any.

OMP also carries the IPsec keys and policies, but those three are the routing part.

Here is what the Controller says it holds:

controller# show omp summary
personality            vsmart
routes-received        18
routes-installed       0
routes-sent            6
tlocs-received         4
tlocs-installed        4
tlocs-sent             4
vedge-peers            2

Four TLOCs: two edges with two transports each. Eighteen routes received and zero installed. The Controller keeps the routes in its OMP table to reflect them, but it does not put them in a forwarding table. That's exactly what a BGP route reflector that isn't in the data path would do.

The lab

The OMP peering

On edge1, there is one OMP peer, and it is the Controller:

edge1#show sdwan omp peers
R -> routes received
I -> routes installed
S -> routes sent
TENANT ID    PEER           TYPE    DOMAIN ID    OVERLAY ID    SITE ID    REGION ID    STATE    UPTIME             R/I/S
0              1.1.1.2     vsmart       1          1          1000        None         up       0:10:28:00        12/2/4

On the Controller, the peers are the two edges:

controller# show omp peers
PEER             TYPE    DOMAIN ID  OVERLAY ID  SITE ID   STATE    UPTIME           R/I/S
10.0.0.1         vedge   1          1           100       up       0:10:29:31       12/0/4
10.0.0.2         vedge   1          1           200       up       0:10:26:18       12/0/4

Notice the Controller column: I is 0. It installs nothing.

The edge config that feeds OMP is short:

edge1#show sdwan running-config sdwan
...
 omp
  no shutdown
  graceful-restart
  no as-dot-notation
  address-family ipv4
   advertise connected
   advertise static
  !

The OMP table on edge1

edge1#show sdwan omp routes
VPN    PREFIX              FROM PEER  PATH ID  LABEL  STATUS    TLOC IP     COLOR         ENCAP
10     192.168.10.0/24     0.0.0.0    66       1004   C,Red,R   10.0.0.1    mpls          ipsec
                           0.0.0.0    68       1004   C,Red,R   10.0.0.1    biz-internet  ipsec
10     192.168.20.0/24     1.1.1.2    3        1004   R         10.0.0.2    -             -
                           1.1.1.2    66       1004   C,I,R     10.0.0.2    mpls          ipsec
                           1.1.1.2    68       1004   C,I,R     10.0.0.2    biz-internet  ipsec

(Some columns trimmed.) Read it like a BGP table:

• 192.168.10.0/24 is edge1's own LAN. FROM PEER 0.0.0.0 means local, like BGP's 0.0.0.0 next hop for locally originated routes. Red means redistributed into OMP.
• 192.168.20.0/24 came from the Controller (1.1.1.2). It has two usable paths, one per edge2 TLOC. Both are C (chosen) and I (installed), so edge1 can use both. The traceroute and capture in the previous post showed exactly that: flows went out on both colors.
• The row with no color and status R only is the system-path entry. Cisco added it for optimized advertisement. It carries the originator's system IP instead of a TLOC, and it isn't the one installed.
• LABEL 1004 marks VPN 10 on the receiving side.

The example: one new prefix, end to end

To watch OMP work, I added a loopback in VPN 10 on edge2. (I removed it after the test.)

edge2(config)# interface Loopback10
edge2(config-if)# vrf forwarding 10
edge2(config-if)# ip address 172.20.20.1 255.255.255.255
edge2(config-if)# commit
Commit complete.

No routing protocol config. advertise connected was already there, so edge2 sent the /32 to the Controller. The Controller then had it, with both of edge2's TLOCs:

controller# show omp routes vpn 10
VPN    PREFIX              FROM PEER   LABEL  STATUS  TLOC IP    COLOR         ENCAP
10     172.20.20.1/32      10.0.0.2    1004   R       10.0.0.2   -             -
                           10.0.0.2    1004   C,R     10.0.0.2   mpls          ipsec
                           10.0.0.2    1004   C,R     10.0.0.2   biz-internet  ipsec

edge1 received it from the Controller, and the detail view shows the attributes. This is where the BGP comparison gets very literal:

edge1#show sdwan omp routes vpn 10 172.20.20.1/32 detail
...
peer            1.1.1.2
path-id         66
label           1004
status          C,I,R
    Attributes:
     originator       10.0.0.2
     type             installed
     tloc             10.0.0.2, mpls, ipsec
     site-id          200
     origin-proto     connected
     origin-metric    0

• peer 1.1.1.2: learned from the reflector.
• originator 10.0.0.2: like BGP's ORIGINATOR_ID, the edge that first sent it.
• tloc 10.0.0.2, mpls, ipsec: the next hop.
• origin-proto connected: like BGP's origin, but it tells you which protocol the route came from on the far side.

When I checked edge1 a few seconds later, the route was 19 seconds old in the VRF table and reachable:

edge1#show ip route vrf 10
      172.20.0.0/32 is subnetted, 1 subnets
m        172.20.20.1 [251/0] via 10.0.0.2, 00:00:19, Sdwan-system-intf
...
edge1#ping vrf 10 172.20.20.1 source 192.168.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/4 ms

The m code is OMP. The distance is 251, and the next hop is edge2's system IP. Behind that system IP sit the two IPsec tunnels, one per color.

Verify OMP on your own overlay

What breaks

• No control connection, no OMP. OMP rides on the DTLS connection to the Controller. If show sdwan control connections has no vsmart row, stop there and fix that first.
• Route in OMP but status is `R` without `I`. The route resolved but wasn't installed. Look at whether its TLOC is reachable (BFD down, or restricted colors that can't form a tunnel).
• Route missing on the far side. Check what the source edge advertises. advertise connected and advertise static cover those two. Cisco's guide says BGP routes and external OSPF routes are not advertised into OMP by default, so they need advertise bgp or advertise ospf external.
• Everything leaks everywhere. Configured at the global level, as in my lab, advertise connected applies to every VRF on the box. (Cisco also lets you set it per VRF with address-family ipv4 vrf <n>.) My edges advertised two prefixes from a VPN I never meant to put in the overlay. The VPN post in this series shows that in detail.
• Controller goes down. With OMP graceful restart (on by default), edges keep forwarding with the routes they have cached. Cisco's default timer is 43,200 seconds, which is 12 hours. Cisco also warns that if you reboot an edge while the Controller is down, it loses that cache. I haven't tested this in my lab yet.
• More than four paths. By default an edge installs at most four OMP paths per prefix (ecmp-limit). Devices also advertise up to four route and TLOC pairs per prefix by default (send-path-limit). Big multi-transport sites can hit both limits.

Key takeaways

• OMP is a BGP-like protocol with a built-in route reflector: the SD-WAN Controller.
• Edges peer only with the Controller. There is no neighbor config. OMP runs over the existing DTLS control connection.
• OMP carries vRoutes, TLOC routes and service routes. A vRoute's next hop is a TLOC.
• In my lab the Controller received 18 routes and installed none. It reflects, it doesn't forward.
• A new connected prefix on edge2 showed up in edge1's VRF table as an m route with distance 251, with no extra config.

Next: TLOCs, the next hop with a label telling you which pipe.