Troubleshooting IP Multicast: Reading show ip mroute, RPF Failures, and Silent Receivers

Multicast fails silently. No ICMP unreachable, no log message, no down interface: the stream just is not there. The good news is that nearly every "multicast is broken" ticket collapses into one of three signatures, and each one is identifiable from show output in under a minute once you know what to look for. Part of the IP multicast guide.

Everything below is real output from CML labs on iol-xe nodes running IOS XE 17.18.2: one built healthy and captured twice, before and after the source went active, and a second deliberately sabotaged one failure at a time. You get both, because broken output is meaningless until you know what the working output should have said. The skill that carries all of it is reading show ip mroute properly: the flags, the incoming interface, the outgoing interface list, and what an empty list means.

The Debugging Model: Follow the Tree

A multicast stream needs three things end to end: membership at the receiver edge (IGMP), a delivery tree between first-hop and last-hop routers (PIM state that passes the RPF check), and agreement on the RP for shared-tree groups. Check the two ends first, because they localize the fault:

• Last-hop router: show ip igmp groups. Is the receiver's join actually registered?
• First-hop router: show ip mroute <G>. Is the source actually sending, and is its (S,G) forwarding or pruned?
• Then walk the middle with show ip mroute and show ip rpf, upstream from the receiver, hunting for a Null OIL or a wrong incoming interface.

The three cases below are the three ways that walk ends. First, the baseline.

The Healthy Baseline: What Working PIM-SM Looks Like

Two commands establish that the control plane is in the game at all. First, RP agreement, captured from R2 which is itself the RP:

R2# show ip pim rp mapping
PIM Group-to-RP Mappings
Group(s): 224.0.0.0/4, Static
    RP: 2.2.2.2 (?)

One static RP covering the whole 224.0.0.0/4 range (the (?) is a failed reverse DNS lookup, not an error). Second, PIM adjacency:

R2# show ip pim neighbor
Neighbor          Interface                Uptime/Expires    Ver   DR Prio/Mode
10.0.12.1         Ethernet0/0              00:01:08/00:01:35 v2    1 / S P G
10.0.23.2         Ethernet0/1              00:01:04/00:01:39 v2    1 / DR S P G

Both core links have a PIM neighbor. That is the check people skip and the cheapest on the list: an interface with no PIM neighbor has no PIM, the tree cannot cross it, and a link missing here is a forgotten ip pim sparse-mode at one end. For why sparse mode builds trees on demand instead of flooding, see how PIM sparse mode actually builds a tree.

How to Read show ip mroute

Every entry has the same four-part anatomy: the entry key, the flags, the incoming interface (IIF), and the outgoing interface list (OIL). Here is the table on the RP before the source has sent a single packet, when the receiver's IGMP join is the only thing driving state:

R2# show ip mroute
(*, 239.1.1.1), 00:01:05/00:03:03, RP 2.2.2.2, flags: SJC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/1, Forward/Sparse, 00:01:05/00:03:03, flags:

(*, 224.0.1.40), 00:01:16/00:03:22, RP 2.2.2.2, flags: SJCL
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/0, Forward/Sparse, ...
    Ethernet0/1, Forward/Sparse, ...
    Loopback0, Forward/Sparse, ...

(*, 239.1.1.1) is the shared tree: any source, this group, rooted at the RP. The timers are uptime and expiry, and RP 2.2.2.2 is which RP this router believes roots the tree, a field that catches an entire class of failure on its own (see case 3).

Now the part that trips people up. The incoming interface is Null and the RPF neighbor is 0.0.0.0, and that is correct here, because this router is the RP: nothing sits upstream of the root of a shared tree for the IIF to point at. The same output on a router that is not the RP would be a real problem, so read the IIF in the context of where you are standing.

The OIL has one entry, Ethernet0/1 toward the receiver, in Forward/Sparse: the router saying "I will replicate this group out this interface." (The 224.0.1.40 entry alongside it is the Auto-RP Discovery group, and its L flag means the router itself is a member.) Now start the source:

R2# show ip mroute
(*, 239.1.1.1), 00:02:24/00:02:41, RP 2.2.2.2, flags: SJC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/1, Forward/Sparse, 00:02:24/00:02:41, flags:

(1.1.1.1, 239.1.1.1), 00:00:49/00:02:44, flags: T
  Incoming interface: Ethernet0/0, RPF nbr 10.0.12.1
  Outgoing interface list:
    Ethernet0/1, Forward/Sparse, 00:00:49/00:02:56, flags:

A second entry appeared, and this is the whole of PIM sparse mode in one screen. (1.1.1.1, 239.1.1.1) is the source tree: this specific source, this group. Its flag is T, the SPT bit, meaning the router switched from pulling this group down the shared tree to receiving it on the shortest path back to the source. Its IIF is a real interface now, Ethernet0/0 with RPF neighbor 10.0.12.1, because something genuinely is upstream: R1.

The flags are the fastest read in the output. The ones you will meet:

An empty OIL is the single most important thing to be able to interpret. "Outgoing interface list: Null" means this router will replicate the group to nobody, and that is not automatically a fault. It has three causes, and telling them apart is the diagnosis:

• Nobody downstream asked. No IGMP membership and no PIM join arrived, so the entry was pruned. Flags include P and the network is behaving correctly. Case 1.
• The router cannot use the traffic it is receiving. Receivers exist and joins arrived, but the entry fails its RPF check, so the OIL is torn down and the entry sits pruned. Case 2.
• The join could never be sent upstream. The router heard the receiver but has no RP, so it has no direction to send a (*,G) join. The OIL may even be populated while the IIF is Null. Case 3.

Finally, prove packets are moving rather than that state merely exists:

R2# show ip mroute count
IP Multicast Statistics
3 routes using 4456 bytes of memory
2 groups, 0.50 average sources per group
Group: 239.1.1.1, Source count: 1, Packets forwarded: 20, Packets received: 21
  RP-tree: Forwarding: 1/0/100/0, Other: 2/1/0
  Source: 1.1.1.1/32, Forwarding: 19/0/100/0, Other: 19/0/0

Twenty packets forwarded, which no amount of correct-looking control plane state can fake. The Other triplet is total, RPF failed, and other drops, in that order. The source entry reads 19/0/0: zero RPF failures. The RP-tree entry reads 2/1/0, one packet failing RPF during the switch to the source tree, which is the normal transient. A counter that keeps climbing in that middle field is the numeric confirmation of the next case.

Case 1: The Silent Receiver (No IGMP Join)

Symptom: the application team swears the receiver is subscribed; no traffic arrives. In the sabotage lab a source was started for 239.2.2.20 with no receiver joined anywhere. The first hop (R3) shows the source is real and registering:

R3# show ip mroute 239.2.2.20
(*, 239.2.2.20), 00:00:27/stopped, RP 2.2.2.2, flags: SPF
  Incoming interface: Ethernet0/0, RPF nbr 10.0.23.1
  Outgoing interface list: Null

(10.0.30.10, 239.2.2.20), 00:00:27/00:02:32, flags: PFT
  Incoming interface: Ethernet0/1, RPF nbr 0.0.0.0
  Outgoing interface list: Null

The RP (R2) also knows the source, and is equally starved of receivers:

R2# show ip mroute 239.2.2.20
(*, 239.2.2.20), 00:00:18/stopped, RP 2.2.2.2, flags: SP
  Outgoing interface list: Null

(10.0.30.10, 239.2.2.20), 00:00:18/00:02:41, flags: PA
  Incoming interface: Ethernet0/1, RPF nbr 10.0.23.2
  Outgoing interface list: Null

Read the signature with the flag key above: source-side state exists everywhere it should (F at the first hop because the source is directly connected, A at the RP), but every OIL is Null and every entry carries P, where the healthy capture named a real interface in Forward/Sparse. The network is working perfectly here; nobody asked for the traffic. The confirming check is at the receiver edge, where show ip igmp groups 239.2.2.20 returns nothing at all.

Root causes, in observed frequency order: the application never actually joined (wrong group, wrong port, bound to the wrong NIC), a host firewall dropping IGMP, IGMP snooping eating reports on a querier-less segment, or a version mismatch where a v2 report for an SSM group is silently ignored. That last trap matters before you deploy source-specific multicast without an RP, and the version behaviour is in the difference between IGMPv2 and IGMPv3. Fix the join and the tree builds itself within a second.

Case 2: RPF Failure (State Exists, Traffic Dies Midpath)

Symptom: joins are fine, source is fine, and somewhere in the middle a router drops every packet without logging a thing. This is the multicast-specific failure, it is the reason traffic is not flowing far more often than anything else on this page, and it is the one most people misdiagnose.

What the RPF check actually does

When a multicast packet arrives, the router does not ask "where is this going." It asks "did this arrive on the interface I would use to get back to the source." If yes, forward out the OIL. If no, drop, silently, with no counter you will notice unless you go looking. That is loop prevention: a multicast packet gets replicated to many neighbors, so a tree with two parents would multiply traffic without bound. The critical detail is which table answers the question, and the healthy output prints it on the page:

R2# show ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
  RPF interface: Ethernet0/0
  RPF neighbor: ? (10.0.12.1)
  RPF route/mask: 1.1.1.1/32
  RPF type: unicast (ospf 1)
  Doing distance-preferred lookups across tables
  RPF topology: ipv4 multicast base, originated from ipv4 unicast base

RPF type: unicast (ospf 1). The multicast forwarding decision was made by the OSPF unicast routing table, and that line explains the entire failure mode: multicast forwarding is a passenger on unicast routing, so anything that makes the unicast best path to the source diverge from the physical path the stream takes breaks multicast while leaving unicast perfectly healthy. Static routes with a better administrative distance, tunnels, asymmetric paths, ECMP, a redistribution boundary, or a route that never made it into the routing table all do it. Note the RPF neighbor 10.0.12.1 too: that address must match the router the stream physically arrives from.

Multicast RPF is not unicast RPF

These get conflated constantly, because both names contain "reverse path forwarding" and both consult the unicast table. They are not the same feature. Multicast RPF is a mandatory, always-on part of the PIM forwarding path: you cannot turn it off, it applies to every multicast packet, and it prevents loops in a replicating topology. Unicast RPF (uRPF) is an optional per-interface anti-spoofing feature you enable in strict or loose mode to drop forged source addresses. So when somebody says "we do not run RPF here," they mean uRPF, and your multicast is still being RPF-checked on every hop regardless.

The broken fingerprint

The sabotage lab breaks it with a bad static route on the last-hop router (the check itself is dissected in how the multicast RPF check decides the incoming interface), producing this:

R1# show ip rpf 10.0.30.10
  RPF interface: Ethernet0/0
  RPF neighbor: ? (192.168.99.100)
  RPF type: unicast (static)

R1# show ip mroute 239.1.1.10
(10.0.30.10, 239.1.1.10), 00:00:43/00:02:15, flags: PJX
  Incoming interface: Ethernet0/0, RPF nbr 192.168.99.100
  Outgoing interface list: Null

The tells, in order of diagnostic value: the (S,G) incoming interface points somewhere the stream cannot possibly arrive from (here, out the receiver-facing edge), the OIL is Null with a P flag despite an active receiver, which is the distinction from case 1, and the RPF type line reads unicast (static) rather than unicast (ospf 1), naming the static route that shadowed the IGP. Traffic arrives on Ethernet0/1, the router demands Ethernet0/0, every packet fails the check.

The fix is either repairing unicast routing or, when the asymmetry is intentional, a static mroute that overrides RPF for multicast only:

R1(config)# ip mroute 10.0.30.0 255.255.255.0 10.0.12.2

R1# show ip rpf 10.0.30.10
  RPF interface: Ethernet0/1
  RPF type: multicast (static)

R1# show ip mroute 239.1.1.10
(10.0.30.10, 239.1.1.10), 00:01:18/00:01:41, flags: JT
  Incoming interface: Ethernet0/1, RPF nbr 10.0.12.2, Mroute

The RPF type changed from unicast (static) to multicast (static): the static mroute table now answers instead of the unicast one. P became JT, matching the healthy T-flag entry from the baseline, the OIL repopulated, and the stream resumed. Suspect this case first whenever tunnels, ECMP, or asymmetric routing exist in the path.

Case 3: Missing or Mismatched RP

Symptom: receivers join, sources send, and the two never meet. For shared-tree (ASM) groups both sides depend on agreeing where the RP is, and a router with no RP mapping cannot even send its join upstream. The lab removes the RP configuration from the last-hop router and has the receiver join 239.5.5.5:

R1# show ip pim rp mapping
PIM Group-to-RP Mappings

R1# show ip mroute 239.5.5.5
(*, 239.5.5.5), 00:00:16/00:02:45, RP 0.0.0.0, flags: SJC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/0, Forward/Sparse, 00:00:16/00:02:45

RP 0.0.0.0 is the entire diagnosis in four characters, and it is worth reading next to the healthy baseline because the two look deceptively similar: both show a (*,G) with a Null incoming interface and RPF nbr 0.0.0.0. On the RP that is correct. Here it is fatal, because this router is not the RP and the RP field reads 0.0.0.0 instead of an address. The rp mapping output agrees: the healthy version listed a group range and an RP, this one is a header with nothing under it. The IGMP join was heard, which is why the group exists with a populated OIL and the C flag, but with no RP to root the shared tree there is no upstream direction to send the (*,G) join.

Check show ip pim rp mapping on every router in the path; they must all resolve the same RP. Partial RP knowledge (some static, some learned, one forgotten) produces the maddening variant where multicast works from some sources and not others. The related failure is an RP that is known but unreachable, its loopback missing from the IGP, which shows up as a valid mapping plus an RPF failure toward the RP address itself; show ip rpf 2.2.2.2 catches that one. Auto-RP and BSR remove the per-router touch points but add a dependency: Auto-RP's 224.0.1.39 and 224.0.1.40 groups must themselves be forwarded, solved by ip pim autorp listener.

When There Is No mroute At All

Everything so far assumes show ip mroute returned something. Sometimes it returns nothing, or a line reading "IP Multicast Forwarding is not enabled," and none of the above applies because the router is not participating in multicast at all. Three checks:

• ip multicast-routing is missing from the global config.
• It was typed but rejected, which is subtler and cost real time in this lab. On iol-xe the distributed keyword is rejected outright, so a config copied from a platform where ip multicast-routing distributed is valid leaves forwarding switched off and never complains again.
• No interface has ip pim sparse-mode, so even with forwarding enabled there is no PIM to build state.

The minimal configuration behind every healthy capture above is four lines:

ip multicast-routing
interface Ethernet0/0
 ip pim sparse-mode
ip pim rp-address 2.2.2.2

Repeat the interface stanza on every interface in the path, including the loopback if the RP address lives on one, and the rp-address line on every router.

The Supporting Cast: Neighbors, Snooping, and Debugs

Three more checks catch the failures that do not fit the big three cleanly. First, PIM adjacency. A single interface missing ip pim sparse-mode breaks the join chain at that hop, and everything downstream looks like case 3 while everything upstream looks healthy. show ip pim neighbor hop by hop finds the gap fast, compared against the healthy output above: one neighbor per core link, uptimes stable rather than resetting.

Second, layer 2. IGMP snooping without a querier is the classic "works for two minutes, then dies" pattern: snooped state ages out with no queries to refresh it and the switch quietly stops forwarding the group. show ip igmp snooping groups on the switch, plus a snooping querier on router-less segments, closes that case. The nastier variant is old firmware that drops IGMPv3 reports entirely, downgrading every SSM receiver behind it. From the router both look exactly like case 1, which is why the L2 check belongs in the routine.

Third, targeted debugs when show output stalls. debug ip igmp shows joins arriving (or not) at the last hop in real time; debug ip pim shows join/prune and register traffic, and running it on the first hop and the RP at once answers "are registers sent" and "are they answered" in one pass. Both are group-filterable with an ACL, so they are safe on busy boxes if scoped.

The Command Kit

A Worked Ticket, Start to Finish

How the method composes on a real ticket: "the 239.1.1.10 dashboard feed stopped for the branch behind R1; other sites are fine." Start at the receiver edge. show ip igmp groups 239.1.1.10 on R1 shows membership present and refreshing, so case 1 is gone in thirty seconds. R1's mroute has a healthy (*,G) but an (S,G) with P, a Null OIL, and an incoming interface that makes no sense for where the source lives: case 2's fingerprint. show ip rpf 10.0.30.10 reports Ethernet0/0 when the stream can only arrive on Ethernet0/1, and the RPF type line reads unicast (static) in a network that runs OSPF, which narrows it to one route before you have looked at the routing table. show ip route 10.0.30.0 then names it: a static with administrative distance 1 shadowing the OSPF route, left over from last month's maintenance.

Remove it (or add a static mroute, had the asymmetry been intentional) and the confirmation is the (S,G) flags flipping from PJX back to JT, the OIL repopulating, and show ip mroute count incrementing "Packets forwarded" while the RPF-failed field stops climbing. Bisecting with pings instead would have burned the afternoon, because unicast pings to the source worked perfectly throughout. Unicast reachability proves nothing about multicast health: the two forwarding planes consult the same table with opposite questions.

Gotchas From the Lab Bench

• Sender TTL. Host stacks default the multicast TTL to 1, which the first router burns, producing a perfect imitation of case 1 (source-side state, nothing anywhere else, and even the first hop's F flag missing since packets never leave the segment). Check the sending socket before blaming the network.
• A wrong keyword can leave multicast silently off. The distributed variant of ip multicast-routing is rejected on iol-xe, and it surfaces much later as an empty mroute table rather than when you typed it.
• Null incoming interface is not automatically a fault. On the RP it is correct for every (*,G). Anywhere else, check the RP field on the same line before you panic. The same goes for a non-zero RPF-failed counter: one or two during an SPT switchover is normal, and only the climbing counter means something.
• Control plane state is not forwarding. An entry with a populated OIL and a T flag can still be moving zero packets, and only show ip mroute count settles it. Keep a healthy capture from your own platform for comparison: half the entries in a broken table look wrong to someone who has never read a correct one.

Key Takeaways

• Read show ip mroute in four parts: the entry key ((*,G) shared tree or (S,G) source tree), the flags, the incoming interface, and the outgoing interface list.
• An empty OIL has three causes: nobody asked, the entry failed RPF, or no RP is known. Telling them apart is the diagnosis.
• The three signatures are silent receiver, RPF failure (wrong IIF, P flag, Null OIL despite active receivers), and missing RP (RP 0.0.0.0 in the (*,G) entry).
• Suspect RPF first. show ip rpf prints the table that answered ("RPF type: unicast (ospf 1)"), naming the culprit routing source as well as the interface. Multicast RPF is mandatory and always on, unlike the optional uRPF anti-spoofing feature it gets confused with.
• Work the two edges first: show ip igmp groups at the last hop, show ip mroute at the first hop. Then verify RP agreement on every hop, and that the RP address itself passes RPF.
• Before any of it: confirm ip multicast-routing is actually enabled and the sender's TTL is greater than 1.

The rest of the cluster, from addressing and IGMP through PIM and SSM, is indexed in the complete multicast guide.