BGP Design for Service Providers: Full Tables, Peering, and Transit

Service provider BGP operates at a scale and complexity that dwarfs enterprise deployments. An SP carries the full internet routing table (950,000+ IPv4 prefixes, 210,000+ IPv6 prefixes as of 2025), peers with dozens to hundreds of external ASes, and manages an internal iBGP mesh across potentially thousands of routers. The design decisions at this level are about scalability, policy flexibility, and economic optimization of traffic flow.

SP Peering Relationships

Service providers classify their BGP neighbors into three categories, and each has fundamentally different routing policy:

Customers

Pay you for transit. You accept their routes and advertise them to the rest of the internet. You also send them your full table (or a default route, depending on the service level).

! Policy: Accept customer routes, advertise to everyone
! Set local-pref high — customer routes preferred over transit/peer routes
route-map CUSTOMER-IN permit 10
 set local-preference 200
 set community 65010:100 additive

Peers

Exchange traffic for free (settlement-free peering). You accept their routes and their customers' routes, but NOT routes they learned from their own upstreams (transit). You advertise your routes and your customers' routes, but NOT routes from your other peers or transit providers.

! Policy: Accept peer routes, advertise only customer routes
route-map PEER-IN permit 10
 set local-preference 150
 set community 65010:200 additive
!
! Outbound: only advertise customer-tagged routes
ip community-list standard CUSTOMERS permit 65010:100
route-map PEER-OUT permit 10
 match community CUSTOMERS

Transit Providers (Upstream)

You pay them for access to the rest of the internet. They send you full tables. You send them your routes and your customers' routes. Transit is expensive — the economic incentive is to peer as much traffic as possible and minimize transit usage.

! Policy: Accept all routes, lowest preference (use only when no better path)
route-map TRANSIT-IN permit 10
 set local-preference 100
 set community 65010:300 additive

Local Preference Hierarchy

The standard SP local-pref scheme ensures proper path selection:

This is economically optimal: traffic destined for a customer goes to them directly (they're paying you), traffic exchangeable via peering avoids transit costs, and only traffic that can't be reached via customers or peers uses paid transit.

Full Table Management

Memory Requirements

A full IPv4+IPv6 BGP table requires approximately 2-4 GB of RAM depending on the number of paths per prefix and the platform. With multipath enabled (multiple paths stored), memory can exceed 8 GB. Plan for growth — the table grows ~10-15% per year.

ISP-A-PE1# show ip bgp summary
BGP router identifier 203.0.113.1, local AS number 65010
BGP table version is 1245892
952417 network entries using 238104250 bytes of memory
2156734 path entries using 285289892 bytes of memory
247831 BGP path/bestpath attribute entries using 67408836 bytes of memory

Route Reflector Design

SPs can't run iBGP full mesh across hundreds of routers. The standard design uses hierarchical route reflectors:

• Top-level RRs: 2-4 dedicated route reflector servers (often virtual machines or dedicated boxes — not in the forwarding path). These peer with all PE and border routers.
• Regional RRs (optional): In very large networks, regional RRs reduce the number of iBGP sessions per device. Regional RRs peer up to top-level RRs and down to local PE/P routers.
• RR placement: Dedicated off-path RR servers are preferred. Using in-path routers as RRs creates the path selection distortion problem (RR chooses best path from its own perspective).

! Top-level RR configuration
router bgp 65010
 neighbor CLIENTS peer-group
 neighbor CLIENTS remote-as 65010
 neighbor CLIENTS update-source Loopback0
 neighbor CLIENTS route-reflector-client
 neighbor CLIENTS next-hop-self all
!
 neighbor 203.0.113.10 peer-group CLIENTS
 neighbor 203.0.113.11 peer-group CLIENTS
 neighbor 203.0.113.12 peer-group CLIENTS
 ! ... hundreds of clients

Dual-Stack: IPv4 and IPv6

Modern SPs must support both address families. Use MP-BGP (Multiprotocol BGP) with separate address families:

router bgp 65010
 neighbor 172.16.0.1 remote-as 65001
!
 address-family ipv4 unicast
  neighbor 172.16.0.1 activate
  neighbor 172.16.0.1 route-map CUSTOMER-V4-IN in
!
 address-family ipv6 unicast
  neighbor 2001:db8::1 activate
  neighbor 2001:db8::1 route-map CUSTOMER-V6-IN in

IPv6 table size is growing faster percentage-wise than IPv4 (210K+ prefixes in 2025), but the absolute size is still smaller. The same design principles apply: local-pref hierarchy, filtering, communities.

Internet Exchange Points (IXPs)

IXPs are where SPs exchange traffic via peering. BGP at an IXP involves:

• Bilateral peering: Direct eBGP sessions between two ASes at the IXP
• Route server peering: eBGP session with the IXP's route server, which reflects routes from all participating ASes. Reduces the number of BGP sessions from O(n^2) to O(n).
• Filtering: Only accept routes from each peer that they're authorized to announce (based on IRR/RPKI). Apply max-prefix limits.

! Route server peering (multihop via IXP LAN)
neighbor 192.0.2.1 remote-as 64512
neighbor 192.0.2.1 ebgp-multihop 2
neighbor 192.0.2.1 description IXP-Route-Server
neighbor 192.0.2.1 route-map IXP-PEER-IN in
neighbor 192.0.2.1 route-map IXP-PEER-OUT out
neighbor 192.0.2.1 maximum-prefix 500000 80

Community-Based Policy Architecture

SPs build their entire routing policy around communities. Customers signal intent with communities, and the SP's route-maps act on them. This scales because adding a new customer doesn't require new route-maps — just new community tags.

Scaling Considerations

• Soft reconfiguration inbound: Don't enable everywhere — it doubles memory per peer. Use route-refresh capability (default on modern IOS XE) instead.
• Update groups: IOS XE automatically groups peers with identical outbound policy into update groups, calculating outbound UPDATEs once per group instead of per peer. Verify with show ip bgp update-group.
• BGP additional-paths: For PIC (Prefix-Independent Convergence), install backup paths in the FIB. Essential for sub-second convergence at scale.
• Graceful restart: Maintains forwarding during BGP process restart or route processor switchover. Critical for non-stop forwarding during software upgrades.

router bgp 65010
 bgp graceful-restart
 bgp graceful-restart restart-time 120
 bgp graceful-restart stalepath-time 360

Security Baseline for SPs

• RPKI ROV on all eBGP sessions — reject Invalid origins (see BGP RPKI)
• IRR-based prefix filtering for all customers and peers
• Maximum-prefix limits on ALL external sessions
• MD5/TCP-AO authentication on all sessions
• GTSM on directly connected eBGP peers
• Bogon filtering on all internet-facing sessions (see BGP Route Filtering)
• RTBH capability for DDoS mitigation
• Regular IRR and RPKI ROA audits

Key Takeaways

• SP BGP design revolves around the customer/peer/transit hierarchy, enforced through local-pref (200/150/100) and communities.
• Route reflectors (hierarchical, off-path) are essential — full mesh iBGP doesn't scale to SP size.
• Community-based policy architecture lets you manage thousands of customers and peers without per-neighbor route-maps.
• Full table management requires careful memory planning (2-4 GB minimum), update group optimization, and PIC for convergence at scale.
• Security is non-negotiable at SP scale: RPKI ROV, IRR filtering, max-prefix, authentication, and RTBH are all baseline requirements.