C9800 Client Roaming Deep Dive: 802.11r, OKC, CCKM, and PMKID Caching

C9800 Client Roaming Deep Dive: 802.11r, OKC, CCKM, and PMKID Caching

When a wireless client roams from one access point to another, what you experience should be seamless—your video call drops for a fraction of a second, your email stays connected, your Slack message sends without hiccup. Behind that smoothness is a complex dance between your client device, multiple APs, and the Catalyst 9800 controller. This article dives into how the 9800 handles client roaming, the four primary fast-roaming techniques you can deploy, and why each one matters for your network.

What Is Roaming, and Why It Matters

Roaming (or mobility) is the ability of a wireless LAN client to maintain a seamless and secure connection while moving from one access point to another with minimal latency. On the C9800 controller, this means:

• The controller maintains and updates client state across access points
• Security contexts (encryption keys, 802.1X session state) are preserved or re-established quickly
• Client IP address is maintained or handled transparently during the handoff
• Quality of Service (QoS) and access policies follow the client across the wireless fabric

Without optimized roaming, your client triggers a full re-authentication (including EAP, key derivation, and 4-way handshake) every time it associates with a new AP. On a typical enterprise deployment with 802.1X/EAP, that re-auth can take 500 milliseconds to several seconds. With fast roaming, you can cut that down to 50–150 milliseconds or less, which is invisible to voice and video.

The Problem: Full-Auth Roaming (Slow Roam)

Before diving into fast-roaming techniques, understand what happens during a full re-authentication roam:

1. Client sends authentication and association request to a new AP
2. AP forwards the request to the controller
3. Controller initiates a full 802.1X/EAP exchange with the RADIUS server
4. RADIUS server challenges the client (often multi-round EAP-TLS or EAP-PEAP)
5. Client and server derive a Master Session Key (MSK)
6. Controller extracts the Pairwise Master Key (PMK) from the MSK
7. Controller and client complete a 4-way WPA/WPA2 handshake to derive the Pairwise Transient Key (PTK)
8. Association completes; traffic flows

Each round-trip to the RADIUS server adds latency. Each cryptographic operation adds latency. On a campus with 802.1X/PEAP and multiple APs, a single roam can consume 500–1000 ms or more. For voice or video, that's noticeable. For user experience on the 9800, it's the roaming performance baseline you want to avoid.

Fast Roaming Technique 1: PMKID Caching (Sticky Key Caching)

PMKID caching is the simplest and most universally supported fast-roaming mechanism. The idea is elegant: the controller caches the Pairwise Master Key (PMK) derived from a successful 802.1X authentication, associates that PMK with a unique identifier (the PMKID), and allows the client to reuse that PMK on subsequent associations without re-authenticating to the RADIUS server.

How PMKID Caching Works on the C9800

When a client completes a full 802.1X authentication on AP1:

1. The RADIUS server returns an MSK to the controller
2. The controller extracts the PMK from the MSK (typically the first 32 bytes)
3. The controller generates a PMKID (a hash of the PMK, AP MAC address, and client MAC address)
4. The PMK and PMKID are cached in the controller's memory, indexed by client and AP
5. During the 4-way handshake, the controller includes the PMKID in the RSNE (Robust Security Network Element) of the EAPOL frame

When the client roams to AP2 (still on the same C9800 controller):

1. Client sends association request with the cached PMKID
2. AP2 forwards the request to the controller
3. Controller looks up the PMKID in its cache; if found, it matches to the cached PMK
4. Controller skips the RADIUS authentication and proceeds directly to the 4-way handshake using the cached PMK
5. Client and controller derive a new PTK from the cached PMK and complete the 4-way handshake
6. Roam completes in ~100–200 ms (depending on the AP and client performance)

PMKID caching reduces roaming latency by eliminating the RADIUS round-trip. However, it has a critical limitation: the PMK is cached locally on the controller. If the client roams to an AP managed by a different controller (inter-controller roaming), the new controller has no knowledge of the cached PMK and must re-authenticate the client to the RADIUS server. This makes PMKID caching effective only within a single controller's APs (intra-controller roaming).

On the C9800, you enable PMKID caching as part of the wireless security configuration. Some clients cache PMKIDs aggressively; others do not. Modern iOS, Windows, Android, and enterprise devices all support it.

Fast Roaming Technique 2: Opportunistic Key Caching (OKC)

Opportunistic Key Caching is an extension to PMKID caching designed to work across multiple access points when they do not share a common controller. OKC allows a client to pre-emptively request and cache the PMK for an AP it has not yet associated with, using the same MSK obtained from a previous authentication.

How OKC Works on the C9800

On a single C9800 with multiple APs, OKC operates as follows:

1. Client authenticates to AP1 via 802.1X, deriving an MSK
2. Controller caches the PMK as with normal PMKID caching
3. During the association with AP1, the controller includes additional AP information (such as neighbors discovered via 802.11k Neighbor Reports)
4. Client proactively derives PMKs for other APs in the list using the same MSK and stores them locally
5. When the client roams to AP2, it presents a PMKID for AP2 without having explicitly authenticated to AP2 previously
6. The controller recognizes the PMKID as derived from the original MSK and allows the fast roam

OKC is more powerful than PMKID caching when used with 802.11k (Neighbor Reports) because the client learns about potential target APs and pre-caches their PMKs, reducing the likelihood of triggering a full re-auth even on the first association to a neighbor AP.

However, OKC also has a security consideration: because it allows key derivation without explicit per-AP authentication, some organizations disable it in favor of full 802.11r FT, which provides more explicit key freshness semantics. OKC is supported on most modern clients and is a standard optimization on the C9800.

Fast Roaming Technique 3: Cisco Centralized Key Management (CCKM)

Cisco Centralized Key Management (CCKM) is a proprietary Cisco fast-roaming protocol that pre-derives and distributes keys to multiple access points in advance of client roaming. Unlike PMKID caching and OKC, which rely on the client to manage PMKIDs, CCKM uses a symmetric key derivation and pre-distribution model orchestrated by the controller.

How CCKM Works on the C9800

CCKM requires that all APs in the roaming domain (typically a controller or cluster) share a common key called the Cisco Centralized Key (CCK). The flow is:

1. Client authenticates to AP1 and derives an MSK from the RADIUS server
2. Controller extracts the PMK and also derives a CCKM-specific Master Key (CMK) from the PMK and the CCK
3. Controller generates roaming keys (analogous to PMKs) for multiple APs by hashing the CMK with each AP's BSSID
4. Controller distributes these roaming keys to all target APs via the controller-to-AP management channel (not the air interface)
5. When the client roams to AP2, AP2 already possesses the roaming key and can complete a fast 4-way handshake without any controller involvement
6. Roam completes in ~50–100 ms because the AP acts autonomously

The advantage of CCKM is speed: by pre-positioning keys at the APs, roaming handshakes are faster because the AP does not need to send a request to the controller and wait for a PMK lookup. This is especially beneficial for fast-moving clients (VoIP users in large campuses) or high client densities where controller load is a concern.

The trade-off is that CCKM is Cisco-proprietary and requires both the controller and the client driver to support it. Not all clients support CCKM (particularly non-Cisco endpoints, Linux, or older devices). Additionally, CCKM is less commonly deployed today than 802.11r because it is not an IEEE standard, and most modern networks now prefer standards-based approaches.

Fast Roaming Technique 4: 802.11r Fast Transition (FT)

802.11r, formally known as IEEE 802.11r-2008 (and refreshed in subsequent amendments), is the standards-based fast roaming protocol. It defines explicit key derivation and pre-authentication mechanisms that work across controllers and vendors.

802.11r Variants: Over-the-Air FT and Over-the-Distribution-System FT

802.11r defines two FT methods:

Over-the-Air FT (FT-OTA): The client performs an FT authentication exchange directly with the target AP before dissociating from the current AP. The APs exchange key material and context information to enable a fast reassociation. This variant reduces the time the client is not associated with any AP (dwell time) because the handshake occurs before the roam.

Over-the-Distribution-System FT (FT-ODS): The controller and APs exchange FT keys and client context via the wired network (the distribution system) before the client roams. When the client associates with the new AP, the AP already has the necessary keys. This variant reduces on-air negotiation latency.

How 802.11r FT Works on the C9800

When 802.11r is enabled on the C9800:

1. Client performs initial 802.1X/EAP authentication to AP1 on the C9800, deriving an MSK
2. Controller extracts the PMK and also derives an FT Master Key (FTK) and FT Intermediate Key (FTIK) from the MSK using HMAC-SHA256
3. Controller stores the FT material (FTK, FTIK, PMKR0, PMKR1) for the client
4. For Over-the-Air FT: Client probes the target AP and requests an FT Key Request using the target AP's BSSID. The controller derives a target-AP-specific PMKR1 and sends it to the target AP. Client performs an FT authentication (FT-Reassociation) with the target AP without re-engaging the RADIUS server.
5. For Over-the-Distribution-System FT: The controller pushes FT key material to candidate APs (typically identified via 802.11k Neighbor Reports). Client associates with the target AP, which uses the pre-positioned key material.
6. Roam completes in ~50–150 ms depending on the variant and client implementation

802.11r is the most robust and standards-compliant fast-roaming method. It provides:

• Explicit key freshness per roam event
• Support for inter-controller roaming (if the controllers coordinate via a shared RADIUS server or PMKR1 distribution mechanism)
• Works across different vendors if all devices are 802.11r-capable
• Explicit protection against key misuse via PMKR0/PMKR1 hierarchy

On the C9800, you enable 802.11r by configuring an FT profile (also called a Mobility Domain or MD-ID) and applying it to your SSIDs. The controller manages all FT key derivation and distribution. Modern clients (iOS 10+, Windows 10+, Android 6.0+, and most enterprise laptops) support 802.11r.

Key Hierarchy: PMK, PTK, GTK, and FT Keys

To understand roaming optimization, you need to understand the key hierarchy that 802.1X and 802.11i define:

The critical insight: all fast-roaming techniques (PMKID, OKC, CCKM, FT) depend on reusing or caching the PMK or a key derived from it, avoiding the expensive RADIUS round-trip. The difference is when and where this reuse happens, and what additional security properties each method provides.

Roaming Optimization Standards: 802.11k, 802.11v

Fast roaming is only part of the roaming story. Two additional 802.11 amendments help the client make smarter roaming decisions and reduce unnecessary handoffs:

802.11k (Neighbor Reports): Allows the AP to advertise a list of neighboring APs with details such as signal strength, channel, and band. The client uses this information to proactively scan for and pre-authenticate to candidate APs, reducing roaming delays.

802.11v (BSS Transition Management): Allows the AP to suggest or request that a client transition to a different AP or band. Useful for load balancing, band steering, and guiding clients away from poor AP associations before they experience disconnection.

The C9800 supports all three of these standards (802.11r, 802.11k, 802.11v). In practice, you enable all three on your SSID configuration to achieve the best roaming performance and network stability.

C9800 Roaming Types: Intra-Controller, Inter-Controller, Layer 2, and Layer 3

On the C9800 platform, roaming can occur across different scopes. Understanding these scopes is crucial for designing your network and configuring mobility domains:

Intracontroller Roaming

Intracontroller roaming occurs when a client moves between access points managed by the same C9800 controller. The controller updates its client database to reflect the new AP association. If necessary, the controller creates new security contexts or reestablishes associations to maintain connectivity.

For intracontroller roaming:

• PMKID caching is highly effective because the controller holds all cached PMKs
• The controller maintains a single, authoritative client database entry, simplifying state management
• Roaming is typically transparent and fast
• Data traffic may be bridged on VLAN X (as determined by policy)

Intercontroller Roaming

Intercontroller roaming occurs when a client moves from an AP managed by one C9800 controller to an AP managed by a different C9800 controller. The controllers exchange mobility messages to update client information (MAC address, IP address, security context, QoS, and access policies).

For intercontroller roaming:

• The original controller is designated the anchor controller; it maintains the primary client database entry and session state
• The new controller is the foreign controller; it receives a copy of the client database entry marked as a foreign entry
• Controllers exchange mobility messages via the CAPWAP tunnel (Controller Attachment Protocol), which uses a control path (UDP 5246) and data path (UDP 5247)
• PMKID caching does not work across controllers; the client must re-authenticate to the RADIUS server (unless OKC or 802.11r FT is enabled)
• The client may experience a full re-auth roam (~300–500 ms) if only PMKID caching is available
• With 802.11r FT, roaming is fast because the FT Master Key is derived from the MSK and can be coordinated between controllers

To support inter-controller roaming, configure mobility peers (the other C9800 controllers in your network) and ensure all controllers can reach the same RADIUS server. The controllers automatically exchange PMK and session information as clients roam.

Layer 2 and Layer 3 Mobility

Layer 2 mobility refers to roaming within the same IP subnet. The client's IP address remains the same before and after the roam. No DHCP renewal or gateway update is required. This is the common case for campus networks with well-designed WLAN deployments (one or a few large subnets per building or floor).

Layer 3 mobility (also called intersubnet roaming) refers to roaming across different IP subnets. The client's IP address changes, or the client must renew its lease. The C9800 uses anchor and foreign controller roles to manage this: the anchor controller maintains session anchoring, and the foreign controller tunnels traffic back to the anchor to preserve the client's original IP context.

For Layer 3 roaming on the C9800:

• The controllers maintain an anchor/foreign relationship
• The client traffic may be tunneled via CAPWAP or a mobility tunnel between the anchor and foreign controllers
• RADIUS authentication is handled at the anchor controller
• VLAN and QoS policies are coordinated between controllers

Mobility Domains, Site Tags, and Policy Tags

To organize your C9800 controllers and APs for efficient roaming, you configure:

Mobility Domains: A logical grouping of C9800 controllers and APs that share the same FT Master Key (FTK) seed material. All controllers in a mobility domain can participate in 802.11r FT fast roaming. Clients can roam within a mobility domain with fast key material reuse.

Site Tags: Labels assigned to groups of APs, often corresponding to physical locations (e.g., "Building-A", "Floor-3"). Used for network management, troubleshooting, and policy application.

Policy Tags: Labels applied to SSID policies, security configurations, and QoS rules. Policy tags can be associated with site tags to ensure consistent configuration across APs in a location.

When you deploy multiple C9800 controllers across a campus, you assign all controllers to the same mobility domain (if fast inter-controller roaming is desired) and use site/policy tags to manage scale and consistency.

Configuring Fast Roaming on the C9800: Practical Approach

Here is a typical configuration sequence to enable fast roaming on the C9800:

Step 1: Enable PMKID Caching and OKC on the SSID Security Profile

Device# configure terminal
Device(config)# wireless profile security MySSID-Security
Device(config-wireless-security)# pmkid-caching
Device(config-wireless-security)# opportunistic-key-caching
Device(config-wireless-security)# exit

Step 2: Configure 802.11r Mobility Domain

Device(config)# wireless mobility-domain MyMobilityDomain
Device(config-wireless-mobility-domain)# mobility-domain-id 0x1234
Device(config-wireless-mobility-domain)# r0-key-holder-address 192.0.2.1
Device(config-wireless-mobility-domain)# exit

Step 3: Assign Mobility Domain to SSID

Device(config)# wireless ssid MySSID
Device(config-wireless-ssid)# security ft-over-the-air enable
Device(config-wireless-ssid)# security ft-over-the-distribution-system enable
Device(config-wireless-ssid)# mobility-domain MyMobilityDomain
Device(config-wireless-ssid)# exit

Step 4: Enable 802.11k and 802.11v

Device(config)# wireless ssid MySSID
Device(config-wireless-ssid)# 802.11k enable
Device(config-wireless-ssid)# 802.11v enable
Device(config-wireless-ssid)# exit

Step 5: Configure Mobility Peers (for Inter-Controller Roaming)

Device(config)# wireless controller-config MyControllerName
Device(config-wireless-controller-config)# mobility-peer 192.0.2.2 ip-address 192.0.2.2
Device(config-wireless-controller-config)# exit

After these configurations are applied, clients will:

1. Perform a full 802.1X authentication on first association
2. Cache the PMKID and use it for subsequent intra-controller roams
3. Participate in 802.11r FT for fast inter-controller roams or pre-authentication
4. Respond to 802.11k Neighbor Reports and 802.11v BSS Transition Management to optimize roaming decisions

Roaming Performance: What to Expect

With proper configuration, here are typical roaming latencies on a C9800:

The variation depends on client device capability, controller load, AP firmware, and network latency between controllers. Voice and video applications typically tolerate up to ~150 ms of handoff latency; beyond that, users notice a dropout.

Troubleshooting Roaming Issues on the C9800

When clients experience slow roaming or frequent re-authentications, check:

• Client support: Verify the client device and driver support the fast-roaming methods you have configured (802.11r, PMKID caching, OKC). Older drivers may not support all methods.
• RADIUS server responsiveness: If clients are triggering full re-auths on every roam, the RADIUS server may be unavailable or slow. Check RADIUS timeout and retry settings on the controller.
• AP capability: Ensure all APs in the roaming domain support the same encryption (WPA2 or WPA3) and fast-roaming methods. Mixed AP models or firmware versions can cause inconsistency.
• 802.11k Neighbor Reports: If 802.11k is enabled but the AP does not advertise neighbor APs, clients may not pre-authenticate to candidate APs. Check the AP's 802.11k configuration and radio band support.
• PMKID cache size: On controllers with high client density, the PMK cache may become full, causing newer clients' PMKIDs to be evicted. Monitor cache hit rates and increase cache size if needed.
• Mobility peer connectivity: For inter-controller roaming, verify that the controllers can reach each other via the CAPWAP tunnel (UDP 5246 and 5247). Firewall rules and NAT can block mobility messages.
• VLAN configuration: If the client's VLAN changes after a roam, the controller may reconfigure the data path, introducing delay. Ensure consistent VLAN assignment across APs for a given SSID.

Security Considerations in Fast Roaming

Fast roaming optimizations trade some re-authentication for speed. Understand the security implications:

• PMKID caching: The PMK is cached in the controller's memory. If the controller is compromised, an attacker gains access to cached PMKs. Mitigate by securing the controller (strong passwords, VPN access, system hardening) and enabling cache expiration timers.
• OKC: Because OKC allows key derivation without explicit per-AP authentication, it assumes the initial 802.1X exchange was secure. If the RADIUS server or initial exchange is compromised, OKC keys are also compromised. Use OKC with a trusted RADIUS infrastructure.
• CCKM: CCKM pre-distributes keys to APs, which increases the risk surface if an AP is compromised. An attacker with access to an AP can obtain client keys. This is less of a concern in enterprise environments with strong AP access controls.
• 802.11r FT: FT provides explicit per-roaming-event key freshness via the FT key hierarchy, making it more resistant to key recovery attacks than PMKID caching alone. FT is the most secure fast-roaming method.

In practice, enable fast roaming on your enterprise WLAN, but ensure your RADIUS server is secure, your controllers are hardened, and your APs are physically and logically protected. Fast roaming with 802.11r and PMKID caching together provides a good balance of speed and security.

SDA Roaming and the Catalyst 9800 in a Fabric

When you deploy a C9800 controller as part of Cisco's Software-Defined Access (SDA) fabric, roaming behavior changes. In SDA, the wireless LAN is integrated with a centrally managed, policy-driven fabric architecture (wired and wireless together).

In an SDA deployment:

• The C9800 acts as the fabric edge for wireless clients
• xTRs (fabric edge nodes) at each wired access switch perform tunnel routing and policy enforcement
• Clients roam within the fabric and maintain their IP address across subnets via anchor and foreign controller roles
• Roaming latency is typically lower in SDA because the fabric handles most of the tunneling and policy application transparently
• Client state is shared across the fabric via the policy plane, reducing re-authentication events

If you are building a greenfield deployment or migrating to SDA, the C9800's native roaming support (PMKID, OKC, 802.11r, and mobility messaging) integrates seamlessly with the fabric architecture, providing both fast roaming and centralized policy control.

Key Takeaways

• Roaming is critical for wireless user experience. Without fast roaming, every client-AP change triggers a full 802.1X re-authentication, consuming 500 ms to several seconds and causing noticeable dropouts in voice and video.
• PMKID caching is the baseline. Always enable PMKID caching and opportunistic key caching; they are simple, widely supported, and eliminate RADIUS round-trips within a controller.
• 802.11r FT is the modern standard. Deploy 802.11r on all your SSIDs. It provides standards-compliant fast roaming that works across controllers, vendors, and roaming scenarios. Pair it with 802.11k and 802.11v for optimal client decisions.
• CCKM is Cisco-only but fast. If your network is 100% Cisco (APs and clients), CCKM can provide the fastest roaming (50–100 ms) because APs pre-cache keys. However, it is not standards-based and limits interoperability.
• Understand your roaming scope. Intra-controller roaming is fast and simple. Inter-controller roaming requires RADIUS coordination and mobility messaging. Use mobility domains to ensure all controllers can share FT material.
• Test roaming on real clients. Roaming performance varies by device, driver, and AP firmware. Conduct roaming tests with your target clients (iOS, Windows, Android) before deploying. Expect best results (50–100 ms) with newer devices and latest drivers; older devices may require full re-auth.
• Secure your RADIUS server and controllers. Fast roaming is only as secure as your RADIUS infrastructure. Protect your authentication server, harden your controllers, and monitor for unauthorized access to cached keys.
• Use 802.11k and 802.11v to guide client decisions. Neighbor Reports (802.11k) and BSS Transition Management (802.11v) allow the controller to steer clients proactively, reducing unnecessary roams and improving overall network stability.
• Monitor roaming latency and cache hit rates. On the C9800, use CLI commands (show wireless client) and controller dashboards to monitor roaming performance. High cache miss rates or slow re-auth times indicate configuration or RADIUS issues.