ISE Live Logs Casebook: Five 802.1X Failures, Reproduced

"802.1X is broken" is never a useful ticket. The switch only knows the port stayed unauthorized. The user only knows the laptop won't connect. The real answer is almost always sitting in Operations > RADIUS > Live Logs on Cisco ISE, as an event code and a failure reason. If you can read those two fields, most tickets close in minutes.

So I broke one access port five different ways and kept the evidence each time. Every case below has the symptom, the Live Logs entry, the root cause, the fix and how I verified it. Then there's a passing authentication to compare against.

I built this in my home lab with Cisco ISE 3.5.0.527, a Cisco IOSv-L2 15.2 access switch and a Linux host running wpa_supplicant 2.11 with PEAP-MSCHAPv2. All the output is real. Addresses and MACs are documentation values, and secrets are placeholders.

If you're setting up from scratch, start with Configuring Cisco ISE as a RADIUS Server for 802.1X and the Cisco ISE 802.1X Wired Configuration Guide. This post is about reading the failures afterwards.

The lab

On ISE there are two internal users, hd-01 (enabled) and hd-02 (disabled for case E), plus a Wired-Access policy set with a Wired-Staff-Permit rule that returns PermitAccess. The switch is a network device with a RADIUS shared secret.

Step 1: Switch configuration

This is the baseline that passes. Every case below is one change away from it.

aaa new-model
!
radius server ISE-1
 address ipv4 192.0.2.10 auth-port 1812 acct-port 1813
 key <RADIUS-SHARED-SECRET>
!
aaa group server radius ISE
 server name ISE-1
 ip vrf forwarding MGMT
 ip radius source-interface GigabitEthernet0/0
!
aaa authentication dot1x default group ISE
aaa authorization network default group ISE
aaa accounting dot1x default start-stop group ISE
!
radius-server attribute 6 on-for-login-auth
radius-server attribute 8 include-in-access-req
radius-server vsa send authentication
radius-server vsa send accounting
!
dot1x system-auth-control
!
vlan 860
 name STAFF
!
interface GigabitEthernet0/1
 switchport mode access
 switchport access vlan 860
 authentication port-control auto
 dot1x pae authenticator
 dot1x timeout tx-period 10
 spanning-tree portfast edge

Step 2: Find the failure in Live Logs

Open Operations > RADIUS > Live Logs and filter on the endpoint MAC or the username. A red status is a failure. The useful columns are off to the right, so scroll: Failure Reason, Network Device and Authorization Policy.

Click the magnifier on a row to open the detail report. Everything in the cases below comes from three parts of that report:

• Overview / Authentication Details: the event, failure reason and username ISE actually saw.
• Resolution and Root cause: ISE's own explanation, which is usually right.
• Steps: the numbered steps ISE ran. The last few lines tell you exactly where it stopped.

Case A: 5405 RADIUS Request dropped, 11036 Message-Authenticator invalid

Symptom. The port never authorizes. The switch says ISE isn't answering, even though ISE is reachable and marked UP.

acc-sw4#show aaa servers | include RADIUS|State|Auth
RADIUS: id 1, priority 1, host 192.0.2.10, auth-port 1812, acct-port 1813
     State: current UP, duration 108s, previous duration 0s
     Authen: request 4, timeouts 4, failover 0, retransmission 3
acc-sw4#show logging | include DOT1X
*Oct 31 06:51:49.037: %DOT1X-5-FAIL: Authentication failed for client (0000.5e00.5301) on Interface Gi0/1 AuditSessionID 0A56FC0A0000000D0003ACC7

Four requests, four timeouts. ISE got every one of them and dropped them silently.

Live Logs. Event 5405 RADIUS Request dropped, failure reason 11036 The Message-Authenticator RADIUS attribute is invalid. The steps are short: 11001 Received RADIUS Access-Request, 11017 new session, 11036, 5405. ISE never got as far as EAP. On the RADIUS Live Logs counters, Misconfigured Network Devices and RADIUS Drops each went up by one.

Root cause. The shared secret on the network device in ISE didn't match the key on the switch. ISE uses the secret to check the Message-Authenticator, so a mismatch fails the check and ISE drops the packet without replying. That's why the switch counts timeouts, not rejects.

Fix. Set the same secret on both sides. I corrected it under Administration > Network Resources > Network Devices > acc-sw4 > RADIUS Authentication Settings. Don't touch the switch unless you know which side is wrong.

Verify. The next attempt reached EAP and showed up with a real failure reason instead of a drop. For a deeper walkthrough of this one, see the 802.1X RADIUS shared secret mismatch lab. If ISE logs 11007 Could not locate Network Device instead, the device or its IP is missing altogether: that's this lab.

Case B: 5400 Authentication failed, 22056 Subject not found

Symptom. One user can't connect. Everyone else on the same switch is fine.

Live Logs. Event 5400 Authentication failed, failure reason 22056 Subject not found in the applicable identity store(s). The username in the report is INVALID, not what the user typed. ISE hides usernames that don't exist, so you can't search Live Logs for the bad name. Search by MAC or switch port instead.

The steps show where it stopped:

15013  Selected Identity Source - Internal Users
24210  Looking up User in Internal Users IDStore - INVALID
24216  The user is not found in the internal users identity store
22056  Subject not found in the applicable identity store(s)
22058  The advanced option that is configured for an unknown user is used
22061  The 'Reject' advanced option is configured in case of a failed authentication request

Root cause. The supplicant was configured with hd-0l (lowercase L) instead of hd-01 (zero). It's a typo, but on a real network it's just as often a user who was never created, or an identity source sequence that doesn't include the store the user lives in.

Fix. Correct the identity on the supplicant. If the name is right, check the identity source in the policy set's authentication rule.

Verify. The next attempt found the user (step 24210 shows the real name) and moved on to the password check.

Case C: 22063 Wrong password

Symptom. Same as case B from the user's side: the laptop prompts again or just gives up.

Live Logs. Failure reason 22063 Wrong password. ISE found the user (24212 Found User in Internal Users IDStore) and then rejected the inner MSCHAPv2 exchange:

24212  Found User in Internal Users IDStore
22063  Wrong password
22057  The advanced option that is configured for a failed authentication request is used
22061  The 'Reject' advanced option is configured in case of a failed authentication request
11823  EAP-MSCHAP authentication attempt failed
5411   Supplicant stopped responding to ISE

Notice the event here isn't 5400. With PEAP the wrong password happens inside the tunnel, ISE sends another challenge, and the supplicant walks away. In my lab that showed up as 5411 Supplicant stopped responding to ISE and 5440 Endpoint abandoned EAP session and started new, both with 22063 as the failure reason. If you filter Live Logs on 5400 only, you'll miss wrong passwords.

Root cause. The supplicant had a stale password for hd-01.

Fix. Update the password on the client. If it's a cached credential on a laptop, clear it, don't just type a new one at the next prompt.

Verify. The final run with the correct password passed (see the pass case below).

Case D: 5400 Authentication failed, 12321 client rejected the ISE certificate

Symptom. The laptop fails instantly, often before the user is asked for anything. On the host side, wpa_supplicant was clear about it:

eth1: CTRL-EVENT-EAP-TLS-CERT-ERROR reason=1 depth=0 subject='/CN=ise35.example.com' err='self-signed certificate'
EAP: Status notification: local TLS alert (param=unknown CA)
eth1: CTRL-EVENT-EAP-FAILURE EAP authentication failed

Live Logs. Event 5400, failure reason 12321 PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate. The steps end with:

12815  Extracted TLS Alert message
12817  TLS handshake failed
12321  PEAP failed SSL/TLS handshake because the client rejected the ISE local-certificate
12307  PEAP authentication failed
11504  Prepared EAP-Failure
11003  Returned RADIUS Access-Reject

The OpenSSLErrorMessage attribute names the reason: SSL alert ... source=remote ... message="unknown CA". Remote means the client sent the alert. Also note the username is USERNAME, not hd-01. The handshake fails before the inner identity is ever sent, so searching for the user's name won't find this row either.

Root cause. The supplicant was set to trust a different CA (an old root) instead of the one that signed the ISE EAP certificate. In this lab ISE still had its self-signed certificate, so the client had nothing to chain it to.

Fix. Either give the client the CA that issued the ISE EAP certificate, or (better) replace the ISE EAP certificate with one from a CA your clients already trust. Don't fix it by turning off server validation on the client. See Configuring PEAP Authentication with Cisco ISE and EAP-TLS certificates for the certificate side.

Verify. With the right trust setting the TLS tunnel came up and the attempt moved on to MSCHAPv2.

Case E: 5400 Authentication failed, 24206 User disabled

Symptom. A user who worked yesterday can't connect today.

Live Logs. Event 5400, failure reason 24206 User disabled. Unlike case B, the username is real (hd-02) because the account exists:

15013  Selected Identity Source - Internal Users
24210  Looking up User in Internal Users IDStore - hd-02
24206  User disabled
22057  The advanced option that is configured for a failed authentication request is used
22061  The 'Reject' advanced option is configured in case of a failed authentication request

Root cause. The account status was Disabled under Administration > Identity Management > Identities > Users. In production this is often deliberate (offboarding, or a password expiry policy), so check why before you re-enable it.

Fix. Enable the account if it should be enabled. Otherwise, the reject is the correct result and the ticket goes to whoever owns that user.

The pass, for comparison

With the right secret, the right username and password, and a supplicant that trusts the ISE certificate, the port authorizes:

acc-sw4#show authentication sessions interface Gi0/1 details
            Interface:  GigabitEthernet0/1
          MAC Address:  0000.5e00.5331
            User-Name:  hd-01
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  single-host
    Common Session ID:  0A56FC0A00000011000C6DFB
       Current Policy:  POLICY_Gi0/1

Method status list:
      Method            State
      dot1x              Authc Success

Live Logs shows event 5200 Authentication succeeded, with Wired-Access >> Wired-Staff-Permit as the authorization policy and PermitAccess as the result. Keep a passing report like this open next to a failing one. Comparing the two Steps lists side by side is the fastest way to see where a failure goes off track.

A side note: the MAC changed and the port err-disabled

Between case A and case B, the host's NIC MAC changed (that's why case A shows 0000.5e00.5301 and the rest show 0000.5e00.5331). The port was in single-host mode, so the switch treated the new MAC as a second device:

%AUTHMGR-5-SECURITY_VIOLATION: Security violation on the interface GigabitEthernet0/1, new MAC address (0000.5e00.5331) is seen.AuditSessionID  Unassigned
%PM-4-ERR_DISABLE: security-violation error detected on Gi0/1, putting Gi0/1 in err-disable state

Nothing reached ISE, so Live Logs had nothing for it. A shutdown / no shutdown on the port cleared it. If a user swaps docks or adapters and the port goes dead with no Live Logs entry, check show logging on the switch first.

Troubleshooting table

Takeaways

• Read the failure reason, then the last few Steps. The Steps tell you exactly where ISE stopped: before EAP, during TLS, at the user lookup or at the password.
• Timeouts on the switch aren't always a reachability problem. A bad shared secret makes ISE drop requests silently (5405 / 11036).
• Don't search only by username. Unknown users show as INVALID, and a failed TLS handshake can show a generic username. Search by MAC or port.
• Don't filter only on 5400. PEAP wrong passwords can land as 5411 or 5440 with 22063.
• No Live Logs entry means the problem is before ISE. Check the switch logs.

If the user passes but ends up with the wrong access, that's an authorization problem, not an authentication one. See the ISE dACL DNS implicit deny lab and the MAB 15039 printer lab.