ASA NAT Troubleshooting: Stale Manual NAT Overrides Static NAT

Symptom: the published website times out

The static NAT rule is still in the ASA configuration. The outside ACL still permits HTTP to the DMZ server. Inside users can reach the outside test host, but requests to the published website time out.

This lab reproduces that failure with one deliberate change: a stale manual NAT rule maps the same published address to an obsolete backend. The ASA selects that rule before the working object NAT rule. The resulting ACL denial names the wrong server address, which is the clue to follow before changing policy.

Three outside HTTP requests fail in the faulted phase. This is one recorded attempt, run from the lab host against the outside client container; the exit-status line is recorder metadata. Source: failure traffic.

$ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/
curl: (28) Connection timed out after 4001 milliseconds
[probe exit status: 28]

The repair removes one NAT rule. The same HTTP probes then succeed without changing the outside ACL.

Topology and addressing

The four-node lab uses ASAv 9.24(1), an inside client, a DMZ web server, and an outside host that acts as both an HTTP client and a test server. The outside segment is an isolated internet stand-in. All probes use the data interfaces.

The inside client tests outbound PAT by contacting 198.51.100.80. The outside host tests the static publish by contacting 198.51.100.50. Those are separate flows, so working PAT does not prove the inbound publish works.

Traffic was captured at the outside host's eth1 and the web server's eth1. Zeek analyzed those captures offline. In the artifact names, baseline means the initial working state, before means the injected fault before repair, and after means the repaired state.

Baseline configuration

The ASA publishes DMZ-WEB with static object NAT and translates INSIDE-NET with dynamic interface PAT. The outside ACL permits TCP/80 to the real server object, DMZ-WEB.

For background on that publish pattern, see Cisco ASA static NAT for a DMZ server.

Apply these data-plane commands in ASA configuration mode on an otherwise prepared lab. This excerpt comes directly from the supplied startup configuration; management configuration is outside this walkthrough.

interface GigabitEthernet0/0
 nameif inside
 security-level 100
 ip address 10.64.10.1 255.255.255.0
 no shutdown
interface GigabitEthernet0/1
 nameif dmz
 security-level 50
 ip address 10.64.20.1 255.255.255.0
 no shutdown
interface GigabitEthernet0/2
 nameif outside
 security-level 0
 ip address 198.51.100.1 255.255.255.0
 no shutdown
object network INSIDE-NET
 subnet 10.64.10.0 255.255.255.0
 nat (inside,outside) dynamic interface
object network DMZ-WEB
 host 10.64.20.10
 nat (dmz,outside) static 198.51.100.50
access-list OUTSIDE-IN extended permit tcp any object DMZ-WEB eq 80 log
access-group OUTSIDE-IN in interface outside
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
logging enable
logging buffered informational
logging timestamp

For this publish, 198.51.100.50 is the mapped address seen outside. 10.64.20.10 is the real destination used by the ACL. The configured static mapping is not limited to HTTP; the outside ACL is what permits TCP/80 here.

Verify the working baseline

Run the HTTP probe from the lab host, using the outside client container. All three baseline publish requests returned HTTP 200; one complete response is shown below. Source: baseline traffic.

$ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.14.8
Date: Fri, 16 Oct 2026 20:26:46 GMT
Content-type: text/html
Content-Length: 42
Last-Modified: Fri, 16 Oct 2026 20:09:33 GMT

PingLabz ASA NAT lab: DMZ web 10.64.20.10
[probe exit status: 0]

At the ASA privileged EXEC prompt, run packet-tracer input outside tcp 198.51.100.80 41080 198.51.100.50 80. The verified baseline selects the DMZ-WEB object NAT, translates the destination to .10, and allows the flow. These are excerpts from the same trace; ... replaces whole omitted lines. Source: baseline verification.

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 29944 ns
Config:
object network DMZ-WEB
 nat (dmz,outside) static 198.51.100.50
Additional Information:
NAT divert to egress interface dmz
Untranslate 198.51.100.50/80 to 10.64.20.10/80

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 6304 ns
Config:
access-group OUTSIDE-IN in interface outside
Additional Information:
...
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: allow
Time Taken: 126868 ns

The relevant show nat detail excerpt places both data-plane rules in section 2:

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 
    translate_hits = 0, untranslate_hits = 5
    Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32
2 (inside) to (outside) source dynamic INSIDE-NET interface 
    translate_hits = 6, untranslate_hits = 0
    Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24

The baseline also records three successful inside pings and HTTP 200 through PAT. Keep those as control tests when introducing the fault.

The fault: add a stale manual NAT rule

In ASA configuration mode, apply the supplied fault commands. They create an obsolete backend object and map it to the address already used by the working publish.

object network STALE-WEB
 host 10.64.20.99
object network PUBLIC-WEB
 host 198.51.100.50
nat (dmz,outside) 1 source static STALE-WEB PUBLIC-WEB

That final command installs the manual rule in section 1. The 1 sets its position within the manual NAT section. The original section 2 static object NAT remains configured.

Repeat the same traffic tests. All three outside publish attempts now time out with curl exit 28. Inside ping still succeeds with 0% loss, and inside HTTP still returns 200. The failure is specific to the published service path.

Evidence: follow the translated destination

Packet-tracer selects the stale rule, then drops at the ACL

Run the same outside packet-tracer command at the ASA privileged EXEC prompt. The faulted trace selects STALE-WEB PUBLIC-WEB and translates the published address to 10.64.20.99. Only then does the access-list phase drop the packet. Source: failure evidence.

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 59888 ns
Config:
nat (dmz,outside) source static STALE-WEB PUBLIC-WEB
Additional Information:
NAT divert to egress interface dmz
Untranslate 198.51.100.50/80 to 10.64.20.99/80

Phase: 2
Type: ACCESS-LIST
Subtype: 
Result: DROP
Elapsed time: 6796 ns
Config:
Implicit Rule
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: drop
Time Taken: 66684 ns
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame snp_classify_table_lookup:6167 flow (NA)/NA

The last line reports acl-drop, but the earlier UN-NAT phase explains why the packet reaches the ACL with the wrong destination. The ACL still permits 10.64.20.10; this packet has become a request for 10.64.20.99.

NAT counters show which rule receives the attempts

Run show nat detail on the ASA. This faulted snapshot shows the stale manual rule above the original object NAT:

Manual NAT Policies (Section 1)
1 (dmz) to (outside) source static STALE-WEB PUBLIC-WEB 
    translate_hits = 0, untranslate_hits = 13
    Source - Origin: 10.64.20.99/32, Translated: 198.51.100.50/32

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 
    translate_hits = 0, untranslate_hits = 5
    Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32
2 (inside) to (outside) source dynamic INSIDE-NET interface 
    translate_hits = 10, untranslate_hits = 0
    Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24

The stale rule has 13 untranslate_hits. The correct publish rule remains at 5, its baseline value. Inside PAT has advanced from 6 to 10 translate_hits, consistent with the working outbound controls.

These counters are snapshots from the recorded investigation. Packet-tracer evaluations can affect them, so 13 hits does not mean 13 curl requests. Use the rule selection and counter movement together with the real traffic evidence.

Syslog 106023 names the wrong real address

Run show logging on the ASA and inspect the denials for the outside client. Two recorded lines are enough to expose the mismatch:

Oct 16 2026 20:27:37: %ASA-4-106023: Deny tcp src outside:198.51.100.80/44722 dst dmz:10.64.20.99/80 by access-group "OUTSIDE-IN" [0x0, 0x0]
Oct 16 2026 20:27:38: %ASA-4-106023: Deny tcp src outside:198.51.100.80/44722 dst dmz:10.64.20.99/80 by access-group "OUTSIDE-IN" [0x0, 0x0]

The destination is dmz:10.64.20.99/80. That agrees with packet-tracer's stale translation. Expanding the ACL to permit .99 would not restore the intended publish: the lab has no server at that address.

Zeek records S0 during the failure

The faulted outside conn.log contains these three publish attempts, copied from the Zeek log:

{"ts":1792182456.935313,"uid":"CMPZmd4ySoTikHkojc","id.orig_h":"198.51.100.80","id.orig_p":44722,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.0824029445648193,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6}
{"ts":1792182461.107519,"uid":"Ca3GPj3sxkeEgs3Zx8","id.orig_h":"198.51.100.80","id.orig_p":49268,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.0702571868896484,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6}
{"ts":1792182465.251724,"uid":"Cv0KGxFVNzAZNcjC4","id.orig_h":"198.51.100.80","id.orig_p":49284,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.085987091064453,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6}

Each entry has conn_state set to S0, a SYN-only history of S, four originator packets, and zero responder packets. Across the three attempts, that is twelve SYNs without a response at the outside capture point.

The DMZ capture recorded zero packets. This adjacent pair of tcpdump summaries is from the failure traffic artifact: outside first, then DMZ.

tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
31 packets captured
31 packets received by filter
0 packets dropped by kernel
tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel

No conn.log was generated for the faulted DMZ capture. Zeek does not see traffic to .99 on that link because the ASA drops the packets before transmitting them toward the DMZ.

S0 describes an unanswered connection attempt at this capture point. It does not identify a NAT fault by itself. Here, the empty DMZ capture, the selected NAT rule, and the denial naming .99 explain where the requests stop.

Diagnosis: section 1 wins before section 2

The two rules overlap on 198.51.100.50, but they live in different NAT sections. The relevant order in this lab is:

For this outside-to-DMZ request, the ASA uses the static mapping in reverse, turning the mapped destination into the real destination. The earlier section 1 match takes precedence; the presence of the correct section 2 rule does not make it the selected rule.

The ACL denial is therefore downstream of the translation mistake. The ACL enforces the intended destination, while the stale NAT rule supplies the obsolete one. Read ASA NAT table order alongside this trace, and ASA NAT ACLs and real IP addresses for the address distinction.

Fix: remove the stale mapping

In ASA configuration mode, remove the single conflicting rule using the supplied fix command:

no nat (dmz,outside) source static STALE-WEB PUBLIC-WEB

The outside ACL and the DMZ-WEB static object NAT stay as configured. The recorded final configuration retains the unused STALE-WEB and PUBLIC-WEB address objects; neither has a NAT rule after this removal.

Re-verify the publish and the PAT control

Repeat the outside HTTP probe from the lab host. All three repaired publish requests returned HTTP 200. One complete response from the repaired traffic run:

$ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.14.8
Date: Fri, 16 Oct 2026 20:29:13 GMT
Content-type: text/html
Content-Length: 42
Last-Modified: Fri, 16 Oct 2026 20:09:33 GMT

PingLabz ASA NAT lab: DMZ web 10.64.20.10
[probe exit status: 0]

Run the same outside packet-tracer again on the ASA. It now selects DMZ-WEB, translates to 10.64.20.10, passes the ACL, and finishes with Action: allow. Source: final ASA verification.

packet-tracer input outside tcp 198.51.100.80 41080 198.51.100.50 80

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 57524 ns
Config:
object network DMZ-WEB
 nat (dmz,outside) static 198.51.100.50
Additional Information:
NAT divert to egress interface dmz
Untranslate 198.51.100.50/80 to 10.64.20.10/80

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 11426 ns
Config:
access-group OUTSIDE-IN in interface outside
Additional Information:
...
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: allow
Time Taken: 243098 ns

The final show nat detail snapshot no longer contains the stale section 1 rule. The relevant section 2 excerpt shows the correct publish counter increasing from 5 to 9 and the inside PAT counter reaching 16:

Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 
    translate_hits = 0, untranslate_hits = 9
    Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32
2 (inside) to (outside) source dynamic INSIDE-NET interface 
    translate_hits = 16, untranslate_hits = 0
    Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24

The repaired inside controls also pass: three pings with 0% loss, HTTP 200 through PAT, and an allowed inside packet-tracer simulation.

Zeek records three SF publish sessions on each side after repair, consistent with established connections that closed normally. These paired examples use client source port 54948. On the outside capture, the responder is the published address:

{"ts":1792182553.39728,"uid":"CROinO36ppAlmNM6x","id.orig_h":"198.51.100.80","id.orig_p":54948,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","service":"http","duration":0.005959033966064453,"orig_bytes":77,"resp_bytes":227,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADadfF","orig_pkts":5,"orig_ip_bytes":345,"resp_pkts":5,"resp_ip_bytes":495,"ip_proto":6}

On the DMZ capture, the responder is the real server:

{"ts":1792182553.398496,"uid":"CATKOz4To9kufE0E8l","id.orig_h":"198.51.100.80","id.orig_p":54948,"id.resp_h":"10.64.20.10","id.resp_p":80,"proto":"tcp","service":"http","duration":0.005201101303100586,"orig_bytes":77,"resp_bytes":227,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADadfF","orig_pkts":5,"orig_ip_bytes":345,"resp_pkts":5,"resp_ip_bytes":495,"ip_proto":6}

The DMZ HTTP log also records status 200 for that request. A later real HTTP probe produces an ACL permit and a built connection with both the real and mapped server addresses. Source: final runtime evidence.

Oct 16 2026 20:30:00: %ASA-6-106100: access-list OUTSIDE-IN permitted tcp outside/198.51.100.80(44254) -> dmz/10.64.20.10(80) hit-cnt 1 first hit [0x4d33a418, 0x00000000]
Oct 16 2026 20:30:00: %ASA-6-302013: Built inbound TCP connection 52 for outside:198.51.100.80/44254 (198.51.100.80/44254) to dmz:10.64.20.10/80 (198.51.100.50/80) -1 -1

After verifying recovery, save the ASA configuration from privileged EXEC. The recorded save completed successfully:

write memory
Building configuration...
Cryptochecksum: 70fb0c3b 640e5ec6 9ee7db71 2f9e5cfe 

11841 bytes copied in 0.80 secs
[OK]

The measured progression is:

Takeaways and troubleshooting checklist

A correct static NAT rule can remain visible while an earlier manual rule captures its traffic. In this lab, removing the stale mapping restores the publish with no ACL change. The useful clue is the real destination selected before the denial.

When a published service fails, work through the evidence in this order:

• Test the publish from outside and keep a separate outbound PAT control.
• Run packet-tracer with the client's source address and the published destination. Read the selected UN-NAT rule and real destination before the final action.
• Inspect show nat detail across the manual and object NAT sections. Look for overlapping mapped addresses and compare counter snapshots.
• Match syslog 106023's destination to the intended real server and the ACL object.
• Compare outside and DMZ captures. Treat Zeek S0 as evidence of unanswered attempts, then use device evidence to explain them.
• Remove the confirmed stale rule, repeat the same probes, and verify the chosen translation, HTTP response, counters, and Zeek sessions.
• Save the repaired configuration after the publish and the control tests pass.