> ## Content Index > Fetch the complete content index at: https://www.pinglabz.com/llms.txt > Use this file to discover other available public pages before exploring further. # ASA NAT Troubleshooting: Stale Manual NAT Overrides Static NAT - URL: https://www.pinglabz.com/asa-nat-troubleshooting-stale-manual-rule/ - Published: 2026-10-16T20:42:31.000Z - Updated: 2026-10-16T20:42:31.000Z - Description: Fix an ASA static NAT publish hijacked by a stale manual rule. Follow packet-tracer, NAT hits, syslog 106023, and Zeek S0-to-SF recovery in this lab. - Author: Jaime - Tags: ASA, Labs, Security ![Lab topology: ASAv with inside, DMZ and outside Alpine hosts; 198.51.100.50 should map to 10.64.20.10 but a stale section 1 manual NAT rule maps it to 10.64.20.99](https://storage.ghost.io/c/ff/28/ff28db4a-11e1-4835-b928-158974cf96c1/content/images/2026/10/asa-nat-topology.png) ## Symptom: the published website times out The static NAT rule is still in the ASA configuration. The outside ACL still permits HTTP to the DMZ server. Inside users can reach the outside test host, but requests to the published website time out. This lab reproduces that failure with one deliberate change: a stale manual NAT rule maps the same published address to an obsolete backend. The ASA selects that rule before the working object NAT rule. The resulting ACL denial names the wrong server address, which is the clue to follow before changing policy. Three outside HTTP requests fail in the faulted phase. This is one recorded attempt, run from the lab host against the outside client container; the exit-status line is recorder metadata. Source: failure traffic. ```text $ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/ curl: (28) Connection timed out after 4001 milliseconds [probe exit status: 28] ``` The repair removes one NAT rule. The same HTTP probes then succeed without changing the outside ACL. ## Topology and addressing The four-node lab uses ASAv 9.24(1), an inside client, a DMZ web server, and an outside host that acts as both an HTTP client and a test server. The outside segment is an isolated internet stand-in. All probes use the data interfaces. | Node or role | Interface | Address | Gateway or translation | | ----------------- | ------------------- | ---------------- | ----------------------------------------------- | | ASA inside | GigabitEthernet0/0 | 10.64.10.1/24 | Security level 100 | | ASA DMZ | GigabitEthernet0/1 | 10.64.20.1/24 | Security level 50 | | ASA outside | GigabitEthernet0/2 | 198.51.100.1/24 | Security level 0 | | Inside client | eth1 | 10.64.10.10/24 | Gateway 10.64.10.1; PAT to 198.51.100.1 | | DMZ web server | eth1 | 10.64.20.10/24 | Gateway 10.64.20.1; static NAT to 198.51.100.50 | | Outside test host | eth1 | 198.51.100.80/24 | Gateway 198.51.100.1 | | Published address | NAT mapping | 198.51.100.50 | Intended backend 10.64.20.10 | | Stale backend | Address object only | 10.64.20.99 | No host exists at this address | The inside client tests outbound PAT by contacting `198.51.100.80`. The outside host tests the static publish by contacting `198.51.100.50`. Those are separate flows, so working PAT does not prove the inbound publish works. Traffic was captured at the outside host's `eth1` and the web server's `eth1`. Zeek analyzed those captures offline. In the artifact names, `baseline` means the initial working state, `before` means the injected fault before repair, and `after` means the repaired state. ## Baseline configuration The ASA publishes `DMZ-WEB` with static object NAT and translates `INSIDE-NET` with dynamic interface PAT. The outside ACL permits TCP/80 to the real server object, `DMZ-WEB`. For background on that publish pattern, see [Cisco ASA static NAT for a DMZ server](https://www.pinglabz.com/cisco-asa-static-nat-dmz/). Apply these data-plane commands in ASA configuration mode on an otherwise prepared lab. This excerpt comes directly from the supplied startup configuration; management configuration is outside this walkthrough. ```cisco interface GigabitEthernet0/0 nameif inside security-level 100 ip address 10.64.10.1 255.255.255.0 no shutdown interface GigabitEthernet0/1 nameif dmz security-level 50 ip address 10.64.20.1 255.255.255.0 no shutdown interface GigabitEthernet0/2 nameif outside security-level 0 ip address 198.51.100.1 255.255.255.0 no shutdown object network INSIDE-NET subnet 10.64.10.0 255.255.255.0 nat (inside,outside) dynamic interface object network DMZ-WEB host 10.64.20.10 nat (dmz,outside) static 198.51.100.50 access-list OUTSIDE-IN extended permit tcp any object DMZ-WEB eq 80 log access-group OUTSIDE-IN in interface outside policy-map global_policy class inspection_default inspect icmp service-policy global_policy global logging enable logging buffered informational logging timestamp ``` For this publish, `198.51.100.50` is the mapped address seen outside. `10.64.20.10` is the real destination used by the ACL. The configured static mapping is not limited to HTTP; the outside ACL is what permits TCP/80 here. ## Verify the working baseline Run the HTTP probe from the lab host, using the outside client container. All three baseline publish requests returned HTTP 200; one complete response is shown below. Source: baseline traffic. ```text $ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/ HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.14.8 Date: Fri, 16 Oct 2026 20:26:46 GMT Content-type: text/html Content-Length: 42 Last-Modified: Fri, 16 Oct 2026 20:09:33 GMT PingLabz ASA NAT lab: DMZ web 10.64.20.10 [probe exit status: 0] ``` At the ASA privileged EXEC prompt, run `packet-tracer input outside tcp 198.51.100.80 41080 198.51.100.50 80`. The verified baseline selects the `DMZ-WEB` object NAT, translates the destination to `.10`, and allows the flow. These are excerpts from the same trace; `...` replaces whole omitted lines. Source: baseline verification. ```text Phase: 1 Type: UN-NAT Subtype: static Result: ALLOW Elapsed time: 29944 ns Config: object network DMZ-WEB nat (dmz,outside) static 198.51.100.50 Additional Information: NAT divert to egress interface dmz Untranslate 198.51.100.50/80 to 10.64.20.10/80 Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Elapsed time: 6304 ns Config: access-group OUTSIDE-IN in interface outside Additional Information: ... Result: input-interface: outside input-status: up input-line-status: up output-interface: dmz output-status: up output-line-status: up Action: allow Time Taken: 126868 ns ``` The relevant `show nat detail` excerpt places both data-plane rules in section 2: ```text Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 translate_hits = 0, untranslate_hits = 5 Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32 2 (inside) to (outside) source dynamic INSIDE-NET interface translate_hits = 6, untranslate_hits = 0 Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24 ``` The baseline also records three successful inside pings and HTTP 200 through PAT. Keep those as control tests when introducing the fault. ## The fault: add a stale manual NAT rule In ASA configuration mode, apply the supplied fault commands. They create an obsolete backend object and map it to the address already used by the working publish. ```cisco object network STALE-WEB host 10.64.20.99 object network PUBLIC-WEB host 198.51.100.50 nat (dmz,outside) 1 source static STALE-WEB PUBLIC-WEB ``` That final command installs the manual rule in section 1\. The `1` sets its position within the manual NAT section. The original section 2 static object NAT remains configured. Repeat the same traffic tests. All three outside publish attempts now time out with curl exit 28\. Inside ping still succeeds with 0% loss, and inside HTTP still returns 200\. The failure is specific to the published service path. ## Evidence: follow the translated destination ### Packet-tracer selects the stale rule, then drops at the ACL Run the same outside `packet-tracer` command at the ASA privileged EXEC prompt. The faulted trace selects `STALE-WEB PUBLIC-WEB` and translates the published address to `10.64.20.99`. Only then does the access-list phase drop the packet. Source: failure evidence. ```text Phase: 1 Type: UN-NAT Subtype: static Result: ALLOW Elapsed time: 59888 ns Config: nat (dmz,outside) source static STALE-WEB PUBLIC-WEB Additional Information: NAT divert to egress interface dmz Untranslate 198.51.100.50/80 to 10.64.20.99/80 Phase: 2 Type: ACCESS-LIST Subtype: Result: DROP Elapsed time: 6796 ns Config: Implicit Rule Additional Information: Result: input-interface: outside input-status: up input-line-status: up output-interface: dmz output-status: up output-line-status: up Action: drop Time Taken: 66684 ns Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame snp_classify_table_lookup:6167 flow (NA)/NA ``` The last line reports `acl-drop`, but the earlier UN-NAT phase explains why the packet reaches the ACL with the wrong destination. The ACL still permits `10.64.20.10`; this packet has become a request for `10.64.20.99`. ### NAT counters show which rule receives the attempts Run `show nat detail` on the ASA. This faulted snapshot shows the stale manual rule above the original object NAT: ```text Manual NAT Policies (Section 1) 1 (dmz) to (outside) source static STALE-WEB PUBLIC-WEB translate_hits = 0, untranslate_hits = 13 Source - Origin: 10.64.20.99/32, Translated: 198.51.100.50/32 Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 translate_hits = 0, untranslate_hits = 5 Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32 2 (inside) to (outside) source dynamic INSIDE-NET interface translate_hits = 10, untranslate_hits = 0 Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24 ``` The stale rule has 13 `untranslate_hits`. The correct publish rule remains at 5, its baseline value. Inside PAT has advanced from 6 to 10 `translate_hits`, consistent with the working outbound controls. These counters are snapshots from the recorded investigation. Packet-tracer evaluations can affect them, so 13 hits does not mean 13 curl requests. Use the rule selection and counter movement together with the real traffic evidence. ### Syslog 106023 names the wrong real address Run `show logging` on the ASA and inspect the denials for the outside client. Two recorded lines are enough to expose the mismatch: ```text Oct 16 2026 20:27:37: %ASA-4-106023: Deny tcp src outside:198.51.100.80/44722 dst dmz:10.64.20.99/80 by access-group "OUTSIDE-IN" [0x0, 0x0] Oct 16 2026 20:27:38: %ASA-4-106023: Deny tcp src outside:198.51.100.80/44722 dst dmz:10.64.20.99/80 by access-group "OUTSIDE-IN" [0x0, 0x0] ``` The destination is `dmz:10.64.20.99/80`. That agrees with packet-tracer's stale translation. Expanding the ACL to permit `.99` would not restore the intended publish: the lab has no server at that address. ### Zeek records S0 during the failure The faulted outside `conn.log` contains these three publish attempts, copied from the Zeek log: ```json {"ts":1792182456.935313,"uid":"CMPZmd4ySoTikHkojc","id.orig_h":"198.51.100.80","id.orig_p":44722,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.0824029445648193,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6} {"ts":1792182461.107519,"uid":"Ca3GPj3sxkeEgs3Zx8","id.orig_h":"198.51.100.80","id.orig_p":49268,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.0702571868896484,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6} {"ts":1792182465.251724,"uid":"Cv0KGxFVNzAZNcjC4","id.orig_h":"198.51.100.80","id.orig_p":49284,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","duration":3.085987091064453,"orig_bytes":0,"resp_bytes":0,"conn_state":"S0","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"S","orig_pkts":4,"orig_ip_bytes":240,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":6} ``` Each entry has `conn_state` set to `S0`, a SYN-only `history` of `S`, four originator packets, and zero responder packets. Across the three attempts, that is twelve SYNs without a response at the outside capture point. The DMZ capture recorded zero packets. This adjacent pair of tcpdump summaries is from the failure traffic artifact: outside first, then DMZ. ```text tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 31 packets captured 31 packets received by filter 0 packets dropped by kernel tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 0 packets captured 0 packets received by filter 0 packets dropped by kernel ``` No `conn.log` was generated for the faulted DMZ capture. Zeek does not see traffic to `.99` on that link because the ASA drops the packets before transmitting them toward the DMZ. `S0` describes an unanswered connection attempt at this capture point. It does not identify a NAT fault by itself. Here, the empty DMZ capture, the selected NAT rule, and the denial naming `.99` explain where the requests stop. ## Diagnosis: section 1 wins before section 2 The two rules overlap on `198.51.100.50`, but they live in different NAT sections. The relevant order in this lab is: | Order | Rule | Result for the incoming publish request | | --------------------- | ----------------------- | ----------------------------------------------------- | | Section 1: manual NAT | STALE-WEB → PUBLIC-WEB | Matches the published address and selects 10.64.20.99 | | Section 2: object NAT | DMZ-WEB → 198.51.100.50 | Correct mapping to 10.64.20.10 remains shadowed | For this outside-to-DMZ request, the ASA uses the static mapping in reverse, turning the mapped destination into the real destination. The earlier section 1 match takes precedence; the presence of the correct section 2 rule does not make it the selected rule. The ACL denial is therefore downstream of the translation mistake. The ACL enforces the intended destination, while the stale NAT rule supplies the obsolete one. Read [ASA NAT table order](https://www.pinglabz.com/asa-nat-table-order/) alongside this trace, and [ASA NAT ACLs and real IP addresses](https://www.pinglabz.com/asa-nat-acl-real-ip/) for the address distinction. ## Fix: remove the stale mapping In ASA configuration mode, remove the single conflicting rule using the supplied fix command: ```cisco no nat (dmz,outside) source static STALE-WEB PUBLIC-WEB ``` The outside ACL and the `DMZ-WEB` static object NAT stay as configured. The recorded final configuration retains the unused `STALE-WEB` and `PUBLIC-WEB` address objects; neither has a NAT rule after this removal. ## Re-verify the publish and the PAT control Repeat the outside HTTP probe from the lab host. All three repaired publish requests returned HTTP 200\. One complete response from the repaired traffic run: ```text $ docker exec clab-pl-asa-nat-internet curl --noproxy '*' -sS -i --connect-timeout 4 --max-time 6 http://198.51.100.50/ HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.14.8 Date: Fri, 16 Oct 2026 20:29:13 GMT Content-type: text/html Content-Length: 42 Last-Modified: Fri, 16 Oct 2026 20:09:33 GMT PingLabz ASA NAT lab: DMZ web 10.64.20.10 [probe exit status: 0] ``` Run the same outside packet-tracer again on the ASA. It now selects `DMZ-WEB`, translates to `10.64.20.10`, passes the ACL, and finishes with `Action: allow`. Source: final ASA verification. ```text packet-tracer input outside tcp 198.51.100.80 41080 198.51.100.50 80 Phase: 1 Type: UN-NAT Subtype: static Result: ALLOW Elapsed time: 57524 ns Config: object network DMZ-WEB nat (dmz,outside) static 198.51.100.50 Additional Information: NAT divert to egress interface dmz Untranslate 198.51.100.50/80 to 10.64.20.10/80 Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Elapsed time: 11426 ns Config: access-group OUTSIDE-IN in interface outside Additional Information: ... Result: input-interface: outside input-status: up input-line-status: up output-interface: dmz output-status: up output-line-status: up Action: allow Time Taken: 243098 ns ``` The final `show nat detail` snapshot no longer contains the stale section 1 rule. The relevant section 2 excerpt shows the correct publish counter increasing from 5 to 9 and the inside PAT counter reaching 16: ```text Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static DMZ-WEB 198.51.100.50 translate_hits = 0, untranslate_hits = 9 Source - Origin: 10.64.20.10/32, Translated: 198.51.100.50/32 2 (inside) to (outside) source dynamic INSIDE-NET interface translate_hits = 16, untranslate_hits = 0 Source - Origin: 10.64.10.0/24, Translated: 198.51.100.1/24 ``` The repaired inside controls also pass: three pings with 0% loss, HTTP 200 through PAT, and an allowed inside packet-tracer simulation. Zeek records three `SF` publish sessions on each side after repair, consistent with established connections that closed normally. These paired examples use client source port `54948`. On the outside capture, the responder is the published address: ```json {"ts":1792182553.39728,"uid":"CROinO36ppAlmNM6x","id.orig_h":"198.51.100.80","id.orig_p":54948,"id.resp_h":"198.51.100.50","id.resp_p":80,"proto":"tcp","service":"http","duration":0.005959033966064453,"orig_bytes":77,"resp_bytes":227,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADadfF","orig_pkts":5,"orig_ip_bytes":345,"resp_pkts":5,"resp_ip_bytes":495,"ip_proto":6} ``` On the DMZ capture, the responder is the real server: ```json {"ts":1792182553.398496,"uid":"CATKOz4To9kufE0E8l","id.orig_h":"198.51.100.80","id.orig_p":54948,"id.resp_h":"10.64.20.10","id.resp_p":80,"proto":"tcp","service":"http","duration":0.005201101303100586,"orig_bytes":77,"resp_bytes":227,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADadfF","orig_pkts":5,"orig_ip_bytes":345,"resp_pkts":5,"resp_ip_bytes":495,"ip_proto":6} ``` The DMZ HTTP log also records status 200 for that request. A later real HTTP probe produces an ACL permit and a built connection with both the real and mapped server addresses. Source: final runtime evidence. ```text Oct 16 2026 20:30:00: %ASA-6-106100: access-list OUTSIDE-IN permitted tcp outside/198.51.100.80(44254) -> dmz/10.64.20.10(80) hit-cnt 1 first hit [0x4d33a418, 0x00000000] Oct 16 2026 20:30:00: %ASA-6-302013: Built inbound TCP connection 52 for outside:198.51.100.80/44254 (198.51.100.80/44254) to dmz:10.64.20.10/80 (198.51.100.50/80) -1 -1 ``` After verifying recovery, save the ASA configuration from privileged EXEC. The recorded save completed successfully: ```text write memory Building configuration... Cryptochecksum: 70fb0c3b 640e5ec6 9ee7db71 2f9e5cfe 11841 bytes copied in 0.80 secs [OK] ``` The measured progression is: | Check | Baseline | Faulted | Repaired | | --------------------------------- | ------------ | ------------------------- | ------------ | | Outside HTTP to 198.51.100.50 | 3 × HTTP 200 | 3 × timeout, exit 28 | 3 × HTTP 200 | | Outside packet-tracer | .10, allow | .99, ACL drop | .10, allow | | Correct publish untranslate\_hits | 5 | 5 | 9 | | Stale rule untranslate\_hits | Absent | 13 | Absent | | Inside HTTP through PAT | HTTP 200 | HTTP 200 | HTTP 200 | | Outside Zeek publish sessions | 3 × SF | 3 × S0 | 3 × SF | | DMZ Zeek publish sessions | 3 × SF | No conn.log; zero packets | 3 × SF | ## Takeaways and troubleshooting checklist A correct static NAT rule can remain visible while an earlier manual rule captures its traffic. In this lab, removing the stale mapping restores the publish with no ACL change. The useful clue is the real destination selected before the denial. When a published service fails, work through the evidence in this order: - Test the publish from outside and keep a separate outbound PAT control. - Run packet-tracer with the client's source address and the published destination. Read the selected UN-NAT rule and real destination before the final action. - Inspect `show nat detail` across the manual and object NAT sections. Look for overlapping mapped addresses and compare counter snapshots. - Match syslog 106023's destination to the intended real server and the ACL object. - Compare outside and DMZ captures. Treat Zeek `S0` as evidence of unanswered attempts, then use device evidence to explain them. - Remove the confirmed stale rule, repeat the same probes, and verify the chosen translation, HTTP response, counters, and Zeek sessions. - Save the repaired configuration after the publish and the control tests pass.