nat44-ed: SIGSEGV in out2in slow path on ICMP error carrying GRE/IPIP inner packet (26.06)

[TeTecko]

Summary

On VPP 26.06-release (nat44-ed, on-a-stick — one interface configured as both NAT in and out,
so the classification feature is active), a worker thread reliably SIGSEGVs in
nat44_ed_out2in_slowpath_node_fn under normal CGNAT traffic. For ~39 s before the crash the log
floods with:

nat_6t_flow_icmp_translate:3955: unexpected NAT protocol value 47' nat_6t_flow_icmp_translate:3955: unexpected NAT protocol value 4'

i.e. an ICMP error message whose quoted inner packet is GRE (proto 47) or IPIP (proto 4) —
generated naturally by subscribers running PPTP (GRE) / IP-in-IP tunnels.

Environment

• VPP 26.06-release (x86_64-v4 build), vpp_plugins/nat_plugin.so
• Debian 13 (trixie), kernel 6.12.94, Intel E810 via DPDK/vfio-pci, 16 worker threads
• nat44-ed (endpoint-dependent), on-a-stick: both interfaces nat44 in out (classification feature)
• CGNAT pool /24 + 375 address-only 1:1 static mappings, ~1.3–1.5M sessions, ~930k pps

Steps to reproduce

1. nat44-ed on-a-stick (one interface configured as both in and out) with a NAT pool / static-mapped
   external address.
2. From outside, send to that NATed external address an ICMP error (dest-unreachable / TTL-exceeded /
   frag-needed) whose quoted inner IPv4 header has protocol = 47 (GRE) or 4 (IPIP).
3. Drive volume through the out2in slow path. The log floods with nat_6t_flow_icmp_translate:3955: unexpected NAT protocol value '47'/'4', then a worker SIGSEGVs in nat44_ed_out2in_slowpath_node_fn.

Crash details

Syslog:
received signal SIGSEGV, PC 0x..b957, faulting address 0x..
#0 nat44_ed_out2in_slowpath_node_fn_x86_64_v4 + 0x9e7 (nat_plugin.so)

Backtrace (gdb on core):
#6 nat44_ed_out2in_slowpath_node_fn_x86_64_v4 () nat_plugin.so
#7 dispatch_node () libvlib
#8 vlib_worker_thread_bootstrap_fn () libvlib

Faulting instruction:
nat44_ed_out2in_slowpath_node_fn_x86_64_v4+2535: mov %ebp,0x4(%rcx,%rbx,4)

followed by lea (%rbp,%rbp,2) (×3) and mov qword [..],-1 — the shape of clib_dlist
manipulation on the per-thread LRU pool (dlist_elt_t = {next,prev,value} = 3×u32, -1 = invalid-index
sentinel).

Source analysis

nat_6t_flow_icmp_translate() (nat44_ed.c) switches on the inner packet's protocol; for anything other
than TCP/UDP/ICMP (GRE 47, IPIP 4) it returns cleanly:

3954    default:
3955      clib_warning ("unexpected NAT protocol value `%d'", inner_proto);
3956      return NAT_ED_TRNSL_ERR_TRANSLATION_FAILED;
The node-fn callers handle that return correctly (next=DROP; b0->error; goto trace0), so this is not
a missing-drop bug.

Suspected mechanism (UNVERIFIED — static analysis + faulting-instruction shape only):
icmp_out2in_ed_slow_path() (nat44_ed_out2in.c) creates a session for the inbound ICMP error and links
it into the per-thread LRU dlist. Back in the node fn (~l.1122-1131) the follow-up
nat_6t_flow_buf_translate_o2i() then fails for the GRE/IPIP inner packet, so the packet is dropped — but
unlike the FLOW_MISMATCH path (~l.936-939, which does nat44_ed_free_session_data + nat_ed_session_delete)
this drop path does not delete the just-created session, leaking it in the LRU dlist. Accumulation
corrupts the dlist → later SIGSEGV in dlist code (consistent with the ~39 s warning flood beforehand).

Candidate fix (unverified): on that drop path, free_session_data + session_delete the just-created
s0 (mirroring l.936-939). Reproducing + verifying on a debug build; will follow up with a PR if confirmed.

Impact
Worker SIGSEGV → VPP restart. Triggered by everyday GRE/IPIP tunnel traffic, so a nat44-ed node cannot
stay up reliably. The crash is in LRU clib_dlist code but the trigger differs from VPP-2117 / CSIT-1947
(not the eviction path — a session leaked into the LRU on an ICMP-error translate failure).

Artifacts
A 12 GB core (VPP 26.06-release, core.vpp_wk_15.*) is available on request.

Question
Does this analysis match a known issue, or is it already fixed on master? Happy to test a patch.

[TeTecko]

Correction / update — deeper core analysis (supersedes the diagnosis in the issue body):

I disassembled the faulting frame in the core. The crash is not a separate bug and not a
leaked/undeleted session — please disregard those parts of the original text.

The faulting code at nat44_ed_out2in_slowpath_node_fn+2499..+2547 is an inlined clib_dlist_remove
on the per-thread LRU pool, i.e. nat44_session_update_lru() (nat44_ed_inlines.h:815):

+2524 mov (%rcx,%rsi,4),%edi ; edi = pool[s->lru_index].next
+2527 mov 0x4(%rcx,%rsi,4),%ebp ; ebp = pool[s->lru_index].prev
+2531 lea (%rdi,%rdi,2),%rbx ; rbx = next*3 (dlist_elt = 3×u32)
=2535 mov %ebp,0x4(%rcx,%rbx,4) ; pool[next].prev = prev <-- SIGSEGV
+2547 movq $-1,(%rcx,%rsi,4) ; pool[s->lru_index].{next,prev} = -1

Crash-frame registers: rdi = rbp = 0xffffffff, i.e. the element at s->lru_index already has
next = prev = -1. Since clib_dlist_remove is exactly what sets a removed element's next/prev to -1
(+2547), and the session was linked via nat_ed_lru_insert (addtail), the -1/-1 state means the
element was already removed once → this is a double clib_dlist_remove (use-after-remove) of the
same session's LRU element; clib_dlist_remove has no guard, so it dereferences pool[-1].

So this is the VPP-2117 / CSIT-1947 family (clib_dlist_remove in nat44_session_update_lru, LRU
list corruption). The ICMP-error packets carrying GRE(47)/IPIP(4) inner are only a trigger/amplifier:
they spawn many short-lived ICMP sessions → session-table / LRU eviction pressure
(clib_dlist_remove_head, nat44_ed_inlines.h:334) that exposes the corruption.

I retract the earlier "add free+delete on the drop path" suggestion — nat44_ed_free_session_data also
calls clib_dlist_remove, so it would hit the same fault. The fix belongs in the LRU session lifecycle.

Is there a known/merged fix for VPP-2117 on master or a later branch? 12 GB core (26.06-release) available.