fix: use semantic qualifiers for stable id_path construction

[hf-kklein]

Summary

Fixes #256. Replaces the positional @sort_path tiebreaker in id_path construction with semantic qualifiers derived from EDIFACT code values, making id_path stable across format versions.

This is based on the excellent bug analysis by @hf-mrdachner in #256, who correctly identified that positional indices in id_path shift when segments are inserted between versions, causing v_ahb_diff to produce false adds/deletes. Their analysis in #258 and #259 also identified important edge cases that informed this fix.

The Problem

When a new segment is inserted in a newer format version, all subsequent positional indices shift:

FV2510: SG4>DTM>@00004-00001-   (position 4)
FV2604: SG4>DTM>@00005-00001-   (position shifted to 5 because new SG inserted before it)

The v_ahb_diff JOIN on id_path fails → false adds + deletes for unchanged lines. For UTILMD/44001 (FV2510→FV2604), this produced 69 "added" + 64 "deleted" instead of just 5 genuinely new lines.

The Fix

Approach: Qualifier injection during CTE construction

Instead of fixing the diff view (as proposed in #257), we fix id_path construction itself at the source — in materialize_ahb_view.sql. This preserves id_path as a single unique column, maintaining all downstream guarantees.

Before (positional, unstable):

SG4>DTM>@00004-00001-          → shifts when segments are inserted
SG4>DTM>C_C507>@00004-00001-00001-
SG2>SG5>SG8>@00003-00003-00006-

After (semantic, stable):

SG4>DTM+92>                    → qualifier 92 = "Beginn zum" (stable across versions)
SG4>DTM+92>C_C507>             → children inherit parent's qualifier prefix
SG2>SG5>SG8+Z36>               → qualifier Z36 identifies the segment group

How it works

1. Pre-compute qualifiers — Before the main CTE, we compute the first code_value in each segment/SG/DE's subtree using temp tables. This is the EDIFACT "qualifier" that semantically identifies the element (e.g., DTM qualifier 92 = delivery date, 93 = validity date).

2. Inject into CTE — The existing recursive CTE is modified to append +qualifier to the element ID when a sibling with the same ID exists under the same parent. Children automatically inherit the qualified prefix from their parent.

3. Fallback counter — For ~0.6% of rows where the qualifier is NULL or shared among siblings, a #N occurrence counter is appended. This uses ROW_NUMBER() pre-computed in a temp table (required because SQLite processes UPDATE rows sequentially, breaking self-referencing counters).

Edge cases handled

Case	Issue	How handled
IFTSTA STS segments (#258)	Flat SG naming, multiple STS with same qualifier	#N counter fallback
PARTIN FII D_3192 (#259)	Repeated "Name des Kontoinhabers"	#N counter fallback
MSCONS segment groups (#259)	Repeated "Referenzangaben"	#N counter fallback
PRICAT IMD (#259)	Repeated IMD segments	#N counter fallback
UTILMD segment insertion (#256)	Shifted positions between FV2510→FV2604	Qualifier resolves it
Segments with no codes	NULL qualifier	#N counter fallback

Why IFTSTA still needs the #N tiebreaker

IFTSTA (prüfi 21045) uses a flat segment group naming scheme where multiple STS segments under "Grund der Privilegierung nach EnFG" share the same structural path AND the same qualifying code value. Since the qualifier can't distinguish them, the #N counter is the only option. This is acceptable because these truly identical elements are stable in order across versions.

Production DB validation

Tested against the full production database (308,993 rows, 595 prüfidentifikators, 4 format versions):

Resolution method	Rows	%
Cascade from parent qualifier	131,716	85.4%
Own qualifier (first code in subtree)	21,665	14.0%
#N counter fallback	940	0.6%
Total disambiguated	154,321	100%

Zero duplicates. UNIQUE INDEX succeeds.

Cross-version stability

For UTILMD/44001 (FV2510→FV2604), >90% of id_path values now match between versions, compared to ~57% with the old @sort_path approach.

Test plan

• All 257 existing tests pass (snapshots updated)
• All 7 diff view tests pass (snapshots updated)
• New regression tests for each problematic message type: IFTSTA, PARTIN, MSCONS, PRICAT, UTILMD
• Cross-version id_path stability test (UTILMD/44001 FV2510→FV2604)
• UNIQUE INDEX constraint verified on production data
• Formatting, linting, type checking pass

🤖 Generated with Claude Code

[hf-mrdachner]

Testing against production data (FV2510 → FV2604)

Built a fresh DB with this branch and compared against the original (broken) view and our (path, segment_code) fix (#257):

Prüfi	Nachricht	Original (broken)	#257 (path,seg)	This PR
44001	UTILMD	+69 -64 ~27	+5 -0 ~1	+6 -1 ~1
31002	INVOIC	+0 -0 ~1	+0 -0 ~1	+0 -0 ~1
13016	MSCONS	+17 -0 ~0	+17 -0 ~0	+43 -26 ~0
37000	PARTIN	+2 -1 ~17	+2 -1 ~17	+2 -1 ~16

UTILMD/44001 — slightly different but close

+6/-1 instead of +5/-0 — one extra add and one delete. Minor.

MSCONS/13016 — worse than original

+43/-26 instead of +17/-0. The issue: the semantic qualifier for SG2 changed between versions. In FV2510 the id_path is SG2+9>NAD>#1, in FV2604 it's SG2+MS>NAD>. Both refer to "MP-ID Absender" but the chosen qualifier is different (9=GS1 code vs MS=NAD qualifier), so the JOIN still fails.

Similarly, SG1 becomes SG1+AGI in FV2604 — the segment group gained a qualifier it didn't have before.

This seems to be a case where the qualifier selection algorithm picks a different "first code" between versions, even though the semantic meaning is the same.

[hf-mrdachner]

Ich mag den Ansatz sehr das zu fixen! Anscheinend gibt es aber noch Probleme. Der id path ist doch aber gar nicht hauptsächlich für den cross version vergleich, oder? Wenn der Innerhalb einer Version konsistent ist, aber für den use case eben probleme hat, müssen wir den auch nicht unbedingt durch die Form pressen. Ich gehe den Weg aber gerne weiter!

[hf-kklein]

Thanks for the thorough testing, @hf-mrdachner! Great analysis.

I investigated the MSCONS/13016 case. The root cause is that the AHB itself changed:

FV2510: NAD>D_3035 (Beteiligter, Qualifier) has no codes listed — so the first code comes from C_C082>D_3055>9 (GS1 code agency)
FV2604: NAD>D_3035 now has code MS (Dokumenten-/Nachrichtenaussteller) — making MS the first code

The qualifying data element was there in both versions, but the AHB authors didn't list codes under it in FV2510. When they added MS in FV2604, our "first code in subtree" qualifier changed from 9 to MS.

This is fundamentally a case where the AHB data structure itself changed (a code was added to a DE that previously had none). No qualifier algorithm can produce the same result when the source data is different.

Options to discuss:

1. Accept that id_path is within-version only — as you noted, id_path guarantees uniqueness within (pruefidentifikator, edifact_format_version). Cross-version matching is a different problem that v_ahb_diff should solve with a smarter matching strategy (like your (path, segment_code) approach in fix: use (path, segment_code) instead of id_path as diff JOIN key #257, or fingerprint matching).

2. Hybrid approach — use semantic qualifiers for id_path (better than @sort_path for human readability and within-version stability), but make v_ahb_diff use a fallback matching strategy when id_path doesn't match.

3. More conservative qualifier selection — e.g., prefer codes from DEGs (composites like C_C082) over direct DEs, since composite qualifiers tend to be more stable. But this doesn't fully solve the problem.

What do you think? Option 1 or 2 seems most pragmatic to me. The current PR already improves id_path readability significantly (SG2+MS> is much more meaningful than SG2>@00003-) even if cross-version matching isn't perfect.

[hf-kklein]

@hf-mrdachner I pushed a fix for the MSCONS cross-version instability you found (c5b9266).

Root cause

The qualifier selection preferred direct DE codes (like D_3035>MS) over DEG-based codes (like C_C082>D_3055>9). In FV2510, D_3035 had no codes listed in the AHB, so the qualifier came from the composite C_C082>D_3055>9. In FV2604, the AHB authors added code MS to D_3035, making it the new "first code" — and breaking cross-version matching.

Fix

Reversed the COALESCE order: now DEG-based codes (from EDIFACT composites/MIG structure) are preferred over direct DE codes (AHB customizations). DEG structure is defined by the MIG and stable across AHB versions, while direct DE codes can be added/removed.

Results after fix

Prüfi	Nachricht	Before fix	After fix
13016	MSCONS	+43 -26 ~0	+22 -5 ~0 (141 unchanged)
44001	UTILMD	+6 -1 ~1	+6 -1 ~1 (211 unchanged)

MSCONS/13016 now has 95% cross-version id_path overlap (was broken). SG2 is stable as SG2+9>#1 / SG2+9>#2 in both versions.

Could you re-test with the latest push? Especially interested in the cases you checked before (INVOIC/31002, PARTIN/37000).

[hf-mrdachner]

Hmmm. Spannend. Ich muss das mal testen finde es aber komisch. NAD+MS war natürlich schon immer NAD+MS und auch in den vorherigen Versionen im EDIFACT und den Menschen AHBs. Dass das nicht in den gekauften Daten vom BDEW drin ist, klingt ja irgendwie eher nach einem entsprechenden Fehler, um den wir herum bauen. Und wir zeigen das doch in ahb-tabellen.de auch an? Da jetzt den Wert zu nehmen, der ja gar nicht die Bedeutung wiederspiegelt, fühlt sich irgendwie komisch an. Auch wenn das Ergebnis stimmt. Irgendwas in mir sagt, wir müssten eigentlich die Daten korrigieren 😅

[hf-kklein]

Irgendwas in mir sagt, wir müssten eigentlich die Daten korrigieren

wäre ja nicht das erste Mal, dass die XMLs Mist sind. Statustransaktionszusatzgrund oder was das letzte Mal das Problem war....

muss zugeben, dass ich in diesem Fall aber auch icht händisch verifiziert habe, ob das stimmt, was claude behauptet.

[hf-mrdachner]

ich hab nochmal nachgeahscut in ahb-tabellen.de und dem PDF für 13016 auf FV2504: die Daten sind einfach falsch. Natürlich ist das NAD+MS. Ein Feature rund um falsche Daten zu bauen, finde ich irgendwie falsch.

Ich sehe also zwei Ansätze:

• wir korrigieren die Daten und nehmen den neuen, guten Algorithmus
• wir nehmen den hybriden Ansatz, meinetwegen auch für explizit als fehlerhaft geflagte Daten (ist zwar auch ein Feature um einen Fehler gebaut, aber irgendwie besser

[hf-kklein]

SG2+9

wir müssten doch eigentlich auf NAD+MS und NAD+MR sehen statt auf die 9.

[hf-kklein]

@hf-mrdachner insgesamt ist es doch aber - unabhängig von den mscons quircks - eine verbesserung, dass man nicht so viel sort-path-artefakte im id-path hat. Das soltle die cross-version-kompatibilität schon verbessern

[hf-kklein]

Good catch, both of you. I reverted the DEG-first change (ecf57f7).

@hf-mrdachner is right — the XML data for FV2510 is simply missing the D_3035 codes that should be there. The PDF confirms NAD+MS and NAD+MR are the correct qualifiers. Building a workaround for broken data was the wrong call.

Reverted to direct-DE-first — this gives the semantically correct NAD+MS/NAD+MR when the data is correct. The MSCONS cross-version instability is a data quality issue, not an algorithm issue.

The algorithm itself is now correct:

• NAD+MS / NAD+MR (when D_3035 has codes) — meaningful, matches PDF
• NAD+9 fallback (when D_3035 is missing) — from C_C082>D_3055, less meaningful but unique

The data fix for FV2510 should be tracked separately — is there an existing issue for XML data corrections, or should we file one?

[hf-mrdachner]

eben. das war aber nicht im XML... Hab schon überlegt, ob wir einen Patch-Mechanismus dafür einbauen, um nicht in den fremden Daten zu schreiben. wobei da jetzt vielleicht auch egal ist. Eigentlich müsste man die beim BDEW reklamieren, als falsch

[hf-kklein]

Filed Hochfrequenz/xml-migs-and-ahbs#80 for the data bug.

Summary: Exactly 2 D_3035 elements are missing codes in MSCONS FV2410/FV2504/FV2510 (the two NAD segments under SG2). No other message type is affected. Fixed in FV2604 by the BDEW.

I reverted the DEG-first workaround — the algorithm now correctly uses direct-DE-first, which gives NAD+MS/NAD+MR when the data is correct. The MSCONS cross-version instability will resolve itself once the data is corrected in Hochfrequenz/xml-migs-and-ahbs#80.

[hf-mrdachner]

great work. bad data

🚀