Audit Unknown TLV Tags When SkipUnknownTLVTags Is Enabled

[alovak]

this is proposed by @FacundoMora

Summary

The current implementation provides two flags for handling unknown TLV tags:

• SkipUnknownTLVTags: When enabled, allows skipping unknown tags instead of returning an error
• StoreUnknownTLVTags: When enabled (and SkipUnknownTLVTags is also enabled), stores the skipped unknown tags in hexadecimal format

This is a great feature for handling messages with unexpected fields. However, there is currently no way to audit or track which tags were skipped/stored during unpacking.

Problem

In production environments, it's important to have visibility into which unknown tags are being encountered. This information is valuable for:

• Debugging: Understanding why certain data might be missing from parsed messages
• Compliance & Auditing: Tracking unexpected fields for regulatory or security purposes
• Spec Evolution: Card networks frequently introduce new tags in their specifications. Issuers may not have implemented these new tags yet, but still receive them in messages. Having visibility into these unknown tags helps identify which new fields from the network specification need to be mapped and implemented.
• Monitoring: Alerting on unexpected message formats from counterparties

Currently, when SkipUnknownTLVTags is enabled, unknown tags are processed without any way to know which tags were encountered or where they occurred in the message structure.

Proposed Solution

Add the ability to record the path IDs of unknown/skipped TLV tags during unpacking. This would allow consumers to audit which fields were ignored.

Possible Implementation Approaches

Option 1: Store in Message struct with dedicated field

Add a slice to iso8583.Message to track skipped tags:

type Message struct {
    // ... existing fields
    skippedTags []string // e.g., ["48.99", "55.DF8101", "112.45.3"]
}

func (m *Message) SkippedTags() []string {
    return m.skippedTags
}

Option 2: Include in MarshalJSON output

When marshaling to JSON, include skipped tags in a separate key:

{
    "0": "0100",
    "2": "4111111111111111",
    "48": {
        "1": "value1",
        "2": "value2"
    },
    "_skipped_tags": ["48.99", "55.DF8101"]
}

Option 3: Callback/Hook mechanism

Provide a callback function that gets invoked when a tag is skipped:

type UnpackOptions struct {
    SkipUnknownTLVTags bool
    OnSkippedTag       func(fullPath string, data []byte)
}

Benefits

• Non-breaking: Can be implemented without changing existing behavior
• Optional: Only consumers who need auditing would use it
• Flexible: Multiple implementation options to fit different use cases

Use Case Example

msg := iso8583.NewMessage(spec)
msg.Unpack(data)

// After unpacking, check what was skipped
if skipped := msg.SkippedTags(); len(skipped) > 0 {
    log.Warn("Unknown TLV tags encountered", 
        "tags", skipped,
        "mti", msg.MTI())
}

[Stashchenko]

To me, both options look good 👍
#404
#405

But I think there are a few points worth to checkout.

Helper-based approach

Pros

• Conceptually simpler.
• It feels like parsing and packing might be faster since there’s no additional state being stored.
• Less extra logic inside the message structure.

Cons

• To retrieve Unknown, we would need to essentially iterate over the whole message.
• If Unknown fields are accessed frequently, this could become a bottleneck.

Approach with storing unknownTags during parsing

Pros

• Accessing Unknown should be faster and simpler since they’re already collected.
• No need to rescan the entire message.

Cons/Questions

• How will this affect parsing performance?
• Is there any noticeable memory overhead?
• Will it impact packing performance?
• Does it complicate the message lifecycle (especially if the message is reused)?

---

I think instead of guessing, it would be better to measure it:

• Benchmark Unpack
• Benchmark Pack
• Benchmark Get of unknownTags

And then choose the approach based on actual numbers.
If the difference is negligible, we might prefer the simpler and more maintainable solution.

As my initial take, I’d assume that storing unknownTags shouldn’t introduce significant overhead during pack/unpack, while at the same time providing the fastest way to access unknownTags. However better to measure :)

[alovak]

@Stashchenko benchmarking it is tricky. Because data everyone working with is different. Someone has all fields defined, someone only part of them. Storing unknown tags adds more memory allocations + some perf impact.

I'll try to build some artificial data to see the difference.