Add HuggingFaceNerRecognizer for direct NER model inference

[ultramancode]

Summary

I'd like to propose adding a HuggingFaceNerRecognizer that uses HuggingFace Transformers pipeline directly for NER, bypassing spaCy tokenizer alignment issues.

Problem

The current approach using spaCy tokenizer with HuggingFace NER models has alignment issues for agglutinative languages (Korean, Japanese, Turkish, etc.):

• Particles/postpositions attach to nouns: "김태웅이고" (Kim Taewoong + particle)
• spaCy tokenizer: "김태웅이고" (includes particle)
• NER model result: "김태웅" (name only)
• char_span() alignment fails with alignment_mode="strict"
• alignment_mode="expand" includes particles, causing downstream issues

Example (Korean)

Step	Result
Input Text	"내 이름은 김태웅이고 전화번호는 010-1234-5678이야"
spaCy Token	"김태웅이고" (name + particle)
NER Result	"김태웅" (name only, start=6, end=9)
char_span(strict)	SKIP (boundary mismatch)
char_span(expand)	"김태웅이고" (includes particle → wrong!)

Proposed Solution

Create HuggingFaceNerRecognizer that:

1. Uses HuggingFace Transformers pipeline directly (bypasses spaCy tokenizer)
2. Returns NER results without char_span alignment
3. Supports any language with a HuggingFace NER model

Key Features

• Language-agnostic: Works with any HuggingFace NER model (English, Korean, Japanese, etc.)
• Direct inference: No spaCy tokenizer dependency for entity boundaries

[omri374]

Hi @ultramancode , thanks for you suggestion. There is already a sample doing that, but it might be a bit outdated: https://github.com/microsoft/presidio/blob/main/docs/samples/python/transformers_recognizer/index.md

If you end up using it, please consider contributing back any enhancements. Adding it to the main repo as an optional recognizer might be a good idea.

[ultramancode]

Thanks @omri374

The sample provided was a good reference and made it much easier to grasp the code logic.

Compared to the sample, the key improvements in this PR are:

• Batch processing: Supports batch inference, with fallback to iterative processing for compatibility.

• Deduplication: The sample uses set(tuple(d.items())) which only removes exact duplicates—if the same span is detected with different scores from overlapping chunks, both are kept. This implementation uses (start, end, entity_type) as the key and retains only the highest-scoring result.

• Built-in Label Mapping: Includes DEFAULT_LABEL_MAPPING with common NER labels, so users don't need a separate configuration file.

• Full YAML Configuration Support: Implemented dynamic argument loading in the YAML loader, allowing users to configure model_name, device and other parameters directly in recognizers.yaml without any code changes.