llm-pii-poc/docs/ner-models.md

NER Models for PII Redaction

This document describes the Named Entity Recognition (NER) ONNX models at the core of the PII Redaction POC. Person-name detection is the only NER responsibility in this solution; structured identifiers (email, phone, PAN, domain IDs) are handled by regex and domain-rule detectors.

For pipeline placement, trust boundaries, and routing diagrams, see architecture.md. For the Tamil/Tanglish implementation plan and success metrics, see tamil-tanglish-ner-plan.md.

---

1. Executive Summary

The POC uses dual-model ONNX NER routing to redact person-name PII before prompts reach an LLM:

Script in prompt	Model invoked	Typical use case
Latin only (LatinOnly)	English (dslim/bert-base-NER)	English names, Indian names in Roman script, Tanglish
Tamil only (TamilOnly)	Tamil (prachuryyaIITG/SampurNER_Tamil_IndicBERTv2)	Tamil-script customer names
Mixed (Mixed)	Both models on the full text; spans merged	Code-mixed Indian CS prompts
No letters (NoLetters)	Neither	Digits-only or symbol-only text

RoutingOnnxNerModelRunner classifies script via ScriptRouter, delegates to EnglishOnnxNerRunner and/or TamilOnnxNerRunner, and merges overlapping PERSON spans (longer span wins). Only PERSON entities are emitted to the redaction pipeline; all other NER labels are discarded.

---

2. English Model

Hugging Face model ID

dslim/bert-base-NER

Architecture

Property	Value
Base	BERT-base (uncased), ~110M parameters
Task	Token classification (NER)
Tokenizer	WordPiece via vocab.txt (BertWordPieceEncoder)
Runtime	ONNX via Microsoft.ML.OnnxRuntime
Export	Hugging Face Optimum (ORTModelForTokenClassification) or pre-exported ONNX from HF

Labels (BIO)

The English model uses standard CoNLL-style BIO tags. The POC maps only person labels to PiiEntityType.Person:

Label	Mapped to PERSON
O	No
B-PER, I-PER	Yes
B-PERSON, I-PERSON	Yes
B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC	No

Full label list is written to ner-labels.txt at download time from the model config.json id2label map (typically 9 labels for this model).

Label matching is implemented in NerLabelConfig.English:

    private static bool IsEnglishPersonLabel(string label) =>
        label is "B-PER" or "I-PER" or "B-PERSON" or "I-PERSON"
        || (label.EndsWith("-PER", StringComparison.Ordinal) &&
            (label.StartsWith("B-", StringComparison.Ordinal) || label.StartsWith("I-", StringComparison.Ordinal)));

Asset paths

File	Primary path (appsettings.json)	Legacy fallback
ONNX model	models/en/ner-model.onnx	models/ner-model.onnx (OnnxModelPath)
Vocabulary	models/en/vocab.txt	models/vocab.txt
Labels	models/en/ner-labels.txt	models/ner-labels.txt

EnglishOnnxNerRunner resolves the model path with primary + legacy fallback:

        var modelPath = OnnxAssetPathResolver.ResolveModelPath(
            options.Value.EnglishOnnxModelPath,
            options.Value.OnnxModelPath);

        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
        var encoder = new BertWordPieceEncoder(modelDirectory, logger);
        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.English, labels, logger);

Note: scripts/download-ner-model.ps1 writes assets to models/ (repository root). For the configured primary path, copy or move them into models/en/, or rely on the OnnxModelPath fallback.

Download script

.\scripts\download-ner-model.ps1

Or with Python directly:

python scripts/download-ner-model.py

Behavior:

1. If Python + Optimum are available → exports dslim/bert-base-NER to ONNX under models/.
2. Otherwise → downloads pre-exported ONNX from https://huggingface.co/dslim/bert-base-NER/resolve/main/onnx/ (model.onnx, vocab.txt, config.json → ner-labels.txt).

---

3. Tamil Model

Hugging Face model ID

prachuryyaIITG/SampurNER_Tamil_IndicBERTv2

(SampurNER Tamil IndicBERTv2 — fine-grained NER for Tamil script.)

Why SampurNER IndicBERTv2 vs MuRIL

Criterion	SampurNER Tamil IndicBERTv2	MuRIL (fallback candidate)
Tamil NER training	Fine-grained SampurNER dataset (Tamil-specific labels)	General multilingual; NER requires separate fine-tune
Model size	~0.3B parameters (IndicBERTv2, ~278M base)	~0.6B parameters
POC fit	Lighter memory footprint; ONNX export path validated in this repo	Reserved for Phase 5 if Tamil recall is insufficient
Indian financial context	Trained on Indian-language NER corpus; person subtypes map cleanly to PERSON	Heavier; eval-driven swap only

See tamil-tanglish-ner-plan.md §3 for the original selection rationale.

Architecture

Property	Value
Base	IndicBERTv2 (AI4Bharat), ~0.3B parameters
Task	Fine-grained token classification
Tokenizer	WordPiece when vocab.txt is present (this repo's export path); SentencePiece fallback if sentencepiece.bpe.model / spiece.model exists
Runtime	Same shared OnnxTokenClassifierRunner as English

Tokenizer: WordPiece, not SentencePiece (in practice)

The Hugging Face repo for this model does not ship a SentencePiece model file. The download scripts extract WordPiece assets from tokenizer.json → vocab.txt. TokenClassifierEncoderFactory prefers vocab.txt:

    public static ITokenClassifierEncoder Create(string modelDirectory, ILogger logger)
    {
        var vocabPath = OnnxAssetPathResolver.ResolveAssetPath(Path.Combine(modelDirectory, "vocab.txt"));
        if (File.Exists(vocabPath))
        {
            logger.LogInformation(
                "Using WordPiece tokenizer (vocab.txt) from {ModelDirectory}.",
                modelDirectory);
            return new BertWordPieceEncoder(modelDirectory, logger);
        }

The PowerShell Tamil download script emits an explicit warning when WordPiece assets are saved instead of SentencePiece.

Labels (fine-grained person tags)

SampurNER uses fine-grained BIO tags (e.g. B-person-politician, I-person-artist, B-location, O). The POC treats any label containing person (case-insensitive) as a person span:

    private static bool IsTamilPersonLabel(string label) =>
        label.Contains("person", StringComparison.OrdinalIgnoreCase);

Unit tests lock this behavior:

    [TestCase("B-person-politician", true)]
    [TestCase("I-person-artist", true)]
    [TestCase("B-location", false)]
    [TestCase("O", false)]
    public void Tamil_IsPersonLabel_MatchesFineGrainedTags(string label, bool expected)
    {
        NerLabelConfig.Tamil.IsPersonLabel(label).Should().Be(expected);
    }

Asset paths

File	Path
ONNX model	models/ta/model.onnx
Tokenizer	models/ta/vocab.txt (WordPiece, preferred) or models/ta/sentencepiece.bpe.model
Labels	models/ta/ner-labels.txt
Optional	models/ta/tokenizer.json (intermediate export artifact)

Download script

.\scripts\download-tamil-ner-model.ps1

Or with Python directly:

python scripts/download-tamil-ner-model.py

Behavior:

1. Python + Optimum → full export to models/ta/ including ONNX, labels, and tokenizer assets.
2. PowerShell fallback → downloads onnx/model.onnx from Hugging Face when published; otherwise requires Python export (pre-exported ONNX may return 404).

TamilOnnxNerRunner wiring:

        var modelPath = OnnxAssetPathResolver.ResolveModelPath(options.Value.TamilOnnxModelPath);
        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
        var encoder = TokenClassifierEncoderFactory.Create(modelDirectory, logger);
        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.Tamil, labels, logger);

---

4. Why These Models

Evidence-based rationale for this Indian financial POC:

Requirement	Decision
English + Indian Latin names	dslim/bert-base-NER is industry-standard, pre-integrated, and handles many Indian names in Roman script (e.g. Ravi Kumar, Anita Sharma)
Tamil script names	English BERT is out-of-vocabulary for Tamil letters (U+0B80–U+0BFF); a Tamil-trained NER model is required
Tanglish (Roman-script Tamil)	Routed to the English model only (ScriptComposition.LatinOnly); no Tamil ONNX on Latin-only text
Code-mixed prompts	Both models run on the full prompt; MergePersonSpans deduplicates overlaps
Deployable size	English ~431 MB ONNX (FP32); Tamil IndicBERTv2 ~0.3B params — smaller than MuRIL ~0.6B
ONNX export path	Both models export via Hugging Face Optimum; English has pre-exported ONNX on HF; Tamil may require local Python export
PERSON-only scope	POC redacts person names via NER; org/location/misc labels are intentionally ignored to limit false positives

---

5. How They Integrate

Configuration (appsettings.json)

{
  "PiiRedaction": {
    "OnnxModelPath": "models/ner-model.onnx",
    "EnglishOnnxModelPath": "models/en/ner-model.onnx",
    "TamilOnnxModelPath": "models/ta/model.onnx",
    "EnableTamilNer": true
  }
}

Options type:

public sealed class PiiRedactionOptions
{
    public const string SectionName = "PiiRedaction";

    public string OnnxModelPath { get; set; } = "models/ner-model.onnx";

    public string EnglishOnnxModelPath { get; set; } = "models/en/ner-model.onnx";

    public string TamilOnnxModelPath { get; set; } = "models/ta/model.onnx";

    public bool EnableTamilNer { get; set; } = true;
}

Set EnableTamilNer to false for English-only routing.

Dependency injection

        services.AddSingleton<EnglishOnnxNerRunner>();
        services.AddSingleton<TamilOnnxNerRunner>();
        services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();

OnnxNerPiiDetector consumes IOnnxNerModelRunner (the router) and returns [] when no model is available — fail-open for person detection.

Script routing (ScriptRouter)

    public ScriptComposition GetComposition(string text)
    {
        ArgumentNullException.ThrowIfNull(text);

        var hasLatin = false;
        var hasTamil = false;

        foreach (var character in text)
        {
            if (IsTamilLetter(character))
            {
                hasTamil = true;
            }
            else if (char.IsAsciiLetter(character))
            {
                hasLatin = true;
            }

            if (hasLatin && hasTamil)
            {
                return ScriptComposition.Mixed;
            }
        }
        // ...
        return hasTamil ? ScriptComposition.TamilOnly : ScriptComposition.LatinOnly;
    }

Routing runner (RoutingOnnxNerModelRunner)

    public IReadOnlyList<PiiEntity> PredictEntities(string text)
    {
        var composition = _scriptRouter.GetComposition(text);
        var entities = new List<PiiEntity>();

        switch (composition)
        {
            case ScriptComposition.LatinOnly:
                if (_englishRunner.IsModelAvailable)
                {
                    entities.AddRange(_englishRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.TamilOnly:
                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
                {
                    entities.AddRange(_tamilRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.Mixed:
                if (_englishRunner.IsModelAvailable)
                {
                    entities.AddRange(_englishRunner.PredictEntities(text));
                }
                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
                {
                    entities.AddRange(_tamilRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.NoLetters:
                break;
        }

        return MergePersonSpans(entities);
    }

Shared inference (OnnxTokenClassifierRunner)

Both runners share:

• Encode → input_ids, attention_mask, optional token_type_ids
• Argmax over per-token logits
• BIO decode → PiiEntityType.Person with PiiDetectionSource.Ner
• Max sequence length: 128 tokens

    private const int MaxSequenceLength = 128;

End-to-end flow

Prompt → CompositePiiDetector → OnnxNerPiiDetector
       → RoutingOnnxNerModelRunner → ScriptRouter
       → EnglishOnnxNerRunner / TamilOnnxNerRunner
       → OnnxTokenClassifierRunner → PERSON entities
       → PlaceholderPiiRedactor → <PERSON_n>

---

6. Evidence from Codebase

Real-model tests (Category=RealModel)

English direct inference — RealNerModelRunnerTests:

[Category("RealModel")]
public sealed class RealNerModelRunnerTests : RealNerModelFixture
{
    [TestCase("Customer Ravi Kumar called about billing.", "Ravi", "Ravi Kumar")]
    [TestCase("Mr. John Smith called about a duplicate debit.", "John", "John Smith")]
    public void PredictEntities_DetectsPersonWithCorrectSpan(...)

English pipeline — RealNerPipelineTests (Category=RealModel): canonical FullFinancialWithCustomer, multi-person, clean-ticket negative.

Fixture skips when model missing:

    public const string ModelMissingMessage =
        "ONNX model not found. Run scripts/download-ner-model.ps1 from the repository root.";

Resolves models/en/ner-model.onnx then models/ner-model.onnx.

Tamil tests (Category=TamilNer)

Direct Tamil runner — RealTamilNerModelRunnerTests:

[Category("TamilNer")]
public sealed class RealTamilNerModelRunnerTests : RealTamilNerModelFixture
{
    [TestCase("வாடிக்கையாளர் ராஜேஷ் குமார் அழைத்தார்.", "ராஜேஷ்", "ராஜேஷ் குமார்")]
    public void PredictEntities_TamilScript_DetectsPersonEntity(...)
    // ...
    [TestCase("Customer Senthil phone 9876543210", "Senthil")]
    public void PredictEntities_TanglishLatinScript_DoesNotInvokeTamilRunner(...)

Routed pipeline — RealTamilPipelineTests covers Tamil-only, Tanglish (English path), mixed, full financial, and clean Tamil negative.

Console samples (SamplePromptCatalog)

Tamil/Tanglish/mixed samples (indices 10–14):

Sample	Category	Input excerpt
TamilCustomerNameOnly	NER (Tamil)	வாடிக்கையாளர் ராஜேஷ் குமார் …
TamilWithPhonePan	NER (Tamil) + Regex	Tamil person + phone + PAN
TanglishCustomer	NER (English/Tanglish)	Customer Senthil phone 9876543210 …
MixedTamilEnglish	NER (Mixed)	வாடிக்கையாளர் Ravi Kumar phone …
TamilFullFinancial	NER (Tamil) + Regex + Domain	Tamil canonical demo

Run all samples (including Tamil) with no flags:

dotnet run --project src/PiiRedaction.ConsoleApp

Or a single Tamil sample:

dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly

Expected console output (English canonical)

When models are loaded, person names appear as [PERSON] with source Ner:

Detected PII:
  [PERSON       ] Ravi Kumar             (Ner)
  [EMAIL        ] ravi.kumar@gmail.com   (Regex)
  ...

Sanitized Prompt:
Customer <PERSON_1> with email <EMAIL_1> ...

---

7. Model Assets Table

All model binaries are gitignored; only .gitkeep placeholders are committed.

Directory	File	Approx. size	Gitignored	Purpose
models/ or models/en/	ner-model.onnx	~431 MB	Yes	English BERT NER (FP32 ONNX from HF)
models/ or models/en/	vocab.txt	~213 KB	Yes	WordPiece vocabulary
models/ or models/en/	ner-labels.txt	< 1 KB	Yes	BIO label index (one per line)
models/ta/	model.onnx	~1.2 GB (FP32 export, varies)	Yes	Tamil IndicBERT NER
models/ta/	vocab.txt	varies	Yes	WordPiece vocab (preferred tokenizer)
models/ta/	tokenizer.json	varies	Yes	HF tokenizer export (optional)
models/ta/	sentencepiece.bpe.model	varies	Yes	SentencePiece (if present instead of vocab)
models/ta/	ner-labels.txt	few KB	Yes	Fine-grained SampurNER labels

.gitignore entries:

models/*.onnx
models/vocab.txt
models/ner-labels.txt
models/*.json
models/en/*
models/ta/*

English size reference: docs/git-xenovex-setup.md notes ~431 MB for the English ONNX file.

---

8. Limitations

Limitation	Detail
Tanglish on English model only	Roman-script Tanglish (Customer Senthil) is classified LatinOnly and handled by English BERT. Recall is best-effort and inconsistent for non-standard spellings. Tamil ONNX is not invoked on Latin-only text.
Fail-open if model missing	OnnxNerPiiDetector and routing runners return [] when models are unavailable. Person names are not redacted; regex/domain layers still run. No regex fallback for names.
128 token limit	OnnxTokenClassifierRunner truncates encoding at 128 tokens. Very long prompts may miss person names beyond the window.
PERSON-only NER scope	Organization, location, and misc NER labels are ignored. Only person spans become <PERSON_n>.
Mixed-script merge	When both models run, overlapping spans are deduped by length; shorter overlapping spans are dropped.
Tamil ONNX availability	Pre-exported Tamil ONNX may not exist on Hugging Face; local Python export is often required.
No fail-closed mode	Missing NER does not block sanitization or LLM calls (optional Phase 5 enhancement).

---

9. How to Reproduce

Download models

From the repository root:

.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1

If Tamil PowerShell download fails with a 404 on onnx/model.onnx, install Python 3.12+ and re-run:

winget install Python.Python.3.12 --accept-package-agreements --accept-source-agreements
.\scripts\download-tamil-ner-model.ps1 -Python "$env:LOCALAPPDATA\Programs\Python\Python312\python.exe"

Optional: copy English assets from models/ to models/en/ to match EnglishOnnxModelPath.

Verify with tests

dotnet build
dotnet test
dotnet test --filter "Category=RealModel"
dotnet test --filter "Category=TamilNer"
dotnet test --logger "console;verbosity=detailed"

Tests skip gracefully when the corresponding ONNX files are absent.

Verify with console

dotnet run --project src/PiiRedaction.ConsoleApp -- --name CustomerNameOnly
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TanglishCustomer

Disable Tamil routing (English-only)

Set in appsettings.json:

"EnableTamilNer": false

---

Related Documentation

• README.md — build, run, and test overview
• architecture.md — dual-model routing diagrams and trust boundary
• tamil-tanglish-ner-plan.md — implementation phases and Tanglish expectations