llm-pii-poc/docs/ner-models.md

# NER Models for PII Redaction

This document describes the **Named Entity Recognition (NER)** ONNX models at the core of the PII Redaction POC. Person-name detection is the only NER responsibility in this solution; structured identifiers (email, phone, PAN, domain IDs) are handled by regex and domain-rule detectors.

For pipeline placement, trust boundaries, and routing diagrams, see [architecture.md](architecture.md). For the Tamil/Tanglish implementation plan and success metrics, see [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md).

---

## 1. Executive Summary

The POC uses **dual-model ONNX NER routing** to redact **person-name PII** before prompts reach an LLM:

| Script in prompt | Model invoked | Typical use case |
|------------------|---------------|------------------|
| Latin only (`LatinOnly`) | English (`dslim/bert-base-NER`) | English names, Indian names in Roman script, **Tanglish** |
| Tamil only (`TamilOnly`) | Tamil (`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`) | Tamil-script customer names |
| Mixed (`Mixed`) | **Both** models on the full text; spans merged | Code-mixed Indian CS prompts |
| No letters (`NoLetters`) | Neither | Digits-only or symbol-only text |

`RoutingOnnxNerModelRunner` classifies script via `ScriptRouter`, delegates to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`, and merges overlapping PERSON spans (longer span wins). Only **PERSON** entities are emitted to the redaction pipeline; all other NER labels are discarded.

---

## 2. English Model

### Hugging Face model ID

**[`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER)**

### Architecture

| Property | Value |
|----------|-------|
| Base | BERT-base (uncased), ~110M parameters |
| Task | Token classification (NER) |
| Tokenizer | **WordPiece** via `vocab.txt` (`BertWordPieceEncoder`) |
| Runtime | ONNX via Microsoft.ML.OnnxRuntime |
| Export | Hugging Face Optimum (`ORTModelForTokenClassification`) or pre-exported ONNX from HF |

### Labels (BIO)

The English model uses standard CoNLL-style BIO tags. The POC maps only **person** labels to `PiiEntityType.Person`:

| Label | Mapped to PERSON |
|-------|------------------|
| `O` | No |
| `B-PER`, `I-PER` | Yes |
| `B-PERSON`, `I-PERSON` | Yes |
| `B-ORG`, `I-ORG`, `B-LOC`, `I-LOC`, `B-MISC`, `I-MISC` | No |

Full label list is written to `ner-labels.txt` at download time from the model `config.json` `id2label` map (typically 9 labels for this model).

Label matching is implemented in `NerLabelConfig.English`:

```19:22:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs
    private static bool IsEnglishPersonLabel(string label) =>
        label is "B-PER" or "I-PER" or "B-PERSON" or "I-PERSON"
        || (label.EndsWith("-PER", StringComparison.Ordinal) &&
            (label.StartsWith("B-", StringComparison.Ordinal) || label.StartsWith("I-", StringComparison.Ordinal)));
```

### Asset paths

| File | Primary path (`appsettings.json`) | Legacy fallback |
|------|----------------------------------|-----------------|
| ONNX model | `models/en/ner-model.onnx` | `models/ner-model.onnx` (`OnnxModelPath`) |
| Vocabulary | `models/en/vocab.txt` | `models/vocab.txt` |
| Labels | `models/en/ner-labels.txt` | `models/ner-labels.txt` |

`EnglishOnnxNerRunner` resolves the model path with primary + legacy fallback:

```15:22:src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs
        var modelPath = OnnxAssetPathResolver.ResolveModelPath(
            options.Value.EnglishOnnxModelPath,
            options.Value.OnnxModelPath);

        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
        var encoder = new BertWordPieceEncoder(modelDirectory, logger);
        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.English, labels, logger);
```

> **Note:** `scripts/download-ner-model.ps1` writes assets to `models/` (repository root). For the configured primary path, copy or move them into `models/en/`, or rely on the `OnnxModelPath` fallback.

### Download script

```powershell
.\scripts\download-ner-model.ps1
```

Or with Python directly:

```bash
python scripts/download-ner-model.py
```

**Behavior:**

1. If Python + Optimum are available → exports `dslim/bert-base-NER` to ONNX under `models/`.
2. Otherwise → downloads pre-exported ONNX from `https://huggingface.co/dslim/bert-base-NER/resolve/main/onnx/` (`model.onnx`, `vocab.txt`, `config.json` → `ner-labels.txt`).

---

## 3. Tamil Model

### Hugging Face model ID

**[`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`](https://huggingface.co/prachuryyaIITG/SampurNER_Tamil_IndicBERTv2)**

(SampurNER Tamil IndicBERTv2 — fine-grained NER for Tamil script.)

### Why SampurNER IndicBERTv2 vs MuRIL

| Criterion | SampurNER Tamil IndicBERTv2 | MuRIL (fallback candidate) |
|-----------|----------------------------|----------------------------|
| Tamil NER training | Fine-grained SampurNER dataset (Tamil-specific labels) | General multilingual; NER requires separate fine-tune |
| Model size | ~0.3B parameters (IndicBERTv2, ~278M base) | ~0.6B parameters |
| POC fit | Lighter memory footprint; ONNX export path validated in this repo | Reserved for Phase 5 if Tamil recall is insufficient |
| Indian financial context | Trained on Indian-language NER corpus; person subtypes map cleanly to PERSON | Heavier; eval-driven swap only |

See [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) §3 for the original selection rationale.

### Architecture

| Property | Value |
|----------|-------|
| Base | IndicBERTv2 (AI4Bharat), ~0.3B parameters |
| Task | Fine-grained token classification |
| Tokenizer | **WordPiece** when `vocab.txt` is present (this repo's export path); SentencePiece fallback if `sentencepiece.bpe.model` / `spiece.model` exists |
| Runtime | Same shared `OnnxTokenClassifierRunner` as English |

### Tokenizer: WordPiece, not SentencePiece (in practice)

The Hugging Face repo for this model does **not** ship a SentencePiece model file. The download scripts extract **WordPiece** assets from `tokenizer.json` → `vocab.txt`. `TokenClassifierEncoderFactory` prefers `vocab.txt`:

```10:19:src/PiiRedaction.Infrastructure/Onnx/TokenClassifierEncoderFactory.cs
    public static ITokenClassifierEncoder Create(string modelDirectory, ILogger logger)
    {
        var vocabPath = OnnxAssetPathResolver.ResolveAssetPath(Path.Combine(modelDirectory, "vocab.txt"));
        if (File.Exists(vocabPath))
        {
            logger.LogInformation(
                "Using WordPiece tokenizer (vocab.txt) from {ModelDirectory}.",
                modelDirectory);
            return new BertWordPieceEncoder(modelDirectory, logger);
        }
```

The PowerShell Tamil download script emits an explicit warning when WordPiece assets are saved instead of SentencePiece.

### Labels (fine-grained person tags)

SampurNER uses fine-grained BIO tags (e.g. `B-person-politician`, `I-person-artist`, `B-location`, `O`). The POC treats **any label containing `person`** (case-insensitive) as a person span:

```24:25:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs
    private static bool IsTamilPersonLabel(string label) =>
        label.Contains("person", StringComparison.OrdinalIgnoreCase);
```

Unit tests lock this behavior:

```20:27:tests/PiiRedaction.Infrastructure.Tests/Onnx/NerLabelConfigTests.cs
    [TestCase("B-person-politician", true)]
    [TestCase("I-person-artist", true)]
    [TestCase("B-location", false)]
    [TestCase("O", false)]
    public void Tamil_IsPersonLabel_MatchesFineGrainedTags(string label, bool expected)
    {
        NerLabelConfig.Tamil.IsPersonLabel(label).Should().Be(expected);
    }
```

### Asset paths

| File | Path |
|------|------|
| ONNX model | `models/ta/model.onnx` |
| Tokenizer | `models/ta/vocab.txt` (WordPiece, preferred) **or** `models/ta/sentencepiece.bpe.model` |
| Labels | `models/ta/ner-labels.txt` |
| Optional | `models/ta/tokenizer.json` (intermediate export artifact) |

### Download script

```powershell
.\scripts\download-tamil-ner-model.ps1
```

Or with Python directly:

```bash
python scripts/download-tamil-ner-model.py
```

**Behavior:**

1. Python + Optimum → full export to `models/ta/` including ONNX, labels, and tokenizer assets.
2. PowerShell fallback → downloads `onnx/model.onnx` from Hugging Face when published; otherwise requires Python export (pre-exported ONNX may return 404).

`TamilOnnxNerRunner` wiring:

```15:19:src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs
        var modelPath = OnnxAssetPathResolver.ResolveModelPath(options.Value.TamilOnnxModelPath);
        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
        var encoder = TokenClassifierEncoderFactory.Create(modelDirectory, logger);
        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.Tamil, labels, logger);
```

---

## 4. Why These Models

Evidence-based rationale for this Indian financial POC:

| Requirement | Decision |
|-------------|----------|
| **English + Indian Latin names** | `dslim/bert-base-NER` is industry-standard, pre-integrated, and handles many Indian names in Roman script (e.g. `Ravi Kumar`, `Anita Sharma`) |
| **Tamil script names** | English BERT is out-of-vocabulary for Tamil letters (U+0B80–U+0BFF); a Tamil-trained NER model is required |
| **Tanglish (Roman-script Tamil)** | Routed to the **English** model only (`ScriptComposition.LatinOnly`); no Tamil ONNX on Latin-only text |
| **Code-mixed prompts** | Both models run on the full prompt; `MergePersonSpans` deduplicates overlaps |
| **Deployable size** | English ~431 MB ONNX (FP32); Tamil IndicBERTv2 ~0.3B params — smaller than MuRIL ~0.6B |
| **ONNX export path** | Both models export via Hugging Face Optimum; English has pre-exported ONNX on HF; Tamil may require local Python export |
| **PERSON-only scope** | POC redacts person names via NER; org/location/misc labels are intentionally ignored to limit false positives |

---

## 5. How They Integrate

### Configuration (`appsettings.json`)

```1:8:src/PiiRedaction.ConsoleApp/appsettings.json
{
  "PiiRedaction": {
    "OnnxModelPath": "models/ner-model.onnx",
    "EnglishOnnxModelPath": "models/en/ner-model.onnx",
    "TamilOnnxModelPath": "models/ta/model.onnx",
    "EnableTamilNer": true
  }
}
```

Options type:

```3:14:src/PiiRedaction.Core/Configuration/PiiRedactionOptions.cs
public sealed class PiiRedactionOptions
{
    public const string SectionName = "PiiRedaction";

    public string OnnxModelPath { get; set; } = "models/ner-model.onnx";

    public string EnglishOnnxModelPath { get; set; } = "models/en/ner-model.onnx";

    public string TamilOnnxModelPath { get; set; } = "models/ta/model.onnx";

    public bool EnableTamilNer { get; set; } = true;
}
```

Set `EnableTamilNer` to `false` for English-only routing.

### Dependency injection

```34:36:src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs
        services.AddSingleton<EnglishOnnxNerRunner>();
        services.AddSingleton<TamilOnnxNerRunner>();
        services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();
```

`OnnxNerPiiDetector` consumes `IOnnxNerModelRunner` (the router) and returns `[]` when no model is available — **fail-open** for person detection.

### Script routing (`ScriptRouter`)

```11:41:src/PiiRedaction.Core/Detection/ScriptRouter.cs
    public ScriptComposition GetComposition(string text)
    {
        ArgumentNullException.ThrowIfNull(text);

        var hasLatin = false;
        var hasTamil = false;

        foreach (var character in text)
        {
            if (IsTamilLetter(character))
            {
                hasTamil = true;
            }
            else if (char.IsAsciiLetter(character))
            {
                hasLatin = true;
            }

            if (hasLatin && hasTamil)
            {
                return ScriptComposition.Mixed;
            }
        }
        // ...
        return hasTamil ? ScriptComposition.TamilOnly : ScriptComposition.LatinOnly;
    }
```

### Routing runner (`RoutingOnnxNerModelRunner`)

```39:79:src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs
    public IReadOnlyList<PiiEntity> PredictEntities(string text)
    {
        var composition = _scriptRouter.GetComposition(text);
        var entities = new List<PiiEntity>();

        switch (composition)
        {
            case ScriptComposition.LatinOnly:
                if (_englishRunner.IsModelAvailable)
                {
                    entities.AddRange(_englishRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.TamilOnly:
                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
                {
                    entities.AddRange(_tamilRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.Mixed:
                if (_englishRunner.IsModelAvailable)
                {
                    entities.AddRange(_englishRunner.PredictEntities(text));
                }
                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
                {
                    entities.AddRange(_tamilRunner.PredictEntities(text));
                }
                break;
            case ScriptComposition.NoLetters:
                break;
        }

        return MergePersonSpans(entities);
    }
```

### Shared inference (`OnnxTokenClassifierRunner`)

Both runners share:

- **Encode** → `input_ids`, `attention_mask`, optional `token_type_ids`
- **Argmax** over per-token logits
- **BIO decode** → `PiiEntityType.Person` with `PiiDetectionSource.Ner`
- **Max sequence length: 128 tokens**

```13:13:src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs
    private const int MaxSequenceLength = 128;
```

### End-to-end flow

```
Prompt → CompositePiiDetector → OnnxNerPiiDetector
       → RoutingOnnxNerModelRunner → ScriptRouter
       → EnglishOnnxNerRunner / TamilOnnxNerRunner
       → OnnxTokenClassifierRunner → PERSON entities
       → PlaceholderPiiRedactor → <PERSON_n>
```

---

## 6. Evidence from Codebase

### Real-model tests (`Category=RealModel`)

English direct inference — `RealNerModelRunnerTests`:

```15:47:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealNerModelRunnerTests.cs
[Category("RealModel")]
public sealed class RealNerModelRunnerTests : RealNerModelFixture
{
    [TestCase("Customer Ravi Kumar called about billing.", "Ravi", "Ravi Kumar")]
    [TestCase("Mr. John Smith called about a duplicate debit.", "John", "John Smith")]
    public void PredictEntities_DetectsPersonWithCorrectSpan(...)
```

English pipeline — `RealNerPipelineTests` (`Category=RealModel`): canonical `FullFinancialWithCustomer`, multi-person, clean-ticket negative.

Fixture skips when model missing:

```5:6:tests/TestSupport.Shared/RealNerModelPaths.cs
    public const string ModelMissingMessage =
        "ONNX model not found. Run scripts/download-ner-model.ps1 from the repository root.";
```

Resolves `models/en/ner-model.onnx` then `models/ner-model.onnx`.

### Tamil tests (`Category=TamilNer`)

Direct Tamil runner — `RealTamilNerModelRunnerTests`:

```9:69:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealTamilNerModelRunnerTests.cs
[Category("TamilNer")]
public sealed class RealTamilNerModelRunnerTests : RealTamilNerModelFixture
{
    [TestCase("வாடிக்கையாளர் ராஜேஷ் குமார் அழைத்தார்.", "ராஜேஷ்", "ராஜேஷ் குமார்")]
    public void PredictEntities_TamilScript_DetectsPersonEntity(...)
    // ...
    [TestCase("Customer Senthil phone 9876543210", "Senthil")]
    public void PredictEntities_TanglishLatinScript_DoesNotInvokeTamilRunner(...)
```

Routed pipeline — `RealTamilPipelineTests` covers Tamil-only, Tanglish (English path), mixed, full financial, and clean Tamil negative.

### Console samples (`SamplePromptCatalog`)

Tamil/Tanglish/mixed samples (indices 10–14):

| Sample | Category | Input excerpt |
|--------|----------|---------------|
| `TamilCustomerNameOnly` | NER (Tamil) | `வாடிக்கையாளர் ராஜேஷ் குமார் …` |
| `TamilWithPhonePan` | NER (Tamil) + Regex | Tamil person + phone + PAN |
| `TanglishCustomer` | NER (English/Tanglish) | `Customer Senthil phone 9876543210 …` |
| `MixedTamilEnglish` | NER (Mixed) | `வாடிக்கையாளர் Ravi Kumar phone …` |
| `TamilFullFinancial` | NER (Tamil) + Regex + Domain | Tamil canonical demo |

Run all samples (including Tamil) with no flags:

```bash
dotnet run --project src/PiiRedaction.ConsoleApp
```

Or a single Tamil sample:

```bash
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly
```

### Expected console output (English canonical)

When models are loaded, person names appear as `[PERSON]` with source `Ner`:

```
Detected PII:
  [PERSON       ] Ravi Kumar             (Ner)
  [EMAIL        ] ravi.kumar@gmail.com   (Regex)
  ...

Sanitized Prompt:
Customer <PERSON_1> with email <EMAIL_1> ...
```

---

## 7. Model Assets Table

All model binaries are **gitignored**; only `.gitkeep` placeholders are committed.

| Directory | File | Approx. size | Gitignored | Purpose |
|-----------|------|--------------|------------|---------|
| `models/` or `models/en/` | `ner-model.onnx` | ~431 MB | Yes | English BERT NER (FP32 ONNX from HF) |
| `models/` or `models/en/` | `vocab.txt` | ~213 KB | Yes | WordPiece vocabulary |
| `models/` or `models/en/` | `ner-labels.txt` | < 1 KB | Yes | BIO label index (one per line) |
| `models/ta/` | `model.onnx` | ~1.2 GB (FP32 export, varies) | Yes | Tamil IndicBERT NER |
| `models/ta/` | `vocab.txt` | varies | Yes | WordPiece vocab (preferred tokenizer) |
| `models/ta/` | `tokenizer.json` | varies | Yes | HF tokenizer export (optional) |
| `models/ta/` | `sentencepiece.bpe.model` | varies | Yes | SentencePiece (if present instead of vocab) |
| `models/ta/` | `ner-labels.txt` | few KB | Yes | Fine-grained SampurNER labels |

`.gitignore` entries:

```
models/*.onnx
models/vocab.txt
models/ner-labels.txt
models/*.json
models/en/*
models/ta/*
```

English size reference: [docs/git-xenovex-setup.md](git-xenovex-setup.md) notes ~431 MB for the English ONNX file.

---

## 8. Limitations

| Limitation | Detail |
|------------|--------|
| **Tanglish on English model only** | Roman-script Tanglish (`Customer Senthil`) is classified `LatinOnly` and handled by English BERT. Recall is best-effort and inconsistent for non-standard spellings. Tamil ONNX is **not** invoked on Latin-only text. |
| **Fail-open if model missing** | `OnnxNerPiiDetector` and routing runners return `[]` when models are unavailable. Person names are **not** redacted; regex/domain layers still run. No regex fallback for names. |
| **128 token limit** | `OnnxTokenClassifierRunner` truncates encoding at 128 tokens. Very long prompts may miss person names beyond the window. |
| **PERSON-only NER scope** | Organization, location, and misc NER labels are ignored. Only person spans become `<PERSON_n>`. |
| **Mixed-script merge** | When both models run, overlapping spans are deduped by length; shorter overlapping spans are dropped. |
| **Tamil ONNX availability** | Pre-exported Tamil ONNX may not exist on Hugging Face; local Python export is often required. |
| **No fail-closed mode** | Missing NER does not block sanitization or LLM calls (optional Phase 5 enhancement). |

---

## 9. How to Reproduce

### Download models

From the repository root:

```powershell
.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1
```

If Tamil PowerShell download fails with a 404 on `onnx/model.onnx`, install Python 3.12+ and re-run:

```powershell
winget install Python.Python.3.12 --accept-package-agreements --accept-source-agreements
.\scripts\download-tamil-ner-model.ps1 -Python "$env:LOCALAPPDATA\Programs\Python\Python312\python.exe"
```

Optional: copy English assets from `models/` to `models/en/` to match `EnglishOnnxModelPath`.

### Verify with tests

```bash
dotnet build
dotnet test
dotnet test --filter "Category=RealModel"
dotnet test --filter "Category=TamilNer"
dotnet test --logger "console;verbosity=detailed"
```

Tests skip gracefully when the corresponding ONNX files are absent.

### Verify with console

```bash
dotnet run --project src/PiiRedaction.ConsoleApp -- --name CustomerNameOnly
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TanglishCustomer
```

### Disable Tamil routing (English-only)

Set in `appsettings.json`:

```json
"EnableTamilNer": false
```

---

## Related Documentation

- [README.md](../README.md) — build, run, and test overview
- [architecture.md](architecture.md) — dual-model routing diagrams and trust boundary
- [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) — implementation phases and Tanglish expectations