Add Tamil NER routing and WPF test harness for POC validation.

Introduce dual-script ONNX NER routing (English/Tamil/mixed), Tamil console samples and integration tests, model download scripts, and a resizable WPF MVVM harness with click-to-load prompts, batch validation, and runtime-adjustable detection panels.
2026-07-07 17:12:38 +05:30
parent a707c6c9cf
commit cf8f5a7232
71 changed files with 4494 additions and 356 deletions
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -10,7 +10,7 @@ The POC validates a compliance-oriented pattern suitable for financial and custo

 ## Canonical Example

-The console application ships with a **sample catalog** (11 prompts). The canonical demo is sample `FullFinancialWithCustomer`. The table below shows the exact strings produced by the production pipeline when the ONNX NER model is loaded (run `scripts/download-ner-model.ps1` first).
+The console application ships with a **sample catalog** (16 prompts). The canonical demo is sample `FullFinancialWithCustomer`. Tamil script, Tanglish, and mixed-script samples run in the **default** `dotnet run` batch (no `--interactive` required). The table below shows the exact strings produced by the production pipeline when the ONNX NER models are loaded (run `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1` first).

 | Stage | Value |
 |-------|-------|
@@ -38,7 +38,7 @@ Running `dotnet run --project src/PiiRedaction.ConsoleApp` executes all samples

 ### NER / person-name samples

-These prompts exercise `OnnxNerPiiDetector` and `OnnxNerModelRunner`. Person names require the ONNX model (`models/ner-model.onnx` plus `vocab.txt` and `ner-labels.txt`). Without the model, person spans are not detected.
+These prompts exercise `OnnxNerPiiDetector` and `RoutingOnnxNerModelRunner`. Person names require ONNX models (`models/en/` for English, `models/ta/` for Tamil script). Without models, person spans are not detected. Legacy `models/ner-model.onnx` is still supported for English.

 | Sample | Input (excerpt) | Detected person | Sanitized (excerpt) |
 |--------|-----------------|-----------------|---------------------|
@@ -50,6 +50,18 @@ These prompts exercise `OnnxNerPiiDetector` and `OnnxNerModelRunner`. Person nam
 | **PersonWithDomainIds** | Customer Meera Iyer holds CID-7070… | Meera Iyer | Customer `<PERSON_1>` holds `<CUSTOMER_ID_1>`… |
 | **PersonWithEmailNoPhone** | Customer Arjun Mehta wrote from arjun.mehta@company.in… | Arjun Mehta | Customer `<PERSON_1>` wrote from `<EMAIL_1>`… |

+### Tamil / Tanglish / mixed samples
+
+These prompts exercise `RoutingOnnxNerModelRunner` script routing. Tamil script uses `models/ta/`; Latin Tanglish uses `models/en/`. Mixed prompts may invoke both models.
+
+| Sample | Input (excerpt) | Detected person | Sanitized (excerpt) |
+|--------|-----------------|-----------------|---------------------|
+| **TamilCustomerNameOnly** | வாடிக்கையாளர் ராஜேஷ் குமார் சேமிப்பு… | ராஜேஷ் குமார் | வாடிக்கையாளர் `<PERSON_1>` சேமிப்பு… |
+| **TamilWithPhonePan** | வாடிக்கையாளர் ராஜேஷ் குமார் தொலைபேசி 9876543210 PAN… | ராஜேஷ் குமார் | `<PERSON_1>` … `<PHONE_1>` … `<PAN_1>` |
+| **TanglishCustomer** | Customer Senthil phone 9876543210… | Senthil | Customer `<PERSON_1>` phone `<PHONE_1>`… |
+| **MixedTamilEnglish** | வாடிக்கையாளர் Ravi Kumar phone 9876543210… | Ravi Kumar | வாடிக்கையாளர் `<PERSON_1>` phone `<PHONE_1>`… |
+| **TamilFullFinancial** | வாடிக்கையாளர் ராஜேஷ் குமார் மின்னஞ்சல் ravi.kumar@gmail.com… | ராஜேஷ் குமார் | Tamil canonical — all placeholder types |
+
 ### Other sample categories

 | Category | Sample | Purpose |
@@ -84,7 +96,9 @@ flowchart TB
    end

    subgraph infra [PiiRedaction.Infrastructure]
-        onnxRunner["OnnxNerModelRunner"]
+        onnxRunner["RoutingOnnxNerModelRunner"]
+        enRunner["EnglishOnnxNerRunner"]
+        taRunner["TamilOnnxNerRunner"]
        mockLlm["MockLlmPromptService"]
        mockChat["MockChatClient"]
    end
@@ -110,6 +124,8 @@ flowchart TB
    di -.-> mockLlm
 ```

+`RoutingOnnxNerModelRunner` selects English and/or Tamil ONNX models based on script composition in the prompt. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english) for the routing decision tree.
+
 ---

 ## Detection to Redaction Detail
@@ -123,7 +139,7 @@ flowchart LR
  subgraph detectPhase [Detection Phase]
    domainDet["DomainRulePiiDetector"]
    regexDet["RegexPiiDetector"]
-    onnxDet["OnnxNerPiiDetector"]
+    onnxDet["OnnxNerPiiDetector<br/>(RoutingOnnxNerModelRunner)"]
    composite["CompositePiiDetector"]
    merge["Overlap merge and source priority"]
    entityList["PiiEntity list"]
@@ -157,14 +173,144 @@ flowchart LR
 2. On overlapping spans, the first registered detector wins.
 3. Tie-breaking uses source priority: Domain (3) > Regex (2) > NER (1).

+The ONNX NER detector delegates to `RoutingOnnxNerModelRunner`, which routes inference to English and/or Tamil models by script composition. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english).
+
 **Placeholder assignment** (applied by `PlaceholderPiiRedactor`):

+
 - Format: `<{TYPE}_{n}>` (e.g. `<EMAIL_1>`, `<PERSON_1>`).
 - Duplicate values of the same type reuse the same placeholder.
 - Replacement proceeds from highest `StartIndex` to lowest to avoid index drift.

 ---

+## Dual-Model NER Routing (Tamil + English)
+
+Person-name detection uses two ONNX token-classifier models: **English** (`models/en/`, BERT WordPiece) and **Tamil** (`models/ta/`, SentencePiece or WordPiece). `OnnxNerPiiDetector` calls `RoutingOnnxNerModelRunner`, which classifies prompt script via `ScriptRouter` and dispatches to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`. Both runners share `OnnxTokenClassifierRunner` for BIO decoding; only **PERSON** spans are emitted.
+
+The diagram below expands the detection and NER branches summarized in [High-Level Data Flow](#high-level-data-flow) and [Detection to Redaction Detail](#detection-to-redaction-detail).
+
+### End-to-end pipeline (with NER branch)
+
+```mermaid
+flowchart TB
+    subgraph Entry["Console entry"]
+        A["Program.cs<br/>Host + AddPiiRedactionServices()"]
+        B["PromptDemoRunner.RunAsync()"]
+        A --> B
+    end
+
+    B --> C["SanitizationRequest(OriginalPrompt)"]
+    C --> D["PromptSanitizer.Sanitize()"]
+
+    subgraph Detect["CompositePiiDetector.Detect() — registration order"]
+        direction TB
+        E1["DomainRulePiiDetector<br/>LOAN_NUMBER, CUSTOMER_ID, ACCOUNT_NUMBER"]
+        E2["RegexPiiDetector<br/>EMAIL, PHONE, AADHAAR, PAN, CREDIT_CARD"]
+        E3["OnnxNerPiiDetector<br/>PERSON (via IOnnxNerModelRunner)"]
+        E1 --> MERGE
+        E2 --> MERGE
+        E3 --> MERGE
+        MERGE["Merge overlapping spans<br/>sort: StartIndex ↑, Length ↓, Source priority ↓<br/>(Domain=3, Regex=2, Ner=1)<br/>first candidate wins on overlap"]
+    end
+
+    D --> Detect
+    MERGE --> F["IReadOnlyList&lt;PiiEntity&gt;"]
+
+    F --> G["PlaceholderPiiRedactor.Redact()<br/>replace spans right-to-left<br/>dedupe by Type|Value → &lt;TYPE_n&gt;"]
+    G --> H["SanitizationResult<br/>SanitizedPrompt, DetectedEntities, PlaceholderMap"]
+
+    H --> I["MockLlmPromptService.SendPromptAsync(SanitizedPrompt)"]
+    I --> J["Mock LLM response<br/>(sanitized text only)"]
+
+    subgraph NerBranch["OnnxNerPiiDetector branch"]
+        E3 --> N1{"RoutingOnnxNerModelRunner<br/>.IsModelAvailable?"}
+        N1 -->|no| N2["return []"]
+        N1 -->|yes| N3["RoutingOnnxNerModelRunner<br/>.PredictEntities()"]
+    end
+```
+
+### RoutingOnnxNerModelRunner decision tree
+
+`ScriptRouter.GetComposition` scans each character once. Tamil letters (U+0B80–U+0BFF) and ASCII Latin letters (`char.IsAsciiLetter`) determine the route. When both scripts appear, classification is **Mixed** (early exit).
+
+```mermaid
+flowchart TB
+    IN["text"] --> SR["ScriptRouter.GetComposition(text)<br/>scan each char"]
+
+    SR --> C1{"LatinOnly?"}
+    SR --> C2{"TamilOnly?"}
+    SR --> C3{"Mixed?"}
+    SR --> C4{"NoLetters?"}
+
+    C1 -->|yes| EN1{"EnglishOnnxNerRunner<br/>.IsModelAvailable?"}
+    EN1 -->|yes| EN_RUN["EnglishOnnxNerRunner.PredictEntities(text)"]
+    EN1 -->|no| SKIP1["skip English"]
+    EN_RUN --> ACC
+    SKIP1 --> ACC
+
+    C2 -->|yes| TA_GATE{"EnableTamilNer<br/>&& TamilOnnxNerRunner<br/>.IsModelAvailable?"}
+    TA_GATE -->|yes| TA_RUN["TamilOnnxNerRunner.PredictEntities(text)"]
+    TA_GATE -->|no| SKIP2["skip Tamil"]
+    TA_RUN --> ACC
+    SKIP2 --> ACC
+
+    C3 -->|yes| EN2{"English available?"}
+    EN2 -->|yes| EN_MIX["EnglishOnnxNerRunner.PredictEntities(text)"]
+    EN2 -->|no| SKIP3["skip English"]
+    EN_MIX --> TA_GATE2{"EnableTamilNer<br/>&& Tamil available?"}
+    SKIP3 --> TA_GATE2
+    TA_GATE2 -->|yes| TA_MIX["TamilOnnxNerRunner.PredictEntities(text)"]
+    TA_GATE2 -->|no| SKIP4["skip Tamil"]
+    TA_MIX --> ACC
+    SKIP4 --> ACC
+
+    C4 -->|yes| EMPTY["no NER inference"]
+    EMPTY --> OUT_EMPTY["return []"]
+
+    subgraph EN_Pipeline["EnglishOnnxNerRunner"]
+        EN_RUN --> EN_ENC["BertWordPieceEncoder<br/>(model dir vocab.txt)"]
+        EN_ENC --> EN_OCR["OnnxTokenClassifierRunner<br/>NerLabelConfig.English<br/>B-PER / I-PER / B-PERSON / I-PERSON"]
+    end
+
+    subgraph TA_Pipeline["TamilOnnxNerRunner"]
+        TA_RUN --> TA_ENC["TokenClassifierEncoderFactory.Create()<br/>vocab.txt → BertWordPieceEncoder<br/>else SentencePiece (*.bpe.model, spiece.model, tokenizer.model)"]
+        TA_ENC --> TA_OCR["OnnxTokenClassifierRunner<br/>NerLabelConfig.Tamil<br/>label contains 'person' (case-insensitive)"]
+    end
+
+    subgraph SharedInference["OnnxTokenClassifierRunner (shared)"]
+        ENC["Encode(text, max 128 tokens)"]
+        ONNX["ONNX InferenceSession.Run<br/>input_ids + attention_mask [+ token_type_ids]"]
+        ARGMAX["Per-token argmax over logits"]
+        BIO["BIO decode → PiiEntityType.Person<br/>PiiDetectionSource.Ner"]
+        ENC --> ONNX --> ARGMAX --> BIO
+    end
+
+    EN_OCR --> SharedInference
+    TA_OCR --> SharedInference
+    BIO --> ACC["accumulate entities"]
+
+    ACC --> MERGE["MergePersonSpans()<br/>sort: Length ↓, StartIndex ↑<br/>drop overlapping spans<br/>(longer span wins)"]
+    MERGE --> OUT["return merged PERSON entities"]
+```
+
+### Routing rules
+
+| Rule | Source | Behavior |
+|------|--------|----------|
+| **Script classification** | `ScriptRouter.GetComposition` | Single pass over characters. Tamil letter = U+0B80–U+0BFF. Latin letter = `char.IsAsciiLetter`. Both seen → `Mixed` (early exit). Neither → `NoLetters`. Tamil only → `TamilOnly`. Latin only → `LatinOnly`. |
+| **LatinOnly** | `RoutingOnnxNerModelRunner` | Run **English only** if `englishRunner.IsModelAvailable`. |
+| **TamilOnly** | `RoutingOnnxNerModelRunner` | Run **Tamil only** if `EnableTamilNer` (default `true` in `PiiRedactionOptions`) **and** `tamilRunner.IsModelAvailable`. |
+| **Mixed** | `RoutingOnnxNerModelRunner` | Run **both** models independently on the **full text** (English if available; Tamil if `EnableTamilNer` and available). |
+| **NoLetters** | `RoutingOnnxNerModelRunner` | No NER inference; returns `[]` from routing (before merge). |
+| **Model availability gate** | `OnnxNerPiiDetector` | If `RoutingOnnxNerModelRunner.IsModelAvailable` is false, NER detector returns `[]` (English OR Tamil available when Tamil enabled). |
+| **Post-route merge** | `MergePersonSpans` | After EN/TA results are concatenated, overlapping PERSON spans are deduped; **longer span wins**, then ordered by `StartIndex`. |
+| **Composite merge** | `CompositePiiDetector` | Domain → Regex → NER all run. Overlaps resolved globally: earlier registration order + longer span + higher source priority (Domain > Regex > Ner). |
+| **Encoder choice** | `EnglishOnnxNerRunner` vs `TamilOnnxNerRunner` | English always uses `BertWordPieceEncoder`. Tamil uses factory: `vocab.txt` → WordPiece; else first SentencePiece file found; fallback WordPiece with warning. |
+| **NER output scope** | `OnnxTokenClassifierRunner` | Only **PERSON** entities decoded from BIO tags; max sequence length **128** tokens. |
+
+---
+
 ## Runtime Sequence

 ```mermaid
@@ -255,7 +401,7 @@ In the POC, `MockChatClient` simulates the external provider without network I/O
 |---------|-------|----------------|
 | `PiiRedaction.ConsoleApp` | Presentation | Application entry point; reads prompt (sample or interactive); bootstraps `IHost` and DI via `AddPiiRedactionServices`; orchestrates sanitization and LLM invocation; renders audit output (detected entities, sanitized text, placeholder map). |
 | `PiiRedaction.Core` | Domain / Application | Defines abstractions (`IPiiDetector`, `IPiiRedactor`, `IPromptSanitizer`, `ILlmPromptService`); implements detection strategies (`RegexPiiDetector`, `DomainRulePiiDetector`, `OnnxNerPiiDetector`, `CompositePiiDetector`); implements `PlaceholderPiiRedactor` and `PromptSanitizer`; owns domain models (`PiiEntity`, `SanitizationResult`, `RedactionResult`) and configuration (`PiiRedactionOptions`). Has no dependency on ONNX Runtime or LLM SDKs. |
-| `PiiRedaction.Infrastructure` | Infrastructure | Implements technical adapters: `OnnxNerModelRunner` (ONNX Runtime inference), `MockChatClient` and `MockLlmPromptService` (`Microsoft.Extensions.AI`); depends on Core abstractions and is swappable without changing domain logic. |
+| `PiiRedaction.Infrastructure` | Infrastructure | Implements technical adapters: `RoutingOnnxNerModelRunner`, `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` (ONNX Runtime inference), `MockChatClient` and `MockLlmPromptService` (`Microsoft.Extensions.AI`); depends on Core abstractions and is swappable without changing domain logic. |
 | `tests/PiiRedaction.Core.Tests` | Test | Unit and integration tests for detectors, redactor, sanitizer, overlap rules, golden prompt scenarios (`PromptScenarioCatalog`), and LLM boundary assertions. |
 | `tests/PiiRedaction.Infrastructure.Tests` | Test | Tests for mock LLM behavior and ONNX runner load semantics. |

@@ -272,7 +418,7 @@ In the POC, `MockChatClient` simulates the external provider without network I/O
 | `IPromptSanitizer` | Core | `PromptSanitizer` | Unlikely to change; orchestrates detect + redact |
 | `ILlmPromptService` | Core | `MockLlmPromptService` | Production adapter with telemetry, retry, policy |
 | `IChatClient` | Microsoft.Extensions.AI | `MockChatClient` | Azure OpenAI, OpenAI, or other provider SDK |
-| `IOnnxNerModelRunner` | Core | `OnnxNerModelRunner` | BERT WordPiece tokenization, ONNX inference, BIO label decoding |
+| `IOnnxNerModelRunner` | Core | `RoutingOnnxNerModelRunner` | Script-based routing to English (BERT WordPiece) and Tamil (SentencePiece) ONNX models |

 ---

@@ -282,14 +428,18 @@ Runtime behavior is controlled via `appsettings.json` under the `PiiRedaction` s

 | Setting | Effect |
 |---------|--------|
-| `OnnxModelPath` | Path to ONNX NER model (`models/ner-model.onnx` by default). Companion files `vocab.txt` and `ner-labels.txt` must live in the same directory. |
+| `OnnxModelPath` | Legacy English model path (`models/ner-model.onnx`). Used as fallback when `models/en/` is absent. |
+| `EnglishOnnxModelPath` | Primary English ONNX model (`models/en/ner-model.onnx`). |
+| `TamilOnnxModelPath` | Tamil ONNX model (`models/ta/model.onnx`). |
+| `EnableTamilNer` | When `false`, routing uses English model only. Default `true`. |

-Download the model assets with `scripts/download-ner-model.ps1` (exports `dslim/bert-base-NER`).
+Download model assets with `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1`.

 ---

 ## Related Documentation

+- [NER models](ner-models.md) — English/Tamil ONNX model IDs, assets, labels, and integration reference
 - [README](../README.md) — build, run, configuration, and testing instructions
 - [ServiceCollectionExtensions.cs](../src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs) — DI registration and detector ordering
 - [PromptScenarioCatalog.cs](../tests/PiiRedaction.Core.Tests/TestSupport/PromptScenarioCatalog.cs) — focused golden pipeline scenarios including the canonical example
--- a/docs/ner-models.md
+++ b/docs/ner-models.md
@@ -0,0 +1,548 @@
+# NER Models for PII Redaction
+
+This document describes the **Named Entity Recognition (NER)** ONNX models at the core of the PII Redaction POC. Person-name detection is the only NER responsibility in this solution; structured identifiers (email, phone, PAN, domain IDs) are handled by regex and domain-rule detectors.
+
+For pipeline placement, trust boundaries, and routing diagrams, see [architecture.md](architecture.md). For the Tamil/Tanglish implementation plan and success metrics, see [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md).
+
+---
+
+## 1. Executive Summary
+
+The POC uses **dual-model ONNX NER routing** to redact **person-name PII** before prompts reach an LLM:
+
+| Script in prompt | Model invoked | Typical use case |
+|------------------|---------------|------------------|
+| Latin only (`LatinOnly`) | English (`dslim/bert-base-NER`) | English names, Indian names in Roman script, **Tanglish** |
+| Tamil only (`TamilOnly`) | Tamil (`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`) | Tamil-script customer names |
+| Mixed (`Mixed`) | **Both** models on the full text; spans merged | Code-mixed Indian CS prompts |
+| No letters (`NoLetters`) | Neither | Digits-only or symbol-only text |
+
+`RoutingOnnxNerModelRunner` classifies script via `ScriptRouter`, delegates to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`, and merges overlapping PERSON spans (longer span wins). Only **PERSON** entities are emitted to the redaction pipeline; all other NER labels are discarded.
+
+---
+
+## 2. English Model
+
+### Hugging Face model ID
+
+**[`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER)**
+
+### Architecture
+
+| Property | Value |
+|----------|-------|
+| Base | BERT-base (uncased), ~110M parameters |
+| Task | Token classification (NER) |
+| Tokenizer | **WordPiece** via `vocab.txt` (`BertWordPieceEncoder`) |
+| Runtime | ONNX via Microsoft.ML.OnnxRuntime |
+| Export | Hugging Face Optimum (`ORTModelForTokenClassification`) or pre-exported ONNX from HF |
+
+### Labels (BIO)
+
+The English model uses standard CoNLL-style BIO tags. The POC maps only **person** labels to `PiiEntityType.Person`:
+
+| Label | Mapped to PERSON |
+|-------|------------------|
+| `O` | No |
+| `B-PER`, `I-PER` | Yes |
+| `B-PERSON`, `I-PERSON` | Yes |
+| `B-ORG`, `I-ORG`, `B-LOC`, `I-LOC`, `B-MISC`, `I-MISC` | No |
+
+Full label list is written to `ner-labels.txt` at download time from the model `config.json` `id2label` map (typically 9 labels for this model).
+
+Label matching is implemented in `NerLabelConfig.English`:
+
+```19:22:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs
+    private static bool IsEnglishPersonLabel(string label) =>
+        label is "B-PER" or "I-PER" or "B-PERSON" or "I-PERSON"
+        || (label.EndsWith("-PER", StringComparison.Ordinal) &&
+            (label.StartsWith("B-", StringComparison.Ordinal) || label.StartsWith("I-", StringComparison.Ordinal)));
+```
+
+### Asset paths
+
+| File | Primary path (`appsettings.json`) | Legacy fallback |
+|------|----------------------------------|-----------------|
+| ONNX model | `models/en/ner-model.onnx` | `models/ner-model.onnx` (`OnnxModelPath`) |
+| Vocabulary | `models/en/vocab.txt` | `models/vocab.txt` |
+| Labels | `models/en/ner-labels.txt` | `models/ner-labels.txt` |
+
+`EnglishOnnxNerRunner` resolves the model path with primary + legacy fallback:
+
+```15:22:src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs
+        var modelPath = OnnxAssetPathResolver.ResolveModelPath(
+            options.Value.EnglishOnnxModelPath,
+            options.Value.OnnxModelPath);
+
+        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
+        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
+        var encoder = new BertWordPieceEncoder(modelDirectory, logger);
+        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.English, labels, logger);
+```
+
+> **Note:** `scripts/download-ner-model.ps1` writes assets to `models/` (repository root). For the configured primary path, copy or move them into `models/en/`, or rely on the `OnnxModelPath` fallback.
+
+### Download script
+
+```powershell
+.\scripts\download-ner-model.ps1
+```
+
+Or with Python directly:
+
+```bash
+python scripts/download-ner-model.py
+```
+
+**Behavior:**
+
+1. If Python + Optimum are available → exports `dslim/bert-base-NER` to ONNX under `models/`.
+2. Otherwise → downloads pre-exported ONNX from `https://huggingface.co/dslim/bert-base-NER/resolve/main/onnx/` (`model.onnx`, `vocab.txt`, `config.json` → `ner-labels.txt`).
+
+---
+
+## 3. Tamil Model
+
+### Hugging Face model ID
+
+**[`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`](https://huggingface.co/prachuryyaIITG/SampurNER_Tamil_IndicBERTv2)**
+
+(SampurNER Tamil IndicBERTv2 — fine-grained NER for Tamil script.)
+
+### Why SampurNER IndicBERTv2 vs MuRIL
+
+| Criterion | SampurNER Tamil IndicBERTv2 | MuRIL (fallback candidate) |
+|-----------|----------------------------|----------------------------|
+| Tamil NER training | Fine-grained SampurNER dataset (Tamil-specific labels) | General multilingual; NER requires separate fine-tune |
+| Model size | ~0.3B parameters (IndicBERTv2, ~278M base) | ~0.6B parameters |
+| POC fit | Lighter memory footprint; ONNX export path validated in this repo | Reserved for Phase 5 if Tamil recall is insufficient |
+| Indian financial context | Trained on Indian-language NER corpus; person subtypes map cleanly to PERSON | Heavier; eval-driven swap only |
+
+See [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) §3 for the original selection rationale.
+
+### Architecture
+
+| Property | Value |
+|----------|-------|
+| Base | IndicBERTv2 (AI4Bharat), ~0.3B parameters |
+| Task | Fine-grained token classification |
+| Tokenizer | **WordPiece** when `vocab.txt` is present (this repo's export path); SentencePiece fallback if `sentencepiece.bpe.model` / `spiece.model` exists |
+| Runtime | Same shared `OnnxTokenClassifierRunner` as English |
+
+### Tokenizer: WordPiece, not SentencePiece (in practice)
+
+The Hugging Face repo for this model does **not** ship a SentencePiece model file. The download scripts extract **WordPiece** assets from `tokenizer.json` → `vocab.txt`. `TokenClassifierEncoderFactory` prefers `vocab.txt`:
+
+```10:19:src/PiiRedaction.Infrastructure/Onnx/TokenClassifierEncoderFactory.cs
+    public static ITokenClassifierEncoder Create(string modelDirectory, ILogger logger)
+    {
+        var vocabPath = OnnxAssetPathResolver.ResolveAssetPath(Path.Combine(modelDirectory, "vocab.txt"));
+        if (File.Exists(vocabPath))
+        {
+            logger.LogInformation(
+                "Using WordPiece tokenizer (vocab.txt) from {ModelDirectory}.",
+                modelDirectory);
+            return new BertWordPieceEncoder(modelDirectory, logger);
+        }
+```
+
+The PowerShell Tamil download script emits an explicit warning when WordPiece assets are saved instead of SentencePiece.
+
+### Labels (fine-grained person tags)
+
+SampurNER uses fine-grained BIO tags (e.g. `B-person-politician`, `I-person-artist`, `B-location`, `O`). The POC treats **any label containing `person`** (case-insensitive) as a person span:
+
+```24:25:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs
+    private static bool IsTamilPersonLabel(string label) =>
+        label.Contains("person", StringComparison.OrdinalIgnoreCase);
+```
+
+Unit tests lock this behavior:
+
+```20:27:tests/PiiRedaction.Infrastructure.Tests/Onnx/NerLabelConfigTests.cs
+    [TestCase("B-person-politician", true)]
+    [TestCase("I-person-artist", true)]
+    [TestCase("B-location", false)]
+    [TestCase("O", false)]
+    public void Tamil_IsPersonLabel_MatchesFineGrainedTags(string label, bool expected)
+    {
+        NerLabelConfig.Tamil.IsPersonLabel(label).Should().Be(expected);
+    }
+```
+
+### Asset paths
+
+| File | Path |
+|------|------|
+| ONNX model | `models/ta/model.onnx` |
+| Tokenizer | `models/ta/vocab.txt` (WordPiece, preferred) **or** `models/ta/sentencepiece.bpe.model` |
+| Labels | `models/ta/ner-labels.txt` |
+| Optional | `models/ta/tokenizer.json` (intermediate export artifact) |
+
+### Download script
+
+```powershell
+.\scripts\download-tamil-ner-model.ps1
+```
+
+Or with Python directly:
+
+```bash
+python scripts/download-tamil-ner-model.py
+```
+
+**Behavior:**
+
+1. Python + Optimum → full export to `models/ta/` including ONNX, labels, and tokenizer assets.
+2. PowerShell fallback → downloads `onnx/model.onnx` from Hugging Face when published; otherwise requires Python export (pre-exported ONNX may return 404).
+
+`TamilOnnxNerRunner` wiring:
+
+```15:19:src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs
+        var modelPath = OnnxAssetPathResolver.ResolveModelPath(options.Value.TamilOnnxModelPath);
+        var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory;
+        var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory);
+        var encoder = TokenClassifierEncoderFactory.Create(modelDirectory, logger);
+        _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.Tamil, labels, logger);
+```
+
+---
+
+## 4. Why These Models
+
+Evidence-based rationale for this Indian financial POC:
+
+| Requirement | Decision |
+|-------------|----------|
+| **English + Indian Latin names** | `dslim/bert-base-NER` is industry-standard, pre-integrated, and handles many Indian names in Roman script (e.g. `Ravi Kumar`, `Anita Sharma`) |
+| **Tamil script names** | English BERT is out-of-vocabulary for Tamil letters (U+0B80–U+0BFF); a Tamil-trained NER model is required |
+| **Tanglish (Roman-script Tamil)** | Routed to the **English** model only (`ScriptComposition.LatinOnly`); no Tamil ONNX on Latin-only text |
+| **Code-mixed prompts** | Both models run on the full prompt; `MergePersonSpans` deduplicates overlaps |
+| **Deployable size** | English ~431 MB ONNX (FP32); Tamil IndicBERTv2 ~0.3B params — smaller than MuRIL ~0.6B |
+| **ONNX export path** | Both models export via Hugging Face Optimum; English has pre-exported ONNX on HF; Tamil may require local Python export |
+| **PERSON-only scope** | POC redacts person names via NER; org/location/misc labels are intentionally ignored to limit false positives |
+
+---
+
+## 5. How They Integrate
+
+### Configuration (`appsettings.json`)
+
+```1:8:src/PiiRedaction.ConsoleApp/appsettings.json
+{
+  "PiiRedaction": {
+    "OnnxModelPath": "models/ner-model.onnx",
+    "EnglishOnnxModelPath": "models/en/ner-model.onnx",
+    "TamilOnnxModelPath": "models/ta/model.onnx",
+    "EnableTamilNer": true
+  }
+}
+```
+
+Options type:
+
+```3:14:src/PiiRedaction.Core/Configuration/PiiRedactionOptions.cs
+public sealed class PiiRedactionOptions
+{
+    public const string SectionName = "PiiRedaction";
+
+    public string OnnxModelPath { get; set; } = "models/ner-model.onnx";
+
+    public string EnglishOnnxModelPath { get; set; } = "models/en/ner-model.onnx";
+
+    public string TamilOnnxModelPath { get; set; } = "models/ta/model.onnx";
+
+    public bool EnableTamilNer { get; set; } = true;
+}
+```
+
+Set `EnableTamilNer` to `false` for English-only routing.
+
+### Dependency injection
+
+```34:36:src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs
+        services.AddSingleton<EnglishOnnxNerRunner>();
+        services.AddSingleton<TamilOnnxNerRunner>();
+        services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();
+```
+
+`OnnxNerPiiDetector` consumes `IOnnxNerModelRunner` (the router) and returns `[]` when no model is available — **fail-open** for person detection.
+
+### Script routing (`ScriptRouter`)
+
+```11:41:src/PiiRedaction.Core/Detection/ScriptRouter.cs
+    public ScriptComposition GetComposition(string text)
+    {
+        ArgumentNullException.ThrowIfNull(text);
+
+        var hasLatin = false;
+        var hasTamil = false;
+
+        foreach (var character in text)
+        {
+            if (IsTamilLetter(character))
+            {
+                hasTamil = true;
+            }
+            else if (char.IsAsciiLetter(character))
+            {
+                hasLatin = true;
+            }
+
+            if (hasLatin && hasTamil)
+            {
+                return ScriptComposition.Mixed;
+            }
+        }
+        // ...
+        return hasTamil ? ScriptComposition.TamilOnly : ScriptComposition.LatinOnly;
+    }
+```
+
+### Routing runner (`RoutingOnnxNerModelRunner`)
+
+```39:79:src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs
+    public IReadOnlyList<PiiEntity> PredictEntities(string text)
+    {
+        var composition = _scriptRouter.GetComposition(text);
+        var entities = new List<PiiEntity>();
+
+        switch (composition)
+        {
+            case ScriptComposition.LatinOnly:
+                if (_englishRunner.IsModelAvailable)
+                {
+                    entities.AddRange(_englishRunner.PredictEntities(text));
+                }
+                break;
+            case ScriptComposition.TamilOnly:
+                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
+                {
+                    entities.AddRange(_tamilRunner.PredictEntities(text));
+                }
+                break;
+            case ScriptComposition.Mixed:
+                if (_englishRunner.IsModelAvailable)
+                {
+                    entities.AddRange(_englishRunner.PredictEntities(text));
+                }
+                if (_enableTamilNer && _tamilRunner.IsModelAvailable)
+                {
+                    entities.AddRange(_tamilRunner.PredictEntities(text));
+                }
+                break;
+            case ScriptComposition.NoLetters:
+                break;
+        }
+
+        return MergePersonSpans(entities);
+    }
+```
+
+### Shared inference (`OnnxTokenClassifierRunner`)
+
+Both runners share:
+
+- **Encode** → `input_ids`, `attention_mask`, optional `token_type_ids`
+- **Argmax** over per-token logits
+- **BIO decode** → `PiiEntityType.Person` with `PiiDetectionSource.Ner`
+- **Max sequence length: 128 tokens**
+
+```13:13:src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs
+    private const int MaxSequenceLength = 128;
+```
+
+### End-to-end flow
+
+```
+Prompt → CompositePiiDetector → OnnxNerPiiDetector
+       → RoutingOnnxNerModelRunner → ScriptRouter
+       → EnglishOnnxNerRunner / TamilOnnxNerRunner
+       → OnnxTokenClassifierRunner → PERSON entities
+       → PlaceholderPiiRedactor → <PERSON_n>
+```
+
+---
+
+## 6. Evidence from Codebase
+
+### Real-model tests (`Category=RealModel`)
+
+English direct inference — `RealNerModelRunnerTests`:
+
+```15:47:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealNerModelRunnerTests.cs
+[Category("RealModel")]
+public sealed class RealNerModelRunnerTests : RealNerModelFixture
+{
+    [TestCase("Customer Ravi Kumar called about billing.", "Ravi", "Ravi Kumar")]
+    [TestCase("Mr. John Smith called about a duplicate debit.", "John", "John Smith")]
+    public void PredictEntities_DetectsPersonWithCorrectSpan(...)
+```
+
+English pipeline — `RealNerPipelineTests` (`Category=RealModel`): canonical `FullFinancialWithCustomer`, multi-person, clean-ticket negative.
+
+Fixture skips when model missing:
+
+```5:6:tests/TestSupport.Shared/RealNerModelPaths.cs
+    public const string ModelMissingMessage =
+        "ONNX model not found. Run scripts/download-ner-model.ps1 from the repository root.";
+```
+
+Resolves `models/en/ner-model.onnx` then `models/ner-model.onnx`.
+
+### Tamil tests (`Category=TamilNer`)
+
+Direct Tamil runner — `RealTamilNerModelRunnerTests`:
+
+```9:69:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealTamilNerModelRunnerTests.cs
+[Category("TamilNer")]
+public sealed class RealTamilNerModelRunnerTests : RealTamilNerModelFixture
+{
+    [TestCase("வாடிக்கையாளர் ராஜேஷ் குமார் அழைத்தார்.", "ராஜேஷ்", "ராஜேஷ் குமார்")]
+    public void PredictEntities_TamilScript_DetectsPersonEntity(...)
+    // ...
+    [TestCase("Customer Senthil phone 9876543210", "Senthil")]
+    public void PredictEntities_TanglishLatinScript_DoesNotInvokeTamilRunner(...)
+```
+
+Routed pipeline — `RealTamilPipelineTests` covers Tamil-only, Tanglish (English path), mixed, full financial, and clean Tamil negative.
+
+### Console samples (`SamplePromptCatalog`)
+
+Tamil/Tanglish/mixed samples (indices 10–14):
+
+| Sample | Category | Input excerpt |
+|--------|----------|---------------|
+| `TamilCustomerNameOnly` | NER (Tamil) | `வாடிக்கையாளர் ராஜேஷ் குமார் …` |
+| `TamilWithPhonePan` | NER (Tamil) + Regex | Tamil person + phone + PAN |
+| `TanglishCustomer` | NER (English/Tanglish) | `Customer Senthil phone 9876543210 …` |
+| `MixedTamilEnglish` | NER (Mixed) | `வாடிக்கையாளர் Ravi Kumar phone …` |
+| `TamilFullFinancial` | NER (Tamil) + Regex + Domain | Tamil canonical demo |
+
+Run all samples (including Tamil) with no flags:
+
+```bash
+dotnet run --project src/PiiRedaction.ConsoleApp
+```
+
+Or a single Tamil sample:
+
+```bash
+dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly
+```
+
+### Expected console output (English canonical)
+
+When models are loaded, person names appear as `[PERSON]` with source `Ner`:
+
+```
+Detected PII:
+  [PERSON       ] Ravi Kumar             (Ner)
+  [EMAIL        ] ravi.kumar@gmail.com   (Regex)
+  ...
+
+Sanitized Prompt:
+Customer <PERSON_1> with email <EMAIL_1> ...
+```
+
+---
+
+## 7. Model Assets Table
+
+All model binaries are **gitignored**; only `.gitkeep` placeholders are committed.
+
+| Directory | File | Approx. size | Gitignored | Purpose |
+|-----------|------|--------------|------------|---------|
+| `models/` or `models/en/` | `ner-model.onnx` | ~431 MB | Yes | English BERT NER (FP32 ONNX from HF) |
+| `models/` or `models/en/` | `vocab.txt` | ~213 KB | Yes | WordPiece vocabulary |
+| `models/` or `models/en/` | `ner-labels.txt` | < 1 KB | Yes | BIO label index (one per line) |
+| `models/ta/` | `model.onnx` | ~1.2 GB (FP32 export, varies) | Yes | Tamil IndicBERT NER |
+| `models/ta/` | `vocab.txt` | varies | Yes | WordPiece vocab (preferred tokenizer) |
+| `models/ta/` | `tokenizer.json` | varies | Yes | HF tokenizer export (optional) |
+| `models/ta/` | `sentencepiece.bpe.model` | varies | Yes | SentencePiece (if present instead of vocab) |
+| `models/ta/` | `ner-labels.txt` | few KB | Yes | Fine-grained SampurNER labels |
+
+`.gitignore` entries:
+
+```
+models/*.onnx
+models/vocab.txt
+models/ner-labels.txt
+models/*.json
+models/en/*
+models/ta/*
+```
+
+English size reference: [docs/git-xenovex-setup.md](git-xenovex-setup.md) notes ~431 MB for the English ONNX file.
+
+---
+
+## 8. Limitations
+
+| Limitation | Detail |
+|------------|--------|
+| **Tanglish on English model only** | Roman-script Tanglish (`Customer Senthil`) is classified `LatinOnly` and handled by English BERT. Recall is best-effort and inconsistent for non-standard spellings. Tamil ONNX is **not** invoked on Latin-only text. |
+| **Fail-open if model missing** | `OnnxNerPiiDetector` and routing runners return `[]` when models are unavailable. Person names are **not** redacted; regex/domain layers still run. No regex fallback for names. |
+| **128 token limit** | `OnnxTokenClassifierRunner` truncates encoding at 128 tokens. Very long prompts may miss person names beyond the window. |
+| **PERSON-only NER scope** | Organization, location, and misc NER labels are ignored. Only person spans become `<PERSON_n>`. |
+| **Mixed-script merge** | When both models run, overlapping spans are deduped by length; shorter overlapping spans are dropped. |
+| **Tamil ONNX availability** | Pre-exported Tamil ONNX may not exist on Hugging Face; local Python export is often required. |
+| **No fail-closed mode** | Missing NER does not block sanitization or LLM calls (optional Phase 5 enhancement). |
+
+---
+
+## 9. How to Reproduce
+
+### Download models
+
+From the repository root:
+
+```powershell
+.\scripts\download-ner-model.ps1
+.\scripts\download-tamil-ner-model.ps1
+```
+
+If Tamil PowerShell download fails with a 404 on `onnx/model.onnx`, install Python 3.12+ and re-run:
+
+```powershell
+winget install Python.Python.3.12 --accept-package-agreements --accept-source-agreements
+.\scripts\download-tamil-ner-model.ps1 -Python "$env:LOCALAPPDATA\Programs\Python\Python312\python.exe"
+```
+
+Optional: copy English assets from `models/` to `models/en/` to match `EnglishOnnxModelPath`.
+
+### Verify with tests
+
+```bash
+dotnet build
+dotnet test
+dotnet test --filter "Category=RealModel"
+dotnet test --filter "Category=TamilNer"
+dotnet test --logger "console;verbosity=detailed"
+```
+
+Tests skip gracefully when the corresponding ONNX files are absent.
+
+### Verify with console
+
+```bash
+dotnet run --project src/PiiRedaction.ConsoleApp -- --name CustomerNameOnly
+dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly
+dotnet run --project src/PiiRedaction.ConsoleApp -- --name TanglishCustomer
+```
+
+### Disable Tamil routing (English-only)
+
+Set in `appsettings.json`:
+
+```json
+"EnableTamilNer": false
+```
+
+---
+
+## Related Documentation
+
+- [README.md](../README.md) — build, run, and test overview
+- [architecture.md](architecture.md) — dual-model routing diagrams and trust boundary
+- [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) — implementation phases and Tanglish expectations
--- a/docs/tamil-tanglish-ner-plan.md
+++ b/docs/tamil-tanglish-ner-plan.md
@@ -0,0 +1,307 @@
+# Tamil / Tanglish NER — Implementation Plan
+
+**Goal:** Raise language coverage from ~15% to production-viable for Tamil script and Tanglish (Roman-script Tamil-English) customer prompts, without changing the secure LLM boundary pattern.
+
+**Status:** Phase 1–3 implemented  
+**Approach:** Dual-model ONNX NER routing (English + Tamil) + lightweight text normalization + optional Tanglish heuristics  
+**Estimated effort:** 4–6 engineering days across 4 phases
+
+---
+
+## 1. Current State vs Gap
+
+| Capability | Today | Tamil script | Tanglish (Latin) |
+|------------|-------|--------------|------------------|
+| Phone, PAN, Aadhaar, email, domain IDs | Regex + domain rules | Works (ASCII digits) | Works |
+| Person names | `dslim/bert-base-NER` (English BERT) | **Fails** — out of vocabulary | **Partial** — inconsistent |
+| Script / language routing | None | N/A | N/A |
+| Tamil numerals (௦–௯) | Not normalized | **May miss** phone/Aadhaar | N/A |
+| Label-aware cues (`பெயர்`, `peru`, `enga peru`) | None | **Misses** contextual names | **Misses** |
+
+**Root cause:** Person detection is a single English-only ONNX model behind `IOnnxNerModelRunner` → `OnnxNerPiiDetector`. Regex/domain layers are already language-agnostic.
+
+---
+
+## 2. Target Architecture
+
+No change to the trust boundary: `PromptSanitizer` → `CompositePiiDetector` → `PlaceholderPiiRedactor` → sanitized text only to LLM.
+
+Only the **NER adapter** expands:
+
+```mermaid
+flowchart TD
+  text["Prompt text"]
+  router["ScriptRouter (Core)"]
+  routing["RoutingOnnxNerModelRunner"]
+  en["EnglishOnnxRunner\nBERT WordPiece"]
+  ta["TamilOnnxRunner\nIndicBERT SentencePiece"]
+  nerDet["OnnxNerPiiDetector"]
+  composite["CompositePiiDetector"]
+
+  text --> composite
+  text --> nerDet
+  nerDet --> routing
+  routing --> router
+  router -->|"LatinOnly / Mixed"| en
+  router -->|"TamilOnly / Mixed"| ta
+  en --> routing
+  ta --> routing
+```
+
+### Routing rules
+
+| `ScriptComposition` | Models invoked | Tanglish note |
+|---------------------|----------------|---------------|
+| `LatinOnly` | English NER only | Tanglish names in Roman script |
+| `TamilOnly` | Tamil NER only | Tamil script names |
+| `Mixed` | **Both**, merge person spans | Common in Indian CS prompts |
+| `NoLetters` | Neither (or English fallback off) | Digits-only prompts |
+
+**Merge inside `RoutingOnnxNerModelRunner`:** dedupe overlapping person spans (prefer longer span; tie-break Tamil vs English by start index order).
+
+---
+
+## 3. Model Selection
+
+| Role | Model | Rationale |
+|------|-------|-----------|
+| English / Tanglish (Latin) | **Keep** `dslim/bert-base-NER` | Already integrated; works for many Indian names in Latin script |
+| Tamil script | **`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`** | Tamil NER; lighter than MuRIL (~0.6B) |
+| Fallback (optional Phase 5) | MuRIL Tamil NER | Only if IndicBERT recall is insufficient on eval set |
+
+### Asset layout
+
+```
+models/
+  en/
+    ner-model.onnx      # or model.onnx (BERT export)
+    vocab.txt
+    ner-labels.txt
+  ta/
+    model.onnx
+    sentencepiece.bpe.model   # or tokenizer.json from HF export
+    ner-labels.txt
+  ner-model.onnx        # legacy path — keep for backward compatibility
+```
+
+### Label mapping (Tamil fine-grained NER)
+
+SampurNER uses fine-grained tags (e.g. `B-person-politician`, `I-person-artist`). Map **any label containing `person`** (case-insensitive) → `PiiEntityType.Person`.
+
+English labels remain: `B-PER`, `I-PER`, `B-PERSON`, `I-PERSON`.
+
+---
+
+## 4. Implementation Phases
+
+### Phase 1 — Generic ONNX token classifier (1–2 days)
+
+**Objective:** Refactor `OnnxNerModelRunner` so BERT and SentencePiece are pluggable.
+
+| Action | Location |
+|--------|----------|
+| Add `ITokenClassifierEncoder` + `EncodedSequence` | `Infrastructure/Onnx/` |
+| `BertWordPieceEncoder` — extract from current runner | Infrastructure |
+| `SentencePieceEncoder` — IndicBERT tokenizer | Infrastructure |
+| `OnnxTokenClassifierRunner` — shared inference + BIO decode | Infrastructure |
+| `NerLabelConfig` — English vs Tamil person label predicates | Infrastructure |
+| `OnnxAssetPathResolver` — resolve model dir from repo root | Infrastructure |
+| Thin wrappers: `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` | Infrastructure |
+
+**Backward compat:** If `models/en/` missing, fall back to `OnnxModelPath` (`models/ner-model.onnx`).
+
+**No behavior change** until Phase 3 wiring — existing tests must pass.
+
+---
+
+### Phase 2 — Tamil model download + config (0.5–1 day)
+
+| Action | Details |
+|--------|---------|
+| `scripts/download-tamil-ner-model.ps1` + `.py` | Mirror `download-ner-model.ps1`; export via `optimum-cli export onnx --task token-classification` |
+| Optional: `scripts/download-all-ner-models.ps1` | Calls English + Tamil scripts |
+| Extend `PiiRedactionOptions` | `EnglishOnnxModelPath`, `TamilOnnxModelPath`, `EnableTamilNer` (default `true`) |
+| Update `appsettings.json` | New paths under `PiiRedaction` section |
+| `.gitignore` | `models/ta/*`, `models/en/*` (same as today for onnx/vocab) |
+
+---
+
+### Phase 3 — Script routing + DI (0.5–1 day)
+
+| Action | Location |
+|--------|----------|
+| `ScriptRouter` + `ScriptComposition` enum | `Core/Detection/` |
+| `RoutingOnnxNerModelRunner` implements `IOnnxNerModelRunner` | Infrastructure |
+| DI registration | `ServiceCollectionExtensions.cs` |
+
+```csharp
+services.AddSingleton<EnglishOnnxNerRunner>();
+services.AddSingleton<TamilOnnxNerRunner>();
+services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();
+```
+
+`OnnxNerPiiDetector` and `CompositePiiDetector` **unchanged**.
+
+---
+
+### Phase 4 — Tests, samples, docs (1 day)
+
+#### Unit tests
+
+| Test class | Coverage |
+|------------|----------|
+| `ScriptRouterTests` | Tamil-only, Latin-only, mixed, no-letters, boundary chars U+0B80/U+0BFF |
+| `RoutingOnnxNerModelRunnerTests` | Fake EN/TA runners; mixed script merges both |
+| `NerLabelConfigTests` | Tamil fine-grained person labels map correctly |
+
+#### Real-model tests (`Category=RealModel` or `Category=TamilNer`)
+
+| Scenario | Input example | Assert |
+|----------|---------------|--------|
+| Tamil name | `வாடிக்கையாளர் ராஜேஷ் தொலைபேசி 9876543210` | `<PERSON_1>`, phone redacted |
+| Tanglish name | `Customer Senthil phone 9876543210` | person + phone (best-effort) |
+| Mixed | `Rajesh மற்றும் Priya` | both persons redacted |
+| Clean Tamil | `பணத்தை திரும்பப் பெறுவது எப்படி?` | no false positives |
+| Canonical English | existing golden tests | no regression |
+
+Skip gracefully when `models/ta/model.onnx` missing (mirror `RealNerModelFixture`).
+
+#### Console samples
+
+Add to `SamplePromptCatalog.cs`:
+
+- `TamilCustomerName` — Tamil script person
+- `TanglishCustomerName` — `enga peru Rajesh` / `Customer Senthil`
+- `MixedTamilEnglish` — code-mixed prompt
+
+#### Docs
+
+- Update `README.md` ONNX setup (dual models)
+- Update `docs/architecture.md` NER section
+- Link this plan from README
+
+---
+
+### Phase 5 — Optional enhancements (post-MVP)
+
+| Enhancement | Benefit | Effort |
+|-------------|---------|--------|
+| **Unicode digit normalization** pre-pass | Tamil numerals → ASCII for regex | 0.5 day |
+| **Label-based regex** (`பெயர்`, `peru`, `peyar`, `enga peru`) | Tanglish recall without ML | 0.5 day |
+| **Tamil name gazetteer** `IPiiDetector` | High precision for top names | 1 day |
+| **Fail-closed policy** when NER unavailable | Compliance option | 0.5 day |
+| MuRIL model swap | Higher Tamil recall | eval-driven |
+
+---
+
+## 5. Tanglish — Realistic Expectations
+
+| Input type | Primary handler | Expected recall |
+|------------|-----------------|-----------------|
+| Tamil script names | Tamil ONNX NER | High (with eval tuning) |
+| Standard Latin Indian names (`Ravi Kumar`) | English ONNX NER | High (already works) |
+| Tanglish spellings (`Senthil`, `senthil`, `Centhil`) | English NER + optional gazetteer | Medium |
+| Code-mixed (`Rajesh oda account ACC-123456`) | English NER + domain regex | Medium–high for IDs; name variable |
+
+**MVP target:** Tamil script person names reliably redacted; Tanglish improved but not 100% without Phase 5 heuristics.
+
+---
+
+## 6. Success Metrics
+
+Before marking language gap closed, run an **eval set of 20–30 real prompts** (anonymized production samples):
+
+| Metric | MVP target |
+|--------|------------|
+| Tamil script person-name recall | ≥ 85% |
+| Tanglish person-name recall | ≥ 70% (with English model + optional heuristics) |
+| False positive rate (clean prompts) | ≤ 5% |
+| Structured PII (phone/PAN/domain) in Tamil prompts | ≥ 95% (regex layer) |
+| Regression on English canonical demo | 100% (existing golden tests) |
+
+---
+
+## 7. Files to Create / Modify (checklist)
+
+### New files
+
+- `src/PiiRedaction.Core/Detection/ScriptRouter.cs`
+- `src/PiiRedaction.Core/Detection/ScriptComposition.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/BertWordPieceEncoder.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/SentencePieceEncoder.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/ITokenClassifierEncoder.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/OnnxAssetPathResolver.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs`
+- `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs`
+- `scripts/download-tamil-ner-model.ps1` / `.py`
+- `tests/.../ScriptRouterTests.cs`
+- `tests/.../RoutingOnnxNerModelRunnerTests.cs`
+- `tests/.../RealTamilNerPipelineTests.cs`
+- `tests/TestSupport.Shared/RealTamilModelFixture.cs`
+
+### Modified files
+
+- `PiiRedactionOptions.cs` — dual model paths
+- `ServiceCollectionExtensions.cs` — routing DI
+- `appsettings.json` — config
+- `SamplePromptCatalog.cs` — Tamil/Tanglish demos
+- `ProductionPipelineFactory.cs` — `CreateWithRoutingRealModel()` for tests
+- `RealNerModelFixture.cs` / paths — support EN + TA
+- `README.md`, `docs/architecture.md`
+- `.gitignore` — `models/en/`, `models/ta/`
+
+### Unchanged (by design)
+
+- `PromptSanitizer`, `PlaceholderPiiRedactor`, `CompositePiiDetector`
+- `RegexPiiDetector`, `DomainRulePiiDetector`
+- `MockLlmPromptService` / LLM boundary
+
+---
+
+## 8. Rollout & Risk
+
+| Risk | Mitigation |
+|------|------------|
+| Tamil model export fails on Windows | PowerShell fallback downloads pre-exported ONNX from Hugging Face |
+| Larger memory (two models) | Lazy-load Tamil runner only when `EnableTamilNer` and Tamil script detected |
+| Fine-grained label mismatch | Load labels from `ner-labels.txt`; unit test label config |
+| Tanglish disappointment | Set stakeholder expectation in README; Phase 5 heuristics |
+| CI without models | Fast tests use fakes; `Category=TamilNer` skips like `RealModel` |
+
+**Feature flag:** `EnableTamilNer=false` reverts to English-only behavior for gradual rollout.
+
+---
+
+## 9. Command Reference (after implementation)
+
+```powershell
+# Download both models
+.\scripts\download-ner-model.ps1
+.\scripts\download-tamil-ner-model.ps1
+
+# Run Tamil-focused console sample
+dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerName
+
+# Tests
+dotnet test
+dotnet test --filter "Category=TamilNer"
+dotnet test --filter "Category=RealModel"
+```
+
+---
+
+## 10. Approval Checklist
+
+- [ ] Stakeholder sign-off on dual-model approach (vs single multilingual model)
+- [ ] Tamil eval prompt set collected (20–30 samples)
+- [ ] Xenovex CI policy: models downloaded in pipeline or tests skip
+- [x] Phase 1–3 implementation PR
+- [ ] Phase 4 eval metrics met
+- [ ] Optional Phase 5 for Tanglish heuristics if recall &lt; 70%
+
+---
+
+**Next step:** Implement Phase 1 in a feature branch (`feature/tamil-tanglish-ner`), open PR to `main` on `xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc`.