Add Tamil NER routing and WPF test harness for POC validation.

Introduce dual-script ONNX NER routing (English/Tamil/mixed), Tamil console samples and integration tests, model download scripts, and a resizable WPF MVVM harness with click-to-load prompts, batch validation, and runtime-adjustable detection panels.
2026-07-07 17:12:38 +05:30
parent a707c6c9cf
commit cf8f5a7232
71 changed files with 4494 additions and 356 deletions
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -10,7 +10,7 @@ The POC validates a compliance-oriented pattern suitable for financial and custo

 ## Canonical Example

-The console application ships with a **sample catalog** (11 prompts). The canonical demo is sample `FullFinancialWithCustomer`. The table below shows the exact strings produced by the production pipeline when the ONNX NER model is loaded (run `scripts/download-ner-model.ps1` first).
+The console application ships with a **sample catalog** (16 prompts). The canonical demo is sample `FullFinancialWithCustomer`. Tamil script, Tanglish, and mixed-script samples run in the **default** `dotnet run` batch (no `--interactive` required). The table below shows the exact strings produced by the production pipeline when the ONNX NER models are loaded (run `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1` first).

 | Stage | Value |
 |-------|-------|
@@ -38,7 +38,7 @@ Running `dotnet run --project src/PiiRedaction.ConsoleApp` executes all samples

 ### NER / person-name samples

-These prompts exercise `OnnxNerPiiDetector` and `OnnxNerModelRunner`. Person names require the ONNX model (`models/ner-model.onnx` plus `vocab.txt` and `ner-labels.txt`). Without the model, person spans are not detected.
+These prompts exercise `OnnxNerPiiDetector` and `RoutingOnnxNerModelRunner`. Person names require ONNX models (`models/en/` for English, `models/ta/` for Tamil script). Without models, person spans are not detected. Legacy `models/ner-model.onnx` is still supported for English.

 | Sample | Input (excerpt) | Detected person | Sanitized (excerpt) |
 |--------|-----------------|-----------------|---------------------|
@@ -50,6 +50,18 @@ These prompts exercise `OnnxNerPiiDetector` and `OnnxNerModelRunner`. Person nam
 | **PersonWithDomainIds** | Customer Meera Iyer holds CID-7070… | Meera Iyer | Customer `<PERSON_1>` holds `<CUSTOMER_ID_1>`… |
 | **PersonWithEmailNoPhone** | Customer Arjun Mehta wrote from arjun.mehta@company.in… | Arjun Mehta | Customer `<PERSON_1>` wrote from `<EMAIL_1>`… |

+### Tamil / Tanglish / mixed samples
+
+These prompts exercise `RoutingOnnxNerModelRunner` script routing. Tamil script uses `models/ta/`; Latin Tanglish uses `models/en/`. Mixed prompts may invoke both models.
+
+| Sample | Input (excerpt) | Detected person | Sanitized (excerpt) |
+|--------|-----------------|-----------------|---------------------|
+| **TamilCustomerNameOnly** | வாடிக்கையாளர் ராஜேஷ் குமார் சேமிப்பு… | ராஜேஷ் குமார் | வாடிக்கையாளர் `<PERSON_1>` சேமிப்பு… |
+| **TamilWithPhonePan** | வாடிக்கையாளர் ராஜேஷ் குமார் தொலைபேசி 9876543210 PAN… | ராஜேஷ் குமார் | `<PERSON_1>` … `<PHONE_1>` … `<PAN_1>` |
+| **TanglishCustomer** | Customer Senthil phone 9876543210… | Senthil | Customer `<PERSON_1>` phone `<PHONE_1>`… |
+| **MixedTamilEnglish** | வாடிக்கையாளர் Ravi Kumar phone 9876543210… | Ravi Kumar | வாடிக்கையாளர் `<PERSON_1>` phone `<PHONE_1>`… |
+| **TamilFullFinancial** | வாடிக்கையாளர் ராஜேஷ் குமார் மின்னஞ்சல் ravi.kumar@gmail.com… | ராஜேஷ் குமார் | Tamil canonical — all placeholder types |
+
 ### Other sample categories

 | Category | Sample | Purpose |
@@ -84,7 +96,9 @@ flowchart TB
    end

    subgraph infra [PiiRedaction.Infrastructure]
-        onnxRunner["OnnxNerModelRunner"]
+        onnxRunner["RoutingOnnxNerModelRunner"]
+        enRunner["EnglishOnnxNerRunner"]
+        taRunner["TamilOnnxNerRunner"]
        mockLlm["MockLlmPromptService"]
        mockChat["MockChatClient"]
    end
@@ -110,6 +124,8 @@ flowchart TB
    di -.-> mockLlm
 ```

+`RoutingOnnxNerModelRunner` selects English and/or Tamil ONNX models based on script composition in the prompt. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english) for the routing decision tree.
+
 ---

 ## Detection to Redaction Detail
@@ -123,7 +139,7 @@ flowchart LR
  subgraph detectPhase [Detection Phase]
    domainDet["DomainRulePiiDetector"]
    regexDet["RegexPiiDetector"]
-    onnxDet["OnnxNerPiiDetector"]
+    onnxDet["OnnxNerPiiDetector<br/>(RoutingOnnxNerModelRunner)"]
    composite["CompositePiiDetector"]
    merge["Overlap merge and source priority"]
    entityList["PiiEntity list"]
@@ -157,14 +173,144 @@ flowchart LR
 2. On overlapping spans, the first registered detector wins.
 3. Tie-breaking uses source priority: Domain (3) > Regex (2) > NER (1).

+The ONNX NER detector delegates to `RoutingOnnxNerModelRunner`, which routes inference to English and/or Tamil models by script composition. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english).
+
 **Placeholder assignment** (applied by `PlaceholderPiiRedactor`):

+
 - Format: `<{TYPE}_{n}>` (e.g. `<EMAIL_1>`, `<PERSON_1>`).
 - Duplicate values of the same type reuse the same placeholder.
 - Replacement proceeds from highest `StartIndex` to lowest to avoid index drift.

 ---

+## Dual-Model NER Routing (Tamil + English)
+
+Person-name detection uses two ONNX token-classifier models: **English** (`models/en/`, BERT WordPiece) and **Tamil** (`models/ta/`, SentencePiece or WordPiece). `OnnxNerPiiDetector` calls `RoutingOnnxNerModelRunner`, which classifies prompt script via `ScriptRouter` and dispatches to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`. Both runners share `OnnxTokenClassifierRunner` for BIO decoding; only **PERSON** spans are emitted.
+
+The diagram below expands the detection and NER branches summarized in [High-Level Data Flow](#high-level-data-flow) and [Detection to Redaction Detail](#detection-to-redaction-detail).
+
+### End-to-end pipeline (with NER branch)
+
+```mermaid
+flowchart TB
+    subgraph Entry["Console entry"]
+        A["Program.cs<br/>Host + AddPiiRedactionServices()"]
+        B["PromptDemoRunner.RunAsync()"]
+        A --> B
+    end
+
+    B --> C["SanitizationRequest(OriginalPrompt)"]
+    C --> D["PromptSanitizer.Sanitize()"]
+
+    subgraph Detect["CompositePiiDetector.Detect() — registration order"]
+        direction TB
+        E1["DomainRulePiiDetector<br/>LOAN_NUMBER, CUSTOMER_ID, ACCOUNT_NUMBER"]
+        E2["RegexPiiDetector<br/>EMAIL, PHONE, AADHAAR, PAN, CREDIT_CARD"]
+        E3["OnnxNerPiiDetector<br/>PERSON (via IOnnxNerModelRunner)"]
+        E1 --> MERGE
+        E2 --> MERGE
+        E3 --> MERGE
+        MERGE["Merge overlapping spans<br/>sort: StartIndex ↑, Length ↓, Source priority ↓<br/>(Domain=3, Regex=2, Ner=1)<br/>first candidate wins on overlap"]
+    end
+
+    D --> Detect
+    MERGE --> F["IReadOnlyList&lt;PiiEntity&gt;"]
+
+    F --> G["PlaceholderPiiRedactor.Redact()<br/>replace spans right-to-left<br/>dedupe by Type|Value → &lt;TYPE_n&gt;"]
+    G --> H["SanitizationResult<br/>SanitizedPrompt, DetectedEntities, PlaceholderMap"]
+
+    H --> I["MockLlmPromptService.SendPromptAsync(SanitizedPrompt)"]
+    I --> J["Mock LLM response<br/>(sanitized text only)"]
+
+    subgraph NerBranch["OnnxNerPiiDetector branch"]
+        E3 --> N1{"RoutingOnnxNerModelRunner<br/>.IsModelAvailable?"}
+        N1 -->|no| N2["return []"]
+        N1 -->|yes| N3["RoutingOnnxNerModelRunner<br/>.PredictEntities()"]
+    end
+```
+
+### RoutingOnnxNerModelRunner decision tree
+
+`ScriptRouter.GetComposition` scans each character once. Tamil letters (U+0B80–U+0BFF) and ASCII Latin letters (`char.IsAsciiLetter`) determine the route. When both scripts appear, classification is **Mixed** (early exit).
+
+```mermaid
+flowchart TB
+    IN["text"] --> SR["ScriptRouter.GetComposition(text)<br/>scan each char"]
+
+    SR --> C1{"LatinOnly?"}
+    SR --> C2{"TamilOnly?"}
+    SR --> C3{"Mixed?"}
+    SR --> C4{"NoLetters?"}
+
+    C1 -->|yes| EN1{"EnglishOnnxNerRunner<br/>.IsModelAvailable?"}
+    EN1 -->|yes| EN_RUN["EnglishOnnxNerRunner.PredictEntities(text)"]
+    EN1 -->|no| SKIP1["skip English"]
+    EN_RUN --> ACC
+    SKIP1 --> ACC
+
+    C2 -->|yes| TA_GATE{"EnableTamilNer<br/>&& TamilOnnxNerRunner<br/>.IsModelAvailable?"}
+    TA_GATE -->|yes| TA_RUN["TamilOnnxNerRunner.PredictEntities(text)"]
+    TA_GATE -->|no| SKIP2["skip Tamil"]
+    TA_RUN --> ACC
+    SKIP2 --> ACC
+
+    C3 -->|yes| EN2{"English available?"}
+    EN2 -->|yes| EN_MIX["EnglishOnnxNerRunner.PredictEntities(text)"]
+    EN2 -->|no| SKIP3["skip English"]
+    EN_MIX --> TA_GATE2{"EnableTamilNer<br/>&& Tamil available?"}
+    SKIP3 --> TA_GATE2
+    TA_GATE2 -->|yes| TA_MIX["TamilOnnxNerRunner.PredictEntities(text)"]
+    TA_GATE2 -->|no| SKIP4["skip Tamil"]
+    TA_MIX --> ACC
+    SKIP4 --> ACC
+
+    C4 -->|yes| EMPTY["no NER inference"]
+    EMPTY --> OUT_EMPTY["return []"]
+
+    subgraph EN_Pipeline["EnglishOnnxNerRunner"]
+        EN_RUN --> EN_ENC["BertWordPieceEncoder<br/>(model dir vocab.txt)"]
+        EN_ENC --> EN_OCR["OnnxTokenClassifierRunner<br/>NerLabelConfig.English<br/>B-PER / I-PER / B-PERSON / I-PERSON"]
+    end
+
+    subgraph TA_Pipeline["TamilOnnxNerRunner"]
+        TA_RUN --> TA_ENC["TokenClassifierEncoderFactory.Create()<br/>vocab.txt → BertWordPieceEncoder<br/>else SentencePiece (*.bpe.model, spiece.model, tokenizer.model)"]
+        TA_ENC --> TA_OCR["OnnxTokenClassifierRunner<br/>NerLabelConfig.Tamil<br/>label contains 'person' (case-insensitive)"]
+    end
+
+    subgraph SharedInference["OnnxTokenClassifierRunner (shared)"]
+        ENC["Encode(text, max 128 tokens)"]
+        ONNX["ONNX InferenceSession.Run<br/>input_ids + attention_mask [+ token_type_ids]"]
+        ARGMAX["Per-token argmax over logits"]
+        BIO["BIO decode → PiiEntityType.Person<br/>PiiDetectionSource.Ner"]
+        ENC --> ONNX --> ARGMAX --> BIO
+    end
+
+    EN_OCR --> SharedInference
+    TA_OCR --> SharedInference
+    BIO --> ACC["accumulate entities"]
+
+    ACC --> MERGE["MergePersonSpans()<br/>sort: Length ↓, StartIndex ↑<br/>drop overlapping spans<br/>(longer span wins)"]
+    MERGE --> OUT["return merged PERSON entities"]
+```
+
+### Routing rules
+
+| Rule | Source | Behavior |
+|------|--------|----------|
+| **Script classification** | `ScriptRouter.GetComposition` | Single pass over characters. Tamil letter = U+0B80–U+0BFF. Latin letter = `char.IsAsciiLetter`. Both seen → `Mixed` (early exit). Neither → `NoLetters`. Tamil only → `TamilOnly`. Latin only → `LatinOnly`. |
+| **LatinOnly** | `RoutingOnnxNerModelRunner` | Run **English only** if `englishRunner.IsModelAvailable`. |
+| **TamilOnly** | `RoutingOnnxNerModelRunner` | Run **Tamil only** if `EnableTamilNer` (default `true` in `PiiRedactionOptions`) **and** `tamilRunner.IsModelAvailable`. |
+| **Mixed** | `RoutingOnnxNerModelRunner` | Run **both** models independently on the **full text** (English if available; Tamil if `EnableTamilNer` and available). |
+| **NoLetters** | `RoutingOnnxNerModelRunner` | No NER inference; returns `[]` from routing (before merge). |
+| **Model availability gate** | `OnnxNerPiiDetector` | If `RoutingOnnxNerModelRunner.IsModelAvailable` is false, NER detector returns `[]` (English OR Tamil available when Tamil enabled). |
+| **Post-route merge** | `MergePersonSpans` | After EN/TA results are concatenated, overlapping PERSON spans are deduped; **longer span wins**, then ordered by `StartIndex`. |
+| **Composite merge** | `CompositePiiDetector` | Domain → Regex → NER all run. Overlaps resolved globally: earlier registration order + longer span + higher source priority (Domain > Regex > Ner). |
+| **Encoder choice** | `EnglishOnnxNerRunner` vs `TamilOnnxNerRunner` | English always uses `BertWordPieceEncoder`. Tamil uses factory: `vocab.txt` → WordPiece; else first SentencePiece file found; fallback WordPiece with warning. |
+| **NER output scope** | `OnnxTokenClassifierRunner` | Only **PERSON** entities decoded from BIO tags; max sequence length **128** tokens. |
+
+---
+
 ## Runtime Sequence

 ```mermaid
@@ -255,7 +401,7 @@ In the POC, `MockChatClient` simulates the external provider without network I/O
 |---------|-------|----------------|
 | `PiiRedaction.ConsoleApp` | Presentation | Application entry point; reads prompt (sample or interactive); bootstraps `IHost` and DI via `AddPiiRedactionServices`; orchestrates sanitization and LLM invocation; renders audit output (detected entities, sanitized text, placeholder map). |
 | `PiiRedaction.Core` | Domain / Application | Defines abstractions (`IPiiDetector`, `IPiiRedactor`, `IPromptSanitizer`, `ILlmPromptService`); implements detection strategies (`RegexPiiDetector`, `DomainRulePiiDetector`, `OnnxNerPiiDetector`, `CompositePiiDetector`); implements `PlaceholderPiiRedactor` and `PromptSanitizer`; owns domain models (`PiiEntity`, `SanitizationResult`, `RedactionResult`) and configuration (`PiiRedactionOptions`). Has no dependency on ONNX Runtime or LLM SDKs. |
-| `PiiRedaction.Infrastructure` | Infrastructure | Implements technical adapters: `OnnxNerModelRunner` (ONNX Runtime inference), `MockChatClient` and `MockLlmPromptService` (`Microsoft.Extensions.AI`); depends on Core abstractions and is swappable without changing domain logic. |
+| `PiiRedaction.Infrastructure` | Infrastructure | Implements technical adapters: `RoutingOnnxNerModelRunner`, `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` (ONNX Runtime inference), `MockChatClient` and `MockLlmPromptService` (`Microsoft.Extensions.AI`); depends on Core abstractions and is swappable without changing domain logic. |
 | `tests/PiiRedaction.Core.Tests` | Test | Unit and integration tests for detectors, redactor, sanitizer, overlap rules, golden prompt scenarios (`PromptScenarioCatalog`), and LLM boundary assertions. |
 | `tests/PiiRedaction.Infrastructure.Tests` | Test | Tests for mock LLM behavior and ONNX runner load semantics. |

@@ -272,7 +418,7 @@ In the POC, `MockChatClient` simulates the external provider without network I/O
 | `IPromptSanitizer` | Core | `PromptSanitizer` | Unlikely to change; orchestrates detect + redact |
 | `ILlmPromptService` | Core | `MockLlmPromptService` | Production adapter with telemetry, retry, policy |
 | `IChatClient` | Microsoft.Extensions.AI | `MockChatClient` | Azure OpenAI, OpenAI, or other provider SDK |
-| `IOnnxNerModelRunner` | Core | `OnnxNerModelRunner` | BERT WordPiece tokenization, ONNX inference, BIO label decoding |
+| `IOnnxNerModelRunner` | Core | `RoutingOnnxNerModelRunner` | Script-based routing to English (BERT WordPiece) and Tamil (SentencePiece) ONNX models |

 ---

@@ -282,14 +428,18 @@ Runtime behavior is controlled via `appsettings.json` under the `PiiRedaction` s

 | Setting | Effect |
 |---------|--------|
-| `OnnxModelPath` | Path to ONNX NER model (`models/ner-model.onnx` by default). Companion files `vocab.txt` and `ner-labels.txt` must live in the same directory. |
+| `OnnxModelPath` | Legacy English model path (`models/ner-model.onnx`). Used as fallback when `models/en/` is absent. |
+| `EnglishOnnxModelPath` | Primary English ONNX model (`models/en/ner-model.onnx`). |
+| `TamilOnnxModelPath` | Tamil ONNX model (`models/ta/model.onnx`). |
+| `EnableTamilNer` | When `false`, routing uses English model only. Default `true`. |

-Download the model assets with `scripts/download-ner-model.ps1` (exports `dslim/bert-base-NER`).
+Download model assets with `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1`.

 ---

 ## Related Documentation

+- [NER models](ner-models.md) — English/Tamil ONNX model IDs, assets, labels, and integration reference
 - [README](../README.md) — build, run, configuration, and testing instructions
 - [ServiceCollectionExtensions.cs](../src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs) — DI registration and detector ordering
 - [PromptScenarioCatalog.cs](../tests/PiiRedaction.Core.Tests/TestSupport/PromptScenarioCatalog.cs) — focused golden pipeline scenarios including the canonical example