llm-pii-poc/docs/tamil-tanglish-ner-plan.md

# Tamil / Tanglish NER — Implementation Plan

**Goal:** Raise language coverage from ~15% to production-viable for Tamil script and Tanglish (Roman-script Tamil-English) customer prompts, without changing the secure LLM boundary pattern.

**Status:** Phase 1–3 implemented  
**Approach:** Dual-model ONNX NER routing (English + Tamil) + lightweight text normalization + optional Tanglish heuristics  
**Estimated effort:** 4–6 engineering days across 4 phases

---

## 1. Current State vs Gap

| Capability | Today | Tamil script | Tanglish (Latin) |
|------------|-------|--------------|------------------|
| Phone, PAN, Aadhaar, email, domain IDs | Regex + domain rules | Works (ASCII digits) | Works |
| Person names | `dslim/bert-base-NER` (English BERT) | **Fails** — out of vocabulary | **Partial** — inconsistent |
| Script / language routing | None | N/A | N/A |
| Tamil numerals (௦–௯) | Not normalized | **May miss** phone/Aadhaar | N/A |
| Label-aware cues (`பெயர்`, `peru`, `enga peru`) | None | **Misses** contextual names | **Misses** |

**Root cause:** Person detection is a single English-only ONNX model behind `IOnnxNerModelRunner` → `OnnxNerPiiDetector`. Regex/domain layers are already language-agnostic.

---

## 2. Target Architecture

No change to the trust boundary: `PromptSanitizer` → `CompositePiiDetector` → `PlaceholderPiiRedactor` → sanitized text only to LLM.

Only the **NER adapter** expands:

```mermaid
flowchart TD
  text["Prompt text"]
  router["ScriptRouter (Core)"]
  routing["RoutingOnnxNerModelRunner"]
  en["EnglishOnnxRunner\nBERT WordPiece"]
  ta["TamilOnnxRunner\nIndicBERT SentencePiece"]
  nerDet["OnnxNerPiiDetector"]
  composite["CompositePiiDetector"]

  text --> composite
  text --> nerDet
  nerDet --> routing
  routing --> router
  router -->|"LatinOnly / Mixed"| en
  router -->|"TamilOnly / Mixed"| ta
  en --> routing
  ta --> routing
```

### Routing rules

| `ScriptComposition` | Models invoked | Tanglish note |
|---------------------|----------------|---------------|
| `LatinOnly` | English NER only | Tanglish names in Roman script |
| `TamilOnly` | Tamil NER only | Tamil script names |
| `Mixed` | **Both**, merge person spans | Common in Indian CS prompts |
| `NoLetters` | Neither (or English fallback off) | Digits-only prompts |

**Merge inside `RoutingOnnxNerModelRunner`:** dedupe overlapping person spans (prefer longer span; tie-break Tamil vs English by start index order).

---

## 3. Model Selection

| Role | Model | Rationale |
|------|-------|-----------|
| English / Tanglish (Latin) | **Keep** `dslim/bert-base-NER` | Already integrated; works for many Indian names in Latin script |
| Tamil script | **`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`** | Tamil NER; lighter than MuRIL (~0.6B) |
| Fallback (optional Phase 5) | MuRIL Tamil NER | Only if IndicBERT recall is insufficient on eval set |

### Asset layout

```
models/
  en/
    ner-model.onnx      # or model.onnx (BERT export)
    vocab.txt
    ner-labels.txt
  ta/
    model.onnx
    sentencepiece.bpe.model   # or tokenizer.json from HF export
    ner-labels.txt
  ner-model.onnx        # legacy path — keep for backward compatibility
```

### Label mapping (Tamil fine-grained NER)

SampurNER uses fine-grained tags (e.g. `B-person-politician`, `I-person-artist`). Map **any label containing `person`** (case-insensitive) → `PiiEntityType.Person`.

English labels remain: `B-PER`, `I-PER`, `B-PERSON`, `I-PERSON`.

---

## 4. Implementation Phases

### Phase 1 — Generic ONNX token classifier (1–2 days)

**Objective:** Refactor `OnnxNerModelRunner` so BERT and SentencePiece are pluggable.

| Action | Location |
|--------|----------|
| Add `ITokenClassifierEncoder` + `EncodedSequence` | `Infrastructure/Onnx/` |
| `BertWordPieceEncoder` — extract from current runner | Infrastructure |
| `SentencePieceEncoder` — IndicBERT tokenizer | Infrastructure |
| `OnnxTokenClassifierRunner` — shared inference + BIO decode | Infrastructure |
| `NerLabelConfig` — English vs Tamil person label predicates | Infrastructure |
| `OnnxAssetPathResolver` — resolve model dir from repo root | Infrastructure |
| Thin wrappers: `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` | Infrastructure |

**Backward compat:** If `models/en/` missing, fall back to `OnnxModelPath` (`models/ner-model.onnx`).

**No behavior change** until Phase 3 wiring — existing tests must pass.

---

### Phase 2 — Tamil model download + config (0.5–1 day)

| Action | Details |
|--------|---------|
| `scripts/download-tamil-ner-model.ps1` + `.py` | Mirror `download-ner-model.ps1`; export via `optimum-cli export onnx --task token-classification` |
| Optional: `scripts/download-all-ner-models.ps1` | Calls English + Tamil scripts |
| Extend `PiiRedactionOptions` | `EnglishOnnxModelPath`, `TamilOnnxModelPath`, `EnableTamilNer` (default `true`) |
| Update `appsettings.json` | New paths under `PiiRedaction` section |
| `.gitignore` | `models/ta/*`, `models/en/*` (same as today for onnx/vocab) |

---

### Phase 3 — Script routing + DI (0.5–1 day)

| Action | Location |
|--------|----------|
| `ScriptRouter` + `ScriptComposition` enum | `Core/Detection/` |
| `RoutingOnnxNerModelRunner` implements `IOnnxNerModelRunner` | Infrastructure |
| DI registration | `ServiceCollectionExtensions.cs` |

```csharp
services.AddSingleton<EnglishOnnxNerRunner>();
services.AddSingleton<TamilOnnxNerRunner>();
services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();
```

`OnnxNerPiiDetector` and `CompositePiiDetector` **unchanged**.

---

### Phase 4 — Tests, samples, docs (1 day)

#### Unit tests

| Test class | Coverage |
|------------|----------|
| `ScriptRouterTests` | Tamil-only, Latin-only, mixed, no-letters, boundary chars U+0B80/U+0BFF |
| `RoutingOnnxNerModelRunnerTests` | Fake EN/TA runners; mixed script merges both |
| `NerLabelConfigTests` | Tamil fine-grained person labels map correctly |

#### Real-model tests (`Category=RealModel` or `Category=TamilNer`)

| Scenario | Input example | Assert |
|----------|---------------|--------|
| Tamil name | `வாடிக்கையாளர் ராஜேஷ் தொலைபேசி 9876543210` | `<PERSON_1>`, phone redacted |
| Tanglish name | `Customer Senthil phone 9876543210` | person + phone (best-effort) |
| Mixed | `Rajesh மற்றும் Priya` | both persons redacted |
| Clean Tamil | `பணத்தை திரும்பப் பெறுவது எப்படி?` | no false positives |
| Canonical English | existing golden tests | no regression |

Skip gracefully when `models/ta/model.onnx` missing (mirror `RealNerModelFixture`).

#### Console samples

Add to `SamplePromptCatalog.cs`:

- `TamilCustomerName` — Tamil script person
- `TanglishCustomerName` — `enga peru Rajesh` / `Customer Senthil`
- `MixedTamilEnglish` — code-mixed prompt

#### Docs

- Update `README.md` ONNX setup (dual models)
- Update `docs/architecture.md` NER section
- Link this plan from README

---

### Phase 5 — Optional enhancements (post-MVP)

| Enhancement | Benefit | Effort |
|-------------|---------|--------|
| **Unicode digit normalization** pre-pass | Tamil numerals → ASCII for regex | 0.5 day |
| **Label-based regex** (`பெயர்`, `peru`, `peyar`, `enga peru`) | Tanglish recall without ML | 0.5 day |
| **Tamil name gazetteer** `IPiiDetector` | High precision for top names | 1 day |
| **Fail-closed policy** when NER unavailable | Compliance option | 0.5 day |
| MuRIL model swap | Higher Tamil recall | eval-driven |

---

## 5. Tanglish — Realistic Expectations

| Input type | Primary handler | Expected recall |
|------------|-----------------|-----------------|
| Tamil script names | Tamil ONNX NER | High (with eval tuning) |
| Standard Latin Indian names (`Ravi Kumar`) | English ONNX NER | High (already works) |
| Tanglish spellings (`Senthil`, `senthil`, `Centhil`) | English NER + optional gazetteer | Medium |
| Code-mixed (`Rajesh oda account ACC-123456`) | English NER + domain regex | Medium–high for IDs; name variable |

**MVP target:** Tamil script person names reliably redacted; Tanglish improved but not 100% without Phase 5 heuristics.

---

## 6. Success Metrics

Before marking language gap closed, run an **eval set of 20–30 real prompts** (anonymized production samples):

| Metric | MVP target |
|--------|------------|
| Tamil script person-name recall | ≥ 85% |
| Tanglish person-name recall | ≥ 70% (with English model + optional heuristics) |
| False positive rate (clean prompts) | ≤ 5% |
| Structured PII (phone/PAN/domain) in Tamil prompts | ≥ 95% (regex layer) |
| Regression on English canonical demo | 100% (existing golden tests) |

---

## 7. Files to Create / Modify (checklist)

### New files

- `src/PiiRedaction.Core/Detection/ScriptRouter.cs`
- `src/PiiRedaction.Core/Detection/ScriptComposition.cs`
- `src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs`
- `src/PiiRedaction.Infrastructure/Onnx/BertWordPieceEncoder.cs`
- `src/PiiRedaction.Infrastructure/Onnx/SentencePieceEncoder.cs`
- `src/PiiRedaction.Infrastructure/Onnx/ITokenClassifierEncoder.cs`
- `src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs`
- `src/PiiRedaction.Infrastructure/Onnx/OnnxAssetPathResolver.cs`
- `src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs`
- `src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs`
- `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs`
- `scripts/download-tamil-ner-model.ps1` / `.py`
- `tests/.../ScriptRouterTests.cs`
- `tests/.../RoutingOnnxNerModelRunnerTests.cs`
- `tests/.../RealTamilNerPipelineTests.cs`
- `tests/TestSupport.Shared/RealTamilModelFixture.cs`

### Modified files

- `PiiRedactionOptions.cs` — dual model paths
- `ServiceCollectionExtensions.cs` — routing DI
- `appsettings.json` — config
- `SamplePromptCatalog.cs` — Tamil/Tanglish demos
- `ProductionPipelineFactory.cs` — `CreateWithRoutingRealModel()` for tests
- `RealNerModelFixture.cs` / paths — support EN + TA
- `README.md`, `docs/architecture.md`
- `.gitignore` — `models/en/`, `models/ta/`

### Unchanged (by design)

- `PromptSanitizer`, `PlaceholderPiiRedactor`, `CompositePiiDetector`
- `RegexPiiDetector`, `DomainRulePiiDetector`
- `MockLlmPromptService` / LLM boundary

---

## 8. Rollout & Risk

| Risk | Mitigation |
|------|------------|
| Tamil model export fails on Windows | PowerShell fallback downloads pre-exported ONNX from Hugging Face |
| Larger memory (two models) | Lazy-load Tamil runner only when `EnableTamilNer` and Tamil script detected |
| Fine-grained label mismatch | Load labels from `ner-labels.txt`; unit test label config |
| Tanglish disappointment | Set stakeholder expectation in README; Phase 5 heuristics |
| CI without models | Fast tests use fakes; `Category=TamilNer` skips like `RealModel` |

**Feature flag:** `EnableTamilNer=false` reverts to English-only behavior for gradual rollout.

---

## 9. Command Reference (after implementation)

```powershell
# Download both models
.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1

# Run Tamil-focused console sample
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerName

# Tests
dotnet test
dotnet test --filter "Category=TamilNer"
dotnet test --filter "Category=RealModel"
```

---

## 10. Approval Checklist

- [ ] Stakeholder sign-off on dual-model approach (vs single multilingual model)
- [ ] Tamil eval prompt set collected (20–30 samples)
- [ ] Xenovex CI policy: models downloaded in pipeline or tests skip
- [x] Phase 1–3 implementation PR
- [ ] Phase 4 eval metrics met
- [ ] Optional Phase 5 for Tanglish heuristics if recall &lt; 70%

---

**Next step:** Implement Phase 1 in a feature branch (`feature/tamil-tanglish-ner`), open PR to `main` on `xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc`.