Add Tamil NER routing and WPF test harness for POC validation.
Introduce dual-script ONNX NER routing (English/Tamil/mixed), Tamil console samples and integration tests, model download scripts, and a resizable WPF MVVM harness with click-to-load prompts, batch validation, and runtime-adjustable detection panels.
This commit is contained in:
Bilal Nazer Ali
307
docs/tamil-tanglish-ner-plan.md
Normal file
307
docs/tamil-tanglish-ner-plan.md
Normal file
@@ -0,0 +1,307 @@
|
||||
# Tamil / Tanglish NER — Implementation Plan
|
||||
|
||||
**Goal:** Raise language coverage from ~15% to production-viable for Tamil script and Tanglish (Roman-script Tamil-English) customer prompts, without changing the secure LLM boundary pattern.
|
||||
|
||||
**Status:** Phase 1–3 implemented
|
||||
**Approach:** Dual-model ONNX NER routing (English + Tamil) + lightweight text normalization + optional Tanglish heuristics
|
||||
**Estimated effort:** 4–6 engineering days across 4 phases
|
||||
|
||||
---
|
||||
|
||||
## 1. Current State vs Gap
|
||||
|
||||
| Capability | Today | Tamil script | Tanglish (Latin) |
|
||||
|------------|-------|--------------|------------------|
|
||||
| Phone, PAN, Aadhaar, email, domain IDs | Regex + domain rules | Works (ASCII digits) | Works |
|
||||
| Person names | `dslim/bert-base-NER` (English BERT) | **Fails** — out of vocabulary | **Partial** — inconsistent |
|
||||
| Script / language routing | None | N/A | N/A |
|
||||
| Tamil numerals (௦–௯) | Not normalized | **May miss** phone/Aadhaar | N/A |
|
||||
| Label-aware cues (`பெயர்`, `peru`, `enga peru`) | None | **Misses** contextual names | **Misses** |
|
||||
|
||||
**Root cause:** Person detection is a single English-only ONNX model behind `IOnnxNerModelRunner` → `OnnxNerPiiDetector`. Regex/domain layers are already language-agnostic.
|
||||
|
||||
---
|
||||
|
||||
## 2. Target Architecture
|
||||
|
||||
No change to the trust boundary: `PromptSanitizer` → `CompositePiiDetector` → `PlaceholderPiiRedactor` → sanitized text only to LLM.
|
||||
|
||||
Only the **NER adapter** expands:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
text["Prompt text"]
|
||||
router["ScriptRouter (Core)"]
|
||||
routing["RoutingOnnxNerModelRunner"]
|
||||
en["EnglishOnnxRunner\nBERT WordPiece"]
|
||||
ta["TamilOnnxRunner\nIndicBERT SentencePiece"]
|
||||
nerDet["OnnxNerPiiDetector"]
|
||||
composite["CompositePiiDetector"]
|
||||
|
||||
text --> composite
|
||||
text --> nerDet
|
||||
nerDet --> routing
|
||||
routing --> router
|
||||
router -->|"LatinOnly / Mixed"| en
|
||||
router -->|"TamilOnly / Mixed"| ta
|
||||
en --> routing
|
||||
ta --> routing
|
||||
```
|
||||
|
||||
### Routing rules
|
||||
|
||||
| `ScriptComposition` | Models invoked | Tanglish note |
|
||||
|---------------------|----------------|---------------|
|
||||
| `LatinOnly` | English NER only | Tanglish names in Roman script |
|
||||
| `TamilOnly` | Tamil NER only | Tamil script names |
|
||||
| `Mixed` | **Both**, merge person spans | Common in Indian CS prompts |
|
||||
| `NoLetters` | Neither (or English fallback off) | Digits-only prompts |
|
||||
|
||||
**Merge inside `RoutingOnnxNerModelRunner`:** dedupe overlapping person spans (prefer longer span; tie-break Tamil vs English by start index order).
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Selection
|
||||
|
||||
| Role | Model | Rationale |
|
||||
|------|-------|-----------|
|
||||
| English / Tanglish (Latin) | **Keep** `dslim/bert-base-NER` | Already integrated; works for many Indian names in Latin script |
|
||||
| Tamil script | **`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`** | Tamil NER; lighter than MuRIL (~0.6B) |
|
||||
| Fallback (optional Phase 5) | MuRIL Tamil NER | Only if IndicBERT recall is insufficient on eval set |
|
||||
|
||||
### Asset layout
|
||||
|
||||
```
|
||||
models/
|
||||
en/
|
||||
ner-model.onnx # or model.onnx (BERT export)
|
||||
vocab.txt
|
||||
ner-labels.txt
|
||||
ta/
|
||||
model.onnx
|
||||
sentencepiece.bpe.model # or tokenizer.json from HF export
|
||||
ner-labels.txt
|
||||
ner-model.onnx # legacy path — keep for backward compatibility
|
||||
```
|
||||
|
||||
### Label mapping (Tamil fine-grained NER)
|
||||
|
||||
SampurNER uses fine-grained tags (e.g. `B-person-politician`, `I-person-artist`). Map **any label containing `person`** (case-insensitive) → `PiiEntityType.Person`.
|
||||
|
||||
English labels remain: `B-PER`, `I-PER`, `B-PERSON`, `I-PERSON`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Phases
|
||||
|
||||
### Phase 1 — Generic ONNX token classifier (1–2 days)
|
||||
|
||||
**Objective:** Refactor `OnnxNerModelRunner` so BERT and SentencePiece are pluggable.
|
||||
|
||||
| Action | Location |
|
||||
|--------|----------|
|
||||
| Add `ITokenClassifierEncoder` + `EncodedSequence` | `Infrastructure/Onnx/` |
|
||||
| `BertWordPieceEncoder` — extract from current runner | Infrastructure |
|
||||
| `SentencePieceEncoder` — IndicBERT tokenizer | Infrastructure |
|
||||
| `OnnxTokenClassifierRunner` — shared inference + BIO decode | Infrastructure |
|
||||
| `NerLabelConfig` — English vs Tamil person label predicates | Infrastructure |
|
||||
| `OnnxAssetPathResolver` — resolve model dir from repo root | Infrastructure |
|
||||
| Thin wrappers: `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` | Infrastructure |
|
||||
|
||||
**Backward compat:** If `models/en/` missing, fall back to `OnnxModelPath` (`models/ner-model.onnx`).
|
||||
|
||||
**No behavior change** until Phase 3 wiring — existing tests must pass.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 — Tamil model download + config (0.5–1 day)
|
||||
|
||||
| Action | Details |
|
||||
|--------|---------|
|
||||
| `scripts/download-tamil-ner-model.ps1` + `.py` | Mirror `download-ner-model.ps1`; export via `optimum-cli export onnx --task token-classification` |
|
||||
| Optional: `scripts/download-all-ner-models.ps1` | Calls English + Tamil scripts |
|
||||
| Extend `PiiRedactionOptions` | `EnglishOnnxModelPath`, `TamilOnnxModelPath`, `EnableTamilNer` (default `true`) |
|
||||
| Update `appsettings.json` | New paths under `PiiRedaction` section |
|
||||
| `.gitignore` | `models/ta/*`, `models/en/*` (same as today for onnx/vocab) |
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Script routing + DI (0.5–1 day)
|
||||
|
||||
| Action | Location |
|
||||
|--------|----------|
|
||||
| `ScriptRouter` + `ScriptComposition` enum | `Core/Detection/` |
|
||||
| `RoutingOnnxNerModelRunner` implements `IOnnxNerModelRunner` | Infrastructure |
|
||||
| DI registration | `ServiceCollectionExtensions.cs` |
|
||||
|
||||
```csharp
|
||||
services.AddSingleton<EnglishOnnxNerRunner>();
|
||||
services.AddSingleton<TamilOnnxNerRunner>();
|
||||
services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();
|
||||
```
|
||||
|
||||
`OnnxNerPiiDetector` and `CompositePiiDetector` **unchanged**.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Tests, samples, docs (1 day)
|
||||
|
||||
#### Unit tests
|
||||
|
||||
| Test class | Coverage |
|
||||
|------------|----------|
|
||||
| `ScriptRouterTests` | Tamil-only, Latin-only, mixed, no-letters, boundary chars U+0B80/U+0BFF |
|
||||
| `RoutingOnnxNerModelRunnerTests` | Fake EN/TA runners; mixed script merges both |
|
||||
| `NerLabelConfigTests` | Tamil fine-grained person labels map correctly |
|
||||
|
||||
#### Real-model tests (`Category=RealModel` or `Category=TamilNer`)
|
||||
|
||||
| Scenario | Input example | Assert |
|
||||
|----------|---------------|--------|
|
||||
| Tamil name | `வாடிக்கையாளர் ராஜேஷ் தொலைபேசி 9876543210` | `<PERSON_1>`, phone redacted |
|
||||
| Tanglish name | `Customer Senthil phone 9876543210` | person + phone (best-effort) |
|
||||
| Mixed | `Rajesh மற்றும் Priya` | both persons redacted |
|
||||
| Clean Tamil | `பணத்தை திரும்பப் பெறுவது எப்படி?` | no false positives |
|
||||
| Canonical English | existing golden tests | no regression |
|
||||
|
||||
Skip gracefully when `models/ta/model.onnx` missing (mirror `RealNerModelFixture`).
|
||||
|
||||
#### Console samples
|
||||
|
||||
Add to `SamplePromptCatalog.cs`:
|
||||
|
||||
- `TamilCustomerName` — Tamil script person
|
||||
- `TanglishCustomerName` — `enga peru Rajesh` / `Customer Senthil`
|
||||
- `MixedTamilEnglish` — code-mixed prompt
|
||||
|
||||
#### Docs
|
||||
|
||||
- Update `README.md` ONNX setup (dual models)
|
||||
- Update `docs/architecture.md` NER section
|
||||
- Link this plan from README
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 — Optional enhancements (post-MVP)
|
||||
|
||||
| Enhancement | Benefit | Effort |
|
||||
|-------------|---------|--------|
|
||||
| **Unicode digit normalization** pre-pass | Tamil numerals → ASCII for regex | 0.5 day |
|
||||
| **Label-based regex** (`பெயர்`, `peru`, `peyar`, `enga peru`) | Tanglish recall without ML | 0.5 day |
|
||||
| **Tamil name gazetteer** `IPiiDetector` | High precision for top names | 1 day |
|
||||
| **Fail-closed policy** when NER unavailable | Compliance option | 0.5 day |
|
||||
| MuRIL model swap | Higher Tamil recall | eval-driven |
|
||||
|
||||
---
|
||||
|
||||
## 5. Tanglish — Realistic Expectations
|
||||
|
||||
| Input type | Primary handler | Expected recall |
|
||||
|------------|-----------------|-----------------|
|
||||
| Tamil script names | Tamil ONNX NER | High (with eval tuning) |
|
||||
| Standard Latin Indian names (`Ravi Kumar`) | English ONNX NER | High (already works) |
|
||||
| Tanglish spellings (`Senthil`, `senthil`, `Centhil`) | English NER + optional gazetteer | Medium |
|
||||
| Code-mixed (`Rajesh oda account ACC-123456`) | English NER + domain regex | Medium–high for IDs; name variable |
|
||||
|
||||
**MVP target:** Tamil script person names reliably redacted; Tanglish improved but not 100% without Phase 5 heuristics.
|
||||
|
||||
---
|
||||
|
||||
## 6. Success Metrics
|
||||
|
||||
Before marking language gap closed, run an **eval set of 20–30 real prompts** (anonymized production samples):
|
||||
|
||||
| Metric | MVP target |
|
||||
|--------|------------|
|
||||
| Tamil script person-name recall | ≥ 85% |
|
||||
| Tanglish person-name recall | ≥ 70% (with English model + optional heuristics) |
|
||||
| False positive rate (clean prompts) | ≤ 5% |
|
||||
| Structured PII (phone/PAN/domain) in Tamil prompts | ≥ 95% (regex layer) |
|
||||
| Regression on English canonical demo | 100% (existing golden tests) |
|
||||
|
||||
---
|
||||
|
||||
## 7. Files to Create / Modify (checklist)
|
||||
|
||||
### New files
|
||||
|
||||
- `src/PiiRedaction.Core/Detection/ScriptRouter.cs`
|
||||
- `src/PiiRedaction.Core/Detection/ScriptComposition.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/BertWordPieceEncoder.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/SentencePieceEncoder.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/ITokenClassifierEncoder.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/OnnxAssetPathResolver.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs`
|
||||
- `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs`
|
||||
- `scripts/download-tamil-ner-model.ps1` / `.py`
|
||||
- `tests/.../ScriptRouterTests.cs`
|
||||
- `tests/.../RoutingOnnxNerModelRunnerTests.cs`
|
||||
- `tests/.../RealTamilNerPipelineTests.cs`
|
||||
- `tests/TestSupport.Shared/RealTamilModelFixture.cs`
|
||||
|
||||
### Modified files
|
||||
|
||||
- `PiiRedactionOptions.cs` — dual model paths
|
||||
- `ServiceCollectionExtensions.cs` — routing DI
|
||||
- `appsettings.json` — config
|
||||
- `SamplePromptCatalog.cs` — Tamil/Tanglish demos
|
||||
- `ProductionPipelineFactory.cs` — `CreateWithRoutingRealModel()` for tests
|
||||
- `RealNerModelFixture.cs` / paths — support EN + TA
|
||||
- `README.md`, `docs/architecture.md`
|
||||
- `.gitignore` — `models/en/`, `models/ta/`
|
||||
|
||||
### Unchanged (by design)
|
||||
|
||||
- `PromptSanitizer`, `PlaceholderPiiRedactor`, `CompositePiiDetector`
|
||||
- `RegexPiiDetector`, `DomainRulePiiDetector`
|
||||
- `MockLlmPromptService` / LLM boundary
|
||||
|
||||
---
|
||||
|
||||
## 8. Rollout & Risk
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Tamil model export fails on Windows | PowerShell fallback downloads pre-exported ONNX from Hugging Face |
|
||||
| Larger memory (two models) | Lazy-load Tamil runner only when `EnableTamilNer` and Tamil script detected |
|
||||
| Fine-grained label mismatch | Load labels from `ner-labels.txt`; unit test label config |
|
||||
| Tanglish disappointment | Set stakeholder expectation in README; Phase 5 heuristics |
|
||||
| CI without models | Fast tests use fakes; `Category=TamilNer` skips like `RealModel` |
|
||||
|
||||
**Feature flag:** `EnableTamilNer=false` reverts to English-only behavior for gradual rollout.
|
||||
|
||||
---
|
||||
|
||||
## 9. Command Reference (after implementation)
|
||||
|
||||
```powershell
|
||||
# Download both models
|
||||
.\scripts\download-ner-model.ps1
|
||||
.\scripts\download-tamil-ner-model.ps1
|
||||
|
||||
# Run Tamil-focused console sample
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerName
|
||||
|
||||
# Tests
|
||||
dotnet test
|
||||
dotnet test --filter "Category=TamilNer"
|
||||
dotnet test --filter "Category=RealModel"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Approval Checklist
|
||||
|
||||
- [ ] Stakeholder sign-off on dual-model approach (vs single multilingual model)
|
||||
- [ ] Tamil eval prompt set collected (20–30 samples)
|
||||
- [ ] Xenovex CI policy: models downloaded in pipeline or tests skip
|
||||
- [x] Phase 1–3 implementation PR
|
||||
- [ ] Phase 4 eval metrics met
|
||||
- [ ] Optional Phase 5 for Tanglish heuristics if recall < 70%
|
||||
|
||||
---
|
||||
|
||||
**Next step:** Implement Phase 1 in a feature branch (`feature/tamil-tanglish-ner`), open PR to `main` on `xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc`.
|
||||
Reference in New Issue
Block a user