llm-pii-poc/docs/tamil-tanglish-ner-plan.md

Tamil / Tanglish NER — Implementation Plan

Goal: Raise language coverage from ~15% to production-viable for Tamil script and Tanglish (Roman-script Tamil-English) customer prompts, without changing the secure LLM boundary pattern.

Status: Phase 1–3 implemented
Approach: Dual-model ONNX NER routing (English + Tamil) + lightweight text normalization + optional Tanglish heuristics
Estimated effort: 4–6 engineering days across 4 phases

---

1. Current State vs Gap

Capability	Today	Tamil script	Tanglish (Latin)
Phone, PAN, Aadhaar, email, domain IDs	Regex + domain rules	Works (ASCII digits)	Works
Person names	dslim/bert-base-NER (English BERT)	Fails — out of vocabulary	Partial — inconsistent
Script / language routing	None	N/A	N/A
Tamil numerals (௦–௯)	Not normalized	May miss phone/Aadhaar	N/A
Label-aware cues (பெயர், peru, enga peru)	None	Misses contextual names	Misses

Root cause: Person detection is a single English-only ONNX model behind IOnnxNerModelRunner → OnnxNerPiiDetector. Regex/domain layers are already language-agnostic.

---

2. Target Architecture

No change to the trust boundary: PromptSanitizer → CompositePiiDetector → PlaceholderPiiRedactor → sanitized text only to LLM.

Only the NER adapter expands:

flowchart TD
  text["Prompt text"]
  router["ScriptRouter (Core)"]
  routing["RoutingOnnxNerModelRunner"]
  en["EnglishOnnxRunner\nBERT WordPiece"]
  ta["TamilOnnxRunner\nIndicBERT SentencePiece"]
  nerDet["OnnxNerPiiDetector"]
  composite["CompositePiiDetector"]

  text --> composite
  text --> nerDet
  nerDet --> routing
  routing --> router
  router -->|"LatinOnly / Mixed"| en
  router -->|"TamilOnly / Mixed"| ta
  en --> routing
  ta --> routing

Routing rules

ScriptComposition	Models invoked	Tanglish note
LatinOnly	English NER only	Tanglish names in Roman script
TamilOnly	Tamil NER only	Tamil script names
Mixed	Both, merge person spans	Common in Indian CS prompts
NoLetters	Neither (or English fallback off)	Digits-only prompts

Merge inside RoutingOnnxNerModelRunner: dedupe overlapping person spans (prefer longer span; tie-break Tamil vs English by start index order).

---

3. Model Selection

Role	Model	Rationale
English / Tanglish (Latin)	Keep dslim/bert-base-NER	Already integrated; works for many Indian names in Latin script
Tamil script	prachuryyaIITG/SampurNER_Tamil_IndicBERTv2	Tamil NER; lighter than MuRIL (~0.6B)
Fallback (optional Phase 5)	MuRIL Tamil NER	Only if IndicBERT recall is insufficient on eval set

Asset layout

models/
  en/
    ner-model.onnx      # or model.onnx (BERT export)
    vocab.txt
    ner-labels.txt
  ta/
    model.onnx
    sentencepiece.bpe.model   # or tokenizer.json from HF export
    ner-labels.txt
  ner-model.onnx        # legacy path — keep for backward compatibility

Label mapping (Tamil fine-grained NER)

SampurNER uses fine-grained tags (e.g. B-person-politician, I-person-artist). Map any label containing person (case-insensitive) → PiiEntityType.Person.

English labels remain: B-PER, I-PER, B-PERSON, I-PERSON.

---

4. Implementation Phases

Phase 1 — Generic ONNX token classifier (1–2 days)

Objective: Refactor OnnxNerModelRunner so BERT and SentencePiece are pluggable.

Action	Location
Add ITokenClassifierEncoder + EncodedSequence	Infrastructure/Onnx/
BertWordPieceEncoder — extract from current runner	Infrastructure
SentencePieceEncoder — IndicBERT tokenizer	Infrastructure
OnnxTokenClassifierRunner — shared inference + BIO decode	Infrastructure
NerLabelConfig — English vs Tamil person label predicates	Infrastructure
OnnxAssetPathResolver — resolve model dir from repo root	Infrastructure
Thin wrappers: EnglishOnnxNerRunner, TamilOnnxNerRunner	Infrastructure

Backward compat: If models/en/ missing, fall back to OnnxModelPath (models/ner-model.onnx).

No behavior change until Phase 3 wiring — existing tests must pass.

---

Phase 2 — Tamil model download + config (0.5–1 day)

Action	Details
scripts/download-tamil-ner-model.ps1 + .py	Mirror download-ner-model.ps1; export via optimum-cli export onnx --task token-classification
Optional: scripts/download-all-ner-models.ps1	Calls English + Tamil scripts
Extend PiiRedactionOptions	EnglishOnnxModelPath, TamilOnnxModelPath, EnableTamilNer (default true)
Update appsettings.json	New paths under PiiRedaction section
.gitignore	models/ta/*, models/en/* (same as today for onnx/vocab)

---

Phase 3 — Script routing + DI (0.5–1 day)

Action	Location
ScriptRouter + ScriptComposition enum	Core/Detection/
RoutingOnnxNerModelRunner implements IOnnxNerModelRunner	Infrastructure
DI registration	ServiceCollectionExtensions.cs

services.AddSingleton<EnglishOnnxNerRunner>();
services.AddSingleton<TamilOnnxNerRunner>();
services.AddSingleton<IOnnxNerModelRunner, RoutingOnnxNerModelRunner>();

OnnxNerPiiDetector and CompositePiiDetector unchanged.

---

Phase 4 — Tests, samples, docs (1 day)

Unit tests

Test class	Coverage
ScriptRouterTests	Tamil-only, Latin-only, mixed, no-letters, boundary chars U+0B80/U+0BFF
RoutingOnnxNerModelRunnerTests	Fake EN/TA runners; mixed script merges both
NerLabelConfigTests	Tamil fine-grained person labels map correctly

Real-model tests (Category=RealModel or Category=TamilNer)

Scenario	Input example	Assert
Tamil name	வாடிக்கையாளர் ராஜேஷ் தொலைபேசி 9876543210	<PERSON_1>, phone redacted
Tanglish name	Customer Senthil phone 9876543210	person + phone (best-effort)
Mixed	Rajesh மற்றும் Priya	both persons redacted
Clean Tamil	பணத்தை திரும்பப் பெறுவது எப்படி?	no false positives
Canonical English	existing golden tests	no regression

Skip gracefully when models/ta/model.onnx missing (mirror RealNerModelFixture).

Console samples

Add to SamplePromptCatalog.cs:

• TamilCustomerName — Tamil script person
• TanglishCustomerName — enga peru Rajesh / Customer Senthil
• MixedTamilEnglish — code-mixed prompt

Docs

• Update README.md ONNX setup (dual models)
• Update docs/architecture.md NER section
• Link this plan from README

---

Phase 5 — Optional enhancements (post-MVP)

Enhancement	Benefit	Effort
Unicode digit normalization pre-pass	Tamil numerals → ASCII for regex	0.5 day
Label-based regex (பெயர், peru, peyar, enga peru)	Tanglish recall without ML	0.5 day
Tamil name gazetteer IPiiDetector	High precision for top names	1 day
Fail-closed policy when NER unavailable	Compliance option	0.5 day
MuRIL model swap	Higher Tamil recall	eval-driven

---

5. Tanglish — Realistic Expectations

Input type	Primary handler	Expected recall
Tamil script names	Tamil ONNX NER	High (with eval tuning)
Standard Latin Indian names (Ravi Kumar)	English ONNX NER	High (already works)
Tanglish spellings (Senthil, senthil, Centhil)	English NER + optional gazetteer	Medium
Code-mixed (Rajesh oda account ACC-123456)	English NER + domain regex	Medium–high for IDs; name variable

MVP target: Tamil script person names reliably redacted; Tanglish improved but not 100% without Phase 5 heuristics.

---

6. Success Metrics

Before marking language gap closed, run an eval set of 20–30 real prompts (anonymized production samples):

Metric	MVP target
Tamil script person-name recall	≥ 85%
Tanglish person-name recall	≥ 70% (with English model + optional heuristics)
False positive rate (clean prompts)	≤ 5%
Structured PII (phone/PAN/domain) in Tamil prompts	≥ 95% (regex layer)
Regression on English canonical demo	100% (existing golden tests)

---

7. Files to Create / Modify (checklist)

New files

• src/PiiRedaction.Core/Detection/ScriptRouter.cs
• src/PiiRedaction.Core/Detection/ScriptComposition.cs
• src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs
• src/PiiRedaction.Infrastructure/Onnx/BertWordPieceEncoder.cs
• src/PiiRedaction.Infrastructure/Onnx/SentencePieceEncoder.cs
• src/PiiRedaction.Infrastructure/Onnx/ITokenClassifierEncoder.cs
• src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs
• src/PiiRedaction.Infrastructure/Onnx/OnnxAssetPathResolver.cs
• src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs
• src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs
• src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs
• scripts/download-tamil-ner-model.ps1 / .py
• tests/.../ScriptRouterTests.cs
• tests/.../RoutingOnnxNerModelRunnerTests.cs
• tests/.../RealTamilNerPipelineTests.cs
• tests/TestSupport.Shared/RealTamilModelFixture.cs

Modified files

• PiiRedactionOptions.cs — dual model paths
• ServiceCollectionExtensions.cs — routing DI
• appsettings.json — config
• SamplePromptCatalog.cs — Tamil/Tanglish demos
• ProductionPipelineFactory.cs — CreateWithRoutingRealModel() for tests
• RealNerModelFixture.cs / paths — support EN + TA
• README.md, docs/architecture.md
• .gitignore — models/en/, models/ta/

Unchanged (by design)

• PromptSanitizer, PlaceholderPiiRedactor, CompositePiiDetector
• RegexPiiDetector, DomainRulePiiDetector
• MockLlmPromptService / LLM boundary

---

8. Rollout & Risk

Risk	Mitigation
Tamil model export fails on Windows	PowerShell fallback downloads pre-exported ONNX from Hugging Face
Larger memory (two models)	Lazy-load Tamil runner only when EnableTamilNer and Tamil script detected
Fine-grained label mismatch	Load labels from ner-labels.txt; unit test label config
Tanglish disappointment	Set stakeholder expectation in README; Phase 5 heuristics
CI without models	Fast tests use fakes; Category=TamilNer skips like RealModel

Feature flag: EnableTamilNer=false reverts to English-only behavior for gradual rollout.

---

9. Command Reference (after implementation)

# Download both models
.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1

# Run Tamil-focused console sample
dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerName

# Tests
dotnet test
dotnet test --filter "Category=TamilNer"
dotnet test --filter "Category=RealModel"

---

10. Approval Checklist

• Stakeholder sign-off on dual-model approach (vs single multilingual model)
• Tamil eval prompt set collected (20–30 samples)
• Xenovex CI policy: models downloaded in pipeline or tests skip
• Phase 1–3 implementation PR
• Phase 4 eval metrics met
• Optional Phase 5 for Tanglish heuristics if recall < 70%

---

Next step: Implement Phase 1 in a feature branch (feature/tamil-tanglish-ner), open PR to main on xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc.