Bilal-Nazer-Ali/llm-pii-poc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Tamil NER routing and WPF test harness for POC validation.

Browse Source 
Introduce dual-script ONNX NER routing (English/Tamil/mixed), Tamil console samples and integration tests, model download scripts, and a resizable WPF MVVM harness with click-to-load prompts, batch validation, and runtime-adjustable detection panels.
This commit is contained in:
Bilal Nazer Ali
2026-07-07 17:12:38 +05:30
parent a707c6c9cf
commit cf8f5a7232
71 changed files with 4494 additions and 356 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
README.md
View File

@@ -15,6 +15,10 @@ Financial and customer-service prompts often contain regulated data (names, gove
For solution design, data-flow diagrams, trust boundaries, and project responsibilities, see **[docs/architecture.md](docs/architecture.md)**.
For English and Tamil ONNX NER model IDs, assets, routing, and reproduction steps, see **[docs/ner-models.md](docs/ner-models.md)**.
**Planned:** Tamil / Tanglish person-name support via dual ONNX NER routing — see **[docs/tamil-tanglish-ner-plan.md](docs/tamil-tanglish-ner-plan.md)**.
## Why Three Detection Strategies?
| Strategy | Used For | Rationale |
@@ -31,15 +35,17 @@ The placeholder map (`<PERSON_1>` → original value) is kept **in-process** for
```
src/
├── PiiRedaction.ConsoleApp/ # Presentation: input/output, DI bootstrap
├── PiiRedaction.Core/ # Business logic: detection, redaction, models
└── PiiRedaction.Infrastructure/ # Technical adapters: ONNX Runtime, mock LLM
models/ # Optional ONNX model files (gitignored)
├── PiiRedaction.ConsoleApp/ # Console demo: input/output, DI bootstrap
├── PiiRedaction.TestHarness.Wpf/ # WPF MVVM test harness for manual POC validation
├── PiiRedaction.Core/ # Business logic: detection, redaction, models
└── PiiRedaction.Infrastructure/ # Technical adapters: ONNX Runtime, mock LLM
models/ # Optional ONNX model files (gitignored)
```
| Project | Responsibility |
|---------|----------------|
| `PiiRedaction.ConsoleApp` | Read prompt, call sanitizer, display results, call LLM service |
| `PiiRedaction.TestHarness.Wpf` | Desktop test harness: preset prompts, redact UI, batch validation |
| `PiiRedaction.Core` | PII detection abstractions, redaction, sanitization orchestration |
| `PiiRedaction.Infrastructure` | ONNX model runner, `IChatClient` mock implementation |
@@ -63,7 +69,7 @@ dotnet build
dotnet run --project src/PiiRedaction.ConsoleApp
```
By default the console app runs **11 curated sample prompts** covering NER/person names, regex identifiers, domain IDs, combined scenarios, and a clean no-PII ticket.
By default the console app runs **16 curated sample prompts** covering English and Tamil/Tanglish/mixed person names, regex identifiers, domain IDs, combined scenarios, and a clean no-PII ticket. No flags are required for Tamil samples — they run in the default batch alongside English.
List available samples:
@@ -78,6 +84,27 @@ dotnet run --project src/PiiRedaction.ConsoleApp -- --sample 2
dotnet run --project src/PiiRedaction.ConsoleApp -- --name MrTitlePerson
```
### WPF Test Harness
A desktop **MVVM** application for interactive POC validation with English and Tamil prompts. Requires **Windows** (`net10.0-windows`).
**Prerequisites:** English and Tamil ONNX models downloaded (see [ONNX Model Setup](#onnx-model-setup)).
```bash
dotnet run --project src/PiiRedaction.TestHarness.Wpf
```
**Workflow:**
1. **Select a test prompt** from the left panel (grouped by language: English, Tamil, Mixed, Tanglish) or type your own prompt in the input box.
2. **Click a test prompt** in the left panel to load it into the input box (previous results are cleared automatically).
3. Click **Redact** to run the full detection pipeline. The status bar shows model availability, script composition (LatinOnly / TamilOnly / Mixed), and elapsed time.
4. Review **Sanitized Output**, detected entities, and the placeholder map in the right panel. A leak warning appears if any detected value remains in the sanitized text.
5. Optionally click **Send Mock LLM** to send only the sanitized prompt to the mock LLM.
6. Click **Run All** to execute all **22 curated scenarios** (16 console samples + 6 harness-only edge cases) and view pass/fail results in the batch panel.
The harness uses the same DI registrations and `IPromptSanitizer` pipeline as the console app, with thin application services (`IRedactionAppService`, `ITestPromptCatalog`, `IScriptAnalysisService`, `IModelStatusService`) following SOLID principles.
Interactive mode (enter your own prompt):
```bash
@@ -100,7 +127,12 @@ Samples are defined in [`SamplePromptCatalog.cs`](src/PiiRedaction.ConsoleApp/Sa
| 7 | PersonWithEmailNoPhone | NER + Regex | `Customer Arjun Mehta` + email |
| 8 | AllRegexTypes | Regex | email, phone, PAN, Aadhaar, card |
| 9 | AllDomainIds | Domain | LN, CID, ACC |
| 10 | NoPiiCleanTicket | Negative | no redaction |
| 10 | TamilCustomerNameOnly | NER (Tamil) | `வாடிக்கையாளர் ராஜேஷ் குமார்` |
| 11 | TamilWithPhonePan | NER (Tamil) + Regex | Tamil person + phone + PAN |
| 12 | TanglishCustomer | NER (English/Tanglish) | `Customer Senthil` + phone |
| 13 | MixedTamilEnglish | NER (Mixed) | `வாடிக்கையாளர் Ravi Kumar` + phone |
| 14 | TamilFullFinancial | NER (Tamil) + Regex + Domain | Tamil canonical demo |
| 15 | NoPiiCleanTicket | Negative | no redaction |
Person names are detected via **ONNX NER** using `dslim/bert-base-NER` (or a compatible token-classification export). A real model is **required** for person-name detection; there is no regex or heuristic fallback.
@@ -147,36 +179,41 @@ Person-name detection requires a token-classification ONNX model and companion t
| File | Purpose |
|------|---------|
| `models/ner-model.onnx` | Exported NER model |
| `models/vocab.txt` | BERT WordPiece vocabulary |
| `models/ner-labels.txt` | One BIO label per line (`O`, `B-PER`, `I-PER`, etc.) |
| `models/en/ner-model.onnx` | English BERT NER model (or legacy `models/ner-model.onnx`) |
| `models/en/vocab.txt` | BERT WordPiece vocabulary |
| `models/en/ner-labels.txt` | One BIO label per line (`O`, `B-PER`, `I-PER`, etc.) |
| `models/ta/model.onnx` | Tamil IndicBERT NER model |
| `models/ta/sentencepiece.bpe.model` | SentencePiece tokenizer for Tamil model |
| `models/ta/ner-labels.txt` | Fine-grained Tamil NER labels |
### Download script
### Download scripts
From the repository root:
```powershell
.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1
```
Or with Python directly:
```bash
python scripts/download-ner-model.py
python scripts/download-tamil-ner-model.py
```
The script exports [`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER) via Hugging Face Optimum when Python is available. Otherwise it downloads the pre-exported ONNX assets from Hugging Face directly.
The English script exports [`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER) via Hugging Face Optimum when Python is available. The Tamil script exports [`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`](https://huggingface.co/prachuryyaIITG/SampurNER_Tamil_IndicBERTv2). Otherwise each script downloads pre-exported ONNX assets from Hugging Face directly.
Set `EnableTamilNer` to `false` in `appsettings.json` to revert to English-only routing.
### Inference pipeline
`OnnxNerModelRunner` performs the full pipeline:
`RoutingOnnxNerModelRunner` classifies script composition and delegates to:
- BERT WordPiece tokenization (`Microsoft.ML.Tokenizers`)
- ONNX Runtime inference (`input_ids`, `attention_mask`, optional `token_type_ids`)
- BIO label decoding (`B-PER` / `I-PER` → `PiiEntityType.Person`)
- Character-span alignment back to the source text
- **`EnglishOnnxNerRunner`** — BERT WordPiece tokenization for Latin script and Tanglish
- **`TamilOnnxNerRunner`** — SentencePiece tokenization for Tamil script (U+0B80–U+0BFF)
When the model or tokenizer files are missing, person detection returns no results.
Both runners share `OnnxTokenClassifierRunner` for ONNX Runtime inference and BIO label decoding. Overlapping person spans from mixed-script prompts are merged (longer span wins).
## Swapping Mock LLM for Azure OpenAI
@@ -251,24 +288,29 @@ The solution includes an **NUnit** test suite across two projects:
dotnet test
dotnet test --filter "FullyQualifiedName~GoldenPromptTests"
dotnet test --filter "Category=RealModel"
dotnet test --filter "Category=TamilNer"
dotnet test --logger "console;verbosity=detailed"
```
Fast CI runs without the ONNX model: fake-based tests always execute; tests marked **`Category=RealModel`** are skipped when `models/ner-model.onnx` is absent. Download the model first:
Fast CI runs without the ONNX model: fake-based tests always execute; tests marked **`Category=RealModel`** or **`Category=TamilNer`** are skipped when the corresponding ONNX models are absent. Download models first:
```powershell
.\scripts\download-ner-model.ps1
.\scripts\download-tamil-ner-model.ps1
```
### Test architecture
- **`PromptScenarioCatalog`** — five focused end-to-end scenarios (canonical demo, multi-regex, duplicate people, overlap stress, no-PII negative)
- **`ProductionPipelineFactory`** — builds the same Domain → Regex → OnnxNer composite stack as production DI; `CreateWithRealModel(runner)` wires a real `OnnxNerModelRunner`
- **`PromptScenarioCatalog`** — five focused end-to-end English scenarios (canonical demo, multi-regex, duplicate people, overlap stress, no-PII negative)
- **`TamilPromptScenarioCatalog`** — five Tamil/Tanglish/mixed golden scenarios (fake NER for person spans)
- **`ProductionPipelineFactory`** — builds the same Domain → Regex → OnnxNer composite stack as production DI; `CreateWithRealModel(runner)` wires a real runner; `CreateWithRoutingRealModels` wires English + Tamil routing
- **`FakeOnnxNerModelRunner`** — unit-test double for NER; golden tests inject person spans per scenario
- **`GoldenPromptTests`** — end-to-end sanitization proof across the catalog (fake NER)
- **`RealNerModelFixture`** — shared fixture that loads `models/ner-model.onnx` once per class; skips when model missing
- **`RealNerModelRunnerTests`** — direct ONNX inference with span accuracy checks
- **`RealNerPipelineTests`** — full pipeline with real NER (canonical, multi-person, clean-ticket negative)
- **`RealNerPipelineTests`** — full pipeline with real English NER (canonical, multi-person, clean-ticket negative)
- **`RealTamilPipelineTests`** — full pipeline with routed English + Tamil NER (`Category=TamilNer`)
- **`RealTamilNerModelRunnerTests`** — direct Tamil ONNX inference (`Category=TamilNer`)
- **`OnnxNerModelRunnerTests`** — unit tests for missing/invalid model paths (no download required)
- **`CompositePiiDetectorTests`** — overlap merge and source-priority rules
- **`LlmBoundaryTests`** — verifies raw PII never appears in outbound LLM messages