diff --git a/README.md b/README.md index 481fbd1..5e26935 100644 --- a/README.md +++ b/README.md @@ -11,13 +11,9 @@ Financial and customer-service prompts often contain regulated data (names, gove 3. Replace values with stable placeholders 4. Send only the **sanitized** prompt to an LLM (mocked for now) -## Architecture +## Documentation -For solution design, data-flow diagrams, trust boundaries, and project responsibilities, see **[docs/architecture.md](docs/architecture.md)**. - -For English and Tamil ONNX NER model IDs, assets, routing, and reproduction steps, see **[docs/ner-models.md](docs/ner-models.md)**. - -**Planned:** Tamil / Tanglish person-name support via dual ONNX NER routing — see **[docs/tamil-tanglish-ner-plan.md](docs/tamil-tanglish-ner-plan.md)**. +Full solution reference (architecture, NER models, routing, Tamil/Tanglish, Git setup, improvement roadmap): **[docs/solution-guide.md](docs/solution-guide.md)** ## Why Three Detection Strategies? @@ -59,7 +55,7 @@ models/ # Optional ONNX model files (gitignored) ## Build and Run -For pushing this repository to Xenovex Git (`xts.xenovex.com`), see **[docs/git-xenovex-setup.md](docs/git-xenovex-setup.md)**. +For pushing this repository to Xenovex Git (`xts.xenovex.com`), see **[docs/solution-guide.md § Git remote setup](docs/solution-guide.md#11-git-remote-setup-xenovex)**. From the repository root: @@ -96,12 +92,12 @@ dotnet run --project src/PiiRedaction.TestHarness.Wpf **Workflow:** -1. **Select a test prompt** from the left panel (grouped by language: English, Tamil, Mixed, Tanglish) or type your own prompt in the input box. +1. **Select a category** from the dropdown (e.g. **Career Guidance**, **Banking & Financial**) or leave **All** to see every prompt. Use the search box for finer filtering. 2. **Click a test prompt** in the left panel to load it into the input box (previous results are cleared automatically). 3. Click **Redact** to run the full detection pipeline. The status bar shows model availability, script composition (LatinOnly / TamilOnly / Mixed), and elapsed time. 4. Review **Sanitized Output**, detected entities, and the placeholder map in the right panel. A leak warning appears if any detected value remains in the sanitized text. 5. Optionally click **Send Mock LLM** to send only the sanitized prompt to the mock LLM. -6. Click **Run All** to execute all **22 curated scenarios** (16 console samples + 6 harness-only edge cases) and view pass/fail results in the batch panel. +6. Click **Run All** to execute scenarios in the **selected category** (or all when **All** is chosen) and view pass/fail results in the batch panel. The harness uses the same DI registrations and `IPromptSanitizer` pipeline as the console app, with thin application services (`IRedactionAppService`, `ITestPromptCatalog`, `IScriptAnalysisService`, `IModelStatusService`) following SOLID principles. diff --git a/docs/architecture.md b/docs/architecture.md deleted file mode 100644 index b33ea2f..0000000 --- a/docs/architecture.md +++ /dev/null @@ -1,445 +0,0 @@ -# PII Redaction POC — Solution Architecture - -## Purpose - -This document describes the architectural design of the **PII Redaction POC**, a .NET proof-of-concept that intercepts user prompts containing regulated personally identifiable information (PII), redacts sensitive values into stable placeholders, and transmits **only sanitized text** across the LLM trust boundary. The solution is structured for enterprise adoption: clear layer separation, interface-driven composition, dependency injection, and swappable infrastructure adapters (ONNX NER, `Microsoft.Extensions.AI` chat clients). - -The POC validates a compliance-oriented pattern suitable for financial and customer-service workloads where raw PII must not leave the application process when invoking external language models. - ---- - -## Canonical Example - -The console application ships with a **sample catalog** (16 prompts). The canonical demo is sample `FullFinancialWithCustomer`. Tamil script, Tanglish, and mixed-script samples run in the **default** `dotnet run` batch (no `--interactive` required). The table below shows the exact strings produced by the production pipeline when the ONNX NER models are loaded (run `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1` first). - -| Stage | Value | -|-------|-------| -| **Input** | `Customer Ravi Kumar with email ravi.kumar@gmail.com and phone 9876543210 has LoanNumber LN-456789 and PAN ABCDE1234F. Please summarize this customer issue.` | -| **Sanitized Output** | `Customer with email and phone has LoanNumber and PAN . Please summarize this customer issue.` | -| **Mock LLM Response** | `[Mock LLM Response] Received sanitized prompt (146 chars). No original PII was transmitted.` | - -Detected entities for this prompt: - -| Type | Value | Detection Source | -|------|-------|------------------| -| PERSON | Ravi Kumar | Ner | -| EMAIL | ravi.kumar@gmail.com | Regex | -| PHONE | 9876543210 | Regex | -| LOAN_NUMBER | LN-456789 | Domain | -| PAN | ABCDE1234F | Regex | - -The internal placeholder map (`` → `Ravi Kumar`, etc.) is retained in-process and is **not** included in the outbound LLM request. - ---- - -## Console Sample Catalog - -Running `dotnet run --project src/PiiRedaction.ConsoleApp` executes all samples sequentially. Use `--list`, `--sample N`, or `--name SampleName` to filter. - -### NER / person-name samples - -These prompts exercise `OnnxNerPiiDetector` and `RoutingOnnxNerModelRunner`. Person names require ONNX models (`models/en/` for English, `models/ta/` for Tamil script). Without models, person spans are not detected. Legacy `models/ner-model.onnx` is still supported for English. - -| Sample | Input (excerpt) | Detected person | Sanitized (excerpt) | -|--------|-----------------|-----------------|---------------------| -| **CustomerNameOnly** | Customer Anita Sharma reported unauthorized… | Anita Sharma | Customer `` reported unauthorized… | -| **MrTitlePerson** | Mr. John Smith called about a duplicate debit… | John Smith | `` called about a duplicate debit… | -| **MrsTitlePerson** | Mrs. Lakshmi Reddy requested a callback regarding LN-112233. | Lakshmi Reddy | `` requested a callback regarding ``. | -| **DrTitlePerson** | Dr. Jane Doe escalated a complaint… | Jane Doe | `` escalated a complaint… | -| **TwoCustomersInOnePrompt** | Customer Ravi Kumar and Customer Priya Nair… | Ravi Kumar, Priya Nair | Customer `` and Customer ``… | -| **PersonWithDomainIds** | Customer Meera Iyer holds CID-7070… | Meera Iyer | Customer `` holds ``… | -| **PersonWithEmailNoPhone** | Customer Arjun Mehta wrote from arjun.mehta@company.in… | Arjun Mehta | Customer `` wrote from ``… | - -### Tamil / Tanglish / mixed samples - -These prompts exercise `RoutingOnnxNerModelRunner` script routing. Tamil script uses `models/ta/`; Latin Tanglish uses `models/en/`. Mixed prompts may invoke both models. - -| Sample | Input (excerpt) | Detected person | Sanitized (excerpt) | -|--------|-----------------|-----------------|---------------------| -| **TamilCustomerNameOnly** | வாடிக்கையாளர் ராஜேஷ் குமார் சேமிப்பு… | ராஜேஷ் குமார் | வாடிக்கையாளர் `` சேமிப்பு… | -| **TamilWithPhonePan** | வாடிக்கையாளர் ராஜேஷ் குமார் தொலைபேசி 9876543210 PAN… | ராஜேஷ் குமார் | `` … `` … `` | -| **TanglishCustomer** | Customer Senthil phone 9876543210… | Senthil | Customer `` phone ``… | -| **MixedTamilEnglish** | வாடிக்கையாளர் Ravi Kumar phone 9876543210… | Ravi Kumar | வாடிக்கையாளர் `` phone ``… | -| **TamilFullFinancial** | வாடிக்கையாளர் ராஜேஷ் குமார் மின்னஞ்சல் ravi.kumar@gmail.com… | ராஜேஷ் குமார் | Tamil canonical — all placeholder types | - -### Other sample categories - -| Category | Sample | Purpose | -|----------|--------|---------| -| NER + Regex + Domain | FullFinancialWithCustomer | End-to-end financial prompt (canonical) | -| Regex only | AllRegexTypes | Email, phone, PAN, Aadhaar, credit card | -| Domain only | AllDomainIds | Loan number, customer ID, account number | -| Negative | NoPiiCleanTicket | Passthrough with no detected PII | - -Sample definitions live in [`SamplePromptCatalog.cs`](../src/PiiRedaction.ConsoleApp/Samples/SamplePromptCatalog.cs). - ---- - -## High-Level Data Flow - -The diagram below traces the canonical example from console input through Core sanitization to the Infrastructure LLM adapter. Data labels reflect the canonical strings at each stage. - -```mermaid -flowchart TB - subgraph consoleApp [PiiRedaction.ConsoleApp] - program["Program.cs"] - di["ServiceRegistration"] - end - - subgraph core [PiiRedaction.Core] - sanitizer["PromptSanitizer"] - composite["CompositePiiDetector"] - regexDet["RegexPiiDetector"] - domainDet["DomainRulePiiDetector"] - onnxDet["OnnxNerPiiDetector"] - redactor["PlaceholderPiiRedactor"] - end - - subgraph infra [PiiRedaction.Infrastructure] - onnxRunner["RoutingOnnxNerModelRunner"] - enRunner["EnglishOnnxNerRunner"] - taRunner["TamilOnnxNerRunner"] - mockLlm["MockLlmPromptService"] - mockChat["MockChatClient"] - end - - rawPrompt["Raw prompt with PII"] - sanitizedPrompt["Sanitized prompt with placeholders"] - llmResponse["Mock LLM acknowledgment"] - - program -->|"Customer Ravi Kumar ... PAN ABCDE1234F"| sanitizer - sanitizer --> composite - composite --> domainDet - composite --> regexDet - composite --> onnxDet - onnxDet --> onnxRunner - sanitizer --> redactor - redactor -->|"Customer PERSON_1 ... PAN PAN_1"| sanitizedPrompt - program -->|"SanitizedPrompt only"| mockLlm - mockLlm --> mockChat - mockChat --> llmResponse - - rawPrompt -.-> program - di -.-> sanitizer - di -.-> mockLlm -``` - -`RoutingOnnxNerModelRunner` selects English and/or Tamil ONNX models based on script composition in the prompt. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english) for the routing decision tree. - ---- - -## Detection to Redaction Detail - -`PromptSanitizer` orchestrates a two-phase pipeline: **detect** then **redact**. `CompositePiiDetector` aggregates spans from all registered detectors, resolves overlaps by registration order and source priority, and returns a merged entity list. `PlaceholderPiiRedactor` replaces spans right-to-left to preserve indices, assigns stable per-type counters, and builds the in-process placeholder map. - -```mermaid -flowchart LR - inputText["Original prompt text"] - - subgraph detectPhase [Detection Phase] - domainDet["DomainRulePiiDetector"] - regexDet["RegexPiiDetector"] - onnxDet["OnnxNerPiiDetector
(RoutingOnnxNerModelRunner)"] - composite["CompositePiiDetector"] - merge["Overlap merge and source priority"] - entityList["PiiEntity list"] - end - - subgraph redactPhase [Redaction Phase] - redactor["PlaceholderPiiRedactor"] - replace["Right-to-left span replacement"] - placeholderMap["Placeholder map in-process"] - sanitizedText["Sanitized text"] - end - - inputText --> domainDet - inputText --> regexDet - inputText --> onnxDet - domainDet --> composite - regexDet --> composite - onnxDet --> composite - composite --> merge - merge --> entityList - entityList --> redactor - inputText --> redactor - redactor --> replace - replace --> sanitizedText - replace --> placeholderMap -``` - -**Overlap resolution rules** (applied by `CompositePiiDetector`): - -1. Detectors run in registration order: **Domain → Regex → ONNX NER**. -2. On overlapping spans, the first registered detector wins. -3. Tie-breaking uses source priority: Domain (3) > Regex (2) > NER (1). - -The ONNX NER detector delegates to `RoutingOnnxNerModelRunner`, which routes inference to English and/or Tamil models by script composition. See [Dual-Model NER Routing (Tamil + English)](#dual-model-ner-routing-tamil--english). - -**Placeholder assignment** (applied by `PlaceholderPiiRedactor`): - - -- Format: `<{TYPE}_{n}>` (e.g. ``, ``). -- Duplicate values of the same type reuse the same placeholder. -- Replacement proceeds from highest `StartIndex` to lowest to avoid index drift. - ---- - -## Dual-Model NER Routing (Tamil + English) - -Person-name detection uses two ONNX token-classifier models: **English** (`models/en/`, BERT WordPiece) and **Tamil** (`models/ta/`, SentencePiece or WordPiece). `OnnxNerPiiDetector` calls `RoutingOnnxNerModelRunner`, which classifies prompt script via `ScriptRouter` and dispatches to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`. Both runners share `OnnxTokenClassifierRunner` for BIO decoding; only **PERSON** spans are emitted. - -The diagram below expands the detection and NER branches summarized in [High-Level Data Flow](#high-level-data-flow) and [Detection to Redaction Detail](#detection-to-redaction-detail). - -### End-to-end pipeline (with NER branch) - -```mermaid -flowchart TB - subgraph Entry["Console entry"] - A["Program.cs
Host + AddPiiRedactionServices()"] - B["PromptDemoRunner.RunAsync()"] - A --> B - end - - B --> C["SanitizationRequest(OriginalPrompt)"] - C --> D["PromptSanitizer.Sanitize()"] - - subgraph Detect["CompositePiiDetector.Detect() — registration order"] - direction TB - E1["DomainRulePiiDetector
LOAN_NUMBER, CUSTOMER_ID, ACCOUNT_NUMBER"] - E2["RegexPiiDetector
EMAIL, PHONE, AADHAAR, PAN, CREDIT_CARD"] - E3["OnnxNerPiiDetector
PERSON (via IOnnxNerModelRunner)"] - E1 --> MERGE - E2 --> MERGE - E3 --> MERGE - MERGE["Merge overlapping spans
sort: StartIndex ↑, Length ↓, Source priority ↓
(Domain=3, Regex=2, Ner=1)
first candidate wins on overlap"] - end - - D --> Detect - MERGE --> F["IReadOnlyList<PiiEntity>"] - - F --> G["PlaceholderPiiRedactor.Redact()
replace spans right-to-left
dedupe by Type|Value → <TYPE_n>"] - G --> H["SanitizationResult
SanitizedPrompt, DetectedEntities, PlaceholderMap"] - - H --> I["MockLlmPromptService.SendPromptAsync(SanitizedPrompt)"] - I --> J["Mock LLM response
(sanitized text only)"] - - subgraph NerBranch["OnnxNerPiiDetector branch"] - E3 --> N1{"RoutingOnnxNerModelRunner
.IsModelAvailable?"} - N1 -->|no| N2["return []"] - N1 -->|yes| N3["RoutingOnnxNerModelRunner
.PredictEntities()"] - end -``` - -### RoutingOnnxNerModelRunner decision tree - -`ScriptRouter.GetComposition` scans each character once. Tamil letters (U+0B80–U+0BFF) and ASCII Latin letters (`char.IsAsciiLetter`) determine the route. When both scripts appear, classification is **Mixed** (early exit). - -```mermaid -flowchart TB - IN["text"] --> SR["ScriptRouter.GetComposition(text)
scan each char"] - - SR --> C1{"LatinOnly?"} - SR --> C2{"TamilOnly?"} - SR --> C3{"Mixed?"} - SR --> C4{"NoLetters?"} - - C1 -->|yes| EN1{"EnglishOnnxNerRunner
.IsModelAvailable?"} - EN1 -->|yes| EN_RUN["EnglishOnnxNerRunner.PredictEntities(text)"] - EN1 -->|no| SKIP1["skip English"] - EN_RUN --> ACC - SKIP1 --> ACC - - C2 -->|yes| TA_GATE{"EnableTamilNer
&& TamilOnnxNerRunner
.IsModelAvailable?"} - TA_GATE -->|yes| TA_RUN["TamilOnnxNerRunner.PredictEntities(text)"] - TA_GATE -->|no| SKIP2["skip Tamil"] - TA_RUN --> ACC - SKIP2 --> ACC - - C3 -->|yes| EN2{"English available?"} - EN2 -->|yes| EN_MIX["EnglishOnnxNerRunner.PredictEntities(text)"] - EN2 -->|no| SKIP3["skip English"] - EN_MIX --> TA_GATE2{"EnableTamilNer
&& Tamil available?"} - SKIP3 --> TA_GATE2 - TA_GATE2 -->|yes| TA_MIX["TamilOnnxNerRunner.PredictEntities(text)"] - TA_GATE2 -->|no| SKIP4["skip Tamil"] - TA_MIX --> ACC - SKIP4 --> ACC - - C4 -->|yes| EMPTY["no NER inference"] - EMPTY --> OUT_EMPTY["return []"] - - subgraph EN_Pipeline["EnglishOnnxNerRunner"] - EN_RUN --> EN_ENC["BertWordPieceEncoder
(model dir vocab.txt)"] - EN_ENC --> EN_OCR["OnnxTokenClassifierRunner
NerLabelConfig.English
B-PER / I-PER / B-PERSON / I-PERSON"] - end - - subgraph TA_Pipeline["TamilOnnxNerRunner"] - TA_RUN --> TA_ENC["TokenClassifierEncoderFactory.Create()
vocab.txt → BertWordPieceEncoder
else SentencePiece (*.bpe.model, spiece.model, tokenizer.model)"] - TA_ENC --> TA_OCR["OnnxTokenClassifierRunner
NerLabelConfig.Tamil
label contains 'person' (case-insensitive)"] - end - - subgraph SharedInference["OnnxTokenClassifierRunner (shared)"] - ENC["Encode(text, max 128 tokens)"] - ONNX["ONNX InferenceSession.Run
input_ids + attention_mask [+ token_type_ids]"] - ARGMAX["Per-token argmax over logits"] - BIO["BIO decode → PiiEntityType.Person
PiiDetectionSource.Ner"] - ENC --> ONNX --> ARGMAX --> BIO - end - - EN_OCR --> SharedInference - TA_OCR --> SharedInference - BIO --> ACC["accumulate entities"] - - ACC --> MERGE["MergePersonSpans()
sort: Length ↓, StartIndex ↑
drop overlapping spans
(longer span wins)"] - MERGE --> OUT["return merged PERSON entities"] -``` - -### Routing rules - -| Rule | Source | Behavior | -|------|--------|----------| -| **Script classification** | `ScriptRouter.GetComposition` | Single pass over characters. Tamil letter = U+0B80–U+0BFF. Latin letter = `char.IsAsciiLetter`. Both seen → `Mixed` (early exit). Neither → `NoLetters`. Tamil only → `TamilOnly`. Latin only → `LatinOnly`. | -| **LatinOnly** | `RoutingOnnxNerModelRunner` | Run **English only** if `englishRunner.IsModelAvailable`. | -| **TamilOnly** | `RoutingOnnxNerModelRunner` | Run **Tamil only** if `EnableTamilNer` (default `true` in `PiiRedactionOptions`) **and** `tamilRunner.IsModelAvailable`. | -| **Mixed** | `RoutingOnnxNerModelRunner` | Run **both** models independently on the **full text** (English if available; Tamil if `EnableTamilNer` and available). | -| **NoLetters** | `RoutingOnnxNerModelRunner` | No NER inference; returns `[]` from routing (before merge). | -| **Model availability gate** | `OnnxNerPiiDetector` | If `RoutingOnnxNerModelRunner.IsModelAvailable` is false, NER detector returns `[]` (English OR Tamil available when Tamil enabled). | -| **Post-route merge** | `MergePersonSpans` | After EN/TA results are concatenated, overlapping PERSON spans are deduped; **longer span wins**, then ordered by `StartIndex`. | -| **Composite merge** | `CompositePiiDetector` | Domain → Regex → NER all run. Overlaps resolved globally: earlier registration order + longer span + higher source priority (Domain > Regex > Ner). | -| **Encoder choice** | `EnglishOnnxNerRunner` vs `TamilOnnxNerRunner` | English always uses `BertWordPieceEncoder`. Tamil uses factory: `vocab.txt` → WordPiece; else first SentencePiece file found; fallback WordPiece with warning. | -| **NER output scope** | `OnnxTokenClassifierRunner` | Only **PERSON** entities decoded from BIO tags; max sequence length **128** tokens. | - ---- - -## Runtime Sequence - -```mermaid -sequenceDiagram - participant User - participant Program as Program.cs - participant DI as ServiceProvider - participant Sanitizer as PromptSanitizer - participant Detector as CompositePiiDetector - participant Redactor as PlaceholderPiiRedactor - participant LlmSvc as MockLlmPromptService - participant Chat as MockChatClient - - User->>Program: Start application - Program->>DI: Resolve IPromptSanitizer, ILlmPromptService - DI-->>Program: Sanitizer, LlmService - - alt Interactive mode - User->>Program: Enter prompt via console - else Default mode - Program->>Program: Load canonical sample prompt - end - - Program->>Sanitizer: Sanitize(SanitizationRequest) - Sanitizer->>Detector: Detect(originalPrompt) - Detector-->>Sanitizer: IReadOnlyList PiiEntity - Sanitizer->>Redactor: Redact(originalPrompt, entities) - Redactor-->>Sanitizer: RedactionResult - Sanitizer-->>Program: SanitizationResult - - Program->>Program: Display detected entities - Program->>Program: Display sanitized prompt - Program->>Program: Display placeholder map in-process - - Program->>LlmSvc: SendPromptAsync(sanitizedPrompt) - Note over Program,LlmSvc: Placeholder map never passed - LlmSvc->>Chat: GetResponseAsync(user message) - Chat-->>LlmSvc: Assistant response - LlmSvc-->>Program: Mock LLM response string - Program->>User: Write LLM response to console -``` - ---- - -## Trust Boundary - -The LLM boundary is the point at which data leaves the application process via `ILlmPromptService` / `IChatClient`. Only the sanitized prompt crosses this boundary. Original PII values, detection metadata, and the placeholder-to-value map remain in-process. - -```mermaid -flowchart TB - subgraph inProcess [In-Process Trust Zone] - originalPrompt["Original prompt with raw PII"] - detectedEntities["Detected PiiEntity list"] - placeholderMap["Placeholder map"] - sanitizationResult["SanitizationResult"] - consoleDisplay["Console audit output"] - end - - subgraph llmBoundary [LLM Trust Boundary] - sanitizedOnly["Sanitized prompt text only"] - end - - subgraph externalLlm [External LLM Provider] - chatClient["IChatClient implementation"] - modelInference["Model inference"] - end - - originalPrompt --> sanitizationResult - detectedEntities --> sanitizationResult - placeholderMap --> sanitizationResult - sanitizationResult --> consoleDisplay - sanitizationResult -->|"SendPromptAsync"| sanitizedOnly - sanitizedOnly --> chatClient - chatClient --> modelInference - - originalPrompt -.-x|"Never transmitted"| chatClient - placeholderMap -.-x|"Never transmitted"| chatClient - detectedEntities -.-x|"Never transmitted"| chatClient -``` - -In the POC, `MockChatClient` simulates the external provider without network I/O. Replacing it with Azure OpenAI or another `IChatClient` implementation does not change the trust model: `MockLlmPromptService` (or a future production adapter) continues to accept only the sanitized string. - ---- - -## Project Responsibilities - -| Project | Layer | Responsibility | -|---------|-------|----------------| -| `PiiRedaction.ConsoleApp` | Presentation | Application entry point; reads prompt (sample or interactive); bootstraps `IHost` and DI via `AddPiiRedactionServices`; orchestrates sanitization and LLM invocation; renders audit output (detected entities, sanitized text, placeholder map). | -| `PiiRedaction.Core` | Domain / Application | Defines abstractions (`IPiiDetector`, `IPiiRedactor`, `IPromptSanitizer`, `ILlmPromptService`); implements detection strategies (`RegexPiiDetector`, `DomainRulePiiDetector`, `OnnxNerPiiDetector`, `CompositePiiDetector`); implements `PlaceholderPiiRedactor` and `PromptSanitizer`; owns domain models (`PiiEntity`, `SanitizationResult`, `RedactionResult`) and configuration (`PiiRedactionOptions`). Has no dependency on ONNX Runtime or LLM SDKs. | -| `PiiRedaction.Infrastructure` | Infrastructure | Implements technical adapters: `RoutingOnnxNerModelRunner`, `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` (ONNX Runtime inference), `MockChatClient` and `MockLlmPromptService` (`Microsoft.Extensions.AI`); depends on Core abstractions and is swappable without changing domain logic. | -| `tests/PiiRedaction.Core.Tests` | Test | Unit and integration tests for detectors, redactor, sanitizer, overlap rules, golden prompt scenarios (`PromptScenarioCatalog`), and LLM boundary assertions. | -| `tests/PiiRedaction.Infrastructure.Tests` | Test | Tests for mock LLM behavior and ONNX runner load semantics. | - -**Dependency direction:** `ConsoleApp` → `Infrastructure` → `Core`. Core references no outer layers, preserving the Dependency Inversion Principle and enabling future hosts (ASP.NET Core API, worker services) to reuse the same Core and Infrastructure assemblies. - ---- - -## Key Abstractions and Extension Points - -| Abstraction | Defined In | Default Implementation | Extension | -|-------------|------------|------------------------|-----------| -| `IPiiDetector` | Core | `CompositePiiDetector` wrapping Domain, Regex, ONNX | Add new detector; register in composite order | -| `IPiiRedactor` | Core | `PlaceholderPiiRedactor` | Replace with hashing, vault-backed tokens, etc. | -| `IPromptSanitizer` | Core | `PromptSanitizer` | Unlikely to change; orchestrates detect + redact | -| `ILlmPromptService` | Core | `MockLlmPromptService` | Production adapter with telemetry, retry, policy | -| `IChatClient` | Microsoft.Extensions.AI | `MockChatClient` | Azure OpenAI, OpenAI, or other provider SDK | -| `IOnnxNerModelRunner` | Core | `RoutingOnnxNerModelRunner` | Script-based routing to English (BERT WordPiece) and Tamil (SentencePiece) ONNX models | - ---- - -## Configuration Surface - -Runtime behavior is controlled via `appsettings.json` under the `PiiRedaction` section: - -| Setting | Effect | -|---------|--------| -| `OnnxModelPath` | Legacy English model path (`models/ner-model.onnx`). Used as fallback when `models/en/` is absent. | -| `EnglishOnnxModelPath` | Primary English ONNX model (`models/en/ner-model.onnx`). | -| `TamilOnnxModelPath` | Tamil ONNX model (`models/ta/model.onnx`). | -| `EnableTamilNer` | When `false`, routing uses English model only. Default `true`. | - -Download model assets with `scripts/download-ner-model.ps1` and `scripts/download-tamil-ner-model.ps1`. - ---- - -## Related Documentation - -- [NER models](ner-models.md) — English/Tamil ONNX model IDs, assets, labels, and integration reference -- [README](../README.md) — build, run, configuration, and testing instructions -- [ServiceCollectionExtensions.cs](../src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs) — DI registration and detector ordering -- [PromptScenarioCatalog.cs](../tests/PiiRedaction.Core.Tests/TestSupport/PromptScenarioCatalog.cs) — focused golden pipeline scenarios including the canonical example diff --git a/docs/git-xenovex-setup.md b/docs/git-xenovex-setup.md deleted file mode 100644 index 19e1172..0000000 --- a/docs/git-xenovex-setup.md +++ /dev/null @@ -1,62 +0,0 @@ -# Git setup — Xenovex (xts.xenovex.com) - -This repository is ready for push to your Xenovex Git server after you create a remote repository. - -## Prerequisites - -- Git 2.x (installed at `C:\Program Files\Git\bin\git.exe`) -- Access to https://xts.xenovex.com/explore/repos -- .NET 10 SDK for build/test - -## 1. Create the remote repository - -1. Sign in to **https://xts.xenovex.com** -2. Open **Explore repos** (or **New repository**) -3. Create a new empty repository, e.g. `llm-pii-poc` -4. Copy the **HTTPS** or **SSH** clone URL (example shapes): - - `https://xts.xenovex.com//llm-pii-poc.git` - - `git@xts.xenovex.com:/llm-pii-poc.git` - -Do **not** initialize the remote with a README if you are pushing an existing local history. - -## 2. Add remote and push (from repository root) - -```powershell -cd C:\Users\bilal.n\Projects\llm-pii-poc - -# Use full path if git is not on PATH -$git = "C:\Program Files\Git\bin\git.exe" - -& $git remote add origin -& $git branch -M main -& $git push -u origin main -``` - -If the remote already has commits (e.g. auto-generated README), either use an empty remote or: - -```powershell -& $git pull origin main --rebase -& $git push -u origin main -``` - -## 3. What is committed vs excluded - -| Included | Excluded (`.gitignore`) | -|----------|-------------------------| -| Source (`src/`), tests, scripts, docs | `bin/`, `obj/`, `.vs/` | -| `README.md`, `PiiRedaction.slnx` | `models/*.onnx`, `vocab.txt`, `ner-labels.txt` (~431MB model) | -| `models/.gitkeep` (empty models folder) | `scratch/` | - -After clone, download the NER model locally: - -```powershell -.\scripts\download-ner-model.ps1 -``` - -## 4. Verify after clone - -```powershell -dotnet build -dotnet test -dotnet run --project src/PiiRedaction.ConsoleApp -- --sample 0 -``` diff --git a/docs/ner-models.md b/docs/ner-models.md deleted file mode 100644 index d861d69..0000000 --- a/docs/ner-models.md +++ /dev/null @@ -1,548 +0,0 @@ -# NER Models for PII Redaction - -This document describes the **Named Entity Recognition (NER)** ONNX models at the core of the PII Redaction POC. Person-name detection is the only NER responsibility in this solution; structured identifiers (email, phone, PAN, domain IDs) are handled by regex and domain-rule detectors. - -For pipeline placement, trust boundaries, and routing diagrams, see [architecture.md](architecture.md). For the Tamil/Tanglish implementation plan and success metrics, see [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md). - ---- - -## 1. Executive Summary - -The POC uses **dual-model ONNX NER routing** to redact **person-name PII** before prompts reach an LLM: - -| Script in prompt | Model invoked | Typical use case | -|------------------|---------------|------------------| -| Latin only (`LatinOnly`) | English (`dslim/bert-base-NER`) | English names, Indian names in Roman script, **Tanglish** | -| Tamil only (`TamilOnly`) | Tamil (`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`) | Tamil-script customer names | -| Mixed (`Mixed`) | **Both** models on the full text; spans merged | Code-mixed Indian CS prompts | -| No letters (`NoLetters`) | Neither | Digits-only or symbol-only text | - -`RoutingOnnxNerModelRunner` classifies script via `ScriptRouter`, delegates to `EnglishOnnxNerRunner` and/or `TamilOnnxNerRunner`, and merges overlapping PERSON spans (longer span wins). Only **PERSON** entities are emitted to the redaction pipeline; all other NER labels are discarded. - ---- - -## 2. English Model - -### Hugging Face model ID - -**[`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER)** - -### Architecture - -| Property | Value | -|----------|-------| -| Base | BERT-base (uncased), ~110M parameters | -| Task | Token classification (NER) | -| Tokenizer | **WordPiece** via `vocab.txt` (`BertWordPieceEncoder`) | -| Runtime | ONNX via Microsoft.ML.OnnxRuntime | -| Export | Hugging Face Optimum (`ORTModelForTokenClassification`) or pre-exported ONNX from HF | - -### Labels (BIO) - -The English model uses standard CoNLL-style BIO tags. The POC maps only **person** labels to `PiiEntityType.Person`: - -| Label | Mapped to PERSON | -|-------|------------------| -| `O` | No | -| `B-PER`, `I-PER` | Yes | -| `B-PERSON`, `I-PERSON` | Yes | -| `B-ORG`, `I-ORG`, `B-LOC`, `I-LOC`, `B-MISC`, `I-MISC` | No | - -Full label list is written to `ner-labels.txt` at download time from the model `config.json` `id2label` map (typically 9 labels for this model). - -Label matching is implemented in `NerLabelConfig.English`: - -```19:22:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs - private static bool IsEnglishPersonLabel(string label) => - label is "B-PER" or "I-PER" or "B-PERSON" or "I-PERSON" - || (label.EndsWith("-PER", StringComparison.Ordinal) && - (label.StartsWith("B-", StringComparison.Ordinal) || label.StartsWith("I-", StringComparison.Ordinal))); -``` - -### Asset paths - -| File | Primary path (`appsettings.json`) | Legacy fallback | -|------|----------------------------------|-----------------| -| ONNX model | `models/en/ner-model.onnx` | `models/ner-model.onnx` (`OnnxModelPath`) | -| Vocabulary | `models/en/vocab.txt` | `models/vocab.txt` | -| Labels | `models/en/ner-labels.txt` | `models/ner-labels.txt` | - -`EnglishOnnxNerRunner` resolves the model path with primary + legacy fallback: - -```15:22:src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs - var modelPath = OnnxAssetPathResolver.ResolveModelPath( - options.Value.EnglishOnnxModelPath, - options.Value.OnnxModelPath); - - var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory; - var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory); - var encoder = new BertWordPieceEncoder(modelDirectory, logger); - _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.English, labels, logger); -``` - -> **Note:** `scripts/download-ner-model.ps1` writes assets to `models/` (repository root). For the configured primary path, copy or move them into `models/en/`, or rely on the `OnnxModelPath` fallback. - -### Download script - -```powershell -.\scripts\download-ner-model.ps1 -``` - -Or with Python directly: - -```bash -python scripts/download-ner-model.py -``` - -**Behavior:** - -1. If Python + Optimum are available → exports `dslim/bert-base-NER` to ONNX under `models/`. -2. Otherwise → downloads pre-exported ONNX from `https://huggingface.co/dslim/bert-base-NER/resolve/main/onnx/` (`model.onnx`, `vocab.txt`, `config.json` → `ner-labels.txt`). - ---- - -## 3. Tamil Model - -### Hugging Face model ID - -**[`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`](https://huggingface.co/prachuryyaIITG/SampurNER_Tamil_IndicBERTv2)** - -(SampurNER Tamil IndicBERTv2 — fine-grained NER for Tamil script.) - -### Why SampurNER IndicBERTv2 vs MuRIL - -| Criterion | SampurNER Tamil IndicBERTv2 | MuRIL (fallback candidate) | -|-----------|----------------------------|----------------------------| -| Tamil NER training | Fine-grained SampurNER dataset (Tamil-specific labels) | General multilingual; NER requires separate fine-tune | -| Model size | ~0.3B parameters (IndicBERTv2, ~278M base) | ~0.6B parameters | -| POC fit | Lighter memory footprint; ONNX export path validated in this repo | Reserved for Phase 5 if Tamil recall is insufficient | -| Indian financial context | Trained on Indian-language NER corpus; person subtypes map cleanly to PERSON | Heavier; eval-driven swap only | - -See [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) §3 for the original selection rationale. - -### Architecture - -| Property | Value | -|----------|-------| -| Base | IndicBERTv2 (AI4Bharat), ~0.3B parameters | -| Task | Fine-grained token classification | -| Tokenizer | **WordPiece** when `vocab.txt` is present (this repo's export path); SentencePiece fallback if `sentencepiece.bpe.model` / `spiece.model` exists | -| Runtime | Same shared `OnnxTokenClassifierRunner` as English | - -### Tokenizer: WordPiece, not SentencePiece (in practice) - -The Hugging Face repo for this model does **not** ship a SentencePiece model file. The download scripts extract **WordPiece** assets from `tokenizer.json` → `vocab.txt`. `TokenClassifierEncoderFactory` prefers `vocab.txt`: - -```10:19:src/PiiRedaction.Infrastructure/Onnx/TokenClassifierEncoderFactory.cs - public static ITokenClassifierEncoder Create(string modelDirectory, ILogger logger) - { - var vocabPath = OnnxAssetPathResolver.ResolveAssetPath(Path.Combine(modelDirectory, "vocab.txt")); - if (File.Exists(vocabPath)) - { - logger.LogInformation( - "Using WordPiece tokenizer (vocab.txt) from {ModelDirectory}.", - modelDirectory); - return new BertWordPieceEncoder(modelDirectory, logger); - } -``` - -The PowerShell Tamil download script emits an explicit warning when WordPiece assets are saved instead of SentencePiece. - -### Labels (fine-grained person tags) - -SampurNER uses fine-grained BIO tags (e.g. `B-person-politician`, `I-person-artist`, `B-location`, `O`). The POC treats **any label containing `person`** (case-insensitive) as a person span: - -```24:25:src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs - private static bool IsTamilPersonLabel(string label) => - label.Contains("person", StringComparison.OrdinalIgnoreCase); -``` - -Unit tests lock this behavior: - -```20:27:tests/PiiRedaction.Infrastructure.Tests/Onnx/NerLabelConfigTests.cs - [TestCase("B-person-politician", true)] - [TestCase("I-person-artist", true)] - [TestCase("B-location", false)] - [TestCase("O", false)] - public void Tamil_IsPersonLabel_MatchesFineGrainedTags(string label, bool expected) - { - NerLabelConfig.Tamil.IsPersonLabel(label).Should().Be(expected); - } -``` - -### Asset paths - -| File | Path | -|------|------| -| ONNX model | `models/ta/model.onnx` | -| Tokenizer | `models/ta/vocab.txt` (WordPiece, preferred) **or** `models/ta/sentencepiece.bpe.model` | -| Labels | `models/ta/ner-labels.txt` | -| Optional | `models/ta/tokenizer.json` (intermediate export artifact) | - -### Download script - -```powershell -.\scripts\download-tamil-ner-model.ps1 -``` - -Or with Python directly: - -```bash -python scripts/download-tamil-ner-model.py -``` - -**Behavior:** - -1. Python + Optimum → full export to `models/ta/` including ONNX, labels, and tokenizer assets. -2. PowerShell fallback → downloads `onnx/model.onnx` from Hugging Face when published; otherwise requires Python export (pre-exported ONNX may return 404). - -`TamilOnnxNerRunner` wiring: - -```15:19:src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs - var modelPath = OnnxAssetPathResolver.ResolveModelPath(options.Value.TamilOnnxModelPath); - var modelDirectory = Path.GetDirectoryName(modelPath) ?? Environment.CurrentDirectory; - var labels = OnnxAssetPathResolver.LoadLabels(modelDirectory); - var encoder = TokenClassifierEncoderFactory.Create(modelDirectory, logger); - _runner = new OnnxTokenClassifierRunner(modelPath, encoder, NerLabelConfig.Tamil, labels, logger); -``` - ---- - -## 4. Why These Models - -Evidence-based rationale for this Indian financial POC: - -| Requirement | Decision | -|-------------|----------| -| **English + Indian Latin names** | `dslim/bert-base-NER` is industry-standard, pre-integrated, and handles many Indian names in Roman script (e.g. `Ravi Kumar`, `Anita Sharma`) | -| **Tamil script names** | English BERT is out-of-vocabulary for Tamil letters (U+0B80–U+0BFF); a Tamil-trained NER model is required | -| **Tanglish (Roman-script Tamil)** | Routed to the **English** model only (`ScriptComposition.LatinOnly`); no Tamil ONNX on Latin-only text | -| **Code-mixed prompts** | Both models run on the full prompt; `MergePersonSpans` deduplicates overlaps | -| **Deployable size** | English ~431 MB ONNX (FP32); Tamil IndicBERTv2 ~0.3B params — smaller than MuRIL ~0.6B | -| **ONNX export path** | Both models export via Hugging Face Optimum; English has pre-exported ONNX on HF; Tamil may require local Python export | -| **PERSON-only scope** | POC redacts person names via NER; org/location/misc labels are intentionally ignored to limit false positives | - ---- - -## 5. How They Integrate - -### Configuration (`appsettings.json`) - -```1:8:src/PiiRedaction.ConsoleApp/appsettings.json -{ - "PiiRedaction": { - "OnnxModelPath": "models/ner-model.onnx", - "EnglishOnnxModelPath": "models/en/ner-model.onnx", - "TamilOnnxModelPath": "models/ta/model.onnx", - "EnableTamilNer": true - } -} -``` - -Options type: - -```3:14:src/PiiRedaction.Core/Configuration/PiiRedactionOptions.cs -public sealed class PiiRedactionOptions -{ - public const string SectionName = "PiiRedaction"; - - public string OnnxModelPath { get; set; } = "models/ner-model.onnx"; - - public string EnglishOnnxModelPath { get; set; } = "models/en/ner-model.onnx"; - - public string TamilOnnxModelPath { get; set; } = "models/ta/model.onnx"; - - public bool EnableTamilNer { get; set; } = true; -} -``` - -Set `EnableTamilNer` to `false` for English-only routing. - -### Dependency injection - -```34:36:src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); -``` - -`OnnxNerPiiDetector` consumes `IOnnxNerModelRunner` (the router) and returns `[]` when no model is available — **fail-open** for person detection. - -### Script routing (`ScriptRouter`) - -```11:41:src/PiiRedaction.Core/Detection/ScriptRouter.cs - public ScriptComposition GetComposition(string text) - { - ArgumentNullException.ThrowIfNull(text); - - var hasLatin = false; - var hasTamil = false; - - foreach (var character in text) - { - if (IsTamilLetter(character)) - { - hasTamil = true; - } - else if (char.IsAsciiLetter(character)) - { - hasLatin = true; - } - - if (hasLatin && hasTamil) - { - return ScriptComposition.Mixed; - } - } - // ... - return hasTamil ? ScriptComposition.TamilOnly : ScriptComposition.LatinOnly; - } -``` - -### Routing runner (`RoutingOnnxNerModelRunner`) - -```39:79:src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs - public IReadOnlyList PredictEntities(string text) - { - var composition = _scriptRouter.GetComposition(text); - var entities = new List(); - - switch (composition) - { - case ScriptComposition.LatinOnly: - if (_englishRunner.IsModelAvailable) - { - entities.AddRange(_englishRunner.PredictEntities(text)); - } - break; - case ScriptComposition.TamilOnly: - if (_enableTamilNer && _tamilRunner.IsModelAvailable) - { - entities.AddRange(_tamilRunner.PredictEntities(text)); - } - break; - case ScriptComposition.Mixed: - if (_englishRunner.IsModelAvailable) - { - entities.AddRange(_englishRunner.PredictEntities(text)); - } - if (_enableTamilNer && _tamilRunner.IsModelAvailable) - { - entities.AddRange(_tamilRunner.PredictEntities(text)); - } - break; - case ScriptComposition.NoLetters: - break; - } - - return MergePersonSpans(entities); - } -``` - -### Shared inference (`OnnxTokenClassifierRunner`) - -Both runners share: - -- **Encode** → `input_ids`, `attention_mask`, optional `token_type_ids` -- **Argmax** over per-token logits -- **BIO decode** → `PiiEntityType.Person` with `PiiDetectionSource.Ner` -- **Max sequence length: 128 tokens** - -```13:13:src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs - private const int MaxSequenceLength = 128; -``` - -### End-to-end flow - -``` -Prompt → CompositePiiDetector → OnnxNerPiiDetector - → RoutingOnnxNerModelRunner → ScriptRouter - → EnglishOnnxNerRunner / TamilOnnxNerRunner - → OnnxTokenClassifierRunner → PERSON entities - → PlaceholderPiiRedactor → -``` - ---- - -## 6. Evidence from Codebase - -### Real-model tests (`Category=RealModel`) - -English direct inference — `RealNerModelRunnerTests`: - -```15:47:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealNerModelRunnerTests.cs -[Category("RealModel")] -public sealed class RealNerModelRunnerTests : RealNerModelFixture -{ - [TestCase("Customer Ravi Kumar called about billing.", "Ravi", "Ravi Kumar")] - [TestCase("Mr. John Smith called about a duplicate debit.", "John", "John Smith")] - public void PredictEntities_DetectsPersonWithCorrectSpan(...) -``` - -English pipeline — `RealNerPipelineTests` (`Category=RealModel`): canonical `FullFinancialWithCustomer`, multi-person, clean-ticket negative. - -Fixture skips when model missing: - -```5:6:tests/TestSupport.Shared/RealNerModelPaths.cs - public const string ModelMissingMessage = - "ONNX model not found. Run scripts/download-ner-model.ps1 from the repository root."; -``` - -Resolves `models/en/ner-model.onnx` then `models/ner-model.onnx`. - -### Tamil tests (`Category=TamilNer`) - -Direct Tamil runner — `RealTamilNerModelRunnerTests`: - -```9:69:tests/PiiRedaction.Infrastructure.Tests/Onnx/RealTamilNerModelRunnerTests.cs -[Category("TamilNer")] -public sealed class RealTamilNerModelRunnerTests : RealTamilNerModelFixture -{ - [TestCase("வாடிக்கையாளர் ராஜேஷ் குமார் அழைத்தார்.", "ராஜேஷ்", "ராஜேஷ் குமார்")] - public void PredictEntities_TamilScript_DetectsPersonEntity(...) - // ... - [TestCase("Customer Senthil phone 9876543210", "Senthil")] - public void PredictEntities_TanglishLatinScript_DoesNotInvokeTamilRunner(...) -``` - -Routed pipeline — `RealTamilPipelineTests` covers Tamil-only, Tanglish (English path), mixed, full financial, and clean Tamil negative. - -### Console samples (`SamplePromptCatalog`) - -Tamil/Tanglish/mixed samples (indices 10–14): - -| Sample | Category | Input excerpt | -|--------|----------|---------------| -| `TamilCustomerNameOnly` | NER (Tamil) | `வாடிக்கையாளர் ராஜேஷ் குமார் …` | -| `TamilWithPhonePan` | NER (Tamil) + Regex | Tamil person + phone + PAN | -| `TanglishCustomer` | NER (English/Tanglish) | `Customer Senthil phone 9876543210 …` | -| `MixedTamilEnglish` | NER (Mixed) | `வாடிக்கையாளர் Ravi Kumar phone …` | -| `TamilFullFinancial` | NER (Tamil) + Regex + Domain | Tamil canonical demo | - -Run all samples (including Tamil) with no flags: - -```bash -dotnet run --project src/PiiRedaction.ConsoleApp -``` - -Or a single Tamil sample: - -```bash -dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly -``` - -### Expected console output (English canonical) - -When models are loaded, person names appear as `[PERSON]` with source `Ner`: - -``` -Detected PII: - [PERSON ] Ravi Kumar (Ner) - [EMAIL ] ravi.kumar@gmail.com (Regex) - ... - -Sanitized Prompt: -Customer with email ... -``` - ---- - -## 7. Model Assets Table - -All model binaries are **gitignored**; only `.gitkeep` placeholders are committed. - -| Directory | File | Approx. size | Gitignored | Purpose | -|-----------|------|--------------|------------|---------| -| `models/` or `models/en/` | `ner-model.onnx` | ~431 MB | Yes | English BERT NER (FP32 ONNX from HF) | -| `models/` or `models/en/` | `vocab.txt` | ~213 KB | Yes | WordPiece vocabulary | -| `models/` or `models/en/` | `ner-labels.txt` | < 1 KB | Yes | BIO label index (one per line) | -| `models/ta/` | `model.onnx` | ~1.2 GB (FP32 export, varies) | Yes | Tamil IndicBERT NER | -| `models/ta/` | `vocab.txt` | varies | Yes | WordPiece vocab (preferred tokenizer) | -| `models/ta/` | `tokenizer.json` | varies | Yes | HF tokenizer export (optional) | -| `models/ta/` | `sentencepiece.bpe.model` | varies | Yes | SentencePiece (if present instead of vocab) | -| `models/ta/` | `ner-labels.txt` | few KB | Yes | Fine-grained SampurNER labels | - -`.gitignore` entries: - -``` -models/*.onnx -models/vocab.txt -models/ner-labels.txt -models/*.json -models/en/* -models/ta/* -``` - -English size reference: [docs/git-xenovex-setup.md](git-xenovex-setup.md) notes ~431 MB for the English ONNX file. - ---- - -## 8. Limitations - -| Limitation | Detail | -|------------|--------| -| **Tanglish on English model only** | Roman-script Tanglish (`Customer Senthil`) is classified `LatinOnly` and handled by English BERT. Recall is best-effort and inconsistent for non-standard spellings. Tamil ONNX is **not** invoked on Latin-only text. | -| **Fail-open if model missing** | `OnnxNerPiiDetector` and routing runners return `[]` when models are unavailable. Person names are **not** redacted; regex/domain layers still run. No regex fallback for names. | -| **128 token limit** | `OnnxTokenClassifierRunner` truncates encoding at 128 tokens. Very long prompts may miss person names beyond the window. | -| **PERSON-only NER scope** | Organization, location, and misc NER labels are ignored. Only person spans become ``. | -| **Mixed-script merge** | When both models run, overlapping spans are deduped by length; shorter overlapping spans are dropped. | -| **Tamil ONNX availability** | Pre-exported Tamil ONNX may not exist on Hugging Face; local Python export is often required. | -| **No fail-closed mode** | Missing NER does not block sanitization or LLM calls (optional Phase 5 enhancement). | - ---- - -## 9. How to Reproduce - -### Download models - -From the repository root: - -```powershell -.\scripts\download-ner-model.ps1 -.\scripts\download-tamil-ner-model.ps1 -``` - -If Tamil PowerShell download fails with a 404 on `onnx/model.onnx`, install Python 3.12+ and re-run: - -```powershell -winget install Python.Python.3.12 --accept-package-agreements --accept-source-agreements -.\scripts\download-tamil-ner-model.ps1 -Python "$env:LOCALAPPDATA\Programs\Python\Python312\python.exe" -``` - -Optional: copy English assets from `models/` to `models/en/` to match `EnglishOnnxModelPath`. - -### Verify with tests - -```bash -dotnet build -dotnet test -dotnet test --filter "Category=RealModel" -dotnet test --filter "Category=TamilNer" -dotnet test --logger "console;verbosity=detailed" -``` - -Tests skip gracefully when the corresponding ONNX files are absent. - -### Verify with console - -```bash -dotnet run --project src/PiiRedaction.ConsoleApp -- --name CustomerNameOnly -dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly -dotnet run --project src/PiiRedaction.ConsoleApp -- --name TanglishCustomer -``` - -### Disable Tamil routing (English-only) - -Set in `appsettings.json`: - -```json -"EnableTamilNer": false -``` - ---- - -## Related Documentation - -- [README.md](../README.md) — build, run, and test overview -- [architecture.md](architecture.md) — dual-model routing diagrams and trust boundary -- [tamil-tanglish-ner-plan.md](tamil-tanglish-ner-plan.md) — implementation phases and Tanglish expectations diff --git a/docs/solution-guide.md b/docs/solution-guide.md new file mode 100644 index 0000000..9b15347 --- /dev/null +++ b/docs/solution-guide.md @@ -0,0 +1,557 @@ +# PII Redaction POC — Solution Guide + +Single reference for architecture, NER models, routing, operations, Tamil/Tanglish support, Git setup, and the post-POC improvement backlog. + +--- + +## Table of contents + +1. [Purpose](#1-purpose) +2. [Canonical example](#2-canonical-example) +3. [Detection strategies](#3-detection-strategies) +4. [Solution architecture](#4-solution-architecture) +5. [Trust boundary](#5-trust-boundary) +6. [NER models](#6-ner-models) +7. [NER routing — English vs Tamil](#7-ner-routing--english-vs-tamil) +8. [Configuration and dependency injection](#8-configuration-and-dependency-injection) +9. [Running and testing](#9-running-and-testing) +10. [Tamil and Tanglish support](#10-tamil-and-tanglish-support) +11. [Git remote setup (Xenovex)](#11-git-remote-setup-xenovex) +12. [Improvement roadmap](#12-improvement-roadmap) +13. [Key source files](#13-key-source-files) + +--- + +## 1. Purpose + +This .NET proof-of-concept intercepts user prompts containing regulated personally identifiable information (PII), redacts sensitive values into stable placeholders, and transmits **only sanitized text** across the LLM trust boundary. + +The solution uses: + +- Clear layer separation (Core / Infrastructure / hosts) +- Interface-driven composition and dependency injection +- Swappable ONNX NER adapters and `Microsoft.Extensions.AI` chat clients + +It targets financial and customer-service workloads where raw PII must not leave the application process when invoking external language models. + +--- + +## 2. Canonical example + +With English and Tamil ONNX models loaded (`scripts/download-ner-model.ps1`, `scripts/download-tamil-ner-model.ps1`): + +| Stage | Value | +|-------|-------| +| **Input** | `Customer Ravi Kumar with email ravi.kumar@gmail.com and phone 9876543210 has LoanNumber LN-456789 and PAN ABCDE1234F. Please summarize this customer issue.` | +| **Sanitized output** | `Customer with email and phone has LoanNumber and PAN . Please summarize this customer issue.` | +| **Mock LLM response** | `[Mock LLM Response] Received sanitized prompt (146 chars). No original PII was transmitted.` | + +| Type | Value | Source | +|------|-------|--------| +| PERSON | Ravi Kumar | NER | +| EMAIL | ravi.kumar@gmail.com | Regex | +| PHONE | 9876543210 | Regex | +| LOAN_NUMBER | LN-456789 | Domain | +| PAN | ABCDE1234F | Regex | + +The placeholder map (`` → `Ravi Kumar`, etc.) stays **in-process** and is never sent to the LLM. + +--- + +## 3. Detection strategies + +| Strategy | Detects | Rationale | +|----------|---------|-----------| +| **Regex** | Email, phone, PAN, Aadhaar, credit card | Deterministic, format-bound, auditable | +| **ONNX NER** | Person names | Contextual; no rigid format | +| **Domain rules** | Loan number (`LN-`), customer ID (`CID-`), account (`ACC-`) | Business-specific identifiers | + +**Overlap resolution** (`CompositePiiDetector`): detectors run in order **Domain → Regex → NER**. On overlapping spans, candidates are sorted by start index, length, and source priority (**Domain 3 > Regex 2 > NER 1**); the first non-overlapping candidate wins. + +**Redaction** (`PlaceholderPiiRedactor`): format `<{TYPE}_{n}>`; duplicate values reuse placeholders; replacement is right-to-left to preserve indices. + +--- + +## 4. Solution architecture + +### Project structure + +``` +src/ +├── PiiRedaction.ConsoleApp/ # Console demo, DI bootstrap +├── PiiRedaction.TestHarness.Wpf/ # WPF MVVM manual test harness +├── PiiRedaction.Core/ # Detection, redaction, abstractions +└── PiiRedaction.Infrastructure/ # ONNX runners, mock LLM +models/ # ONNX assets (gitignored) +tests/ # NUnit unit and integration tests +``` + +| Project | Layer | Responsibility | +|---------|-------|----------------| +| `PiiRedaction.ConsoleApp` | Presentation | Samples, interactive mode, audit output | +| `PiiRedaction.TestHarness.Wpf` | Presentation | Category-filtered prompts, redact UI, batch runner | +| `PiiRedaction.Core` | Domain | `IPiiDetector`, `IPromptSanitizer`, detectors, models | +| `PiiRedaction.Infrastructure` | Infrastructure | `RoutingOnnxNerModelRunner`, `MockLlmPromptService` | +| `tests/*` | Test | ~112 tests; `RealModel` and `TamilNer` categories | + +**Dependency direction:** `ConsoleApp` / `Wpf` → `Infrastructure` → `Core`. Core has no ONNX or LLM SDK references. + +### Key abstractions + +| Abstraction | Default implementation | Extension | +|-------------|------------------------|-----------| +| `IPiiDetector` | `CompositePiiDetector` | Add detector; register in composite order | +| `IPiiRedactor` | `PlaceholderPiiRedactor` | Hashing, vault tokens | +| `IPromptSanitizer` | `PromptSanitizer` | Orchestrates detect + redact | +| `IOnnxNerModelRunner` | `RoutingOnnxNerModelRunner` | Script-based EN/TA routing | +| `ILlmPromptService` | `MockLlmPromptService` | Production adapter | +| `IChatClient` | `MockChatClient` | Azure OpenAI, etc. | + +### Data flow + +```mermaid +flowchart TB + subgraph hosts [Hosts] + console[ConsoleApp / WpfHarness] + end + + subgraph core [PiiRedaction.Core] + sanitizer[PromptSanitizer] + composite[CompositePiiDetector] + domain[DomainRulePiiDetector] + regex[RegexPiiDetector] + onnxDet[OnnxNerPiiDetector] + redactor[PlaceholderPiiRedactor] + end + + subgraph infra [PiiRedaction.Infrastructure] + router[RoutingOnnxNerModelRunner] + en[EnglishOnnxNerRunner] + ta[TamilOnnxNerRunner] + llm[MockLlmPromptService] + end + + console --> sanitizer + sanitizer --> composite + composite --> domain + composite --> regex + composite --> onnxDet + onnxDet --> router + router --> en + router --> ta + sanitizer --> redactor + console -->|"sanitized text only"| llm +``` + +**Pipeline:** `Sanitize` → `Detect` (all detectors) → `Redact` → `SanitizationResult`. Optional: `SendPromptAsync(sanitizedPrompt)` to LLM. + +--- + +## 5. Trust boundary + +Only the **sanitized prompt string** crosses `ILlmPromptService` / `IChatClient`. Original PII, entity metadata, and the placeholder map remain in-process. + +```mermaid +flowchart LR + subgraph inProcess [In-Process] + raw[Original prompt] + entities[Detected entities] + map[Placeholder map] + audit[Console / WPF display] + end + + subgraph boundary [LLM boundary] + sanitized[Sanitized prompt only] + end + + subgraph external [External LLM] + chat[IChatClient] + end + + raw --> audit + entities --> audit + map --> audit + sanitized --> chat + raw -.->|never sent| chat + map -.->|never sent| chat +``` + +--- + +## 6. NER models + +Person-name detection is the **only** NER responsibility. Structured PII uses regex and domain rules. + +### Routing summary + +| Script in prompt | Model(s) | Typical use | +|------------------|----------|-------------| +| `LatinOnly` | English | English names, Indian names in Roman script, **Tanglish** | +| `TamilOnly` | Tamil | Tamil-script names | +| `Mixed` | Both; merge spans | Code-mixed prompts | +| `NoLetters` | Neither | Digits/symbols only | + +### English model + +| Property | Value | +|----------|-------| +| Hugging Face ID | [`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER) | +| Tokenizer | WordPiece (`vocab.txt`, `BertWordPieceEncoder`) | +| Person labels | `B-PER`, `I-PER`, `B-PERSON`, `I-PERSON` | +| Primary path | `models/en/ner-model.onnx` | +| Legacy fallback | `models/ner-model.onnx` | +| Download | `.\scripts\download-ner-model.ps1` | + +### Tamil model + +| Property | Value | +|----------|-------| +| Hugging Face ID | [`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`](https://huggingface.co/prachuryyaIITG/SampurNER_Tamil_IndicBERTv2) | +| Tokenizer | WordPiece when `vocab.txt` present (export path); SentencePiece fallback | +| Person labels | Any BIO tag containing `person` (case-insensitive) | +| Path | `models/ta/model.onnx` | +| Download | `.\scripts\download-tamil-ner-model.ps1` | + +**Why SampurNER over MuRIL:** Tamil-specific NER training, smaller footprint (~0.3B vs ~0.6B), validated ONNX export in this repo. MuRIL reserved as eval-driven fallback. + +### Model assets (gitignored) + +| Directory | Key files | Approx. size | +|-----------|-----------|--------------| +| `models/en/` | `ner-model.onnx`, `vocab.txt`, `ner-labels.txt` | ~431 MB ONNX | +| `models/ta/` | `model.onnx`, `vocab.txt`, `ner-labels.txt` | ~1 GB ONNX | + +Only `models/en/.gitkeep` and `models/ta/.gitkeep` are committed. + +### Shared inference + +Both runners use `OnnxTokenClassifierRunner`: + +- Max sequence length: **128 tokens** (long prompts truncate silently) +- BIO decode → `PiiEntityType.Person` only +- Missing model → fail-open: `[]` from NER (person names not redacted) + +--- + +## 7. NER routing — English vs Tamil + +### Call chain + +``` +PromptSanitizer → CompositePiiDetector → OnnxNerPiiDetector + → RoutingOnnxNerModelRunner.PredictEntities() + → ScriptRouter.GetComposition(text) + → switch (composition) { English / Tamil / both } + → MergePersonSpans() +``` + +### Step 1: `ScriptRouter` (Core) + +**File:** `src/PiiRedaction.Core/Detection/ScriptRouter.cs` + +Single pass over characters: + +- Tamil letter: Unicode **U+0B80 – U+0BFF** +- Latin letter: `char.IsAsciiLetter` +- Both seen → `Mixed` (early exit) +- Neither → `NoLetters` +- Otherwise → `TamilOnly` or `LatinOnly` + +```csharp +// ScriptComposition enum: LatinOnly, TamilOnly, Mixed, NoLetters +public ScriptComposition GetComposition(string text) { /* scan chars */ } +``` + +**Tanglish** in Roman script → `LatinOnly` → **English model only**. + +### Step 2: `RoutingOnnxNerModelRunner` (Infrastructure) + +**File:** `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs` + +This is the **branch decision**: + +| `ScriptComposition` | English | Tamil | +|---------------------|---------|-------| +| `LatinOnly` | Yes if available | No | +| `TamilOnly` | No | Yes if `EnableTamilNer` and available | +| `Mixed` | Yes if available | Yes if `EnableTamilNer` and available | +| `NoLetters` | No | No | + +Both models run on the **full prompt text** for `Mixed` (no script segmentation). + +### Step 3: Merge + +`MergePersonSpans`: overlapping PERSON spans → **longer span wins**; ordered by `StartIndex`. + +### Routing diagram + +```mermaid +flowchart TD + text[Prompt text] --> sr[ScriptRouter] + sr --> latin[LatinOnly] + sr --> tamil[TamilOnly] + sr --> mixed[Mixed] + sr --> none[NoLetters] + latin --> en[EnglishOnnxNerRunner] + tamil --> ta[TamilOnnxNerRunner] + mixed --> en + mixed --> ta + en --> merge[MergePersonSpans] + ta --> merge + none --> empty[No NER] +``` + +### Worked examples + +| Input style | Composition | Models | +|-------------|---------------|--------| +| `Customer Ravi Kumar…` | `LatinOnly` | English | +| `வாடிக்கையாளர் ராஜேஷ் குமார்…` | `TamilOnly` | Tamil | +| `Naan Suresh, phone 9003789456…` | `LatinOnly` | English (Tanglish) | +| `வாடிக்கையாளர் Ravi Kumar phone…` | `Mixed` | Both | +| `Callback on 9123456780…` | `NoLetters` | Neither (phone via Regex) | + +--- + +## 8. Configuration and dependency injection + +### `appsettings.json` + +```json +{ + "PiiRedaction": { + "OnnxModelPath": "models/ner-model.onnx", + "EnglishOnnxModelPath": "models/en/ner-model.onnx", + "TamilOnnxModelPath": "models/ta/model.onnx", + "EnableTamilNer": true + } +} +``` + +| Setting | Effect | +|---------|--------| +| `EnglishOnnxModelPath` | Primary English model | +| `OnnxModelPath` | Legacy English fallback | +| `TamilOnnxModelPath` | Tamil model | +| `EnableTamilNer` | `false` = English-only routing | + +### DI registration + +```csharp +services.AddSingleton(); +services.AddSingleton(); +services.AddSingleton(); +// OnnxNerPiiDetector receives IOnnxNerModelRunner (the router) +``` + +Same pattern in `PiiRedaction.ConsoleApp` and `PiiRedaction.TestHarness.Wpf` `ServiceCollectionExtensions.cs`. + +Detector registration order in composite: **Domain → Regex → ONNX NER**. + +--- + +## 9. Running and testing + +### Build + +```bash +dotnet restore +dotnet build +``` + +### Console app + +```bash +# All 16 samples (English + Tamil/Tanglish/mixed) +dotnet run --project src/PiiRedaction.ConsoleApp + +dotnet run --project src/PiiRedaction.ConsoleApp -- --list +dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerNameOnly +dotnet run --project src/PiiRedaction.ConsoleApp -- --interactive +``` + +Samples: `src/PiiRedaction.ConsoleApp/Samples/SamplePromptCatalog.cs` + +### WPF test harness (Windows) + +```bash +dotnet run --project src/PiiRedaction.TestHarness.Wpf +``` + +- **Category dropdown:** Career Guidance, Banking & Financial, Negative, Edge & Harness +- Click prompt → loads input; **Redact** runs pipeline +- **Run All** executes scenarios in the selected category +- Script badge shows predicted routing (`LatinOnly`, `TamilOnly`, `Mixed`) + +### Download models + +```powershell +.\scripts\download-ner-model.ps1 +.\scripts\download-tamil-ner-model.ps1 +``` + +If Tamil PowerShell download 404s, use Python 3.12+: + +```powershell +.\scripts\download-tamil-ner-model.ps1 -Python "$env:LOCALAPPDATA\Programs\Python\Python312\python.exe" +``` + +### Tests + +```bash +dotnet test +dotnet test --filter "Category=RealModel" +dotnet test --filter "Category=TamilNer" +dotnet test --filter "FullyQualifiedName~ScriptRouterTests|FullyQualifiedName~RoutingOnnxNerModelRunnerTests" +``` + +Tests skip gracefully when ONNX files are absent. + +### Console sample categories + +| Category | Examples | +|----------|----------| +| NER (English) | `CustomerNameOnly`, `MrTitlePerson`, `TwoCustomersInOnePrompt` | +| NER (Tamil) | `TamilCustomerNameOnly`, `TamilFullFinancial` | +| Tanglish / Mixed | `TanglishCustomer`, `MixedTamilEnglish` | +| Regex / Domain | `AllRegexTypes`, `AllDomainIds` | +| Negative | `NoPiiCleanTicket` | + +--- + +## 10. Tamil and Tanglish support + +### Implementation status + +| Phase | Status | Scope | +|-------|--------|-------| +| **1** Generic ONNX token classifier | Done | `OnnxTokenClassifierRunner`, encoders, `NerLabelConfig` | +| **2** Tamil download + config | Done | Scripts, dual paths, `EnableTamilNer` | +| **3** Script routing + DI | Done | `ScriptRouter`, `RoutingOnnxNerModelRunner` | +| **4** Tests, samples, docs | Partial | Tests/samples done; eval metrics open | +| **5** Optional enhancements | Not started | See below | + +### Remaining gaps (Phase 4–5) + +| Gap | Impact | +|-----|--------| +| Tamil numeral normalization (௦–௯ → 0–9) | May miss phone/Aadhaar in Tamil script | +| Tanglish heuristics (`peru`, `enga peru`) | Better Latin-name recall in Tamil context | +| Label-aware regex cues | Contextual name detection | +| Fail-closed when NER missing | Compliance hardening | +| MuRIL model swap | If Tamil recall insufficient | + +### Tanglish expectations + +| Input | Handler | Expected recall | +|-------|---------|-----------------| +| Tamil script names | Tamil ONNX | High (with tuning) | +| Latin Indian names (`Ravi Kumar`) | English ONNX | High | +| Tanglish spellings (`Senthil`) | English NER + optional heuristics | Medium | +| Code-mixed prompts | Both models + merge | Medium–high for IDs; names variable | + +### Success metrics (eval set target) + +| Metric | MVP target | +|--------|------------| +| Tamil script person recall | ≥ 85% | +| Tanglish person recall | ≥ 70% | +| False positives on clean prompts | ≤ 5% | +| Structured PII in Tamil prompts (regex) | ≥ 95% | +| English canonical regression | 100% | + +--- + +## 11. Git remote setup (Xenovex) + +Remote: `https://xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc.git` + +### Create and push + +```powershell +cd C:\Users\bilal.n\Projects\llm-pii-poc +$git = "C:\Program Files\Git\bin\git.exe" + +& $git remote add origin +& $git branch -M main +& $git push -u origin main +``` + +Use an **empty** remote repository (no README) when pushing existing history. + +### Committed vs gitignored + +| Committed | Gitignored | +|-----------|------------| +| `src/`, `tests/`, `scripts/`, `docs/` | `bin/`, `obj/`, `.vs/` | +| `README.md`, `PiiRedaction.slnx` | `models/*.onnx`, `vocab.txt`, `models/en/*`, `models/ta/*` | +| `models/en/.gitkeep`, `models/ta/.gitkeep` | `scratch/` | + +After clone, run model download scripts locally. + +--- + +## 12. Improvement roadmap + +Post-POC items **not yet implemented**. Current maturity: strong architecture and tests; production needs fail-closed policy, API host, and observability. + +### P0 — Security and correctness + +| Item | Proposal | +|------|----------| +| Fail-closed when NER unavailable | `RequireNerOnStartup`, `BlockLlmWhenNerUnavailable` options | +| Outbound LLM guard | Verify sanitized text before `IChatClient` | +| Truncation warning | Surface 128-token limit in `SanitizationResult` | +| NER confidence | Populate `PiiEntity.Confidence` or hide UI column | +| WPF leak check | Use placeholder map, not naive `Contains` | + +### P1 — Platform + +| Item | Proposal | +|------|----------| +| `PiiRedaction.Composition` | Shared `AddPiiRedactionServices` (Console + WPF duplicate today) | +| Unified prompt catalog | Single source for console, WPF, golden tests | +| Async `IPromptSanitizer` | Replace WPF `Task.Run` wrapper | +| `PiiRedaction.Application` | Shared orchestration for API/WPF | +| Minimal API + health checks | `POST /v1/prompts/sanitize`, model readiness | +| Tamil Phase 4 | Numeral normalization, Tanglish heuristics | + +### P2 — Operations + +| Item | Proposal | +|------|----------| +| Placeholder audit store | TTL, encryption, correlation ID | +| Observability | Metrics, OpenTelemetry traces | +| ONNX session pool | Concurrency strategy for API load | +| CI pipeline | Fast tests without models; nightly real-model job | + +### Suggested phases + +1. **Production hardening** — fail-closed, composition root, API, truncation metadata (~1–2 weeks) +2. **Detection quality** — normalizer, Tamil Phase 4, regex hardening (~1 week) +3. **Operations** — audit store, metrics, session pool (~1–2 weeks) + +--- + +## 13. Key source files + +| Topic | Path | +|-------|------| +| Script classification | `src/PiiRedaction.Core/Detection/ScriptRouter.cs` | +| **Model branch decision** | `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs` | +| English NER | `src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs` | +| Tamil NER | `src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs` | +| Shared ONNX inference | `src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs` | +| NER detector | `src/PiiRedaction.Core/Detection/OnnxNerPiiDetector.cs` | +| Sanitizer | `src/PiiRedaction.Core/Sanitization/PromptSanitizer.cs` | +| Composite merge | `src/PiiRedaction.Core/Detection/CompositePiiDetector.cs` | +| Options | `src/PiiRedaction.Core/Configuration/PiiRedactionOptions.cs` | +| DI | `src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs` | +| Console samples | `src/PiiRedaction.ConsoleApp/Samples/SamplePromptCatalog.cs` | +| WPF catalog | `src/PiiRedaction.TestHarness.Wpf/Services/TestPromptCatalog.cs` | +| Routing tests | `tests/PiiRedaction.Infrastructure.Tests/Onnx/RoutingOnnxNerModelRunnerTests.cs` | +| Script tests | `tests/PiiRedaction.Core.Tests/Detection/ScriptRouterTests.cs` | + +--- + +*Last consolidated: July 2026. Replaces separate architecture, NER models, routing reference, Tamil plan, improvement roadmap, and Git setup documents.* diff --git a/docs/tamil-tanglish-ner-plan.md b/docs/tamil-tanglish-ner-plan.md deleted file mode 100644 index 088e5b6..0000000 --- a/docs/tamil-tanglish-ner-plan.md +++ /dev/null @@ -1,307 +0,0 @@ -# Tamil / Tanglish NER — Implementation Plan - -**Goal:** Raise language coverage from ~15% to production-viable for Tamil script and Tanglish (Roman-script Tamil-English) customer prompts, without changing the secure LLM boundary pattern. - -**Status:** Phase 1–3 implemented -**Approach:** Dual-model ONNX NER routing (English + Tamil) + lightweight text normalization + optional Tanglish heuristics -**Estimated effort:** 4–6 engineering days across 4 phases - ---- - -## 1. Current State vs Gap - -| Capability | Today | Tamil script | Tanglish (Latin) | -|------------|-------|--------------|------------------| -| Phone, PAN, Aadhaar, email, domain IDs | Regex + domain rules | Works (ASCII digits) | Works | -| Person names | `dslim/bert-base-NER` (English BERT) | **Fails** — out of vocabulary | **Partial** — inconsistent | -| Script / language routing | None | N/A | N/A | -| Tamil numerals (௦–௯) | Not normalized | **May miss** phone/Aadhaar | N/A | -| Label-aware cues (`பெயர்`, `peru`, `enga peru`) | None | **Misses** contextual names | **Misses** | - -**Root cause:** Person detection is a single English-only ONNX model behind `IOnnxNerModelRunner` → `OnnxNerPiiDetector`. Regex/domain layers are already language-agnostic. - ---- - -## 2. Target Architecture - -No change to the trust boundary: `PromptSanitizer` → `CompositePiiDetector` → `PlaceholderPiiRedactor` → sanitized text only to LLM. - -Only the **NER adapter** expands: - -```mermaid -flowchart TD - text["Prompt text"] - router["ScriptRouter (Core)"] - routing["RoutingOnnxNerModelRunner"] - en["EnglishOnnxRunner\nBERT WordPiece"] - ta["TamilOnnxRunner\nIndicBERT SentencePiece"] - nerDet["OnnxNerPiiDetector"] - composite["CompositePiiDetector"] - - text --> composite - text --> nerDet - nerDet --> routing - routing --> router - router -->|"LatinOnly / Mixed"| en - router -->|"TamilOnly / Mixed"| ta - en --> routing - ta --> routing -``` - -### Routing rules - -| `ScriptComposition` | Models invoked | Tanglish note | -|---------------------|----------------|---------------| -| `LatinOnly` | English NER only | Tanglish names in Roman script | -| `TamilOnly` | Tamil NER only | Tamil script names | -| `Mixed` | **Both**, merge person spans | Common in Indian CS prompts | -| `NoLetters` | Neither (or English fallback off) | Digits-only prompts | - -**Merge inside `RoutingOnnxNerModelRunner`:** dedupe overlapping person spans (prefer longer span; tie-break Tamil vs English by start index order). - ---- - -## 3. Model Selection - -| Role | Model | Rationale | -|------|-------|-----------| -| English / Tanglish (Latin) | **Keep** `dslim/bert-base-NER` | Already integrated; works for many Indian names in Latin script | -| Tamil script | **`prachuryyaIITG/SampurNER_Tamil_IndicBERTv2`** | Tamil NER; lighter than MuRIL (~0.6B) | -| Fallback (optional Phase 5) | MuRIL Tamil NER | Only if IndicBERT recall is insufficient on eval set | - -### Asset layout - -``` -models/ - en/ - ner-model.onnx # or model.onnx (BERT export) - vocab.txt - ner-labels.txt - ta/ - model.onnx - sentencepiece.bpe.model # or tokenizer.json from HF export - ner-labels.txt - ner-model.onnx # legacy path — keep for backward compatibility -``` - -### Label mapping (Tamil fine-grained NER) - -SampurNER uses fine-grained tags (e.g. `B-person-politician`, `I-person-artist`). Map **any label containing `person`** (case-insensitive) → `PiiEntityType.Person`. - -English labels remain: `B-PER`, `I-PER`, `B-PERSON`, `I-PERSON`. - ---- - -## 4. Implementation Phases - -### Phase 1 — Generic ONNX token classifier (1–2 days) - -**Objective:** Refactor `OnnxNerModelRunner` so BERT and SentencePiece are pluggable. - -| Action | Location | -|--------|----------| -| Add `ITokenClassifierEncoder` + `EncodedSequence` | `Infrastructure/Onnx/` | -| `BertWordPieceEncoder` — extract from current runner | Infrastructure | -| `SentencePieceEncoder` — IndicBERT tokenizer | Infrastructure | -| `OnnxTokenClassifierRunner` — shared inference + BIO decode | Infrastructure | -| `NerLabelConfig` — English vs Tamil person label predicates | Infrastructure | -| `OnnxAssetPathResolver` — resolve model dir from repo root | Infrastructure | -| Thin wrappers: `EnglishOnnxNerRunner`, `TamilOnnxNerRunner` | Infrastructure | - -**Backward compat:** If `models/en/` missing, fall back to `OnnxModelPath` (`models/ner-model.onnx`). - -**No behavior change** until Phase 3 wiring — existing tests must pass. - ---- - -### Phase 2 — Tamil model download + config (0.5–1 day) - -| Action | Details | -|--------|---------| -| `scripts/download-tamil-ner-model.ps1` + `.py` | Mirror `download-ner-model.ps1`; export via `optimum-cli export onnx --task token-classification` | -| Optional: `scripts/download-all-ner-models.ps1` | Calls English + Tamil scripts | -| Extend `PiiRedactionOptions` | `EnglishOnnxModelPath`, `TamilOnnxModelPath`, `EnableTamilNer` (default `true`) | -| Update `appsettings.json` | New paths under `PiiRedaction` section | -| `.gitignore` | `models/ta/*`, `models/en/*` (same as today for onnx/vocab) | - ---- - -### Phase 3 — Script routing + DI (0.5–1 day) - -| Action | Location | -|--------|----------| -| `ScriptRouter` + `ScriptComposition` enum | `Core/Detection/` | -| `RoutingOnnxNerModelRunner` implements `IOnnxNerModelRunner` | Infrastructure | -| DI registration | `ServiceCollectionExtensions.cs` | - -```csharp -services.AddSingleton(); -services.AddSingleton(); -services.AddSingleton(); -``` - -`OnnxNerPiiDetector` and `CompositePiiDetector` **unchanged**. - ---- - -### Phase 4 — Tests, samples, docs (1 day) - -#### Unit tests - -| Test class | Coverage | -|------------|----------| -| `ScriptRouterTests` | Tamil-only, Latin-only, mixed, no-letters, boundary chars U+0B80/U+0BFF | -| `RoutingOnnxNerModelRunnerTests` | Fake EN/TA runners; mixed script merges both | -| `NerLabelConfigTests` | Tamil fine-grained person labels map correctly | - -#### Real-model tests (`Category=RealModel` or `Category=TamilNer`) - -| Scenario | Input example | Assert | -|----------|---------------|--------| -| Tamil name | `வாடிக்கையாளர் ராஜேஷ் தொலைபேசி 9876543210` | ``, phone redacted | -| Tanglish name | `Customer Senthil phone 9876543210` | person + phone (best-effort) | -| Mixed | `Rajesh மற்றும் Priya` | both persons redacted | -| Clean Tamil | `பணத்தை திரும்பப் பெறுவது எப்படி?` | no false positives | -| Canonical English | existing golden tests | no regression | - -Skip gracefully when `models/ta/model.onnx` missing (mirror `RealNerModelFixture`). - -#### Console samples - -Add to `SamplePromptCatalog.cs`: - -- `TamilCustomerName` — Tamil script person -- `TanglishCustomerName` — `enga peru Rajesh` / `Customer Senthil` -- `MixedTamilEnglish` — code-mixed prompt - -#### Docs - -- Update `README.md` ONNX setup (dual models) -- Update `docs/architecture.md` NER section -- Link this plan from README - ---- - -### Phase 5 — Optional enhancements (post-MVP) - -| Enhancement | Benefit | Effort | -|-------------|---------|--------| -| **Unicode digit normalization** pre-pass | Tamil numerals → ASCII for regex | 0.5 day | -| **Label-based regex** (`பெயர்`, `peru`, `peyar`, `enga peru`) | Tanglish recall without ML | 0.5 day | -| **Tamil name gazetteer** `IPiiDetector` | High precision for top names | 1 day | -| **Fail-closed policy** when NER unavailable | Compliance option | 0.5 day | -| MuRIL model swap | Higher Tamil recall | eval-driven | - ---- - -## 5. Tanglish — Realistic Expectations - -| Input type | Primary handler | Expected recall | -|------------|-----------------|-----------------| -| Tamil script names | Tamil ONNX NER | High (with eval tuning) | -| Standard Latin Indian names (`Ravi Kumar`) | English ONNX NER | High (already works) | -| Tanglish spellings (`Senthil`, `senthil`, `Centhil`) | English NER + optional gazetteer | Medium | -| Code-mixed (`Rajesh oda account ACC-123456`) | English NER + domain regex | Medium–high for IDs; name variable | - -**MVP target:** Tamil script person names reliably redacted; Tanglish improved but not 100% without Phase 5 heuristics. - ---- - -## 6. Success Metrics - -Before marking language gap closed, run an **eval set of 20–30 real prompts** (anonymized production samples): - -| Metric | MVP target | -|--------|------------| -| Tamil script person-name recall | ≥ 85% | -| Tanglish person-name recall | ≥ 70% (with English model + optional heuristics) | -| False positive rate (clean prompts) | ≤ 5% | -| Structured PII (phone/PAN/domain) in Tamil prompts | ≥ 95% (regex layer) | -| Regression on English canonical demo | 100% (existing golden tests) | - ---- - -## 7. Files to Create / Modify (checklist) - -### New files - -- `src/PiiRedaction.Core/Detection/ScriptRouter.cs` -- `src/PiiRedaction.Core/Detection/ScriptComposition.cs` -- `src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs` -- `src/PiiRedaction.Infrastructure/Onnx/BertWordPieceEncoder.cs` -- `src/PiiRedaction.Infrastructure/Onnx/SentencePieceEncoder.cs` -- `src/PiiRedaction.Infrastructure/Onnx/ITokenClassifierEncoder.cs` -- `src/PiiRedaction.Infrastructure/Onnx/NerLabelConfig.cs` -- `src/PiiRedaction.Infrastructure/Onnx/OnnxAssetPathResolver.cs` -- `src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs` -- `src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs` -- `src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs` -- `scripts/download-tamil-ner-model.ps1` / `.py` -- `tests/.../ScriptRouterTests.cs` -- `tests/.../RoutingOnnxNerModelRunnerTests.cs` -- `tests/.../RealTamilNerPipelineTests.cs` -- `tests/TestSupport.Shared/RealTamilModelFixture.cs` - -### Modified files - -- `PiiRedactionOptions.cs` — dual model paths -- `ServiceCollectionExtensions.cs` — routing DI -- `appsettings.json` — config -- `SamplePromptCatalog.cs` — Tamil/Tanglish demos -- `ProductionPipelineFactory.cs` — `CreateWithRoutingRealModel()` for tests -- `RealNerModelFixture.cs` / paths — support EN + TA -- `README.md`, `docs/architecture.md` -- `.gitignore` — `models/en/`, `models/ta/` - -### Unchanged (by design) - -- `PromptSanitizer`, `PlaceholderPiiRedactor`, `CompositePiiDetector` -- `RegexPiiDetector`, `DomainRulePiiDetector` -- `MockLlmPromptService` / LLM boundary - ---- - -## 8. Rollout & Risk - -| Risk | Mitigation | -|------|------------| -| Tamil model export fails on Windows | PowerShell fallback downloads pre-exported ONNX from Hugging Face | -| Larger memory (two models) | Lazy-load Tamil runner only when `EnableTamilNer` and Tamil script detected | -| Fine-grained label mismatch | Load labels from `ner-labels.txt`; unit test label config | -| Tanglish disappointment | Set stakeholder expectation in README; Phase 5 heuristics | -| CI without models | Fast tests use fakes; `Category=TamilNer` skips like `RealModel` | - -**Feature flag:** `EnableTamilNer=false` reverts to English-only behavior for gradual rollout. - ---- - -## 9. Command Reference (after implementation) - -```powershell -# Download both models -.\scripts\download-ner-model.ps1 -.\scripts\download-tamil-ner-model.ps1 - -# Run Tamil-focused console sample -dotnet run --project src/PiiRedaction.ConsoleApp -- --name TamilCustomerName - -# Tests -dotnet test -dotnet test --filter "Category=TamilNer" -dotnet test --filter "Category=RealModel" -``` - ---- - -## 10. Approval Checklist - -- [ ] Stakeholder sign-off on dual-model approach (vs single multilingual model) -- [ ] Tamil eval prompt set collected (20–30 samples) -- [ ] Xenovex CI policy: models downloaded in pipeline or tests skip -- [x] Phase 1–3 implementation PR -- [ ] Phase 4 eval metrics met -- [ ] Optional Phase 5 for Tanglish heuristics if recall < 70% - ---- - -**Next step:** Implement Phase 1 in a feature branch (`feature/tamil-tanglish-ner`), open PR to `main` on `xts.xenovex.com/Bilal-Nazer-Ali/llm-pii-poc`. diff --git a/scripts/tamil-ner-diagnostic/Program.cs b/scripts/tamil-ner-diagnostic/Program.cs index 66a3a5d..ac8917f 100644 --- a/scripts/tamil-ner-diagnostic/Program.cs +++ b/scripts/tamil-ner-diagnostic/Program.cs @@ -21,7 +21,7 @@ foreach (var t in tokens) Console.WriteLine($"{t.Id}\t{t.Value}"); var runnerOptions = Options.Create(new PiiRedactionOptions { TamilOnnxModelPath = Path.Combine(modelDir, "model.onnx") }); using var runner = new TamilOnnxNerRunner(runnerOptions, NullLogger.Instance); Console.WriteLine($"Available: {runner.IsModelAvailable}"); -foreach (var e in runner.PredictEntities(text)) +foreach (var e in runner.PredictEntities(text).Entities) { Console.WriteLine($"Entity: '{e.Value}' [{e.StartIndex},{e.Length}]"); } diff --git a/src/PiiRedaction.Core/Detection/CompositePiiDetector.cs b/src/PiiRedaction.Core/Detection/CompositePiiDetector.cs index fab2a98..3970851 100644 --- a/src/PiiRedaction.Core/Detection/CompositePiiDetector.cs +++ b/src/PiiRedaction.Core/Detection/CompositePiiDetector.cs @@ -7,15 +7,20 @@ namespace PiiRedaction.Core.Detection; /// Aggregates multiple PII detectors and merges overlapping spans. /// Detectors are applied in registration order; earlier detectors win on overlap. /// -public sealed class CompositePiiDetector : IPiiDetector +public sealed class CompositePiiDetector : IPiiDetector, INerRoutingSource { private readonly IReadOnlyList _detectors; + private readonly INerRoutingSource? _nerRoutingSource; public CompositePiiDetector(IEnumerable detectors) { _detectors = detectors.ToList(); + _nerRoutingSource = _detectors.OfType().FirstOrDefault(); } + public IReadOnlyList LastInvokedModels => + _nerRoutingSource?.LastInvokedModels ?? []; + public IReadOnlyList Detect(string text) { ArgumentException.ThrowIfNullOrWhiteSpace(text); diff --git a/src/PiiRedaction.Core/Detection/INerRoutingSource.cs b/src/PiiRedaction.Core/Detection/INerRoutingSource.cs new file mode 100644 index 0000000..d421494 --- /dev/null +++ b/src/PiiRedaction.Core/Detection/INerRoutingSource.cs @@ -0,0 +1,8 @@ +using PiiRedaction.Core.Models; + +namespace PiiRedaction.Core.Detection; + +public interface INerRoutingSource +{ + IReadOnlyList LastInvokedModels { get; } +} diff --git a/src/PiiRedaction.Core/Detection/IOnnxNerModelRunner.cs b/src/PiiRedaction.Core/Detection/IOnnxNerModelRunner.cs index 41094c7..fe37643 100644 --- a/src/PiiRedaction.Core/Detection/IOnnxNerModelRunner.cs +++ b/src/PiiRedaction.Core/Detection/IOnnxNerModelRunner.cs @@ -7,5 +7,5 @@ public interface IOnnxNerModelRunner { bool IsModelAvailable { get; } - IReadOnlyList PredictEntities(string text); + NerPredictionResult PredictEntities(string text); } diff --git a/src/PiiRedaction.Core/Detection/NerEntityTagging.cs b/src/PiiRedaction.Core/Detection/NerEntityTagging.cs new file mode 100644 index 0000000..b9c7747 --- /dev/null +++ b/src/PiiRedaction.Core/Detection/NerEntityTagging.cs @@ -0,0 +1,11 @@ +using PiiRedaction.Core.Models; + +namespace PiiRedaction.Core.Detection; + +public static class NerEntityTagging +{ + public static IReadOnlyList WithOrigin( + IReadOnlyList entities, + NerModelOrigin origin) => + entities.Select(entity => entity with { ModelOrigin = origin }).ToList(); +} diff --git a/src/PiiRedaction.Core/Detection/NerPredictionResult.cs b/src/PiiRedaction.Core/Detection/NerPredictionResult.cs new file mode 100644 index 0000000..2641ff3 --- /dev/null +++ b/src/PiiRedaction.Core/Detection/NerPredictionResult.cs @@ -0,0 +1,10 @@ +using PiiRedaction.Core.Models; + +namespace PiiRedaction.Core.Detection; + +public sealed record NerPredictionResult( + IReadOnlyList Entities, + IReadOnlyList InvokedModels) +{ + public static NerPredictionResult Empty { get; } = new([], []); +} diff --git a/src/PiiRedaction.Core/Detection/OnnxNerPiiDetector.cs b/src/PiiRedaction.Core/Detection/OnnxNerPiiDetector.cs index 05d5e42..f8bbac5 100644 --- a/src/PiiRedaction.Core/Detection/OnnxNerPiiDetector.cs +++ b/src/PiiRedaction.Core/Detection/OnnxNerPiiDetector.cs @@ -8,7 +8,7 @@ namespace PiiRedaction.Core.Detection; /// NER is used for entities that lack rigid formats and vary in surface form across prompts. /// Requires a loaded ONNX model; returns no person entities when the model is unavailable. /// -public sealed class OnnxNerPiiDetector : IPiiDetector +public sealed class OnnxNerPiiDetector : IPiiDetector, INerRoutingSource { private readonly IOnnxNerModelRunner _modelRunner; @@ -17,15 +17,20 @@ public sealed class OnnxNerPiiDetector : IPiiDetector _modelRunner = modelRunner; } + public IReadOnlyList LastInvokedModels { get; private set; } = []; + public IReadOnlyList Detect(string text) { ArgumentException.ThrowIfNullOrWhiteSpace(text); if (!_modelRunner.IsModelAvailable) { + LastInvokedModels = []; return []; } - return _modelRunner.PredictEntities(text); + var result = _modelRunner.PredictEntities(text); + LastInvokedModels = result.InvokedModels; + return result.Entities; } } diff --git a/src/PiiRedaction.Core/Models/NerModelOrigin.cs b/src/PiiRedaction.Core/Models/NerModelOrigin.cs new file mode 100644 index 0000000..8f70eef --- /dev/null +++ b/src/PiiRedaction.Core/Models/NerModelOrigin.cs @@ -0,0 +1,7 @@ +namespace PiiRedaction.Core.Models; + +public enum NerModelOrigin +{ + English, + Tamil +} diff --git a/src/PiiRedaction.Core/Models/PiiEntity.cs b/src/PiiRedaction.Core/Models/PiiEntity.cs index a642546..05432d9 100644 --- a/src/PiiRedaction.Core/Models/PiiEntity.cs +++ b/src/PiiRedaction.Core/Models/PiiEntity.cs @@ -6,7 +6,8 @@ public sealed record PiiEntity( int StartIndex, int Length, PiiDetectionSource Source, - double? Confidence = null) + double? Confidence = null, + NerModelOrigin? ModelOrigin = null) { public int EndIndex => StartIndex + Length; } diff --git a/src/PiiRedaction.Core/Models/SanitizationResult.cs b/src/PiiRedaction.Core/Models/SanitizationResult.cs index 82a745b..3bc9d42 100644 --- a/src/PiiRedaction.Core/Models/SanitizationResult.cs +++ b/src/PiiRedaction.Core/Models/SanitizationResult.cs @@ -4,4 +4,5 @@ public sealed record SanitizationResult( string OriginalPrompt, string SanitizedPrompt, IReadOnlyList DetectedEntities, - RedactionResult Redaction); + RedactionResult Redaction, + IReadOnlyList NerModelsInvoked); diff --git a/src/PiiRedaction.Core/Sanitization/PromptSanitizer.cs b/src/PiiRedaction.Core/Sanitization/PromptSanitizer.cs index e5341d9..3998862 100644 --- a/src/PiiRedaction.Core/Sanitization/PromptSanitizer.cs +++ b/src/PiiRedaction.Core/Sanitization/PromptSanitizer.cs @@ -1,4 +1,5 @@ using PiiRedaction.Core.Abstractions; +using PiiRedaction.Core.Detection; using PiiRedaction.Core.Models; namespace PiiRedaction.Core.Sanitization; @@ -21,11 +22,15 @@ public sealed class PromptSanitizer : IPromptSanitizer var entities = _detector.Detect(request.OriginalPrompt); var redaction = _redactor.Redact(request.OriginalPrompt, entities); + var nerModelsInvoked = _detector is INerRoutingSource routingSource + ? routingSource.LastInvokedModels + : []; return new SanitizationResult( request.OriginalPrompt, redaction.SanitizedText, entities, - redaction); + redaction, + nerModelsInvoked); } } diff --git a/src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs b/src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs index d70d2d9..37b8202 100644 --- a/src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs +++ b/src/PiiRedaction.Infrastructure/Onnx/EnglishOnnxNerRunner.cs @@ -29,7 +29,18 @@ public class EnglishOnnxNerRunner : IOnnxNerModelRunner, IDisposable public bool IsModelAvailable => _runner.IsAvailable; - public IReadOnlyList PredictEntities(string text) => _runner.PredictEntities(text); + public NerPredictionResult PredictEntities(string text) + { + if (!IsModelAvailable) + { + return NerPredictionResult.Empty; + } + + var entities = _runner.PredictEntities(text); + return new NerPredictionResult( + NerEntityTagging.WithOrigin(entities, NerModelOrigin.English), + [NerModelOrigin.English]); + } public void Dispose() => _runner.Dispose(); } diff --git a/src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs b/src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs index 0601ce1..ffc98e4 100644 --- a/src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs +++ b/src/PiiRedaction.Infrastructure/Onnx/OnnxTokenClassifierRunner.cs @@ -50,7 +50,23 @@ public sealed class OnnxTokenClassifierRunner : IDisposable } var predictedLabelIds = RunInference(encoded); - return DecodePersonEntities(text, predictedLabelIds, encoded); + var entities = DecodePersonEntities(text, predictedLabelIds, encoded); + + if (entities.Count == 0) + { + _logger.LogDebug("Decoded 0 person span(s) from {ModelPath}.", _modelPath); + } + else + { + var spanSummary = string.Join(", ", entities.Select(entity => $"\"{entity.Value}\"@{entity.StartIndex}")); + _logger.LogDebug( + "Decoded {Count} person span(s) from {ModelPath}: {Spans}", + entities.Count, + _modelPath, + spanSummary); + } + + return entities; } private int[] RunInference(EncodedSequence encoded) diff --git a/src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs b/src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs index db8196d..427b74d 100644 --- a/src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs +++ b/src/PiiRedaction.Infrastructure/Onnx/RoutingOnnxNerModelRunner.cs @@ -1,3 +1,5 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; using PiiRedaction.Core.Configuration; using PiiRedaction.Core.Detection; @@ -14,39 +16,53 @@ public sealed class RoutingOnnxNerModelRunner : IOnnxNerModelRunner private readonly IOnnxNerModelRunner _englishRunner; private readonly IOnnxNerModelRunner _tamilRunner; private readonly bool _enableTamilNer; + private readonly ILogger _logger; public RoutingOnnxNerModelRunner( EnglishOnnxNerRunner englishRunner, TamilOnnxNerRunner tamilRunner, - IOptions options) - : this(englishRunner, tamilRunner, options.Value.EnableTamilNer) + IOptions options, + ILogger logger) + : this(englishRunner, tamilRunner, options.Value.EnableTamilNer, logger) { } internal RoutingOnnxNerModelRunner( IOnnxNerModelRunner englishRunner, IOnnxNerModelRunner tamilRunner, - bool enableTamilNer) + bool enableTamilNer, + ILogger? logger = null) { _englishRunner = englishRunner; _tamilRunner = tamilRunner; _enableTamilNer = enableTamilNer; + _logger = logger ?? NullLogger.Instance; } public bool IsModelAvailable => _englishRunner.IsModelAvailable || (_enableTamilNer && _tamilRunner.IsModelAvailable); - public IReadOnlyList PredictEntities(string text) + public NerPredictionResult PredictEntities(string text) { var composition = _scriptRouter.GetComposition(text); + _logger.LogDebug("Script composition: {Composition}", composition); + var entities = new List(); + var invoked = new List(); switch (composition) { case ScriptComposition.LatinOnly: if (_englishRunner.IsModelAvailable) { - entities.AddRange(_englishRunner.PredictEntities(text)); + _logger.LogDebug("Invoking English NER model."); + var englishResult = _englishRunner.PredictEntities(text); + entities.AddRange(englishResult.Entities); + invoked.AddRange(englishResult.InvokedModels); + } + else + { + _logger.LogDebug("Skipping English NER model (not available)."); } break; @@ -54,7 +70,18 @@ public sealed class RoutingOnnxNerModelRunner : IOnnxNerModelRunner case ScriptComposition.TamilOnly: if (_enableTamilNer && _tamilRunner.IsModelAvailable) { - entities.AddRange(_tamilRunner.PredictEntities(text)); + _logger.LogDebug("Invoking Tamil NER model."); + var tamilResult = _tamilRunner.PredictEntities(text); + entities.AddRange(tamilResult.Entities); + invoked.AddRange(tamilResult.InvokedModels); + } + else if (!_enableTamilNer) + { + _logger.LogDebug("Skipping Tamil NER model (disabled in options)."); + } + else + { + _logger.LogDebug("Skipping Tamil NER model (not available)."); } break; @@ -62,21 +89,46 @@ public sealed class RoutingOnnxNerModelRunner : IOnnxNerModelRunner case ScriptComposition.Mixed: if (_englishRunner.IsModelAvailable) { - entities.AddRange(_englishRunner.PredictEntities(text)); + _logger.LogDebug("Invoking English NER model (mixed script)."); + var englishResult = _englishRunner.PredictEntities(text); + entities.AddRange(englishResult.Entities); + invoked.AddRange(englishResult.InvokedModels); + } + else + { + _logger.LogDebug("Skipping English NER model (not available, mixed script)."); } if (_enableTamilNer && _tamilRunner.IsModelAvailable) { - entities.AddRange(_tamilRunner.PredictEntities(text)); + _logger.LogDebug("Invoking Tamil NER model (mixed script)."); + var tamilResult = _tamilRunner.PredictEntities(text); + entities.AddRange(tamilResult.Entities); + invoked.AddRange(tamilResult.InvokedModels); + } + else if (!_enableTamilNer) + { + _logger.LogDebug("Skipping Tamil NER model (disabled in options, mixed script)."); + } + else + { + _logger.LogDebug("Skipping Tamil NER model (not available, mixed script)."); } break; case ScriptComposition.NoLetters: + _logger.LogDebug("No letters detected; skipping all NER models."); break; } - return MergePersonSpans(entities); + var merged = MergePersonSpans(entities); + _logger.LogInformation( + "NER routing complete: {EntityCount} person span(s) from [{InvokedModels}].", + merged.Count, + invoked.Count == 0 ? "none" : string.Join(", ", invoked)); + + return new NerPredictionResult(merged, invoked); } internal static IReadOnlyList MergePersonSpans(IReadOnlyList entities) diff --git a/src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs b/src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs index 54c4150..943da87 100644 --- a/src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs +++ b/src/PiiRedaction.Infrastructure/Onnx/TamilOnnxNerRunner.cs @@ -21,7 +21,18 @@ public sealed class TamilOnnxNerRunner : IOnnxNerModelRunner, IDisposable public bool IsModelAvailable => _runner.IsAvailable; - public IReadOnlyList PredictEntities(string text) => _runner.PredictEntities(text); + public NerPredictionResult PredictEntities(string text) + { + if (!IsModelAvailable) + { + return NerPredictionResult.Empty; + } + + var entities = _runner.PredictEntities(text); + return new NerPredictionResult( + NerEntityTagging.WithOrigin(entities, NerModelOrigin.Tamil), + [NerModelOrigin.Tamil]); + } public void Dispose() => _runner.Dispose(); } diff --git a/src/PiiRedaction.TestHarness.Wpf/App.xaml b/src/PiiRedaction.TestHarness.Wpf/App.xaml index 77fe965..5b4b5ce 100644 --- a/src/PiiRedaction.TestHarness.Wpf/App.xaml +++ b/src/PiiRedaction.TestHarness.Wpf/App.xaml @@ -9,6 +9,7 @@ + diff --git a/src/PiiRedaction.TestHarness.Wpf/App.xaml.cs b/src/PiiRedaction.TestHarness.Wpf/App.xaml.cs index bdb531d..83c89dd 100644 --- a/src/PiiRedaction.TestHarness.Wpf/App.xaml.cs +++ b/src/PiiRedaction.TestHarness.Wpf/App.xaml.cs @@ -3,7 +3,9 @@ using System.Windows; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; using PiiRedaction.TestHarness.Wpf.DependencyInjection; +using PiiRedaction.TestHarness.Wpf.Logging; using PiiRedaction.TestHarness.Wpf.Services; using PiiRedaction.TestHarness.Wpf.ViewModels; @@ -24,6 +26,11 @@ public partial class App : Application configuration.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true); configuration.AddEnvironmentVariables(); }) + .ConfigureLogging((context, logging) => + { + logging.AddConfiguration(context.Configuration.GetSection("Logging")); + logging.AddFilter("PiiRedaction.Infrastructure.Onnx", LogLevel.Debug); + }) .ConfigureServices((context, services) => { services.AddPiiRedactionServices(context.Configuration); @@ -38,6 +45,9 @@ public partial class App : Application Directory.SetCurrentDirectory(AppContext.BaseDirectory); + var loggerFactory = _host.Services.GetRequiredService(); + loggerFactory.AddProvider(new UiLoggerProvider(_host.Services.GetRequiredService())); + await _host.StartAsync().ConfigureAwait(true); var mainWindow = _host.Services.GetRequiredService(); diff --git a/src/PiiRedaction.TestHarness.Wpf/Converters/ValueConverters.cs b/src/PiiRedaction.TestHarness.Wpf/Converters/ValueConverters.cs index cf7d4af..c02f547 100644 --- a/src/PiiRedaction.TestHarness.Wpf/Converters/ValueConverters.cs +++ b/src/PiiRedaction.TestHarness.Wpf/Converters/ValueConverters.cs @@ -49,6 +49,31 @@ public sealed class StringNotEmptyToVisibilityConverter : IValueConverter throw new NotSupportedException(); } +public sealed class NerInvokedToBrushConverter : IValueConverter +{ + public object Convert(object? value, Type targetType, object? parameter, CultureInfo culture) + { + if (value is not string summary) + { + return new SolidColorBrush(Color.FromRgb(107, 114, 128)); + } + + return summary switch + { + "—" or "None" => new SolidColorBrush(Color.FromRgb(107, 114, 128)), + "English" => new SolidColorBrush(Color.FromRgb(37, 99, 235)), + "Tamil" => new SolidColorBrush(Color.FromRgb(124, 58, 237)), + _ when summary.Contains("English", StringComparison.Ordinal) + && summary.Contains("Tamil", StringComparison.Ordinal) => + new SolidColorBrush(Color.FromRgb(217, 119, 6)), + _ => new SolidColorBrush(Color.FromRgb(107, 114, 128)) + }; + } + + public object ConvertBack(object? value, Type targetType, object? parameter, CultureInfo culture) => + throw new NotSupportedException(); +} + public sealed class PassFailBrushConverter : IValueConverter { public object Convert(object? value, Type targetType, object? parameter, CultureInfo culture) diff --git a/src/PiiRedaction.TestHarness.Wpf/DependencyInjection/ServiceCollectionExtensions.cs b/src/PiiRedaction.TestHarness.Wpf/DependencyInjection/ServiceCollectionExtensions.cs index 24fd93d..4d63de5 100644 --- a/src/PiiRedaction.TestHarness.Wpf/DependencyInjection/ServiceCollectionExtensions.cs +++ b/src/PiiRedaction.TestHarness.Wpf/DependencyInjection/ServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.AI; +using PiiRedaction.TestHarness.Wpf.Services; using PiiRedaction.Core.Abstractions; using PiiRedaction.Core.Configuration; using PiiRedaction.Core.Detection; @@ -15,6 +16,7 @@ public static class ServiceCollectionExtensions { public static IServiceCollection AddPiiRedactionServices(this IServiceCollection services, IConfiguration configuration) { + services.AddSingleton(); services.Configure(configuration.GetSection(PiiRedactionOptions.SectionName)); services.AddSingleton(); diff --git a/src/PiiRedaction.TestHarness.Wpf/Logging/UiLogger.cs b/src/PiiRedaction.TestHarness.Wpf/Logging/UiLogger.cs new file mode 100644 index 0000000..3da63a2 --- /dev/null +++ b/src/PiiRedaction.TestHarness.Wpf/Logging/UiLogger.cs @@ -0,0 +1,43 @@ +using Microsoft.Extensions.Logging; +using PiiRedaction.TestHarness.Wpf.Services; + +namespace PiiRedaction.TestHarness.Wpf.Logging; + +public sealed class UiLogger : ILogger +{ + private readonly string _category; + private readonly INerLogService _logService; + + public UiLogger(string category, INerLogService logService) + { + _category = category; + _logService = logService; + } + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => logLevel != LogLevel.None; + + public void Log( + LogLevel logLevel, + EventId eventId, + TState state, + Exception? exception, + Func formatter) + { + if (!IsEnabled(logLevel)) + { + return; + } + + var message = formatter(state, exception); + if (exception is not null) + { + message = $"{message} ({exception.Message})"; + } + + var timestamp = DateTime.Now.ToString("HH:mm:ss.fff"); + var line = $"[{timestamp}] [{logLevel}] {_category}: {message}"; + _logService.Append(line); + } +} diff --git a/src/PiiRedaction.TestHarness.Wpf/Logging/UiLoggerProvider.cs b/src/PiiRedaction.TestHarness.Wpf/Logging/UiLoggerProvider.cs new file mode 100644 index 0000000..31b5893 --- /dev/null +++ b/src/PiiRedaction.TestHarness.Wpf/Logging/UiLoggerProvider.cs @@ -0,0 +1,26 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using PiiRedaction.TestHarness.Wpf.Services; + +namespace PiiRedaction.TestHarness.Wpf.Logging; + +public sealed class UiLoggerProvider : ILoggerProvider +{ + private const string NerCategoryPrefix = "PiiRedaction.Infrastructure.Onnx"; + + private readonly INerLogService _logService; + + public UiLoggerProvider(INerLogService logService) => _logService = logService; + + public ILogger CreateLogger(string categoryName) => + IsNerCategory(categoryName) + ? new UiLogger(categoryName, _logService) + : NullLogger.Instance; + + internal static bool IsNerCategory(string categoryName) => + categoryName.StartsWith(NerCategoryPrefix, StringComparison.Ordinal); + + public void Dispose() + { + } +} diff --git a/src/PiiRedaction.TestHarness.Wpf/MainWindow.xaml b/src/PiiRedaction.TestHarness.Wpf/MainWindow.xaml index 66c80e8..57fc737 100644 --- a/src/PiiRedaction.TestHarness.Wpf/MainWindow.xaml +++ b/src/PiiRedaction.TestHarness.Wpf/MainWindow.xaml @@ -35,6 +35,14 @@ Foreground="White" FontWeight="SemiBold" /> + + + + @@ -67,10 +75,15 @@ FontSize="16" FontWeight="SemiBold" Margin="0,0,0,8" /> + + ToolTip="Filter by name, topic, category, language, or description" />