Add PII redaction POC for secure LLM prompting.
Implements detect-redact-sanitize pipeline with regex, domain rules, and ONNX NER before the LLM boundary, plus NUnit tests and Xenovex push documentation. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Bilal Nazer Ali
281
README.md
Normal file
281
README.md
Normal file
@@ -0,0 +1,281 @@
|
||||
# PII Redaction POC
|
||||
|
||||
A proof-of-concept .NET solution that redacts personally identifiable information (PII) from user prompts **before** sending them to a large language model (LLM). The design demonstrates enterprise-grade separation of concerns using SOLID principles, dependency injection, and the `Microsoft.Extensions.AI` abstractions.
|
||||
|
||||
## Purpose
|
||||
|
||||
Financial and customer-service prompts often contain regulated data (names, government IDs, account numbers). This POC shows how to:
|
||||
|
||||
1. Accept a console prompt
|
||||
2. Detect PII using **Regex**, **ONNX NER**, and **domain rules**
|
||||
3. Replace values with stable placeholders
|
||||
4. Send only the **sanitized** prompt to an LLM (mocked for now)
|
||||
|
||||
## Architecture
|
||||
|
||||
For solution design, data-flow diagrams, trust boundaries, and project responsibilities, see **[docs/architecture.md](docs/architecture.md)**.
|
||||
|
||||
## Why Three Detection Strategies?
|
||||
|
||||
| Strategy | Used For | Rationale |
|
||||
|----------|----------|-----------|
|
||||
| **Regex** | Email, phone, PAN, Aadhaar, credit card | Deterministic, format-bound identifiers with stable rules that are easy to audit and test |
|
||||
| **ONNX NER** | Person names | Contextual entities without rigid formats; names vary widely in surface form |
|
||||
| **Domain rules** | Loan number, customer ID, account number | Business-specific identifiers defined by internal systems, not inferable from generic models alone |
|
||||
|
||||
## Why the LLM Receives Only Sanitized Text
|
||||
|
||||
The placeholder map (`<PERSON_1>` → original value) is kept **in-process** for audit or downstream de-tokenization. Only the sanitized prompt crosses the LLM boundary. This reduces data-exposure risk and supports compliance requirements for regulated workloads.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── PiiRedaction.ConsoleApp/ # Presentation: input/output, DI bootstrap
|
||||
├── PiiRedaction.Core/ # Business logic: detection, redaction, models
|
||||
└── PiiRedaction.Infrastructure/ # Technical adapters: ONNX Runtime, mock LLM
|
||||
models/ # Optional ONNX model files (gitignored)
|
||||
```
|
||||
|
||||
| Project | Responsibility |
|
||||
|---------|----------------|
|
||||
| `PiiRedaction.ConsoleApp` | Read prompt, call sanitizer, display results, call LLM service |
|
||||
| `PiiRedaction.Core` | PII detection abstractions, redaction, sanitization orchestration |
|
||||
| `PiiRedaction.Infrastructure` | ONNX model runner, `IChatClient` mock implementation |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [.NET SDK](https://dotnet.microsoft.com/download) 10.x (or compatible SDK for `net10.0`)
|
||||
- **ONNX NER model** for person-name detection (see [ONNX Model Setup](#onnx-model-setup))
|
||||
- Python 3.10+ (only for the model download script)
|
||||
|
||||
> **Note:** This environment targets `net10.0` because .NET 10 SDK is installed. The architecture is identical to the planned .NET 9 layout; change `TargetFramework` in `.csproj` files if you use .NET 9 SDK.
|
||||
|
||||
## Build and Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
dotnet restore
|
||||
dotnet build
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp
|
||||
```
|
||||
|
||||
By default the console app runs **11 curated sample prompts** covering NER/person names, regex identifiers, domain IDs, combined scenarios, and a clean no-PII ticket.
|
||||
|
||||
List available samples:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp -- --list
|
||||
```
|
||||
|
||||
Run a single sample by index or name:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp -- --sample 2
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp -- --name MrTitlePerson
|
||||
```
|
||||
|
||||
Interactive mode (enter your own prompt):
|
||||
|
||||
```bash
|
||||
dotnet run --project src/PiiRedaction.ConsoleApp -- --interactive
|
||||
```
|
||||
|
||||
### Console sample catalog
|
||||
|
||||
Samples are defined in [`SamplePromptCatalog.cs`](src/PiiRedaction.ConsoleApp/Samples/SamplePromptCatalog.cs).
|
||||
|
||||
| # | Name | Category | NER / Person example |
|
||||
|---|------|----------|----------------------|
|
||||
| 0 | FullFinancialWithCustomer | NER + Regex + Domain | `Customer Ravi Kumar` + email, phone, loan, PAN |
|
||||
| 1 | CustomerNameOnly | NER | `Customer Anita Sharma` |
|
||||
| 2 | MrTitlePerson | NER | `Mr. John Smith` |
|
||||
| 3 | MrsTitlePerson | NER | `Mrs. Lakshmi Reddy` |
|
||||
| 4 | DrTitlePerson | NER | `Dr. Jane Doe` |
|
||||
| 5 | TwoCustomersInOnePrompt | NER | `Customer Ravi Kumar` and `Customer Priya Nair` |
|
||||
| 6 | PersonWithDomainIds | NER + Domain | `Customer Meera Iyer` + CID / ACC |
|
||||
| 7 | PersonWithEmailNoPhone | NER + Regex | `Customer Arjun Mehta` + email |
|
||||
| 8 | AllRegexTypes | Regex | email, phone, PAN, Aadhaar, card |
|
||||
| 9 | AllDomainIds | Domain | LN, CID, ACC |
|
||||
| 10 | NoPiiCleanTicket | Negative | no redaction |
|
||||
|
||||
Person names are detected via **ONNX NER** using `dslim/bert-base-NER` (or a compatible token-classification export). A real model is **required** for person-name detection; there is no regex or heuristic fallback.
|
||||
|
||||
## NuGet Packages
|
||||
|
||||
```bash
|
||||
# Infrastructure
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.ML.OnnxRuntime
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.ML.Tokenizers
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.Extensions.AI.Abstractions
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.Extensions.AI
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.Extensions.Logging.Abstractions
|
||||
dotnet add src/PiiRedaction.Infrastructure package Microsoft.Extensions.Options
|
||||
|
||||
# Core
|
||||
dotnet add src/PiiRedaction.Core package Microsoft.Extensions.Options
|
||||
|
||||
# ConsoleApp
|
||||
dotnet add src/PiiRedaction.ConsoleApp package Microsoft.Extensions.Hosting
|
||||
dotnet add src/PiiRedaction.ConsoleApp package Microsoft.Extensions.DependencyInjection
|
||||
dotnet add src/PiiRedaction.ConsoleApp package Microsoft.Extensions.Configuration.Json
|
||||
dotnet add src/PiiRedaction.ConsoleApp package Microsoft.Extensions.Configuration.EnvironmentVariables
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
[`appsettings.json`](src/PiiRedaction.ConsoleApp/appsettings.json):
|
||||
|
||||
```json
|
||||
{
|
||||
"PiiRedaction": {
|
||||
"OnnxModelPath": "models/ner-model.onnx"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| `OnnxModelPath` | Path to ONNX NER model (relative to working directory or discovered by walking up from the current directory) |
|
||||
|
||||
## ONNX Model Setup
|
||||
|
||||
Person-name detection requires a token-classification ONNX model and companion tokenizer files in the `models/` directory:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `models/ner-model.onnx` | Exported NER model |
|
||||
| `models/vocab.txt` | BERT WordPiece vocabulary |
|
||||
| `models/ner-labels.txt` | One BIO label per line (`O`, `B-PER`, `I-PER`, etc.) |
|
||||
|
||||
### Download script
|
||||
|
||||
From the repository root:
|
||||
|
||||
```powershell
|
||||
.\scripts\download-ner-model.ps1
|
||||
```
|
||||
|
||||
Or with Python directly:
|
||||
|
||||
```bash
|
||||
python scripts/download-ner-model.py
|
||||
```
|
||||
|
||||
The script exports [`dslim/bert-base-NER`](https://huggingface.co/dslim/bert-base-NER) via Hugging Face Optimum when Python is available. Otherwise it downloads the pre-exported ONNX assets from Hugging Face directly.
|
||||
|
||||
### Inference pipeline
|
||||
|
||||
`OnnxNerModelRunner` performs the full pipeline:
|
||||
|
||||
- BERT WordPiece tokenization (`Microsoft.ML.Tokenizers`)
|
||||
- ONNX Runtime inference (`input_ids`, `attention_mask`, optional `token_type_ids`)
|
||||
- BIO label decoding (`B-PER` / `I-PER` → `PiiEntityType.Person`)
|
||||
- Character-span alignment back to the source text
|
||||
|
||||
When the model or tokenizer files are missing, person detection returns no results.
|
||||
|
||||
## Swapping Mock LLM for Azure OpenAI
|
||||
|
||||
The application depends on `ILlmPromptService` (Core) and `IChatClient` (Microsoft.Extensions.AI). To use Azure OpenAI later, replace the mock registration in [`ServiceCollectionExtensions.cs`](src/PiiRedaction.ConsoleApp/DependencyInjection/ServiceCollectionExtensions.cs):
|
||||
|
||||
```csharp
|
||||
// Remove:
|
||||
// services.AddSingleton<IChatClient, MockChatClient>();
|
||||
|
||||
// Add (example — package and API may vary by provider SDK version):
|
||||
// services.AddAzureOpenAIChatClient(
|
||||
// new Uri(configuration["AzureOpenAI:Endpoint"]!),
|
||||
// configuration["AzureOpenAI:ApiKey"]!,
|
||||
// configuration["AzureOpenAI:DeploymentName"]!);
|
||||
|
||||
services.AddSingleton<ILlmPromptService, MockLlmPromptService>(); // unchanged
|
||||
```
|
||||
|
||||
`MockLlmPromptService` already uses `IChatClient`, so it works with any registered chat client implementation.
|
||||
|
||||
## Sample Execution Output
|
||||
|
||||
```
|
||||
=== PII Redaction POC ===
|
||||
|
||||
Original Prompt:
|
||||
Customer Ravi Kumar with email ravi.kumar@gmail.com and phone 9876543210 has LoanNumber LN-456789 and PAN ABCDE1234F. Please summarize this customer issue.
|
||||
|
||||
Detected PII:
|
||||
[PERSON ] Ravi Kumar (Ner)
|
||||
[EMAIL ] ravi.kumar@gmail.com (Regex)
|
||||
[PHONE ] 9876543210 (Regex)
|
||||
[LOAN_NUMBER ] LN-456789 (Domain)
|
||||
[PAN ] ABCDE1234F (Regex)
|
||||
|
||||
Sanitized Prompt:
|
||||
Customer <PERSON_1> with email <EMAIL_1> and phone <PHONE_1> has LoanNumber <LOAN_NUMBER_1> and PAN <PAN_1>. Please summarize this customer issue.
|
||||
|
||||
Internal Placeholder Map (not sent to LLM):
|
||||
<EMAIL_1> -> ravi.kumar@gmail.com
|
||||
<LOAN_NUMBER_1> -> LN-456789
|
||||
<PAN_1> -> ABCDE1234F
|
||||
<PERSON_1> -> Ravi Kumar
|
||||
<PHONE_1> -> 9876543210
|
||||
|
||||
Mock LLM Response:
|
||||
[Mock LLM Response] Received sanitized prompt (146 chars). No original PII was transmitted.
|
||||
```
|
||||
|
||||
## SOLID Principles Applied
|
||||
|
||||
| Principle | Application |
|
||||
|-----------|-------------|
|
||||
| **Single Responsibility** | Each detector, redactor, and runner has one job; `Program.cs` only orchestrates |
|
||||
| **Open/Closed** | Add new `IPiiDetector` implementations without changing merge logic |
|
||||
| **Liskov Substitution** | All detectors are interchangeable via `IPiiDetector` |
|
||||
| **Interface Segregation** | Separate interfaces for detection, redaction, sanitization, and LLM |
|
||||
| **Dependency Inversion** | Core defines abstractions; Infrastructure implements them |
|
||||
|
||||
## Testing
|
||||
|
||||
The solution includes an **NUnit** test suite across two projects:
|
||||
|
||||
| Project | Focus |
|
||||
|---------|-------|
|
||||
| `tests/PiiRedaction.Core.Tests` | Detectors, redactor, sanitizer, golden pipeline scenarios (fake NER), real-model integration tests |
|
||||
| `tests/PiiRedaction.Infrastructure.Tests` | Mock LLM, ONNX runner unit tests, real-model NER runner tests |
|
||||
|
||||
### Run tests
|
||||
|
||||
```bash
|
||||
dotnet test
|
||||
dotnet test --filter "FullyQualifiedName~GoldenPromptTests"
|
||||
dotnet test --filter "Category=RealModel"
|
||||
dotnet test --logger "console;verbosity=detailed"
|
||||
```
|
||||
|
||||
Fast CI runs without the ONNX model: fake-based tests always execute; tests marked **`Category=RealModel`** are skipped when `models/ner-model.onnx` is absent. Download the model first:
|
||||
|
||||
```powershell
|
||||
.\scripts\download-ner-model.ps1
|
||||
```
|
||||
|
||||
### Test architecture
|
||||
|
||||
- **`PromptScenarioCatalog`** — five focused end-to-end scenarios (canonical demo, multi-regex, duplicate people, overlap stress, no-PII negative)
|
||||
- **`ProductionPipelineFactory`** — builds the same Domain → Regex → OnnxNer composite stack as production DI; `CreateWithRealModel(runner)` wires a real `OnnxNerModelRunner`
|
||||
- **`FakeOnnxNerModelRunner`** — unit-test double for NER; golden tests inject person spans per scenario
|
||||
- **`GoldenPromptTests`** — end-to-end sanitization proof across the catalog (fake NER)
|
||||
- **`RealNerModelFixture`** — shared fixture that loads `models/ner-model.onnx` once per class; skips when model missing
|
||||
- **`RealNerModelRunnerTests`** — direct ONNX inference with span accuracy checks
|
||||
- **`RealNerPipelineTests`** — full pipeline with real NER (canonical, multi-person, clean-ticket negative)
|
||||
- **`OnnxNerModelRunnerTests`** — unit tests for missing/invalid model paths (no download required)
|
||||
- **`CompositePiiDetectorTests`** — overlap merge and source-priority rules
|
||||
- **`LlmBoundaryTests`** — verifies raw PII never appears in outbound LLM messages
|
||||
|
||||
Assertions use **FluentAssertions** for readable failures on long prompt strings.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- ASP.NET Core API host with request/response middleware
|
||||
- Persistent audit log of redaction events (without storing raw PII)
|
||||
- Secure vault for reversible tokenization
|
||||
- Real Azure OpenAI / OpenAI provider registration
|
||||
Reference in New Issue
Block a user