AI PII Scanner

An internal security tool for discovering Personally Identifiable Information (PII) in PostgreSQL databases and Jira projects

Overview

The AI PII Scanner is a comprehensive, security-focused tool designed for Application Security and Privacy teams to discover and catalog PII. It supports two scan targets:

• PostgreSQL databases — column-level PII discovery across your application schemas
• Jira — real-time or scheduled scanning of issues and comments for accidental PII exposure

Detection uses a three-layer pipeline:

• Regex-based detection — Deterministic patterns for email, phone, SSN, credit cards, etc.
• NER/ML detection — Microsoft Presidio or spaCy for entity recognition (names, locations, dates)
• Optional LLM classification — Self-hosted models for ambiguous free-text analysis

Key Features

• Multi-layered PII Detection — Combines regex, ML, and optional LLM for comprehensive coverage
• Jira Integration — Scan new issues and comments via polling or real-time webhooks
• Smart Exception Filters — Suppress expected content: @mentions and company email addresses
• Timestamped Reports — Jira scan results auto-saved to a dated JSON or CSV file on every run
• Security-First Design — Read-only database access, no external API calls, minimal PII retention
• Intelligent Sampling — Analyzes representative data samples to minimize performance impact
• Comprehensive Audit Trail — Every scan logged with configuration, timestamps, and model versions
• Privacy-Safe Logging — Automatic PII redaction in all logs and stored samples

Security & Privacy

CRITICAL: This tool is designed for internal use only within a secure, controlled environment.

Security Control	Implementation
Data Exfiltration Prevention	No third-party API calls; LLM endpoints must be self-hosted
Least Privilege Access	Read-only database role with no write access to application data
PII Minimization	Findings store only metadata and heavily redacted samples
Audit Trail	All scans logged with who/what/when/how details
Secure Logging	Structured logs with automatic PII field redaction
Encrypted Transit	SSL/TLS required for all database connections
Webhook Verification	HMAC-SHA256 signature validation for incoming Jira events

Table of Contents

• Installation — Database Schema Scan
• Installation — Jira Scan
• Configuration
• Usage — Database Scanning
• Usage — Jira Integration
• Architecture
• Detection Methods
• Database Schema
• Examples
• Troubleshooting
• Advanced Topics
• Contributing

---

Installation — Database Schema Scan

Use this if you want to scan a PostgreSQL database for PII across columns.

Step 1 — Base install

# Clone the repository
cd PIIHunter

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Download spaCy NER model (~800MB)
python -m spacy download en_core_web_lg

Step 2 — Create a read-only database role

Run this as a PostgreSQL superuser:

CREATE ROLE pii_scanner_readonly WITH LOGIN PASSWORD 'your_secure_password';
GRANT CONNECT ON DATABASE your_database TO pii_scanner_readonly;
GRANT USAGE ON SCHEMA public, app_data TO pii_scanner_readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA public, app_data TO pii_scanner_readonly;

Step 3 — Configure environment

cp .env.example .env
# Fill in DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD

Step 4 — Initialize metadata schema

Creates the security_pii schema where scan results are stored. Run once:

python -m pii_scanner.cli init-schema

Step 5 — Run your first scan

# Preview candidate columns (no data read)
python -m pii_scanner.cli discover

# Run PII detection
python -m pii_scanner.cli scan --sample-size 1000

# View results
python -m pii_scanner.cli report

Verify connectivity at any time

python -m pii_scanner.cli test-connection

---

Installation — Jira Scan

Use this if you want to scan Jira issues and comments for accidental PII exposure.

The base install (Step 1 above) is required first. Database setup (Steps 2–4) is not required if you only want Jira scanning.

Subsection A — REST API (Polling)

Polls Jira on demand for activity in the last N hours. No server needed.

Step 1 — Base install (if not done already)

cd PIIHunter
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_lg

Step 2 — Generate a Jira API token

1. Go to https://id.atlassian.com → Security → API tokens
2. Click Create API token, give it a name, copy the token

Step 3 — Configure environment

cp .env.example .env

Edit .env and fill in your Jira details:

JIRA_URL=https://yourcompany.atlassian.net
[email protected]
JIRA_API_TOKEN=your_api_token_here

# Optional — leave empty to scan all projects
# JIRA_PROJECT_KEYS=ENG,OPS,SUPPORT

# Emails from these domains will NOT be flagged as PII
COMPANY_EMAIL_DOMAINS=yourcompany.com

Step 4 — Test the connection

Verifies your credentials, lists accessible projects, and fetches a sample of recent issues:

python -m pii_scanner.cli jira test

Expected output:

Testing connection to https://yourcompany.atlassian.net...

  Auth          ✓  logged in as Jane Smith ([email protected])
  Projects      ✓  3 accessible: ENG, OPS, SUPPORT
  Project ENG   ✓  accessible

  Recent issues (no time filter):
    ENG-101  Fix login timeout on mobile
    ENG-100  Update password policy

If this fails, check your API token and that JIRA_EMAIL matches the account the token was generated for.

Step 5 — Run a scan

# Scan last 24 hours
python -m pii_scanner.cli jira scan

# Custom lookback window
python -m pii_scanner.cli jira scan --hours 48

---

Subsection B — Webhook (Real-Time)

Starts an HTTP server that receives Jira events and scans them instantly as they arrive. Requires a publicly reachable endpoint.

Step 1 — Base install + REST API setup (complete Subsection A steps 1–4 first)

Step 2 — Install webhook dependencies

pip install fastapi uvicorn

Step 3 — Add webhook config to .env

# Secret used to verify requests genuinely come from Jira
JIRA_WEBHOOK_SECRET=your_webhook_secret
JIRA_WEBHOOK_PORT=8080

Step 4 — Start the webhook server

python -m pii_scanner.cli jira webhook

# Custom port
python -m pii_scanner.cli jira webhook --port 9090

Step 5 — Register the webhook in Jira

1. In Jira go to Project Settings → Webhooks → Create webhook
2. Set URL to http://yourserver:8080/webhook
3. Under Events select:
   • Issue → created
   • Comment → created
4. Set the secret to match JIRA_WEBHOOK_SECRET in your .env
5. Save

Jira will now POST to your server for every new issue and comment. The scanner processes each one immediately.

Available endpoints:

Endpoint	Purpose
POST /webhook	Receives Jira events
GET /health	Confirms server is running

Production note: Do not expose uvicorn directly to the internet. See the Production Webhook Setup section under Advanced Topics.

---

Configuration