Python CLI tool for rapid IOC analysis (IPs, Domains, CVEs) using 6 free Threat Intel APIs. Outputs: Color-coded Excel, JSON, CSV. Uses: VT, Shodan, AbuseIPDB.
Investigate IPs, domains, hashes, and CVEs across 6 free threat intel APIs — without switching between browser tabs.
Quick Start · Usage · Architecture · API Keys · Screenshots · Contributing
🚀 Proudly featured in the official Awesome OSINT repository.
ThreatLens is a single command-line tool that unifies threat intelligence lookups across the most trusted free OSINT sources. Instead of pasting an IP into five different websites, ThreatLens queries them all in parallel, normalizes the results, and gives you a clear verdict — in the terminal, or in a polished, color-coded Excel/JSON/CSV report.
Built for SOC analysts, incident responders, threat hunters, and anyone who wants fast, reliable IOC enrichment without leaving the shell.
|
Why ThreatLens
|
Not for
|
| Feature | Details |
|---|---|
| 🎯 IOC Types | IP, Domain, URL, File Hash (MD5 / SHA1 / SHA256), CVE |
| 🔌 Integrated APIs | AbuseIPDB, VirusTotal, AlienVault OTX, Shodan, URLScan.io, NVD |
| 📄 Log Parsing | Automatically extracts every IOC type from any log or text file |
| 📊 Reports | Excel (color-coded), JSON, CSV |
| 💾 Local Cache | SQLite cache with configurable TTL — skip re-querying known IOCs |
| 🛡️ Security | Redirect blocking, host allow-listing, API-key redaction in logs, spreadsheet-formula neutralisation |
| 🔒 Lockfile | requirements.lock with SHA-256 hashes for reproducible installs |
| 💻 CLI Experience | Rich progress bars, colored tables, and a clean verdict summary |
| 🧩 Architecture | Modular enrichers, typed models, strict separation of concerns |
| ✅ Tested | 60 unit & integration tests with pytest; CI via GitHub Actions |
| ⚡ Resilient | One failing API never blocks the others — errors are isolated and logged |
# 1. Clone & install
git clone https://github.com/AbdaullahAG/threatlens.git
cd threatlens
pip install -r requirements.txt
# 2. Configure your API keys
cp config/keys.env.example config/keys.env
# → edit config/keys.env and fill in your keys
# 3. Run your first scan
python main.py -i 45.33.32.156
💡 NVD (CVE lookups) works out of the box with no API key. Every other API offers a free tier that takes under 2 minutes to sign up for — see API Keys below.
pip install --require-hashes -r requirements.lock
| Basic single-IOC lookup |
| Mix and match IOC types in one run |
| Bulk investigate straight from raw logs |
| Machine-readable output for pipelines |
| Restrict enrichment to selected sources |
| Excel + JSON + CSV in a single run |
| CVE enrichment via NIST NVD (free, no key) |
| Full request/response logging for troubleshooting |
| Flag | Description |
|---|---|
-i, --ip | IP address(es) to investigate |
-d, --domain | Domain(s) to investigate |
-s, --hash | File hash(es) — MD5 / SHA1 / SHA256 |
-c, --cve | CVE ID(s), e.g. CVE-2021-44228 |
--file | Path to a log/text file to auto-extract IOCs from |
--apis | Restrict enrichment to a specific set of APIs |
--format | Output format: excel (default) | json | csv | all |
--output | Directory to save reports (default: ./output) |
--no-report | Print results to terminal only, skip saving a file |
--cache-path | SQLite path for local cache (default: .threatlens/investigations.db) |
--cache-ttl | Cache lifetime in seconds (default: 3600) |
--no-cache | Bypass the local cache entirely |
--max-requests | Cap on external API calls per run (default: 250) |
--max-iocs | Maximum unique IOCs per run (default: 1000) |
--allow-private-iocs | Allow private/loopback IPs (disabled by default) |
--delay | Delay between API calls, for rate-limit tuning |
-v, --verbose | Enable debug logging |
threat_intel_tool/
├── main.py # CLI entry point & argument parser
├── requirements.txt # Runtime dependencies
├── requirements-dev.txt # Dev/CI tooling (ruff, bandit, pip-audit, pip-tools)
├── requirements.lock # Pinned lockfile with SHA-256 hashes
├── pytest.ini # pytest configuration (marks, etc.)
├── config/
│ └── keys.env # API keys (copy from keys.env.example)
├── output/ # Generated reports land here
├── src/
│ ├── engine.py # Main orchestrator (collect → enrich → report)
│ ├── models.py # IOC & EnrichmentResult dataclasses
│ ├── storage.py # SQLite cache & investigation history
│ ├── parsers/
│ │ └── ioc_parser.py # Regex-based IOC extractor with validation
│ ├── enrichers/
│ │ ├── base.py # Abstract base — safe HTTP client (redirect-block, budget, retry)
│ │ ├── registry.py # Enricher dispatcher
│ │ ├── abuseipdb.py # AbuseIPDB (IP)
│ │ ├── virustotal.py # VirusTotal (IP / Domain / URL / Hash)
│ │ ├── otx.py # AlienVault OTX (IP / Domain / URL / Hash)
│ │ ├── shodan.py # Shodan (IP)
│ │ ├── urlscan.py # URLScan.io (URL / Domain)
│ │ └── nvd.py # NVD / NIST (CVE — no key required)
│ ├── reporters/
│ │ ├── excel_reporter.py # Color-coded Excel reports
│ │ ├── other_reporters.py # JSON & CSV output
│ │ └── terminal_display.py # Rich terminal tables
│ └── utils/
│ ├── config.py # API key loader & runtime config
│ ├── logger.py # Rich logging setup
│ ├── banner.py # ASCII banner
│ ├── quota.py # Per-run request budget (thread-safe)
│ └── security.py # IOC validation, formula neutralisation, secret redaction
└── tests/
├── conftest.py # pytest fixtures & --run-e2e flag
├── test_core.py # IOC parser, verdict logic, cache round-trip (34 tests)
├── test_enrichers.py # BaseEnricher HTTP edge-cases — mock only (9 tests)
├── test_reporters.py # Excel/CSV formula protection + SQLite integration (17 tests)
└── test_cli_e2e.py # Full CLI run against real NVD API (opt-in, --run-e2e)
Design principles
src/enrichers/ that subclasses BaseEnricher. No changes needed elsewhere.BaseEnricher.get() enforces HTTPS-only, host allow-listing, redirect blocking, 429/Retry-After handling, and request budget capping in one place.config/keys.env with a fallback to system environment variables.--delay) keeps you within each API's free-tier limits.result.errors; a single failing API never brings down the whole scan.=, +, -, @ prefixes).| Provider | Sign Up | Free Tier |
|---|---|---|
| AbuseIPDB | Free | 1,000 checks/day |
| VirusTotal | Free | 4 req/min · 500 req/day |
| AlienVault OTX | Free | Unlimited (public feed) |
| Shodan | Free | Limited lookups |
| URLScan.io | Free | 5,000 req/day (search is free) |
| NVD / NIST | Optional | No key required |
# Run all unit and integration tests (no network required)
pytest tests/ -v --ignore=tests/test_cli_e2e.py
# With coverage report
pytest tests/ -v --ignore=tests/test_cli_e2e.py --cov=src --cov-report=term-missing
# Run the end-to-end CLI test (makes a real NVD request)
pytest tests/test_cli_e2e.py --run-e2e -v
| Test file | Coverage |
|---|---|
test_core.py | IOC parser (all types + edge cases), verdict logic, SQLite cache round-trip |
test_enrichers.py | BaseEnricher.get() — redirect blocking, budget exhaustion, 429+Retry-After, API-key redaction in logs, non-JSON response, invalid JSON, host allow-list, HTTP scheme block |
test_reporters.py | Excel & CSV formula-injection neutralisation (7 prefix variants), numeric passthrough, SQLite TTL expiry, upsert, investigation recording |
test_cli_e2e.py | Full subprocess run: python main.py -c CVE-2021-44228 --apis nvd --format json → exit 0, valid JSON, correct verdict |
| Control | Implementation |
|---|---|
| HTTPS-only | BaseEnricher.get() rejects any non-https:// URL before making a request |
| Host allow-list | Each enricher declares allowed_hosts; requests to unknown hosts are silently dropped |
| Redirect blocking | All requests use allow_redirects=False |
| 429 / Retry-After | Single automatic retry respecting the Retry-After header (capped at 15 s) |
| Request budget | --max-requests hard-caps total API calls per run |
| API-key redaction | Exceptions and log lines have raw key values replaced with [REDACTED] |
| Formula injection | All Excel and CSV cell values are sanitised with spreadsheet_value() |
| IOC validation | Every CLI-supplied IOC is validated and normalised before enrichment |
| Private IP guard | Private/loopback addresses are rejected by default (--allow-private-iocs to override) |
| Dependency audit | pip-audit runs in CI; requirements.lock pins all hashes for reproducible installs |
Terminal:
╭──────────────────────────── IOC Collection ─────────────────────────────╮
│ Found 4 IOCs to investigate │
│ CVE: 1 Domain: 1 Hash: 1 IP: 1 │
╰──────────────────────────────────────────────────────────────────────────╯
✓ Active APIs: abuseipdb, virustotal, otx, shodan, urlscan, nvd
🌐 IP Address Results
┌─────────────────┬──────────────┬──────────┬─────────┬────────────────────┐
│ IP Address │ Verdict │ Abuse % │ Country │ ISP / Org │
├─────────────────┼──────────────┼──────────┼─────────┼────────────────────┤
│ 45.33.32.156 │ Suspicious │ 42 │ US │ Linode │
└─────────────────┴──────────────┴──────────┴─────────┴────────────────────┘
⚠️ CVE Results
┌──────────────────┬──────────┬──────┬──────────────┐
│ CVE ID │ Severity │ CVSS │ Published │
├──────────────────┼──────────┼──────┼──────────────┤
│ CVE-2021-44228 │ Critical │ 10.0 │ 2021-12-10 │
└──────────────────┴──────────┴──────┴──────────────┘
Excel Report: Multi-sheet workbook with color-coded verdicts (🔴 malicious · 🟡 suspicious · 🟢 clean), saved to output/ThreatLens_Report_<timestamp>.xlsx
Have an idea? Open an issue — contributions and suggestions are welcome.
Contributions are welcome and appreciated!
git checkout -b feature/my-featurepytest tests/ -v --ignore=tests/test_cli_e2e.py passes and ruff check . is cleanNew enrichers, bug fixes, documentation improvements, and test coverage are all great first contributions — see Architecture for how enrichers are structured.
This project is licensed under the PolyForm Noncommercial License 1.0.0.
You're free to use, study, modify, and share this code for personal, educational, or research purposes. Commercial use is not permitted without prior written permission from the author ([email protected]).
This tool is intended for educational and authorized security testing purposes only. The user is solely responsible for complying with the terms of service of the integrated APIs and all applicable laws. The author assumes no liability and is not responsible for any misuse, illegal activity, or damage caused by this program.
If ThreatLens saved you time, consider giving it a ⭐ — it helps others discover the project.