blevene/standalone_tdoExtracts structured Cyber Threat Intelligence (CTI) from PDF, DOCX, and TXT documents using LLMs. Generates MITRE ATT&CK attack flows, detection opportunities, and comprehensive entity-relationship graphs.
A powerful, standalone command-line tool for extracting Cyber Threat Intelligence (CTI) from documents using Large Language Models with advanced structured output capabilities.
# Clone or download the standalone-tdo folder
cd standalone-tdo
# Create virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
The tool uses environment variables loaded from a .env file for configuration:
# Copy the example configuration
cp env.example .env
# Edit .env with your settings
# Required variables:
GEMINI_API_KEY=your-google-ai-api-key-here
GEMINI_MODEL=gemini-2.5-flash
Getting a Gemini API Key:
.env fileAvailable Models:
gemini-2.5-flash-preview-05-20 (recommended - fast and cost-effective)gemini-2.5-pro-preview-06-05 (more powerful, slower)# Process a single file with all features
python tdo_bulk.py report.pdf --flow --opps
# Process multiple files
python tdo_bulk.py file1.pdf file2.docx file3.txt
# Process all files in a directory with parallel workers
python tdo_bulk.py reports/ -j 4
# Generate comprehensive analysis with evaluation
python tdo_bulk.py report.pdf --flow --opps --eval-opps
usage: tdo_bulk.py [options] FILE [FILE ...]
Positional Arguments:
FILE One or more files or directories to process
Core Options:
-o, --output DIR Output directory (default: ./extracted_data)
--csv FILE Export summary to CSV file
-j, --jobs N Number of parallel workers (default: 1)
--retries N LLM retry count (default: 2)
--backoff SEC Exponential back-off base (default: 1.5)
Analysis Features:
--flow Generate Attack-Flow JSON
--opps Generate Threat Detection Opportunities
--eval-opps Evaluate detection opportunities quality
--debug-opps Show debug information for opportunity generation
Output Control:
-q, --quiet Minimal console output
-v, --verbose Debug-level logging
-h, --help Show help message and exit
# Basic CTI extraction
python tdo_bulk.py threat_report.pdf
# Full analysis with all features
python tdo_bulk.py apt_report.pdf --flow --opps --eval-opps
# Batch processing with parallel workers
python tdo_bulk.py reports_folder/ -j 8 --flow --opps
# Export results to CSV
python tdo_bulk.py *.pdf --csv results.csv
# Quiet mode for automation
python tdo_bulk.py reports/ -q -o /var/soc/cti --opps
For each processed file, the tool generates:
{filename}_extracted.json: Structured CTI data with comprehensive schema{filename}_{timestamp}.md: Human-readable markdown report with all analysis--flow)--opps)| Entity Type | Description | Key Properties |
|---|---|---|
| ThreatActor | Cyber threat groups | aliases, primary_motivation, first_seen |
| Tool | Legitimate software | family, capabilities, kill_chain_phases |
| Malware | Malicious software | family, capabilities, first_seen, last_seen |
| Technique | MITRE ATT&CK techniques | id (T1234), description, kill_chain_phases |
| Tactic | MITRE ATT&CK tactics | id (TA0001), description |
| Infrastructure | IPs, domains, URLs | type, tags, first_seen, last_seen |
| Indicator | File hashes, patterns | value, pattern, valid_from, valid_until |
| Vulnerability | CVE entries | id, cvss_score, affected_software |
| Campaign | Named attack campaigns | objective, status, first_seen |
| Identity | Target organizations | type, description |
| CourseOfAction | Mitigations, patches | type, description |
| Source | Document provenance | filename, , |
Attribution & Actor Relationships:
USES_TOOL, USES_TECHNIQUE, CONDUCTS, ATTRIBUTED_TO, TARGETSTechnical Relationships:
TOOL_IMPLEMENTS_TECHNIQUE, HOSTS_ON, COMMUNICATES_WITH, VARIANT_OFDROPS, DOWNLOADS, INDICATES, OBSERVED_ON, EXPLOITSAdvanced Relationships:
MITIGATES, DETECTS, IS_SUBTECHNIQUE_OF, FLOW_CONTAINS_STEPFLOW_USED_BY_ACTOR, OBSERVES, SOURCED_FROMAll relationships support properties like confidence, source, first_seen, last_seen.
The tool generates evidence-based detection opportunities with:
{
"id": "opp-001",
"name": "Detect PowerShell Process Injection",
"technique_id": "T1055",
"artefacts": ["powershell.exe with WriteProcessMemory calls"],
"behaviours": ["Process injection into legitimate processes"],
"rationale": "APT groups commonly use PowerShell for process injection",
"confidence": 0.8,
"source": "PowerShell used for process injection (T1055)",
"evidence": [
"• PowerShell executes WriteProcessMemory calls (line 45)",
"• Relationship: ThreatActor USES_TECHNIQUE T1055"
]
}
Enhanced attack flows provide:
{
"flow": {
"label": "AttackFlow",
"pk": "attack-flow--uuid",
"properties": {
"name": "APT29 Multi-stage Attack",
"description": "Sophisticated spear-phishing to data exfiltration flow"
}
},
"steps": [
{
"order": 1,
"entity": {"label": "Technique", "pk": "T1566.001"},
"description": "Spear-phishing attachment delivery",
"reason": "Initial access method cited in report section 2.1"
}
]
}
tdo_bulk.py).env configuration using python-dotenvtdo_bulk_runner.py)tdo_core/extractors/report_processor.py)tdo_core/detection/opportunity_generator.py)tdo_core/flows/attack_flow_synthesizer.py)tdo_core/report/md_export.py)tdo_core/llm/prompts.py)Document Input → Text Extraction → LLM Processing → Structured Output
↓ ↓ ↓ ↓
PDF/DOCX Clean Text Gemini API JSON + Fallback
↓ ↓ ↓ ↓
Multi-format Preprocessing Pydantic Schema Validation
↓ ↓ ↓ ↓
File Support Text Cleaning Structured Data CTI Graph
↓
Post-Processing
↓ ↓
Attack Flows Detection Opps
↓ ↓
Markdown Report Export
| Variable | Required | Description | Example |
|---|---|---|---|
GEMINI_API_KEY | ✅ | Google AI API key | AIza... |
GEMINI_MODEL | ✅ | Gemini model to use | gemini-2.5-flash-preview-05-20 |
| Model | Speed | Quality | Cost | Use Case |
|---|---|---|---|---|
gemini-2.5-flash-preview-05-20 | ⚡ Fast | 🎯 Good | 💰 Low | Production, bulk processing |
gemini-2.5-pro-preview-06-05 | 🐌 Slow | 🏆 Excellent | 💸 High | Complex analysis, research |
Error: Required environment variables missing: GEMINI_API_KEY, GEMINI_MODEL
Solution: Create .env file with both required variables
Skipping unsupported file: document.rtf
Solution: Convert to PDF, DOCX, TXT, or MD format
Structured Gemini API error: ...
Solutions:
--verbose for detailed error logsThe tool automatically handles JSON parsing issues with:
-j flag for multiple files-q for automation scripts# Enable verbose logging
python tdo_bulk.py report.pdf -v
# Debug detection opportunities
python tdo_bulk.py report.pdf --opps --debug-opps
# Export detailed logs
python tdo_bulk.py report.pdf -v > processing.log 2>&1
The tool requires Python 3.9+ and these key packages:
google-generativeai: Google AI (Gemini) API client with structured outputPyMuPDF: High-performance PDF text extractionpython-docx: Microsoft Word document processingpandas: Data manipulation and CSV exportpydantic: Schema validation and structured outputpython-dotenv: Environment variable managementcleantext: Text preprocessing utilitiesThis standalone version:
However, it includes all core CTI extraction and analysis capabilities needed for most use cases.
.env file management with dotenv.env file - it's excluded by .gitignoreGEMINI_API_KEY securely using environment variablesExtracted JSON and Markdown files may contain:
Always review outputs before public sharing.
Please report security vulnerabilities responsibly. See SECURITY.md for details.
This project is licensed under the MIT License - see LICENSE for details.
Contributions are welcome! Please read CONTRIBUTING.md for guidelines.
Need Help? Run python tdo_bulk.py --help for quick reference or check the troubleshooting section above.
document_titlefile_size_mb