AIL Framework

Open-source framework for the collection, crawling, processing, and analysis of unstructured information.

AIL framework is an open-source platform to collect, crawl, process and analyse unstructured data from the clear web, Tor, I2P, chats, files and external feeds.

Originally developed at CIRCL, AIL helps analysts transform raw, messy content into structured intelligence through extraction, tagging, detection, correlation and investigation workflows.

What is AIL? https://ail-project.org

AIL (Analysis of Information Leaks) is an open-source framework for the collection, crawling, processing, and analysis of unstructured information. It supports threat intelligence, leak analysis, and investigative workflows by helping analysts extract, detect, correlate, and share relevant information from a wide range of sources.

AIL includes:

• an extensible Python-based framework for processing and analysing unstructured information,
• a crawler manager for continuous and authenticated collection,
• feeders for communication platforms and external streams,
• a detection and retro-hunt engine based on keywords, regex and YARA,
• search, correlation and investigation capabilities to pivot across extracted data,
• and export/integration features for platforms such as MISP.

AIL intelligence lifecycle

AIL follows a practical intelligence workflow:

1. Collection Continuous ingestion from chats, websites, hidden services, files and feeds.
2. Processing Extraction, decoding, OCR, QR/barcode parsing, enrichment and tagging.
3. Detection Real-time tracking with words, sets, regex, typo-squatting and YARA rules.
4. Analysis Search, pivoting, correlation graphs and investigations.
5. Dissemination Export of findings and objects to MISP intelligence-sharing platforms.

What’s new in AIL v6.7

AIL is now at v6.7 and recent releases significantly expanded search, image analysis, crawling and document-processing capabilities.

Highlights include:

• Unified search interface with best-match and most-recent ordering
• Date range filtering and improved advanced search workflows
• Image and screenshot descriptions for faster visual analysis and searchability
• Expanded OCR and QR extraction, including support for more difficult image cases
• Full PDF processing pipeline, including metadata extraction and translation support
• I2P crawling support in addition to clear web and Tor collection
• Passive SSH correlation for infrastructure analysis and deanonymization workflows
• Improved chat exploration for platforms such as Discord, Telegram and Matrix

Features

Collection

• Modular architecture to handle streams of unstructured information
• Multiple feeder and importer support
• Feeders for chat and stream sources such as Discord, Telegram and other providers
• Crawling support for the clear web, darknet, Tor hidden services (.onion), and I2P
• Authenticated crawling with browser sessions, cookies and local storage reuse
• Continuous or on-demand monitoring of websites and hidden services over time
• UI submission/import capabilities

Processing and enrichment

• Full-text indexing of unstructured information (chats, crawled contents)
• Extraction of URLs, hostnames, email addresses and credentials
• Detection of phone numbers, API keys, IBANs, certificates and private keys
• Detection of Bitcoin addresses, private keys and related cryptocurrency artifacts
• File extraction and decoding from encoded content (Base64, hex)
• OCR processing for screenshots and images
• QR code and barcode extraction with reprocessing of embedded content
• AI-assisted descriptions for images, screenshots and domains
• PDF metadata extraction, ingestion and translation
• Tagging system using MISP Galaxy and MISP Taxonomies

Detection and tracking

Trackers are user-defined rules or patterns that automatically detect, tag and notify analysts about relevant information collected by AIL.

Supported tracker types:

• word tracking
• set-of-words tracking
• regex tracking
• YARA rules
• typo-squatting detection

Detection capabilities include:

• real-time tagging and classification
• object occurrence tracking
• webhook or email notification workflows
• built-in YARA editor

AIL also supports Retro Hunts, enabling analysts to run newly created YARA rules against historical data to uncover previously missed content.

Search, correlation and investigation

• Unified search interface with recency and relevancy ordering
• Search by date range and specialized advanced search for selected data types
• Search across chats, crawled domains, titles, filenames and AI-generated descriptions
• Correlation engine and graph visualisation for relationships between:
  • decoded files and hashes
  • PGP metadata
  • domains, titles, dom-hash, favicons, cookie-names
  • usernames and user-accounts
  • CVEs
  • SSH keys
  • cryptocurrencies
  • PDF metadata
  • ...
• Investigation workflow to group, enrich and follow analyst findings

Export and integrations

• Alerting and sharing to MISP
• Export of AIL objects and investigations to MISP formats
• Automatic exports on selected detections and tags
• Integrations supporting collaborative intelligence and incident-response workflows

Why AIL?