• Modular architecture to handle streams of unstructured or structured information
• Default support for external ZMQ feeds, such as provided by CIRCL or other providers
• Multiple feed support
• Each module can process and reprocess the information already processed by AIL
• Detecting and extracting URLs including their geographical location (e.g. IP address location)
• Extracting and validating potential leak of credit cards numbers, credentials, ...
• Extracting and validating email addresses leaked including DNS MX validation
• Module for extracting Tor .onion addresses (to be further processed for analysis)
• Keep tracks of duplicates (and diffing between each duplicate found)
• Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
• A full-text indexer module to index unstructured information
• Statistics on modules and web
• Real-time modules manager in terminal
• Global sentiment analysis for each providers based on nltk vader module
• Terms, Set of terms and Regex tracking and occurrence
• Many more modules for extracting phone numbers, credentials and others
• Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
• Detect and decode encoded file (Base64, hex encoded or your own decoding scheme) and store files
• Detect Amazon AWS and Google API keys
• Detect Bitcoin address and Bitcoin private keys
• Detect private keys, certificate, keys (including SSH, OpenVPN)
• Detect IBAN bank accounts
• Tagging system with MISP Galaxy and MISP Taxonomies tags
• UI paste submission
• Create events on MISP and cases on The Hive
• Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags
• Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
• Graph relationships between decoded file (hashes), similar PGP UIDs and addresses of cryptocurrencies
• Tor hidden services crawler to crawl and parse output
• Tor onion availability is monitored to detect up and down of hidden services
• Browser hidden services are screenshot and integrated in the analysed output including a blurring screenshot interface (to avoid "burning the eyes" of the security analysis with specific content)
• Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
• Generic web crawler to trigger crawling on demand or at regular interval URL or Tor hidden services

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh

cd ~/AIL-framework/
cd bin/
./LAUNCH.sh -l

• Python 3.5+

cd bin/
./LAUNCH -l

https://localhost:7000/

@inproceedings{mokaddem2018ail,
  title={AIL-The design and implementation of an Analysis Information Leak framework},
  author={Mokaddem, Sami and Wagener, G{\'e}rard and Dulaunoy, Alexandre},
  booktitle={2018 IEEE International Conference on Big Data (Big Data)},
  pages={5049--5057},
  year={2018},
  organization={IEEE}
}