pedramamini/iocextractDefanged Indicator of Compromise (IOC) Extractor.
Indicator of Compromise (IOC) extractor for some of the most commonly ingested artifacts.
The iocextract package is a library and command line interface (CLI) for extracting URLs, IP addresses, MD5/SHA hashes, email addresses, and YARA rules from text corpora. It allows for you to extract encoded and "defanged" IOCs and optionally decode or refang them.
It is common practice for malware analysts or endpoint software to "defang" IOCs such as URLs and IP addresses, in order to prevent accidental exposure to live malicious content. Being able to extract and aggregate these IOCs is often valuable for analysts. Unfortunately, existing "IOC extraction" tools often pass right by them, as they are not caught by standard regex.
For example, the simple defanging technique of surrounding periods with brackets:
127[.]0[.]0[.]1
Existing tools that use a simple IP address regex will ignore this IOC entirely.
By combining specially crafted regex with some custom post-processing, we are able to both detect and deobfuscate "defanged" IOCs. This saves time and effort for the analyst, who might otherwise have to manually find and convert IOCs into machine-readable format.
Many Twitter users post C2s or other valuable IOC information with defanged URLs. For example, this tweet from @InQuest:
Recommended reading and great work from @unit42_intel:
https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ ...
InQuest customers have had detection for threats delivered from hotfixmsupload[.]com
since 6/3/2017 and cdnverify[.]net since 2/1/18.
If we run this through the extractor, we can easily pull out the URLs:
https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/
hotfixmsupload[.]com
cdnverify[.]net
Passing in refang=True at extraction time would remove the obfuscation, but since these are real IOCs, let's leave them defanged in our documentation.
You may need to install the Python development headers in order to install the regex dependency. On Ubuntu/Debian-based systems, try:
sudo apt-get install python-dev
Then install iocextract from pip:
pip install iocextract
If you have problems installing on Windows, try installing regex directly by downloading the appropriate wheel from PyPI and installing via pip:
pip install regex-2018.06.21-cp27-none-win_amd64.whl
Try extracting some defanged URLs:
import iocextract
content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""
for url in iocextract.extract_urls(content):
print(url)
# Output
# hxxp://example.com/bad/url
# tcp://example[.]com:8989/bad
# example[.]com
# tcp://example[.]com:8989/bad
NOTE: Some URLs may show up twice if they are caught by multiple regexes.
If you want, you can also "refang", or remove common obfuscation methods from IOCs:
import iocextract
for url in iocextract.extract_urls(content, refang=True):
print(url)
# Output
# http://example.com/bad/url
# http://example.com:8989/bad
# http://example.com
# http://example.com:8989/bad
If you don't want to defang the extracted IOCs at all during extraction, you can disable this as well:
import iocextract
content = \
"""
http://example.com/bad/url
http://example.com:8989/bad
http://example.com
http://example.com:8989/bad
"""
for url in iocextract.extract_urls(content, defang=False):
print(url)
# Output
# http://example.com/bad/url
# http://example.com:8989/bad
# http://example.com
# http://example.com:8989/bad
All extract_* functions in this library return iterators, not lists. The benefit of this behavior is that iocextract can process extremely large inputs, with a very low overhead. However, if for some reason you need to iterate over the IOCs more than once, you will have to save the results as a list:
import iocextract
content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""
print(list(iocextract.extract_urls(content)))
# ['hxxp://example.com/bad/url', 'tcp://example[.]com:8989/bad', 'example[.]com', 'tcp://example[.]com:8989/bad']
A command-line tool is also included:
$ iocextract -h
usage: iocextract [-h] [--input INPUT] [--output OUTPUT] [--extract-emails]
[--extract-ips] [--extract-ipv4s] [--extract-ipv6s]
[--extract-urls] [--extract-yara-rules] [--extract-hashes]
[--custom-regex REGEX_FILE] [--refang] [--strip-urls]
[--wide]
Advanced Indicator of Compromise (IOC) extractor. If no arguments are
specified, the default behavior is to extract all IOCs.
optional arguments:
-h, --help show this help message and exit
--input INPUT default: stdin
--output OUTPUT default: stdout
--extract-emails
--extract-ips
--extract-ipv4s
--extract-ipv6s
--extract-urls
--extract-yara-rules
--extract-hashes
--custom-regex REGEX_FILE file with custom regex strings, one per line, with one capture group each
--refang default: no
--strip-urls remove possible garbage from the end of urls. default: no
--wide preprocess input to allow wide-encoded character matches. default: no
NOTE: Only URLs, emails, and IPv4 addresses can be "refanged".
Are you...
Q. Extracting possibly-defanged IOCs from plain text, like the contents of tweets or blog posts?
A. Yes! This is exactly what iocextract was designed for, and where it performs best. Want to go a step farther and automate extraction and storage? Check out ThreatIngestor.
Q. Extracting URLs that have been hex or base64 encoded?
A. Yes, but the CLI might not give you the best results. Try writing a Python script and calling
iocextract.extract_encoded_urlsdirectly.
Note: You will most likely end up with extra garbage at the end of URLs.