hurricanelabs/machinaeAutomated security intelligence collector that queries public feeds and APIs for threat data on IPs, domains, URLs, hashes, and SSL fingerprints, with YAML-based configuration and JSON output.
Machinae is a tool for collecting intelligence from public sites/feeds about various security-related pieces of data: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. It was inspired by [Automater][1], another excellent tool for collecting information. The Machinae project was born from wishing to improve Automater in 4 areas:
Machinae can be installed using pip3:
pip3 install machinae
Or, if you're feeling adventurous, can be installed directly from github:
pip3 install git+https://github.com/HurricaneLabs/machinae.git
You will need to have whatever dependencies are required on your system for
compiling Python modules (on Debian based systems, python3-dev), as well as
the libyaml development package (on Debian based systems, libyaml-dev).
You'll also want to grab the [latest configuration file][2] and place it in
/etc/machinae.yml.
Machinae supports a simple configuration merging system to allow you to make
adjustments to the configuration without modifying the machinae.yml we provide
you, making configuration updates a snap. This is done by finding a system-wide
default configuration (default /etc/machinae.yml), merging into that a
system-wide local configuration (/etc/machinae.local.yml) and finally a
per-user local configuration (~/.machinae.yml). The system-wide configuration
can also be located in the current working directory, can be set using the
MACHINAE_CONFIG environment variable, or of course by using the -c or
--config command line options. Configuration merging can be disabled by
passing the --nomerge option, which will cause Machinae to only load the
default system-wide configuration (or the one passed on the command line).
As an example of this, say you'd like to enable the Fortinet Category site,
which is disabled by default. You could modify /etc/machinae.yml, but these
changes would be overwritten by an update. Instead, you can put the following
in either /etc/machinae.local.yml or ~/.machinae.yml:
fortinet_classify:
default: true
Or, conversely, to disable a site, such as Virus Total pDNS:
vt_ip:
default: false
vt_domain:
default: false
Machinae usage is very similar to Automater:
usage: machinae [-h] [-c CONFIG] [--nomerge] [-d DELAY] [-f FILE] [-i INFILE] [-v]
[-o {D,J,N,S}] [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q]
[-s SITES] [-a AUTH] [-H HTTP_PROXY]
[--dump-config | --detect-otype]
...
See above for details on the -c/--config and --nomerge options.
Machinae supports a -d/--delay option, like Automater. However, Machinae
uses 0 by default.
Machinae output is controlled by two arguments:
-o controls the output format, and can be followed by a single character
to indicated the desired type of output:
-f/--file specifies the file where output should be written. The default
is "-" for stdout.Machinae will attempt to auto-detect the type of target passed in (Machinae
refers to targets as "observables" and the type as "otype"). This detection can
be overridden with the -O/--otype option. The choices are listed in the
usage
By default, Machinae operates in verbose mode. In this mode, it will output
status information about the services it is querying on the console as they are
queried. This output will always be written to stdout, regardless of the output
setting. To disable verbose mode, use -q
By default, Machinae will run through all services in the configuration that apply to each target's otype and are not marked as "default: false". To modify this behavior, you can:
all to run through all services including those
marked as "default: false"Note that in both cases, otype validation is still applied.
Machinae supports passing an HTTP proxy on the command line using the
-H/--http-proxy argument. If no proxy is specified, machinae will search the
standard HTTP_PROXY and HTTPS_PROXY environment variables, as well as the
less standard http_proxy and https_proxy environment variables.
Lastly, a list of targets should be passed. All arguments other than the options listed above will be interpreted as targets.
Machinae comes with out-of-the-box support for the following data sources:
With additional data sources on the way.
Machinae supports HTTP Basic Auth for sites that require it through the --auth/-a
flag. You will need to create a YAML file with your credentials, which will include
a key to the site that requires the credentials and a list of two items, username
and password or API key. For example, for the included PassiveTotal site this might
look like:
passivetotal: ['[email protected]', 'my_api_key']
Inside the site configuration under request you will see a key such as:
json:
request:
url: '...'
auth: passivetotal
The auth: passivetotal points to the key inside the authentication config passed
via the command line.
The following sites are disabled by default