Check network assumptions reproducibly

This project uses Ansible to generate BATS test files verifying security assuptions about network infrastructure. Tests are run from probe machines ("prober nodes") to which Prober is deployed.

This tool is meant for reproducible, automated testing of own networks. It supports multiple prober nodes running tests, each with a different view of the probed network — for example, a view from the external zone, and from an internal zone, and a view from inside the DMZ.

Using Prober to probe networks you are not authorized to might be illegal.

Requirements

On Ansible master used to deploy tests to prober nodes:

• ansible (duh!)
• python*-netaddr (on GNU/Linux) / py*-netaddr (on FreeBSD)

On prober nodes themselves:

• bash
• ssh
• nmap
• dig/kdig
• testssl
• and any other dependencies that are installed by the Ansible role.

On a system you're reviewing the results (*.tap files), you might want to install tappy. Results are kept on each prober node, in local git repositories (by default, in /var/run/prober/results/, see below for configuration options), with commits tagged with the date of each scanning run finished on, for historical record and easy comparison.

Operation

Ansible is used to generate tests to prober nodes. A prober node can be any FreeBSD or GNU/Linux host, as long as it is possible to install Prober's dependencies. It does not need to be dedicated to testing, but it is recommended — a scan of a reasonably-sized network will take hours, use a lot of CPU, and generate quite a lot of traffic.

It makes sense to deploy Prober on a number of different machines to check how your network looks from different vantage points (for example: internal network, DMZ, the Internet).

In the test directory on prober nodes a run-tests.sh script is created in order to facilitate running the tests. Tests are run in a way to ensure <simultaneous_tests_count> tests are running simultaneously at all times — this variable is the main way of controlling how much resources a probing scan uses and how much bandwidth it needs.

Test results are then saved in a local git repository, with commits tagged with the date a given run finished on (which might be different than the start date for some long probing runs).

If dealing with an infrastructure larger than just a few hosts, it might make sense to use a tool like Mitogen to dramatically speed up test generation.

Quickstart

1. Prepare a Debian or FreeBSD server for use as a prober node (let's call it prober.example.com), make sure you have SSH access to it and can sudo on it.

2. Copy the example inventory as inventories/test/.

3. In that test inventory:

   1. edit group_vars/all.yml and set zone_nameserver, zone_domains, address_blocks to your liking; let's assume you add example.com as the only domain one to be probed
   2. edit the hosts file, configuring your prober.example.com prober node in [prober-nodes] section; let's say you set the to .

You can now ssh into the prober node and inspect the tests generated in /opt/prober/. Once you are satisfied, you can run them by executing: /opt/prober/run_tests.sh. Results will be saved in /var/run/prober/results.

Configuration

Configuration variables (defined in probers.yml):

• tests_directory (default: /opt/prober):
  directory on the prober nodes where the test files (*.bats files) are generated to.

• results_repo_directory (default: /var/run/prober/results) directory of the repository for test results (*.tap files); a git repository is initialized there and a tap/ subdirectory created for the actual results; results are committed to the git repository, commits are tagged with a date in yyyy-mm-dd format (the commit with results of a test that finished March 19th, 2020 is tagged as 2020-03-19, for instance).

• simultaneous_tests_count (default: 20):
  how many tests are run simultaneously.

Zones

Zones are defined in ./data/<zone_name>.yml files with the apex key being the string "tested_zone_settings", which then contains keys:

• name: the name of the zone, matching the filename (without extension); for example: "internal", "external", "dmz"
• default (optional): the default settings for all hosts in this zone
• regex keys (necessarily starting with a "^" character) matching multiple FQDNs
• key per each FQDN that needs explicit configuration

The default key, each regex key, and each domain key in turn can contain these keys:

• resolve: should the host resolve to the relevant IP address (bool true/false, or the string "skip")
• ping: should the host respond to ping (bool true/false, or the string "skip")
• ports: list of TCP and UDP ports allowed to be open (sub-keys "tcp" and "udp", each containing an array of integers defining ports allowed to be open, or the string "skip"), and TLS-enabled ports to do a deeper inspection on (the tlssub-key, containing a dict with port numbers as keys, and type of TLS service or the word "skip" as value) is an abbreviation of: TLS service types are whatever supports for the option, for HTTPS test (including headers), or "" for generic TLS test. : TLS tests are run unless the port is also marked as open in the "" key.

You can see the example one config here.

Global defaults are defined in probers.yml. For each probed host these are then combined with the relevant zone defaults, then with regex keys that match a given hostname, and finally with the specific host settings.

This means that specific hosts settings in a given zone take priority over settings from regex matched keys, which in turn take precedence over zone defaults, which in turn take priority over global defaults.

The ports key is treated a bit specially: if certain ports are listed somewhere in the configuration inheritance chain, it's impossible to de-list them lower in the chain, only to add additional port numbers, or to "skip" tests for all ports of a given protocol altogether.

Zone files are selected based on the tested_zone variable set for each prober node.

Per-IP tests vs. per-hostname tests

Some tests make sense in the context of individual IP addresses, regardless of how many domains/hostnames resolve to them; for example, checking if certain ports are open. For example, let's say a.example.com and b.example.com point to the same IP address. With the config from the example above, that means that ports 8080/tcp and 8443/tcp are both supposed to be open, and that HTTPS is expected on port 8443/tcp.

Some tests make sense in the context of particular domain/hostname and IP address combinations; for example, if the TLS certificate presented for a particular domain name on a particular IP address is valid.

These twokinds of tests are implemented by generating two separate lists of targets:

• per-ip list, where each element is of the form:
  <ip-address> <hostname1> (<hostname2> <hostname3> ... <hostnameN>)
  IP addresses should not repeat in this list;
• per-hostname list, where each element is of the form:
  <ip-address> <hostname>
  IP addresses can be expected to repeat in this list.

Some test files are then generated only for targets from the first list (for example, open ports testing), and some only for targets from the second list (for example, TLS-related).

Ports and tests

Some tests do not make sense if certain ports are not open. TLS-related tests are generated for a given host only if any relevant TCP ports are configured as open in the given zone.

By default, if any of the well-known TLS-enabled ports is configured as open in the ports.tcp key, TLS tests will be generated for them. List of well-known TLS-enabled ports is defined in the default_host_settings variable.

FAQ

In general, the difference between Prober and many other similar tools or services usually boils down to some combination of:

• being self-hosted, with Prober nodes being deployable in any location inside or outside of your infrastructure;
• focus on regular, reproducible scans that verify well-defined, configurable assumptions;
• giving you control over the schedule of testing runs, and access to raw scan results.

How is this different from Shodan?

Shodan crawls the Internet to find exposed systems. Prober crawls your own infrastructure to verify your assumptions about it, from as many vantage points as you need. It focuses on reproducibility of results via well-defined tests and regular, scheduled runs, and attempts to deliver a boolean "pass/fail" result for each of the tested assumptions (with the full scan result data available for inspection).

How is this different from BitSight?

BitSight verifies their idea of reasonable assumptions about your infrastructure, without giving you control over or information on when exactly a scan will take place, nor providing the raw scan data. Prober verifies your assumptions about your infrastructure, does it on your schedule, and provides full raw scan data for inspection.

How is this different from Natlas?

Natlas seems to be closer to Shodan (crawling to find exposed hosts, showing results in response to queries), but with the ability to self-host (and thus to also run the Natlas Agent in different locations within and outside of your infrastructure, like Prober nodes). Prober runs regular scans of your infrastructure to verify your own assumptions about it, as expressed in the config.

Large questions

• switch to a more powerful testing system?
  • BATS has annoying limitations, for example: there is no way to do conditional testing ("if Test A fails then skip Test B", etc)
  • drop testing system altogether?
    • https://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html
    • https://github.com/benwebber/ansible-tap
• IPv6 full-block scans are impossible to complete within a millennium
  • provide a way to configure the heuristic of selecting IPs to scan (random, start from bottom with a configured "step", etc)?

Acknowledgements

Logo based on Magnifying Glass by verry obito, ID; CC-By and Network by Creative Stall, PK; CC-By, via the Noun Project.