dtls-fuzzer is a Java tool which performs protocol state fuzzing of DTLS servers. More concretely, it supports the following functionality:

1. given an alphabet, it can automatically generate a model of a local DTLS server implementation;
2. given a test (sequence of inputs) and an alphabet, it can execute the test on a DTLS server implementation;
3. it can run a batch learning task, involving multiple learning runs.

dtls-fuzzer uses [TLS-Attacker][tlsattacker] to generate/parse DTLS messages as well as to maintain state. To that end, TLS-Attacker has been extended with support for DTLS. dtls-fuzzer relies on [version 3.0b][tlsattackerver] of TLS-Attacker, a version implementing the DTLS enhancement.

Artifact contents

The artifact contains:

1. a description of the file structure of dtls-fuzzer, including source code and experimental data consistent with that displayed in the paper;
2. a walkthrough for evaluating dtls-fuzzer on a chosen SUT (System Under Test)/ DTLS server implementation.

dtls-fuzzer file structure

The most important folders in dtls-fuzzer's root directory are:

1. 'src', directory containing the Java source code of dtls-fuzzer;
2. 'examples', directory containing examples of alphabets, tests, specifications (i.e. models), and, argument files that can be supplied to dtls-fuzzer in order to launch learning experiments. Files in this directory are used as inputs for learning experiments;
3. 'experiments', directory containing data relating to experiments. Some of this data also serves as input for learning experiments. The most notable folders are:
   1. 'suts', with binaries for Java SUTs. These SUTs are custom-made DTLS server programs whose source code is publicly available;
   2. 'patches', patches that were applied to some SUTs (particularly to utilities) before the source code was compiled. The primary purpose of these patches was to prevent timing induced-behavior during learning, enable/disable functionality in the SUT, and configure parameters such as the pre-shared key;
   3. 'keystore', key material (e.g. public-private key pairs, Java keystores) used during learning;
   4. 'results', experimental results.

Experimental results

'experiments/results' contains experimental results, which are the main output of the work. In particular

• 'all_ciphers' contains output folders for all the experiments run;
  • 'mapper' contains experimental results which help justify some of the mapper decisions made (see Section 5.2)
• 'included' contains output folders for converging experiments (converging means that learning successfully generates a model).
  • note that not all experiments in 'all_ciphers' were successful/terminated with a final model (we say in such cases that learning did not converge)

Output folders

Output folders are named based on the experiment configuration, that is:

• the SUT/implementation tested;
• the alphabet used, in terms of key exchange algorithms covered, where 'all' indicates all 4 key exchange algorithms were used;
• where applicable whether client certification was required (req), optional (nreq) or disabled (none);
• the testing algorithm: random walk (rwalk) or an adaptation of it (stests);
  • experiments using the adaptation have not been included in the paper
• optionally, whether retransmissions were included in/excluded from outputs (incl or excl).
  • retransmissions were included by default

As an example, the folder name 'jsse-12_rsa_cert_none_rwalk_incl' indicates an experiment on the JSSE 12 implementation of DTLS, using an alphabet including inputs for performing RSA handshakes, client certificate authentication is disabled, the test algorithm is random walk and retransmissions are included.

An output folder contains:

• 'alphabet.xml', the input alphabet;
• 'command.args', the arguments file used containing various experiment parameters, most notably:
  • queries, the bound on the number of random walk tests which need to pass for a hypothesis to be deemed final
  • equivalenceAlgorithms, model-based test algorithms employed
  • runWait and timeout, the start and response timeout respectively (we touch on them later)
• 'sul.config', SUT-dependent configuration for TLS-Attacker, the same configuration can be used to execute workflow traces on the SUT using TLS-Attacker alone;
• 'hyp[0-9]+.dot', intermediate hypotheses;
• 'statistics.txt', experiment statistics such as the total number of tests, learning time;
  • Table 4 displays this data
• 'nondet.log', logs of encountered non-deterministic behavior;
• 'learnedModel.dot', in case learning converged, the learned model (i.e. final hypothesis);
• 'error.msg', an error message generated in case the experiment failed/learning was stopped and hence, did not converge to a final model.
  • the main culprit is time-related non-determinism (same inputs lead to different results).

The evaluator can check (for example) that experimental results in 'included' correspond to those displayed in Table 4, or that configurations tested in Table 2 also appear in 'all_ciphers'. Note that models appearing in the paper were the result of significant pruning/triming, whereas the models appearing in output folders are unaltered.

dtls-fuzzer evaluation steps

For the purpose of evaluating dtls-fuzzer it is necessary to perform the following steps:

1. Ensure pre-requisites are met
2. Install dtls-fuzzer
3. Setup SUT
4. Use dtls-fuzzer to generate models for the SUT
5. Analyze results

This evaluation section is followed by a guide on using dtls-fuzzer which introduces its main use cases.

Ensuring pre-requisites

dtls-fuzzer has been tested on Ubuntu 18.04 and Debian 9 distributions of Linux. It should work on any recent Linux distribution. Support for other platforms has not been tested. This guide assumes a Debian-based distribution is used (which has 'apt-get').

A Java 8 JDK (Java Development Kit) Virtual Machine (VM) is required. The version used to run experiments is 1.8.0_222, though later Java 8 versions should also work. Note that the tool does not build on Java 9 or later. We also rely on maven (the 'mvn' utility) for dependency management/deployment.

We recommend using a sufficiently powerful machine, otherwise sensitive timing parameters such as response waiting time, might become too low, causing different outputs to the ones obtained in the paper. Worse yet, they can cause learning experiments to fail. The original experiments were run on a many-core server, however, we expect (though haven't tested thoroughly) that learning should be possible on a desktop with an i7 processor. Learning is also possible on weaker systems if timing parameters are tuned accordingly. Finally, visualizing .dot models by exporting them to .pdf requires installing the [graphviz library][graphviz]. It is assumed that the 'dot' utility provided by graphviz is located in the system PATH.

In a nutshell, the advised pre-requisites are:

• recent Linux distribution, preferably Debian-based
• desktop/server machine for experiment reproduction/reliable learning
• (>=) 4 GB RAM
• Java 8 JDK
• maven
• graphviz

Setting up the environment

Java 8 JDK

dtls-fuzzer requires Java 8 JDK (Java Development Kit). If Java is not installed, we install the OpenJDK implementation (via 'apt-get' on Ubuntu), and can skip the rest of this subsection.

> sudo apt-get install openjdk-8-jdk

If a version of java is installed, we can check which version it is by running:

> java -version

The version code should start with 1.8 (e.g. 1.8.0_242), and the Virtual Machine should be "Server VM" (indicating that the full JDK is installed, rather than only the runtime environment). If it does, we are done with Java. If it is not we can check if Java 8 JDK is installed on our platform but not currently selected, by listing installed Java VMs via: