LUCX: LUCID network traffic parser eXtended

LUCX (LUCID network traffic parser eXtended) is a network traffic parser that works with both pre-recorded traffic traces in pcap format and live network traffic captured from a Network Interface. LUCX is an extended version of the LUCID dataset parser (https://github.com/doriguzzi/lucid-ddos). Like the original version, LUCX produces traffic observations consistent with those collected in existing online systems, where the detection algorithms must cope with segments of traffic flows collected over pre-defined time windows. Furthermore, LUCX extracts more features (e.g., TLS-specific features) in two different formats: Packet-Level and Statistical, described below in this README.

The traffic pre-processing operations are divided into two different stages: the first extracts the features from the packets and organises them into flows and time windows, the second performs feature normalisation, padding (or flattening, depending on the desired output format) and saves training, validation and test sets in hdf5 format.

The sample Network Intrusion Detection System (NIDS), provided in this repository as a Jupyter notebook, can be used to train and evaluate Deep Learning models (specifically a Multi-Layer Perceptron (MLP) or a Convolutional Neural Network (CNN)) using both packet-level and statistical network traffic representations generated by LUCX.

Installation

LUCX is implemented in Python v3.X, Numpy and Pyshark. LUCX requires the installation of a number of Python tools and libraries. This can be done by using the conda software environment (https://docs.conda.io/projects/conda/en/latest/). We suggest the installation of miniconda, a light version of conda. miniconda is available for MS Windows, MacOSX and Linux and can be installed by following the guidelines available at https://docs.conda.io/en/latest/miniconda.html#.

MS Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11 are required. In MS Windows, first install the Windows subsystem for Linux from the command line: wsl.exe —install (see here for more info). This command will install an Ubuntu Linux OS on your computer. You can access the Linux command line by executing the WSL app from the Windows menu, then follow the guidelines for the Linux OS below.

Open a terminal, execute one of the following commands (based on your operating system) and follow the on-screen instructions:

bash Miniconda3-latest-Linux-x86_64.sh (on Linux operating systems)
bash Miniconda3-latest-MacOSX-x86_64.sh (on macOS)

Then create a new conda environment (called python313) based on Python 3.13.5:

conda create -n python313 python=3.13.5

Activate the new python313 environment:

conda activate python313

And configure the environment with the installation of a few more packages:

(python39)$ pip install scikit-learn h5py pyshark matplotlib graphviz pandas scikeras pydot seaborn

Pyshark is used for data pre-processing. It is just Python wrapper for tshark, meaning that tshark must be also installed. On an Ubuntu-based OS, use the following command:

sudo apt install tshark

In macOS, just download Wireshark from here and install it.

Please note that the current parser code works with wireshark/tshark version 4.4.9 or above.

Traffic feature representations

LUCX considers bidirectional network traffic flows, defined as groups of packets that share the same 5‑tuple identifier or its transposed 5‑tuple. The 5‑tuple is a common way to uniquely characterise a communication stream in IP networks, and it consists of:

• Source IP address
• Destination IP address
• Source port number
• Destination port number
• Transport protocol (e.g., TCP, UDP)

When packets have identical values for these five fields, they are considered part of the same flow. The concept of a transposed 5‑tuple accounts for bidirectional communication: if the source and destination addresses/ports are swapped but the protocol remains the same, the packets are still considered part of the same flow. This ensures that both directions of a conversation (client → server and server → client) are grouped together.

Example: Suppose a client at IP 192.168.1.10 sends a TCP request to a web server at IP 203.0.113.5 on port 80. The client uses source port 54321.

• Forward direction (client → server):
  • Source IP: 192.168.1.10
  • Destination IP: 203.0.113.5
  • Source port: 54321
  • Destination port: 80
  • Protocol: TCP
• Reverse direction (server → client):
  • Source IP: 203.0.113.5
  • Destination IP: 192.168.1.10
  • Source port: 80
  • Destination port: 54321
  • Protocol: TCP

Both sets of packets belong to the same traffic flow because the reverse direction matches the transposed 5‑tuple of the forward direction. In addition, LUCX splits each traffic flow extracted from network traffic traces of a dataset (or the incoming network traffic) into sub‑flows based on time windows. This segmentation simulates real‑world scenarios in which traffic is collected within static or dynamic time‑windows and then forwarded to a classifier for analysis. By doing so, the system mirrors the operational practices of intrusion detection systems (IDS), which often rely on time‑bounded packet aggregation to identify short‑term behaviors and detect attacks.

After the packets are grouped into flows and the flows are further split into sub‑flows, LUCX converts the sub‑flows into samples that can be used to train, validate, and test ML‑based classifiers. There are two possible output formats: arrays (as in LUCID) or vectors. The first format preserves the features extracted directly from packets, while the second produces statistical representations of the packet‑level features. These two operational modes are referred to as packet‑level features and statistical features, respectively.

Packet-level features

This follows the same format as the original LUCID dataset parser, which converts a sub‑flow into an array where each column represents a packet‑level feature (e.g., timestamp, packet length, TCP SYN flag, ICMP type), and each row corresponds to a packet in chronological order. Although the structure remains the same, the number of columns has been increased by introducing additional features or by expanding existing features into binary representations (e.g., IP and TCP flags). The first row contains the first packet of a sub‑flow, which is also the first packet of a flow collected within a time window. The first column is the packet timestamp: for the first packet, the timestamp is set to zero, while the timestamps of subsequent packets are expressed as the difference between their arrival time and the original timestamp of the first packet. This approach avoids using absolute values for packet arrival times and preserves the relative timing patterns of packets belonging to the same sub‑flow.

timestamp	packet_length	IP_ttl	IP_flags_df	...	ICMP_code
0	64	128	0	...	0
1.2	128	64	1	...	8
...	...	...	...	...	...
0	0	0	0	...	0
0	0	0	0	...	0