7h3ram/flowinspectA Network Inspection Tool
A network traffic inspection tool
It uses libnids (via its python bindings from Jon Oberheide: pynids) to defragment IP and reassemble TCP packets (UDP is inspected on a per-packet basis) to generate network flows. These flows are then inspected using the one of the four inspection modes:
Regex matches are performed using the re2 library and its Python bindings, pyre2, that supports PCRE, case-insensitive, invert and multiline matches, etc. It has enormous performance gains compared to the built-in re module in Python (which is used as a fallback in case re2 is not installed).
Fuzzy string matching features are carried out via the fuzzywuzzy module. It helps to perform both an exact and relative string mathing. A default match threshold of 75 is used as a default and can be overridden through cli.
Libemu and its Python bindings, pylibemu, are used for shellcode detection. The GetPC heuristics used by libemu provide a decent detection ratio. There are a few cases where libemu simply fails but for most usecases it is good enough.
Yara is a signature-based malware identification and classification tool. Its yara-python bindings provide an API to use existing/custom signature files on an input buffer which in this case is a network stream.
Inspection could be requested for any of the CTS/STC/ANY directions or their combinations. Inspection buffers are populated as network traffic arrives and as such CTS matches (CTS or ANY) happen first. If more than one mode of inspection is requested, flows are inspected in the following order: regex, fuzzy, libemu, and finally yara. For TCP, if any of the inspection modes succeed, the matched flow won't be inspected any further. This is an optimistic approach and is enabled by default. However if for a certain usecase a TCP stream has to inspected multiple times, it can be requested explicitly using a cli.
Inspection could be completely disabled if required via the linemode cli option. This mode is really helpful and when combined with a suitable outmode helps to have a look at network communication as-is while its happening over wire. Linemode is auto enabled as a fallback if no inspection mode is provided via cli.
For UDP, matches happen on a per-packet basis and as such subsequent packets will be tested even after a match has already been found on a UDP flow. Since only subsequent packets and their content is inspected, it ensures that the data already matched during earlier inspection cycles is not inspected again.
Match scope could be limited through BPF expressions, Snort-like offset-depth content modifiers or via packets/streams inspection limit cli options. For TCP, matched flows could also be killed if need be. Flows could also be logged to files in addition to being dumped on stdout. A few useful output modes (quite, meta, hex, print, raw) help with further analysis. The meta outmode is expecially useful as it shows some really important match specific details like the total size of matched content, offset of the start of a match in the network stream, the packet ids a match spans, the direction of the packet on which a match happened, etc.
Pcap generation for matching flows is also supported. If enabled, it would dump all the packets from the start upto the end of the flow. Matched TCP flows are dumped as soon as a close/reset is seen and for those flows where we don't see a close/reset, they are dumped before the tool exits. For UDP, since there is no close/reset like state information available, they are dumped only when the tool exits. This ensure that all the packets, even those that arrive post match, are captured in the flow pcap. Except for the custom pcap global header, per-packet pcap header and the Ethernet II L2 header (which is not seen by flowinspect), everything above remains as-is in the dumped packet captures.
______ _ __
/ __/ /___ _ __(_)___ _________ ___ _____/ /_
/ /_/ / __ \ | /| / / / __ \/ ___/ __ \/ _ \/ ___/ __/
/ __/ / /_/ / |/ |/ / / / / (__ ) /_/ / __/ /__/ /_
/_/ /_/\____/|__/|__/_/_/ /_/____/ .___/\___/\___/\__/
/_/
flowinspect v0.2 - A network inspection tool
Ankur Tyagi (7h3rAm [at] gmail [dot] com)
usage: flowinspect.py [-h] (-p --pcap | -d --device) [-c --cregex]
[-s --sregex] [-a --aregex] [-i] [-m] [-G --cfuzz]
[-H --sfuzz] [-I --afuzz] [-r fuzzminthreshold]
[-C --cdfa] [-S --sdfa] [-A --adfa] [-l] [-X --dfaexpr]
[-g [graphdir]] [-P --cyararules] [-Q --syararules]
[-R --ayararules] [-M] [-y] [-Y --emuprofileoutsize]
[-O --offset] [-D --depth] [-T --maxinspstreams]
[-U --maxinsppackets] [-t --maxdispstreams]
[-u --maxdisppackets] [-b --maxdispbytes] [-w [logdir]]
[-o {quite,meta,hex,print,raw}] [-f --bpf] [-v] [-V]
[-e] [-k] [-j] [-Z] [-n] [-L]
optional arguments:
-h, --help show this help message and exit
-p --pcap input pcap file
-d --device listening device
RegEx per Direction:
-c --cregex regex to match against CTS data
-s --sregex regex to match against STC data
-a --aregex regex to match against ANY data
RegEx Options:
-i ignore case
-m disable multiline match
Fuzzy Patterns per Direction:
-G --cfuzz string to fuzzy match against CTS data
-H --sfuzz string to fuzzy match against STC data
-I --afuzz string to fuzzy match against ANY data
Fuzzy Options:
-r fuzzminthreshold threshold for fuzzy match (1-100) - default 75
DFAs per Direction ('m[0-9][1-9]=<dfa>'):
-C --cdfa DFA expression to match against CTS data
-S --sdfa DFA expression to match against STC data
-A --adfa DFA expression to match against ANY data
DFA Options:
-l switch default boolean operator to 'or'
-X --dfaexpr expression to test chain members
-g [graphdir] generate DFA transitions graph
Yara Rules per Direction:
-P --cyararules Yara rules to match on CTS data
-Q --syararules Yara rules to match on STC data
-R --ayararules Yara rules to match on ANY data
Shellcode Detection:
-M enable shellcode detection
-y generate emulator profile for detected shellcode
-Y --emuprofileoutsize
emulator profile memory size (default 1024K | max:
10240K)
Content Modifiers:
-O --offset bytes to skip before matching
-D --depth bytes to look at while matching (starting from offset)
Inspection Limits:
-T --maxinspstreams max streams to inspect
-U --maxinsppackets max packets to inspect
Display Limits:
-t --maxdispstreams max streams to display
-u --maxdisppackets max packets to display
-b --maxdispbytes max bytes to display
Output Options:
-w [logdir] write matching packets/streams
-o {quite,meta,hex,print,raw}
match output modes