pentagridsec/pentagridresponseoverviewResponse Overview Extension for BurpSuite - Find exotic responses by grouping response bodies
Response Overview Extension for BurpSuite
Author: Tobias "floyd" Ospelt, @floyd_ch, http://www.floyd.ch
Pentagrid AG, https://www.pentagrid.ch
This extension groups all response bodies by similarity and shows a summary, one request/response per group. The extension will allow a tester to get an overview of the tested website's responses from all tools (scanner, proxy, etc.). It provides an additional "semi-automated detection method" (compared to the usual detection methods response-based, time-based, interaction-based, etc.). There are various optimisations, mainly for performance reasons. The "remove parameters" feature removes reflected request parameters from the response before comparison.
So far:
Usage is very simple:
This extension analyses HTTP responses if:
When the above filter constraints are satisfied, the incoming response bodies (from all Burp tools!) are compared to all the groups we already created. A group is defined by its very first member, which we keep in memory. If the response we are processing is 95% similar to that first member, it belongs to that group and only the "Group Size" counter is increased. That also means we won't store that response. If the response is not 95% similar to any group, it forms a new group and it is its first member.
The first version of such a response overview that allows you to find anomalies I proposed in 2010 to the w3af project (see also a discussion here: https://github.com/andresriancho/w3af/issues/17345) and was written in Python. It never made it into mainline w3af, I don't remember why. I learned a lot about Python's difflib back then and optimized performance for the use case. I wrote a similar extension in Python for Burp in 2019 and again learned a lot about Python's difflib when I was working for modzero AG https://github.com/modzero/burp-ResponseClusterer. However, at one point I realized that the extension might be eating some of Burp's performance, which I at first ignored. In 2021 the extension broke completely as it wouldn't run with newer Jython versions anymore and while revisiting the code I realized that I coded a very memory-inefficient extension. So here we are, this is a new extension in 2021 in Kotlin with new features, again learning a lot from Python's difflib and different features. I did a performance improvement because I realized certain calculations are not necessary if we always compare against the same strings (as already implemented in the Python code). Reponse Clusterer is dead, long live Response Overview.
In theory, the default settings could result in not-that-great performance of Burp. However, this is highly unlikely to occur. A test where all responses had maximum default response size (around 1MB) had to be compared against each other, the comparison took up to 30ms with precalculated bByteCount optimization (regular case) and up to 80ms without this optimization (happens only once for each entry). Multiplied by the maximum amount of groups (1000) this means in the absolute worst case this could mean up to 80 seconds of processing time. As there is a separate thread, this would be bearable, as the next comparison would then only take a maximum time of 30 seconds. But which web applications has many different responses of around 1MB? Let's hope not too many. Also don't forget that if the length of the responses differ more than 2%, the similarity matching takes no time at all due to the veryQuickRatio optimization.
