In the realm of cybersecurity, accurately identifying and characterizing web servers is crucial for threat detection, vulnerability assessment, and network mapping. We introduce HTTP-Basma, a novel active fingerprinting algorithm that unveils unique server profiles through a multi-layered approach.
HTTP-Basma is live at https://httpbasma.netomize.ca/
In the realm of cybersecurity, accurately identifying and characterizing web servers is crucial for threat detection, vulnerability assessment, and network mapping. We introduce HTTP-Basma, a novel active fingerprinting algorithm that unveils unique server profiles through a multi-layered approach, thereby addressing this challenge.
Key Features: Crafted Requests, Revealing Responses: HTTP-Basma sends 8 meticulously designed HTTP probes, eliciting distinctive responses that reflect server configurations. Dual Hashing for Versatility. The algorithm generates two hashes:
Clustering and Hunting: These hashes empower server clustering, identification of unique and similar servers, and the pursuit of malicious actors with heightened confidence.
Modular Design for Expansion: The algorithm's architecture fosters the addition of new hashing variants, encouraging collaboration and adaptability.
In this paper, we first survey notable existing work on HTTP fingerprinting and then explore the algorithm's functionality, design, architecture, and outcomes. Additionally, we will showcase compelling findings from scanning the top 1 million Majestic websites, including the identification and clustering of C&C HTTP servers for various malware families.
HTTP-Basma’s algorithm's core idea centers on sending 8 specially crafted HTTP requests with varying requirements to elicit different responses from the server. Once the server response is retrieved, the HTTP status line is surgically dissected for all elements and encoded optimally. Additionally, select headers from the server response are checked for encoding as well.
The requests it sends are of these types:
Following each request, the server's response is analyzed to extract specific headers and their values. This extracted data undergoes further processing, including dissection and encoding, to generate a reversible fingerprint.
The full technical details of how the algorithm works are in the attached paper.
This modular design philosophy treats each request fingerprint as a building block, allowing for elegant refactoring, with the possibility to add and subtract the fingerprint of any request.
Sample of fingerprints:
- verbosus fp: 011420958a0014514bd5221420958a221420958a221420958a2200001420958a22000000001f
- pacto fp: 02464ae8b7d86f82c9918e2c2b9d6b91
- note: false-positive rate (72/986,910)
- verbosus fp: 01142494d60914514bd522142494d6221420958a701420958a220000140e04922032c37f1609
- pacto fp: 020769322f3d94ac2f258ddf5ce08502
- note-1: false-positive rate 0
- note-2: tevedadav.site/43.209.165.126:443 (TLS)
- sample-(sha-256): 9aa1dec8dd12f8adc7fc1274e1958f3613450109ee8b4ec6442a0fcf06df0972
- verbosus fp: 01140a85e40014512f3612140a85e422140a85e422140a85e4220000140a85e4220000000001
- pacto fp: 0207292309a7a7e798e417d69df5f2a5
- note: false-positive rate (73/986,910)
- verbosus fp: 01140a85e4001320958a22142494d62214254c5e2214254c5e22080014254c5e220000000000
- pacto fp: 0202be780e1eaae0eaa6184e20c909b6
- note: false-positive rate (4/986,910)
- verbosus fp: 01140a85e4011320958a22142494d67214254c5e2214254c5e22080014254c5e220000000000
- pacto fp: 02cc5be6d05192e17de041538508bc22
- note: false-positive rate (38/986,910)
- verbosus fp: 01140a85e40914514bd522140a85e4721420958a701420958a220800140a85e4720000001609
- pacto fp: 0221b4e46bbd0e5c037f5a852ca3fdc0
- note: false-positive rate (6/986,910)
HTTP-Basma is a C++ tool I developed to showcase the practicality and viability of this algorithm. It leverages Chilkat's library for all HTTP socket interactions and utilizes other supporting classes within the library. Additionally, the tool includes a demangler feature that can dissect and reverse the verbosus fuzzy-hash, outputting a comprehensive JSON object, and a comparator function that outputs the differences between two verbosus fingerprints.
Be aware that some output of the tool might use slightly different probe numbers, but the underlying order remains consistent: P1->P1, P2->P2, P3->P3, P4->P4, P->P5, P6->P6F, P7->P6L, P8->P7a.
Usage:
HTTP-Basma [OPTION...]
-d, --domain arg domains/IPs (you may query multiple domains, comma separated)
-p, --port arg port number
-s, --ssl does the HTTP connection have to be carried over SSL/TLS?
-q, --qpath check domain with url path included (not recommended)
-w, --redirect enable/disable HTTP redirects. If disabled/false, only the next redirect is followed,
otherwise, all redirects are followed (default: true)
-t, --ctimeout arg socket connection timeout value in seconds (default: 1)
-g, --rtimeout arg socket read (from the server) timeout value in seconds (default: 1)
-e, --sleep arg the duration (in milliseconds) to pause between each request (default: 100)
-x, --proxy arg proxy config: <"socks4|socks5|http">,<domain>,<port>,<bool:direct_tls>,<login>,<pass>
all values are comma-separated. <direct_tls> is ignored with a non-HTTP proxy
-f, --file arg file with list of domains/IPs (requires "-c/--csv" or "-j/--json")
-P, --parallel Scan list of domains passed via the "-f/--file" option in parallel
-c, --csv save to csv file; if the option 'n' is not specified, the CSV filename will be auto
generated
-n, --csvfile arg name of the CSV file
-j, --json save to json file; if the option 'l' is not specified, the JSON filename will be auto
generated
-l, --jsonfile arg name of the JSON file
-r, --saveh save request response headers
-o, --pjson display fingerprint dissection to the console as a JSON object
-i, --demangle_json arg demangle a fingerprint into a detailed json format (you can have more than one, comma
separated)
-u, --demangle_txt arg output a concise text format of the fingerprint, comma-separated for multiple results
-C, --compare arg compare two verbosus fingerprints (comma-separated)
-a, --pacto arg obtain the Pacto fingerprint using Verbosus
-h, --help print usage