thermoptic

"I don't believe it, thermoptic camouflage!"

What is it?

This is an HTTP proxy designed to bypass services that use fingerprinting such as JA4+ to block certain HTTP clients. Using this proxy, you can use your preferred HTTP clients like curl and still have magically indistinguishable fingerprints from a real (Chrome/Chromium) web browser. thermoptic also comes with some fun features to mitigate JavaScript-based fingerprinting. It also makes it easy to do hybrid scraping using both a web browser and low-level HTTP clients together.

Even if you’re unfamiliar with JA4+ fingerprinting, if you’ve done any scraping you’ve probably been blocked by it before. Popular services such as Cloudflare use such techniques (and other tricks) to detect use of "non-human" HTTP clients to block requests. These services can also use this fingerprinting to detect if you start a session with a real browser and then switch to a low-level client like curl later. thermoptic solves all of these problems by presenting a unified "real" browser fingerprint for all scraping requests.

Example

Here's an example JA4H (HTTP) fingerprint of curl without the proxy:

$ curl https://ja4db.com/id/ja4h/
ge11nn090000_b6a016211e8a_000000000000_e3b0c44298fc

This is quite different from the fingerprint that Chrome produces when you visit the URL directly:

ge11cn19enus_f2808f0d04cf_9a10d4221160_7068f58def6e

However, when we use the proxy to make the request, our JA4H fingerprint is magically identical:

$ curl --proxy http://thermoptic:1234 https://ja4db.com/id/ja4h/
ge11cn19enus_f2808f0d04cf_9a10d4221160_7068f58def6e

(The same goes for our JA4 TLS fingerprint as well, etc).

Setup

To start a thermoptic proxy which cloaks your traffic through a containerized Chrome instance on Ubuntu 22.04:

Regular Docker setup (works on hosts without a GPU runtime):

docker compose up --build

That's all, now you can proxy traffic through it:

curl --proxy http://127.0.0.1:1234 --insecure https://ja4db.com/id/ja4h/

Important notes:

• By default the proxy runs without authentication. If you plan on exposing the proxy externally please make sure to set authentication with the environment variables PROXY_USERNAME and PROXY_PASSWORD.
• If you don't want to use ---insecure you need to use the generated CA file located in ./ssl/rootCA.crt. This is generated the first time you run thermoptic.
• You can connect thermoptic to any Chrome/Chromium instance launched with the --remote-debugging-port flag. This is essential as you'll want to set up and proxy through more commonly used environments to keep your fingerprint as low profile as possible (e.g. Chrome on Windows).
• The GPU compose override is intended for NVIDIA hosts that already have Docker's NVIDIA runtime/toolkit installed. It reserves the GPU device, mounts /dev/dri, and lets the bundled Chrome container switch to the NVIDIA/Vulkan rendering path. To use this, run docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build.

Features

• 🕵️ Browser-parity proxying that replays requests through a real Chrome session to match JA4 fingerprints byte-for-byte.
• 🤝 Little to no custom code required to integrate your HTTP client (e.g. curl, requests, etc) with thermoptic, just set the proxy and your fingerprints are taken care of.
• 🪝 Hook framework for before-request/after-request/on-start automation so you can drive the full browser to solve challenges, or capture artifacts.
  • 📘 An example Cloudflare turnstile solving hook can be seen in ./hooks/onstart.js.
• 🖥️ Web-browser control UI (at http://127.0.0.1:14111) to control the Dockerized Chrome browser window. Useful to manually log into sites manually and then seemlessly use the proxy to make requests as your logged-in session (and for debugging).
• 🔌 Set an upstream HTTP or SOCKS proxy URI via the UPSTREAM_PROXY environment variable in docker-compose.yml.
• 🛡️ Built-in health checks and restart control loop to detect frozen browsers and recover automatically without operator babysitting.
• ⚡ Supports HTTP/1.1 and HTTP/2, allowing traffic to be proxied over either protocol. (Note that you may talk HTTP/1.1 to the proxy, and the puppeted Chrome may negotiate with the end site over a different protocol.)

How does this cloaking work exactly?

• An HTTP request is made using an HTTP client such as curl with thermoptic set as a proxy.
• thermoptic analyzes the request to best determine what type of browser request it's supposed to be (e.g. manual URL visit? Form submission? A fetch() request?).
• thermoptic uses the Chrome Debugging Protocol (CDP) to puppet the browser and set up a page that mocks the request exactly as it normally would occur in a real web browser.
• thermoptic triggers the request via the mocked context and captures the HTTP response.
• thermoptic sends the HTTP response back to the client.

Due to the fact that the browser is actually making the request using its full stack, the resulting JA4 fingerprints are identitical.

NOTE: Due to many WAFs employing JavaScript-level fingerprinting of web browsers, thermoptic also exposes hooks to utilize the browser for key steps of the scraping process. See this section for more information on this.

Why this approach vs other solutions?

To put it bluntly: other approaches have fundamental flaws which prevent them from being a practical long term solution to the browser fingerprinting problem.

Many other attempts to "beat" browser JA4+ fingerprinting do so by reimplementing the various layers of the browser stack. This approach has a number of serious drawbacks, such as:

• Requiring great care to match the behavior of the "real" browser implementation perfectly. As a result, any quirks or discrepancies can be used to differentiate these clients from the "real" browser implementation.
• Attempting to only solve the problem at one layer of the stack. Chrome utilizes multiple protocols to provide a web browsing experience. As a result, even if you've made a perfectly-matching TLS layer your HTTP layer may give you away if it's not byte-perfect.
• As the "real" browsers regularly change their behavior, their fingerprints change, and as a result these tools constantly require more intensive development work to compensate.

In contrast, because thermoptic uses the browser itself to perform HTTP requests:

• Each layer of the stack such as TCP, TLS, HTTP are indistinguishable from the real browser because the request is made using a real browser in the way it would normally occur.
• Changes to browser behavior at various layers are minimally disruptive, the browser thermoptic controls just needs to be updated in order to match the latest set of fingerprints.

Of course, no solution is without drawbacks. See the DOWNSIDES.md docs for a fleshed out list of downsides to the thermoptic approach.

FAQ