Bypass Url Parser

该工具测试多种URL绕过技术，以访问40X保护的页面。

如果你好奇为什么这段代码只不过是一个简陋的curl封装，原因如下：

• 大多数Python请求库会对URL/路径/参数进行编码/解码，而我讨厌这一点。
• 如果我提交原始字符，我就要发送原始字符。
• 如果我发送一个奇怪的路径，我就想要它保持奇怪，而不是被标准化。

在Python中，要在不丢失库的所有好处（如解析、SSL/TLS封装等）的情况下实现这一点，出乎意料地困难。
所以，像我一样，使用curl作为后端，这就行了。

此外，该工具也可以作为库使用，参见lib_sample_usage.py

安装

我们推荐使用pipx来安装此工具：```bash pipx install bypass-url-parser

or for the latest dev version

pipx install git+https://github.com/laluka/bypass-url-parser

或者，你可以使用 `pip`：```bash
pip install bypass-url-parser

用法```

Bypass Url Parser, made with love by @TheLaluka A tool that tests MANY url bypasses to reach a 40X protected page.

Usage: bypass-url-parser (-u | -R ) [-m ] [-o ] [-S ] [ (-H

Program options: -u, --url URL (path is optional) to run bypasses against -R, --request Load HTTP raw request from a file -H, --header

General options: -h, --help Show help, you are here :) -v, --verbose Verbose output -d, --debug Show more details like curl commands generated by this tool -dd, --debug Print Debug level 2 (with all classes debug_class output) -V, --version Show version info

Misc options: --spoofip-replace Disable list of default internal IPs in 'http_headers_ip' bypass mode --spoofport-replace Disable list of default internal ports in 'http_headers_port' bypass mode --request-tls Force usage of TLS/HTTPS for the request load with the '-R, --request' option --dump-payloads Print all payloads (curls) generated by this tool. --jsonl Print results in JSON lines format (pipe command output)

Examples: bypass-url-parser -u "http://127.0.0.1/juicy_403_endpoint/" -s 8.8.8.8 -d bypass-url-parser -u /path/urls -t 30 -T 5 -H "Cookie: me_iz=admin" -H "User-agent: test" bypass-url-parser -R /path/request_file --request-tls -m "mid_paths, end_paths"

## 预期结果```bash
bypass-url-parser -u http://127.0.0.1:8000/foo/bar
2022-08-09 14:52:40 lalu-perso bup[361559] WARNING Trying to bypass 'http://127.0.0.1:8000/foo/bar' url (3213 payloads)...
2022-08-09 14:52:40 lalu-perso bup[361559] INFO Doing: 50 / 3213
[...]
2022-08-09 14:52:54 lalu-perso bup[361559] INFO Doing: 3200 / 3213
2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (1/3) the '16' failed curl commands with 10 threads and 10s timeout
2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (2/3) the '16' failed curl commands with 5 threads and 20s timeout
2022-08-09 14:52:54 lalu-perso bup[361559] INFO Retry (3/3) the '16' failed curl commands with 1 threads and 30s timeout
2022-08-09 14:52:55 lalu-perso bup[361559] INFO
[#####] [bypass_method] [payload] => [status_code] [content_type] [content_length] [lines_count] [word_counts] [title] [server] [redirect_url]
[GROUP (1587)] [original_request] [http://127.0.0.1:8000/foo/bar] => [404] [text/html] [469] [14] [95] [Error response] [SimpleHTTP/0.6 Python/3.8.10] []
[GROUP (10)] [http_methods] [-X CONNECT http://127.0.0.1:8000/foo/bar] => [501] [text/html] [500] [14] [96] [Error response] [SimpleHTTP/0.6 Python/3.8.10] []
[SINGLE] [mid_paths] [http://127.0.0.1:8000/???foo/bar] => [200] [text/html] [913] [26] [27] [Directory listing for /???foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
[SINGLE] [mid_paths] [http://127.0.0.1:8000//???foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/???foo/bar]
[SINGLE] [mid_paths] [http://127.0.0.1:8000/??foo/bar] => [200] [text/html] [911] [26] [27] [Directory listing for /??foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
[SINGLE] [mid_paths] [http://127.0.0.1:8000//??foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/??foo/bar]
[SINGLE] [mid_paths] [http://127.0.0.1:8000/?foo/bar] => [200] [text/html] [909] [26] [27] [Directory listing for /?foo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
[SINGLE] [mid_paths] [http://127.0.0.1:8000//?foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/?foo/bar]
[SINGLE] [mid_paths] [http://127.0.0.1:8000///?anythingfoo/bar] => [200] [text/html] [929] [26] [27] [Directory listing for ///?anythingfoo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
[SINGLE] [mid_paths] [http://127.0.0.1:8000////?anythingfoo/bar] => [200] [text/html] [931] [26] [27] [Directory listing for ////?anythingfoo/bar] [SimpleHTTP/0.6 Python/3.8.10] []
[GROUP (2)] [mid_paths] [http://127.0.0.1:8000/#?foo/bar] => [200] [text/html] [893] [26] [27] [Directory listing for /] [SimpleHTTP/0.6 Python/3.8.10] []
[GROUP (2)] [mid_paths] [http://127.0.0.1:8000//#?foo/bar] => [301] [] [] [0] [0] [] [SimpleHTTP/0.6 Python/3.8.10] [/]

设置

LINUX```bash

Deps

sudo apt install -y bat curl virtualenv python3

Tool

virtualenv -p python3 .py3 source .py3/bin/activate PDM_BUILD_SCM_VERSION="$(git describe --abbrev=0)-dev" pip install .

If bup installed globally, use

python src/bypass_url_parser/init.py -u https://thinkloveshare.com/juicy_403_endpoint/

Else this should work

bypass-url-parser -u https://thinkloveshare.com/juicy_403_endpoint/ cat /tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json | jq -r '.results[].request_curl_cmd' cat /tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json | jq -r '.results[].response_data'

### DOCKER```bash
docker run --rm -it -v "$PWD:/host" -w /host ghcr.io/laluka/bypass-url-parser:latest bash -il
# Then bup -h, keep the docker open as the output is saved by default in /tmp
# Or specify the output to the current directory, and consult them later! :)

关于支持的参数的更多信息

参数解析

Bypass_url_parser 允许以多种方式定义一些参数：

• -m, --mode, -s, --spoofip 和 -p, --spoofport 参数可以是文件名、字符串、逗号分隔的字符串列表或列表（当 Bypasser 作为库使用时）；
• -u, --url 参数可以是文件名、字符串或列表（当 Bypasser 作为库使用时）；
• 所有这些参数都支持 stdin（使用 -）。

例如，如果你想定义多个目标 URL（-u, --url），以下所有命令都会产生相同的结果：```bash bypass-url-parser -u http://thinkloveshare.com/test bypass-url-parser -u /path/urls cat /path/urls | bypass-url-parser -u - echo 'http://thinkloveshare.com/test' | bypass-url-parser -u -

### 目标定义

必须为工具定义一个目标才能工作。有2个选项：

- `-u, --url`: URL(s)，使用GET方法
- `-R, --request`: 请求文件。无法从文件中猜测协议，因此默认为`http`，如果存在`--request-tls`选项则为`https`。

### 绕过模式

如果指定了`-m, --mode`，您可以选择所需的绕过模式来运行特定测试（或多个测试），从而减少工具发送的请求数量。

目前，支持以下绕过模式：```
all, mid_paths, end_paths, case_substitution, char_encode, http_methods, http_versions, http_headers_method, http_headers_scheme, http_headers_ip, http_headers_port, http_headers_url, user_agent

示例```bash bypass-url-parser -u /path/urls -m "case_substitution, char_encode, http_headers_scheme"

### Spoofip / Spoofport

为了自定义绕过尝试中使用的IP地址和端口，该工具支持以下选项：

- 使用 `-s, --spoofip` 可以设置一些IP地址注入到`ip-specific`头（`X-Forwarded-For`、`X-Real-Ip`等）
- 使用 `-p, --spoofport` 可以设置一些端口注入到`port-specific`头（`X-Forwarded-Port`）

默认情况下，这些自定义条目会被添加到内部IP/端口列表中。如果你只想使用自己的IP/端口，可以使用 `--spoofip-replace` 和/或 `--spoofport-replace` 参数。

Example:```bash
bypass-url-parser -u /path/urls -s /path/custom_ip --spoofip-replace
bypass-url-parser -u /path/urls -p "3000, 9443, 10443"

JSON-Lines 输出与命令管道

使用 --jsonl 选项，可以将结果以 JSON-Lines 格式输出到 stdout 上。标准工具的输出和结果通过日志记录器显示在 stderr 上，因此可以将 JSON-Lines 输出格式与其他工具进行管道传递：```bash bypass-url-parser -u "https://thinkloveshare.com/juicy_403_endpoint/" -t 20 -S 0 -m case_substitution,char_encode --jsonl | jq

***Note:** With `-S 2` or `-S 3`, the JSON-Lines output also includes the path and the name of saved html files.*

### Results saving

By default, if target url is unique, the tool saves a copy of the results in `/tmp/tmpXXX-bypass-url-parser/` directory.

***Notes:** If multiple target urls are passed to `-u`, results are prefixed with the url as directory (`/tmp/tmpXXX-bypass-url-parser/http-target-com-8080-api-users/`).*

There are two arguments to customize this behavior:

- `-o, --outdir` to set a custom output directory
- `-S, --save-level` to choose a saving level

The saving levels are:

- `0` (NONE): Disable output saving and output directory creation;
- `1` (MINIMAL): Only save the program log file which contains the results: `triaged-bypass.log`;
- `2` (PERTINENT): Save the program log file `triaged-bypass.log` and **pertinent (results)** curl responses in `triaged-bypass.json` file and separate html files (Default);
- `3` (FULL): Save the program log file `triaged-bypass.log` and **all** curl responses in `triaged-bypass.json` file and separate html files.

#### Example```bash
bypass-url-parser -S 0
bypass-url-parser -S 1 -o /tmp/bypass-res
bypass-url-parser -S 2 -o /tmp/bypass-res2 -H "User-Agent: curl 7.74.0" -u http://thinkloveshare.com/juicy_403_endpoint/
tree /tmp/bypass-res2/
├── bypass-2469eecf6c38b5817d2248e911ad4382.html
├── bypass-6f7cce7caf0a0a4b440859fa189d496d.html
├── bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html
├── bypass-93079abffe63d34f79ac4a511cd6b5e6.html
├── bypass-945822230d58d1ad4680d5dfbc470ecb.html
├── bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html
├── triaged-bypass.json
└── triaged-bypass.log

0 directories, 8 files

结果导出

从 MINIMAL 级别开始，程序显示的结果会保存在 triaged-bypass.log 文件中。

JSON 导出

在 PERTINENT 和 FULL 保存级别下，程序还会将所有结果导出到 triaged-bypass.json 文件中：```json { "url": "http://thinkloveshare.com/juicy_403_endpoint/", "bypass_modes": "all", "results": [ { "request_curl_cmd": "/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -H 'X-BlueCoat-Via: localhos[...SNIP...]", "request_curl_payload": "-H X-BlueCoat-Via: localhost http://thinkloveshare.com/juicy_403_endpoint/", "response_headers": "HTTP/1.1 301 Moved Permanently\nConnection: keep-alive\nContent-Length: 162\nServer: GitHub.c[...SNIP...]", "response_data": "\n301 Moved Permanently\n\n

301 Moved Permane[...SNIP...]", "response_status_code": 301, "response_content_type": "text/html", "response_content_length": 162, "response_lines_count": 7, "response_words_count": 4, "response_title": "301 Moved Permanently", "response_server_type": "GitHub.com", "response_redirect_url": "", "response_html_filename": "bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html" }, { "request_curl_cmd": "/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -X PROPFIND http://thinklove[...SNIP...]", "request_curl_payload": "-X PROPFIND ", "response_headers": "HTTP/1.1 405 Method Not Allowed\nConnection: close\nContent-Length: 131\nServer: Varnish\nRet[...SNIP...]", "response_data": "\n405 Not Allowed\n<body bgcolor="white">\n\nYour browser sent an invalid request.\n\n", "response_status_code": 400, "response_content_type": "text/html", "response_content_length": 90, "response_lines_count": 3, "response_words_count": 7, "response_title": "", "response_server_type": "", "response_redirect_url": "", "response_html_filename": "bypass-2469eecf6c38b5817d2248e911ad4382.html" } ] } 19 │ Your browser sent an invalid request. 20 │ ───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ File: /tmp/bypass-res2/bypass-6f7cce7caf0a0a4b440859fa189d496d.html ───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ /usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -X CONNECT 2 │ 3 │ HTTP/1.1 400 Bad Request 4 │ Connection: close 5 │ Content-Length: 15 6 │ content-type: text/plain; charset=utf-8 7 │ x-served-by: cache-par-lfpg1960083 8 │ 9 │ invalid request ───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ File: /tmp/bypass-res2/bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html ───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ /usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is 2 │ 3 │ HTTP/1.1 400 Bad Request 4 │ Connection: keep-alive 5 │ Content-Length: 9121 6 │ Server: GitHub.com 7 │ Content-Type: text/html; charset=utf-8 8 │ ETag: "64417b9f-23a1" 9 │ Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self' 10 │ X-GitHub-Request-Id: 598E:F13E:26EE27D:284F5D2:64486731 11 │ Accept-Ranges: bytes 12 │ Date: Tue, 25 Apr 2023 23:50:11 GMT 13 │ Via: 1.1 varnish 14 │ X-Served-By: cache-par-lfpg1960046-PAR 15 │ X-Cache: MISS 16 │ X-Cache-Hits: 0 17 │ X-Timer: S1682466611.396077,VS0,VE101 18 │ Vary: Accept-Encoding 19 │ X-Fastly-Request-ID: 30bd6af5892c40da130ee49bbeacd147a1a6b3c3 20 │ 21 │ 22 │ 23 │ 24 │ 25 │ 26 │ Bad request · GitHub Pages 27 │ [...SNIP...]