Skip to content
KitploitKITPLOIT
工具漏洞利用博客
Log in
提交
工具漏洞利用博客
提交

黑客、渗透测试和网络安全工具,武装您的安全武器库!

Kitploit 是一个黑客、网络安全和渗透测试工具的目录。发现最新的项目更新,查找漏洞、分析系统、自动化测试并加强你的安全。

订阅源联系隐私© 2026 Kitploit

工具目录

分类

查看所有分类
Loading categories
duplicut — Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking) | Kitploit
工具/GitHubGitHub/nil0x42/duplicut
General Purpose UtilitiesPassword CrackingUtilities & Frameworks
GitHubnil0x42/duplicut

duplicut

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

查看仓库
98096398天前Kitploit 审核通过
分享
内容在请求的语言中不可用。显示英文版本。

Duplicut :scissors:

Quickly dedupe massive wordlists, without changing the order tweet


github tests codacy code quality github codeql codecov coverage

Mentioned in awesome-pentest Rawsec tool inventory Follow on X

Created by nil0x42 and contributors


Order and memory tradeoffs of sort and awk

📖 Overview

Duplicut removes duplicate lines from massive wordlists without changing their order. Keep the first occurrence of each word, even when the file exceeds RAM.

When combining password wordlists, the most likely candidates should stay first. Sorting changes that priority; keeping every unique word in memory can exhaust RAM. Duplicut is written in optimized C and automatically chooses between chunk-based scans and disk partitions to fit the available resources.


Duplicut terminal demonstration

💡 Quick start

git clone https://github.com/nil0x42/duplicut  # download
cd duplicut && make                         # compile
./duplicut wordlist.txt > clean-wordlist.txt # dedupe

Prefer a ready-to-run binary? Download the matching Linux or macOS binary from the latest release. Linux and macOS builds are available for x64 and ARM64. Build and installation details.

🔧 Options

Run ./duplicut --help for command-line help. Input comes from a named file or, when omitted, from stdin. Results go to stdout unless -o is given; progress and diagnostics go to stderr.

UseOptionBehavior
Output-o, --outfile FILEWrite to a file instead of stdout.
-D, --dupfile FILESave removed duplicate occurrences; adds I/O.
Resources-m, --memlimit VALUEWorking-memory budget, e.g. 100M or 2G; defaults to available memory.
-t, --threads NUMMaximum worker threads; defaults to online CPUs.
Filtering-l, --line-max-size NUMMaximum line length in bytes; default 1024, maximum 4095.
-p, --printableKeep only printable ASCII lines.
Case-c, --lowercaseConvert to lowercase before deduplication.
-C, --uppercaseConvert to uppercase before deduplication.
Engine-e, --engine ENGINEauto (default), mmap-chunks or disk-partitions.
Information-h, --helpShow help.
-v, --versionShow version and license information.

💡 Common uses

# Combine lists while preserving their priority
cat preferred.txt extra.txt | ./duplicut > combined.txt

# Normalize case and keep printable ASCII
./duplicut -cp wordlist.txt > clean-wordlist.txt

# Set a working-memory budget and save duplicate occurrences
./duplicut -m 100M -D duplicates.txt wordlist.txt -o unique.txt

# Append the result of this run to an existing file
./duplicut wordlist.txt >> combined.txt

Append mode deduplicates the new input, not the existing destination contents. See the usage guide for line handling and output behavior, and memory and disk use for what the memory budget covers.

⚡ Observed performance

On a laptop with 16 GB RAM and an NVMe SSD:

WorkloadObserved time
Real wordlist: about 285 million lines, 2.58 GiB~36 seconds
Generated wordlist: 20 GiB, larger than RAM~2 min 20 seconds

These are measured workloads, not a throughput guarantee. Workloads, options and measurement details.

📖 Documentation

Documentation index · How it works · Troubleshooting · Development

下载工具

最受欢迎

查看全部 →

发现我们社区最常用的工具。

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 终极 OSINT 工具包:调查图谱、置信度评分、别名家族、地图、时间线和报告。49 个模块、400+ 个站点,无需 API 密钥。

    640
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock 一键执行应用(CVE-2026-43499)

    479
  • 3

    ARTEX

    baarai/artex

    基于LLM的多智能体自主渗透测试平台,具备规划器/执行器智能体、资产与探索图谱、MITM流量记录以及漏洞复测功能。

    391
  • 4

    redforge

    cybersecurity91/redforge

    浏览器端红队工具包,包含 102 个离线工具,用于侦察、载荷生成、凭据攻击、加密、编码和交战报告。

    335
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    AI原生网络安全的行动系统——在这里,意图转化为受治理的执行,证据沉淀为运营记忆,每一次操作都让下一次更出色。

    258
  • 6

    antifox

    antifox-ai/antifox

    AI multi-agent autonomous penetration testing platform that runs recon, attack-surface discovery, exploitation validation and evidence-backed report delivery.

    249

探索所有工具

浏览我们的工具集合

查看所有工具 →