Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Einreichen
ToolsExploitsBlog
Einreichen

Hacking-, PenTest- und Cybersicherheits-Tools für Ihr Sicherheitsarsenal!

Kitploit ist ein Verzeichnis von Hacking-, Cybersicherheits- und Pentesting-Tools. Entdecken Sie die neuesten Projekt-Updates, um Schwachstellen zu finden, Systeme zu analysieren, Tests zu automatisieren und Ihre Sicherheit zu stärken.

FeedsKontaktDatenschutz© 2026 Kitploit

Tool-Verzeichnis

Kategorien

Alle Kategorien anzeigen
Loading categories
duplicut — Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking) | Kitploit
Tools/GitHubGitHub/nil0x42/duplicut
General Purpose UtilitiesPassword CrackingUtilities & Frameworks
GitHubnil0x42/duplicut

duplicut

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

Repository anzeigen
9809639vor 8 TagenVon Kitploit geprüft
Teilen
Inhalt in der angeforderten Sprache nicht verfügbar. Englische Version wird angezeigt.

Duplicut :scissors:

Quickly dedupe massive wordlists, without changing the order tweet


github tests codacy code quality github codeql codecov coverage

Mentioned in awesome-pentest Rawsec tool inventory Follow on X

Created by nil0x42 and contributors


Order and memory tradeoffs of sort and awk

📖 Overview

Duplicut removes duplicate lines from massive wordlists without changing their order. Keep the first occurrence of each word, even when the file exceeds RAM.

When combining password wordlists, the most likely candidates should stay first. Sorting changes that priority; keeping every unique word in memory can exhaust RAM. Duplicut is written in optimized C and automatically chooses between chunk-based scans and disk partitions to fit the available resources.


Duplicut terminal demonstration

💡 Quick start

git clone https://github.com/nil0x42/duplicut  # download
cd duplicut && make                         # compile
./duplicut wordlist.txt > clean-wordlist.txt # dedupe

Prefer a ready-to-run binary? Download the matching Linux or macOS binary from the latest release. Linux and macOS builds are available for x64 and ARM64. Build and installation details.

🔧 Options

Run ./duplicut --help for command-line help. Input comes from a named file or, when omitted, from stdin. Results go to stdout unless -o is given; progress and diagnostics go to stderr.

UseOptionBehavior
Output-o, --outfile FILEWrite to a file instead of stdout.
-D, --dupfile FILESave removed duplicate occurrences; adds I/O.
Resources-m, --memlimit VALUEWorking-memory budget, e.g. 100M or 2G; defaults to available memory.
-t, --threads NUMMaximum worker threads; defaults to online CPUs.
Filtering-l, --line-max-size NUMMaximum line length in bytes; default 1024, maximum 4095.
-p, --printableKeep only printable ASCII lines.
Case-c, --lowercaseConvert to lowercase before deduplication.
-C, --uppercaseConvert to uppercase before deduplication.
Engine-e, --engine ENGINEauto (default), mmap-chunks or disk-partitions.
Information-h, --helpShow help.
-v, --versionShow version and license information.

💡 Common uses

# Combine lists while preserving their priority
cat preferred.txt extra.txt | ./duplicut > combined.txt

# Normalize case and keep printable ASCII
./duplicut -cp wordlist.txt > clean-wordlist.txt

# Set a working-memory budget and save duplicate occurrences
./duplicut -m 100M -D duplicates.txt wordlist.txt -o unique.txt

# Append the result of this run to an existing file
./duplicut wordlist.txt >> combined.txt

Append mode deduplicates the new input, not the existing destination contents. See the usage guide for line handling and output behavior, and memory and disk use for what the memory budget covers.

⚡ Observed performance

On a laptop with 16 GB RAM and an NVMe SSD:

WorkloadObserved time
Real wordlist: about 285 million lines, 2.58 GiB~36 seconds
Generated wordlist: 20 GiB, larger than RAM~2 min 20 seconds

These are measured workloads, not a throughput guarantee. Workloads, options and measurement details.

📖 Documentation

Documentation index · How it works · Troubleshooting · Development

Tool herunterladen

Beliebteste

Alle anzeigen →

Entdecken Sie die meistgenutzten Tools unserer Community.

Noch keine Beliebtheitsdaten.

Alle Tools erkunden

Durchsuchen Sie unsere Tool-Sammlung

Alle Tools anzeigen →