Skip to content
KitploitKITPLOIT
OutilsExploitsBlog
Log in
Soumettre
OutilsExploitsBlog
Soumettre

Outils de Hacking, PenTest et Cybersécurité pour votre Arsenal de Sécurité !

Kitploit est un répertoire d'outils de hacking, de cybersécurité et de pentesting. Découvrez les dernières mises à jour des projets pour trouver des vulnérabilités, analyser des systèmes, automatiser les tests et renforcer votre sécurité.

FluxContactConfidentialité© 2026 Kitploit

Répertoire d'outils

Catégories

Voir toutes les catégories
Loading categories
duplicut — Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking) | Kitploit
Outils/GitHubGitHub/nil0x42/duplicut
General Purpose UtilitiesPassword CrackingUtilities & Frameworks
GitHubnil0x42/duplicut

duplicut

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

Voir le dépôt
9809639il y a 8 joursVérifié par Kitploit
Partager
Contenu non disponible dans la langue demandée. Affichage de la version anglaise.

Duplicut :scissors:

Quickly dedupe massive wordlists, without changing the order tweet


github tests codacy code quality github codeql codecov coverage

Mentioned in awesome-pentest Rawsec tool inventory Follow on X

Created by nil0x42 and contributors


Order and memory tradeoffs of sort and awk

📖 Overview

Duplicut removes duplicate lines from massive wordlists without changing their order. Keep the first occurrence of each word, even when the file exceeds RAM.

When combining password wordlists, the most likely candidates should stay first. Sorting changes that priority; keeping every unique word in memory can exhaust RAM. Duplicut is written in optimized C and automatically chooses between chunk-based scans and disk partitions to fit the available resources.


Duplicut terminal demonstration

💡 Quick start

git clone https://github.com/nil0x42/duplicut  # download
cd duplicut && make                         # compile
./duplicut wordlist.txt > clean-wordlist.txt # dedupe

Prefer a ready-to-run binary? Download the matching Linux or macOS binary from the latest release. Linux and macOS builds are available for x64 and ARM64. Build and installation details.

🔧 Options

Run ./duplicut --help for command-line help. Input comes from a named file or, when omitted, from stdin. Results go to stdout unless -o is given; progress and diagnostics go to stderr.

UseOptionBehavior
Output-o, --outfile FILEWrite to a file instead of stdout.
-D, --dupfile FILESave removed duplicate occurrences; adds I/O.
Resources-m, --memlimit VALUEWorking-memory budget, e.g. 100M or 2G; defaults to available memory.
-t, --threads NUMMaximum worker threads; defaults to online CPUs.
Filtering-l, --line-max-size NUMMaximum line length in bytes; default 1024, maximum 4095.
-p, --printableKeep only printable ASCII lines.
Case-c, --lowercaseConvert to lowercase before deduplication.
-C, --uppercaseConvert to uppercase before deduplication.
Engine-e, --engine ENGINEauto (default), mmap-chunks or disk-partitions.
Information-h, --helpShow help.
-v, --versionShow version and license information.

💡 Common uses

# Combine lists while preserving their priority
cat preferred.txt extra.txt | ./duplicut > combined.txt

# Normalize case and keep printable ASCII
./duplicut -cp wordlist.txt > clean-wordlist.txt

# Set a working-memory budget and save duplicate occurrences
./duplicut -m 100M -D duplicates.txt wordlist.txt -o unique.txt

# Append the result of this run to an existing file
./duplicut wordlist.txt >> combined.txt

Append mode deduplicates the new input, not the existing destination contents. See the usage guide for line handling and output behavior, and memory and disk use for what the memory budget covers.

⚡ Observed performance

On a laptop with 16 GB RAM and an NVMe SSD:

WorkloadObserved time
Real wordlist: about 285 million lines, 2.58 GiB~36 seconds
Generated wordlist: 20 GiB, larger than RAM~2 min 20 seconds

These are measured workloads, not a throughput guarantee. Workloads, options and measurement details.

📖 Documentation

Documentation index · How it works · Troubleshooting · Development

Télécharger l’outil

Populaires

Voir tout →

Découvrez les outils les plus utilisés par notre communauté.

Pas encore de données de popularité.

Explorer tous les outils

Parcourez notre collection d'outils

Voir tous les outils →