Magic Extractor

Description

Magic Extractor is a universal extraction tool for Windows that identifies a file with several detectors and routes it to the right bundled extractor. It aims to cover mainstream compression formats, the installers you actually see today, and a range of less common archivers.

It auto-detects 80+ formats — archives, installers, disk images, forensic images (EWF/AFF/AD1), disc images, mail stores and modern codecs. See formats.md for the full list.

Quick Start

Download the latest release, unzip it, and run:

magic-extractor.exe extract mystery.bin

See Examples for identify, list, carve, --recursive and --bruteforce.

Project Structure

• cli: source code.
  • bin: bundled detector and extractor binaries.
    • detectors: DIE, Magika, binwalk (TrID's defs are converted to data/signatures.json).
    • extractors: 7z, unrar, unace, unshield, lessmsi, dark (WiX), and more.
  • data: runtime configuration, loaded dynamically (see below).
  • formats: one handler module per format family.
• gui: optional tkinter front-end that wraps the CLI (see GUI).
• test: sample files per format (fixtures for the extraction/detection tests).
• tools: developer tooling (generate_data.py — builds the data files from handlers).

The compiled build keeps bin/, data/ and config.ini external to the exe so they can be updated with a file swap; the path resolver in main.py finds them beside the executable (frozen) or under cli/ (dev).

How detection works

For normal extraction, detectors run in this order with early-exit — the first one that yields a known handler wins (cheapest first, so the ML model is usually skipped):

1. puremagic — pure-python, no subprocess; a cheap MIME check for well-formed archives.
2. built-in signatures — magic-byte patterns in data/signatures.json; names archivers the engines miss (bcm, dgca, kgb, uharc, alzip, freearc, ...) with no external process.
3. DIE (Detect It Easy) — signature engine; the specialist for installers, PE and SFX.
4. binwalk — short type keys (cpio, lzma, ...) and embedded content.
5. Magika — Google's AI content-type detector, as a catch-all.

Each detector contributes uniquely (they are complementary, not redundant): the signature DB names archivers the engines miss, DIE handles installers/PE, binwalk catches a few types the others miss, puremagic/Magika cover MIME.

PureMagic 2.x also provides content-aware deep scanning when it receives the full file. Full-file analysis is the default; the optional --fast-check modifier passes only the first 64 KiB when startup speed matters more than deep inspection.

Each handler declares its own indicators via detection_mimes() / detection_names() / detection_signatures(); tools/generate_data.py compiles these into data/handlers.json and data/signatures.json (with an optional data/extra_detections.json merged on top). TrID is not used.

Note: any format whose signature is missing from puremagic's magic_data.json (or that puremagic reports only as a generic application/octet-stream) must declare a custom detection_signatures() entry in its handler — otherwise it will not be detected by content.

• --bruteforce disables early-exit: every detector runs and each detected handler is tried in turn (useful when the first guess is wrong).
• Executables that no detector identifies fall back to the wrapped-exe installer handlers (BitRock, Clickteam, Inno, ...), which self-validate.
• The carve subcommand additionally uses binwalk's offset map to extract archives embedded at arbitrary offsets (e.g. inside firmware images).

The detection → handler routing map lives in data/handlers.json (hand-curated, loaded at runtime); a generic-token blacklist lives in data/detection_blacklist.json.

Supported Formats

See formats.md for the full list of formats and their handlers.

Adding a format

To add support for a new format, see docs/adding-a-handler.md — the end-to-end guide (handler class, detection declaration, DIE/TrID lookup, magic signatures, regenerating the routing data, bundling the tool, and testing).

Installation (from source)

Most users just download the release (see Quick Start). To run from source: Python 3.12 or newer is required.

git clone <repo-url>
cd magic-extractor
pip install -r cli/requirements.txt

Usage

Magic Extractor uses subcommands:

python cli/main.py extract  <path> [output_dir] [options]   # detect and extract
python cli/main.py identify <path>                          # report type + candidate handlers
python cli/main.py list     <path>                          # list archive contents
python cli/main.py carve    <path> [output_dir] [options]   # carve embedded archives (binwalk offsets)

A bare path with no subcommand defaults to extract (backward compatible):

python cli/main.py <path> <output_dir> [options]

extract options:

• --password <password>: password for encrypted archives.

• -r, --recursive: extract archives found inside the output (bounded by --max-depth, default 5).

• -b, --bruteforce: try every handler detected instead of stopping at the first.

• --open-output-folder <true|false>: open the output folder when done.

• --check-free-space <true|false>: warn if the output volume may lack room.

• --check-unicode <true|false>: warn about non-ASCII extracted names.

• --fix-file-extensions <true|false>: give extensionless extracted files a content-based extension (never overwrites an existing one).

• --create-log-files <true|false>: write a per-run log to the output dir.

  (Each defaults to its config.ini value when omitted; combine with --update-defaults to persist the given value — e.g. --open-output-folder false --update-defaults turns a previously-saved default off.)

• --fast-check: check only the first 64 KiB instead of running full-file detection.

• --update-defaults: persist the given settings as defaults in config.ini.

carve options: --list (print the binwalk fragment table), --fragment N (carve one fragment by index), --raw (carve every fragment, not only handler-known ones).

In the examples below, magic-extractor is the built .exe. From source, replace it with python cli/main.py — the arguments are identical.

Examples

Extract an archive — you don't need to know its type; it is auto-detected:

magic-extractor extract mystery.bin
# extracts into mystery_extracted/ next to the file

Identify a file without touching it — shows what each detector saw and which handler would run:

magic-extractor identify setup.exe
File: setup.exe
  [DIE] detect   inno setup installer
Candidate handlers (in order):
  - FormatInnoSetupHandler

List an archive's contents (no extraction):

magic-extractor list backup.7z

Recursive — extract archives found inside the output (e.g. a .tar.gz, or an installer that contains more archives), up to --max-depth levels:

magic-extractor extract app-1.0.tar.gz --recursive

Bruteforce — when detection is unsure, try every handler that matched instead of stopping at the first:

magic-extractor extract weird-archive.dat --bruteforce

Carve — pull archives that are embedded at some offset inside a bigger file (classic for firmware images). Inspect first, then carve:

magic-extractor carve router-firmware.bin --list
IDX      OFFSET          SIZE  NAME       DESCRIPTION
  0  0x00000000       793,720  pe         Windows PE binary
  1  0x000c1c78     2,495,983  lzma       LZMA compressed data

magic-extractor carve router-firmware.bin              # carve + extract the known blobs
magic-extractor carve router-firmware.bin --fragment 1 # carve only fragment #1

GUI

An optional tkinter front-end (in gui/) wraps the CLI — a Universal-Extractor-style window with extract, scan and carve modes, drag-and-drop, a batch queue, run history and a Preferences dialog. Bruteforce is available under Run options. It shells out to the same main.py, so detection and extraction behave identically.

python gui/main.py                 # launch the window
python gui/main.py <file> [outdir] # prefill the source (and destination)
python gui/main.py <file> /scan    # prefill and start in identify mode

Drag-and-drop needs the optional tkinterdnd2 package (pip install -r gui/requirements.txt); without it the window still works, minus drop support. It can also register an Explorer context-menu entry from its Preferences dialog.

Building (Windows)

cd cli
pyinstaller --onefile main.py --name magic-extractor --collect-data puremagic

Then copy bin/, data/ and config.ini next to dist/magic-extractor.exe. CI does this automatically — see .github/workflows/release.yml.

License

MIT — see LICENSE.txt. Note: the bundled third-party extractor/detector binaries under cli/bin/ keep their own licenses (some proprietary freeware) and are not covered by MIT; verify their redistribution terms before shipping.

Authors

• Lead Developer: DSR! — [email protected]
• Thanks to all contributors.