binref/refineryHigh Octane Triage Analysis
[
][docs]
[
][tests]
[
][codecov]
[
][pypi]
__ __ High Octane Triage Analysis __
|| _||______ __ __________ _____ ||
|| \||___ \__| ____/ ______/___ / ____\ ||
==||=====|| | __/ |/ \ /==| / __ \ __\===]|
'======|| | \ | | \_ _| \ ___/| | ||
||____ /__|___|__/ / | \____]| | ||
=========''====\/=========/ /==|__|=====|__|======'
\ /
\/
The Binary Refinery™ is a collection of Python scripts that implement transformations of binary data such as compression and encryption.
We will often refer to it simply by refinery, which is also the name of the corresponding package.
The scripts are designed to exclusively read input from stdin and write output to stdout.
The main philosophy is that every script should be a unit in the sense that it does one job,
and individual units can be combined into pipelines with the piping operator | on the commandline to perform more complex tasks.
The project's main focus is malware triage,
and is an attempt to implement something like CyberChef on the commandline.
Make a Python virtual environment. You need Python 3.11 or later. Install refinery like this:
python -m pip install -U pip
python -m pip install -U binary-refinery[extended]
Run units with -h to learn how they work, grep through the [docs][] or use the command binref to find them.
Watch [a recent video][VOD3] if you want to see it in action.
But also, read the rest of this readme.
There is no release schedule, but releases happen very frequently and it is recommended to update periodically. Bugfixes are not documented outside of GIT, but all other changes (i.e. new features) are documented in the changelog. Follow me on [Mastodon][] for updates about particularly impactful releases.
The help text that is displayed when executing a unit with the -h or --help switch is its main documentation.
The [automatically generated documentation][docs] contains a compilation of that output for each unit at the top level,
but also contains the specification for the three fundamental concepts of the toolkit:
[framing][frame], [multibin arguments][argformats], and [meta variables][meta].
Full-text search of the description and help text for every unit is also available on the command line,
via the provided binref command. In recognition of the fact that reference documentation can be somewhat dry,
there is an ongoing effort to produce a series of tutorials; I very much recommend to check them out.
On top of that, I collect additional resources (including some produced by third parties) below.
[!NOTE]
Refinery is still in alpha and the interface can sometimes change, i.e. units and parameters can be removed or renamed. Hence, it can happen that specific command lines from older videos and blog posts don't work any more.
2021/08] [OALabs][OA] was kind enough to let me [demo the toolkit in a feature video][VOD1].
In the video, I essentially work through the contents of
the first tutorial.2021/11] [Johannes Bader][JB] wrote an amazing [blog post][BLOG] about analyzing malspam with binary refinery.2024/03] [Malware Analysis For Hedgehogs][MH] made [a video about unpacking an XWorm sample][VOD2] using refinery.2024/11] [the CyberYeti][CY] had me [on stream presenting refinery][VOD3].2025/06] I was [on stream again][VOD4] with [the CyberYeti][CY], this one is a little more raw.
All bugs you can see here were fixed. 😉Showcases again include samples from the example section below and the tutorials.
The Binary Refinery is (c) 2019 Jesko Hüttenhain, and published under a [3-Clause BSD License][license]. This repository also contains a copy of the full license text. If you want to do something with it that's not covered by this license, please feel free to contact the author.
The refinery requires at least Python 3.11. It is recommended to install it into its own [virtual environment][venv]: The package can pull in a lot of dependencies, and installing it into your global Python is somewhat prone to version conflicts. Also, since the toolkit introduces a large number of new commands, there is a good chance that some of these will clash on some systems, and keeping them in their own separate virtual environment is one way to prevent that.
If you want to have all refinery commands available in your shell at all times (i.e. without having to switch to a custom virtual environment),
you also have the option to choose a prefix for the installation,
which will be put in front of every command shim that is installed.
For example, if you choose r. as your prefix, then the [emit][] unit will be installed as the command r.emit.
An added benefit is that you can type r. and hammer Tab twice to get a list of all available refinery commands.
Note however that no prefix is assumed in documentation and it is a development goal of refinery to not clash on most systems.
The author does not use a prefix and provides this option as a safety blanket.
The most straightforward way to install and update refinery is via pip. Make sure you run the latest version first:
python -m pip install -U pip
Then just install the refinery package:
pip install -U binary-refinery
If you want to choose a prefix for all units, you can specify it via the environment variable REFINERY_PREFIX.
For example, the following command will install refinery into the current Python environment with prefix r. on Linux:
REFINERY_PREFIX=r. pip install -U binary-refinery
On Windows, you would have to run the following commands:
set REFINERY_PREFIX=r.
pip install -U binary-refinery
Specifying the special prefix ! will have the effect that no shell commands are created at all,
and binary refinery will be installed only as a library.
If you want to install the current refinery HEAD, you can repeat all of the above steps, specifying this repository instead of the pip package.
For example, the following will install the very latest refinery commit:
pip install -U git+git://github.com/binref/refinery.git
Finally, if you are using [REMnux][remnux-main], you can use their [refinery docker container][remnux].
If you want to teach your local malware analysis [claude][] to use binary refinery, check out the [Binary Refinery Skill][agent].
The following is a summary of how well various shell environments are currently supported: