HashR: Generate your own set of hashes

Table of Contents

• HashR: Generate your own set of hashes
  • Table of Contents
  • About
  • Requirements
  • Building HashR binary and running tests
  • Setting up HashR
    • HashR using docker
    • OS configuration & required 3rd party tooling
    • Setting up storage for processing tasks
      • Setting up PostgreSQL storage
      • Setting up Cloud Spanner
    • Setting up importers
      • GCP (Google Cloud Platform)
      • AWS (Amazon Web Services)
      • GCR (Google Container Registry)
      • Windows
      • WSUS
      • TarGz
      • Deb
      • RPM
      • Zip (and other zip-like formats)
      • ISO 9660
    • Setting up exporters
      • Setting up Postgres exporter
      • Setting up GCP exporter
    • Additional flags

About

HashR allows you to build your own hash sets based on your data sources. It's a tool that extracts files and hashes out of input sources (e.g. raw disk image, GCE disk image, ISO file, Windows update package, .tar.gz file, etc.).

HashR consists of the following components:

1. Importers, which are responsible for copying the source to local storage and doing any required preprocessing.
2. Core, which takes care of extracting the content from the source using image_export.py (Plaso), caching and repository level deduplication and preparing the extracted files for the exporters.
3. Exporters, which are responsible for exporting files, metadata and hashes to given data sinks.

Currently implemented importers:

1. GCP, which extracts file from base GCP disk images
2. Windows, which extracts files from Windows installation media in ISO-13346 format.
3. WSUS, which extracts files from Windows Update packages.
4. GCR, which extracts file from container images stored in Google Container Registry.
5. TarGz, which extracts files from .tar.gz archives.
6. Deb, which extracts files Debian software packages.
7. RPM, which extracts files from RPM software packages.
8. Zip, which extracts files from .zip (and zip-like) archives.

Once files are extracted and hashed results will be passed to the exporters, currently implemented exporters:

1. PostgreSQL, which upload the data to PostgreSQL instance.
2. Cloud Spanner, which uploads the data to GCP Spanner instance.

You can choose which importers you want to run, each one have different requirements. More about this can be found in sections below.

Requirements

HashR requires Linux OS to run, this can be a physical, virtual or cloud machine. Below are optimal hardware requirements:

1. 8-16 cores
2. 128GB memory
3. 2TB fast local storage (SSDs preferred)

HashR can likely run how machines with lower specifications, however this was not thoroughly tested.

Building HashR binary and running tests

In order to build a hashr binary run the following command:

env GOOS=linux GOARCH=amd64 go build hashr.go

In order to run tests for the core hashR package you need to run Spanner emulator:

gcloud emulators spanner start

Then to execute all tests run the following command:

go test -timeout 2m ./...

Setting up HashR

HashR in OSDFIR Infrastructure

You can deploy HashR as part of the OSDFIR Infrastructure project This deployment will run HashR as kubernetes cronjobs and allows for an easy integration with Timesketch.

HashR using docker

To run HashR in a docker container visit the docker specific guide

OS configuration & required 3rd party tooling

HashR takes care of the heavy lifting (parsing disk images, volumes, file systems) by using Plaso. You need to pull the Plaso docker container using the following command:

docker pull log2timeline/plaso

We also need 7z, which is used by WSUS importer for recursive extraction of Windows Update packages, to be installed on the machine running HashR:

sudo apt install p7zip-full

You need to allow the user, under which HashR will run, to run certain commands via sudo. Assuming that your user is hashr create a file /etc/sudoers.d/hashr and put in:

hashr ALL = (root) NOPASSWD: /bin/mount,/bin/umount,/sbin/losetup,/bin/rm

The user under which HashR will run will also need to be able to run docker. Assuming that your user is hashr, add them to the docker group like this:

sudo usermod -aG docker hashr

Setting up storage for processing tasks

HashR needs to store information about processed sources. It also stores additional telemetry about processing tasks: processing times, number of extracted files, etc. You can choose between using:

1. PostgreSQL
2. Cloud (GCP) Spanner

Setting up PostgreSQL storage

There are many ways you can run and maintain your PostgreSQL instance, one of the simplest ways would be to run it in a Docker container. Follow the steps below to set up a PostgreSQL Docker container.

Step 1: Pull the PostgreSQL docker image.

docker pull postgres

Step 2: Initialize and run the PostgreSQL container in the background. Make sure to adjust the password.

docker run -itd -e POSTGRES_DB=hashr -e POSTGRES_USER=hashr -e POSTGRES_PASSWORD=hashr -p 5432:5432 -v /data:/var/lib/postgresql/data --name hashr_postgresql postgres

Step 3: Create a table that will be used to store processing jobs.

cat scripts/CreateJobsTable.sql | docker exec -i hashr_postgresql psql -U hashr -d hashr

In order to use PostgreSQL to store information about processing tasks you need to specify the following flags: -storage postgres -postgres_host <host> -postgres_port <port> -postgres_user <user> -postgres_password <pass> -postgres_db <db_name>

Setting up Cloud Spanner

You can choose the store the data about processing jobs in Cloud Spanner. You'll need a Google Cloud project for that. The main advantage of this setup is that you can easily create dashboard(s) using Google Data Studio and directly connect to the Cloud Spanner instance that allows monitoring and debugging without running queries against your PostgreSQL instance.

Assuming that your gcloud tool is configured with your target hashr GCP project, you'll need to follow the steps below to enable Cloud Spanner.

Create HashR service account:

gcloud iam service-accounts create hashr --description="HashR SA key." --display-name="hashr"

Create service account key and store in your home directory. Set <project_name> to your project name.

gcloud iam service-accounts keys create ~/hashr-sa-private-key.json --iam-account=hashr-sa@<project_name>.iam.gserviceaccount.com

Point GOOGLE_APPLICATION_CREDENTIALS env variable to your service account key:

export GOOGLE_APPLICATION_CREDENTIALS=/home/hashr/hashr-sa-private-key.json

Create Spanner instance, adjust the config and processing-units value if needed: