git-all-secrets

About

git-all-secrets is a tool that can:

• Clone multiple public/private github repositories of an organization and scan them,
• Clone multiple public/private github repositories of a user that belongs to an organization and scan them,
• Clone a single public/private repository of an organization and scan it,
• Clone a single public/private repository of a user and scan it,
• Clone a single public/secret gist of a user and scan it
• Clone a team's repositories in an organization and scan them,
• All of the above together!! Oh yeah!! Simply provide an organization name and get all their secrets. If you also want to get secrets of a team within an organization, just mention the team name along with the org.
• Clone and scan Github Enterprise repositories and gists as well.

Scanning is done by multiple open source tools such as:

• truffleHog - scans commits for high entropy strings and user provided regular expressions,
• repo-supervisor - scans for high entropy strings in .js and .json files

NOTE - More such tools can be added in future, if desired! NOTE - Scanning can be done by all the tools or any one of them by specifying the toolName flag.

If all the tools are used to scan, the final output from the tool combines the output from all files from all the tools into one consolidated output file.

Getting started

The easiest way to run git-all-secrets is via Docker and I highly recommend installing Docker if you don't already have it. Once you have Docker installed,

• Type docker run --rm -it abhartiya/tools_gitallsecrets --help to understand the different flags it can take as inputs.
• Once you know what you want to scan, type something like docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<>. You can also specify a particular tool to use for scanning by typing something like docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<> -toolName=<>. Options are thog and repo-supervisor.
• If you want to run truffleHog with the default regex AND the high entropy settings, provide the thogEntropy flag like this - docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<> -toolName=thog -thogEntropy.
• After the container finishes running, retrieve the container ID by typing docker ps -a.
• Once you have the container ID, get the results file from the container to the host by typing docker cp <container-id>:/root/results.txt .

Flags/Options

• -token = Github personal access token. We need this because unauthenticated requests to the Github API can hit the rate limiting pretty soon!

• -org = Name of the Organization to scan. This will scan all public repos in the org + all the repos & gists of all users in the org. If you are using a token of a user who is a part of this org, it will also clone and scan all the secret gists belonging to that user as well as all the private repos in that org that the user has access to. However, it will NOT clone and scan any private repositories of this user belonging to this org. To scan private repositories of users, please use the scanPrivateReposOnly flag with the user flag along with the SSH key mounted on a volume.

• -user = Name of the User to scan. This will scan all the repos & gists of this user. If the token provided is the token of the user, secret gists will also be cloned and scanned. But, only public repos will be cloned and scanned. To scan private repositories of this user, please use the scanPrivateReposOnly flag with the user flag along with the SSH key mounted on a volume.

• -repoURL = HTTPS URL of the Repo to scan. This will scan this repository only. For public repos, mentioning the https URL of the repo will suffice. However, if you wish to scan a private repo, then you need to provide the ssh URL along with the SSH key mounted on a volume and the scanPrivateReposOnly flag.

• -gistURL = HTTPS URL of the Gist to scan. This will scan this gist only. There is no concept of public or secret gist as long as you have the URL. Even if you have a secret gist, if someone knows the HTTPS URL of your secret gist, they can access it too.

• -output = This is the name of the file where all the results will get stored. By default, this is results.txt.

• -cloneForks = This is the optional boolean flag to clone forks of org and user repositories. By default, this is set to 0 i.e. no cloning of forks. If forks are to be cloned, this value needs to be set to 1. Or, simply mention -cloneForks along with other flags.

• -orgOnly = This is the optional boolean flag to skip cloning user repositories belonging to an org. By default, this is set to 0 i.e. regular behavior. If user repo's are not to be scanned and only the org repositories are to be scanned, this value needs to be set to 1. Or, simply mention -orgOnly along with other flags.

• -toolName = This is the optional string flag to specify which tool to use for scanning. By default, this is set to all i.e. thog and repo-supervisor will all be used for scanning. Values are either thog or repo-supervisor.

• -teamName = Name of the Organization Team which has access to private repositories for scanning. This flag is not fully tested so I can't guarantee the functionality.

• -scanPrivateReposOnly = This is the optional boolean flag to specify if you want to scan private user repositories or not. Mentioning this will NOT scan public user repositories. And, you need to provide the SSH key by mounting the volume onto the container. Also, this only works with either the user flag, the repoURL flag or the org flag.

  When the org flag is mentioned along with the scanPrivateReposOnly flag and without the orgOnly flag, it will scan the public AND the private repos belonging to this org to which the user has access to (whose token is provided). It will then continue to scan ONLY the private repositories of the user (whose token is provided). Finally, it will continue to scan all public and secret gists of this user (whose token is provided). In a nutshell, the scanPrivateReposOnly flag only really affects the user and the repoURL flag.

• -enterpriseURL = Optional flag to provide the enterprise Github URL, if you wish to scan enterprise repositories. It should be something like https://github.org.com/api/v3 along with the SSH key mounted onto the container. Refer to scanning github enterprise below.

• -threads = Default value is 10. This is to limit the number of threads if your system is not beefy enough. For the most part, leaving this to 10 should be okay.

• -thogEntropy = This is an optional flag that basically tells if you want to get back high entropy based secrets from truffleHog or not. The high entropy secrets from truffleHog produces a LOT of noise so if you don't really want all that noise and if you are running git-all-secrets on a big organization, I'd recommend not to mention this flag. By default, this is set to False which means truffleHog will only produce result based on the Regular expressions in the rules.json file. If you are scanning a fairly small org with a limited set of repos or a user with a few repos, mentioning this flag makes more sense.

• -mergeOutput = Optional flag to merge and deduplicate the ouput of the tools used (currently truffleHog and repo-supervisor). Default value is False.

• -blacklist = Repo names provided as comma separated values that should NOT be scanned.

Note

• The token flag is compulsory. This can't be empty.