detect-secrets

About

detect-secrets is an aptly named module for (surprise, surprise) detecting secrets within a code base.

However, unlike other similar packages that solely focus on finding secrets, this package is designed with the enterprise client in mind: providing a backwards compatible, systematic means of:

1. Preventing new secrets from entering the code base,
2. Detecting if such preventions are explicitly bypassed, and
3. Providing a checklist of secrets to roll, and migrate off to a more secure storage.

This way, you create a separation of concern: accepting that there may currently be secrets hiding in your large repository (this is what we refer to as a baseline), but preventing this issue from getting any larger, without dealing with the potentially gargantuan effort of moving existing secrets away.

It does this by running periodic diff outputs against heuristically crafted regex statements, to identify whether any new secret has been committed. This way, it avoids the overhead of digging through all git history, as well as the need to scan the entire repository every time.

For a look at recent changes, please see CHANGELOG.md.

If you are looking to contribute, please see CONTRIBUTING.md.

For more detailed documentation, check out our other documentation.

Examples

Quickstart:

Create a baseline of potential secrets currently found in your git repository.

$ detect-secrets scan > .secrets.baseline

or, to run it from a different directory:

$ detect-secrets -C /path/to/directory scan > /path/to/directory/.secrets.baseline

Scanning non-git tracked files:

$ detect-secrets scan test_data/ --all-files > .secrets.baseline

Adding New Secrets to Baseline:

This will rescan your codebase, and:

1. Update/upgrade your baseline to be compatible with the latest version,
2. Add any new secrets it finds to your baseline,
3. Remove any secrets no longer in your codebase

This will also preserve any labelled secrets you have.

$ detect-secrets scan --baseline .secrets.baseline

For baselines older than version 0.9, just recreate it.

Alerting off newly added secrets:

Scanning Staged Files Only:

$ git diff --staged --name-only -z | xargs -0 detect-secrets-hook --baseline .secrets.baseline

Scanning All Tracked Files:

$ git ls-files -z | xargs -0 detect-secrets-hook --baseline .secrets.baseline

Viewing All Enabled Plugins:

$ detect-secrets scan --list-all-plugins
ArtifactoryDetector
AWSKeyDetector
AzureStorageKeyDetector
BasicAuthDetector
CloudantDetector
DiscordBotTokenDetector
GitHubTokenDetector
GitLabTokenDetector
Base64HighEntropyString
HexHighEntropyString
IbmCloudIamDetector
IbmCosHmacDetector
IPPublicDetector
JwtTokenDetector
KeywordDetector
MailchimpDetector
NpmDetector
OpenAIDetector
PrivateKeyDetector
PypiTokenDetector
SendGridDetector
SlackDetector
SoftlayerDetector
SquareOAuthDetector
StripeDetector
TelegramBotTokenDetector
TwilioKeyDetector

Disabling Plugins:

$ detect-secrets scan --disable-plugin KeywordDetector --disable-plugin AWSKeyDetector

If you want to only run a specific plugin, you can do:

$ detect-secrets scan --list-all-plugins | \
    grep -v 'BasicAuthDetector' | \
    sed "s#^#--disable-plugin #g" | \
    xargs detect-secrets scan test_data

Auditing a Baseline:

This is an optional step to label the results in your baseline. It can be used to narrow down your checklist of secrets to migrate, or to better configure your plugins to improve its signal-to-noise ratio.

$ detect-secrets audit .secrets.baseline

Usage in Other Python Scripts

Basic Use:

from detect_secrets import SecretsCollection
from detect_secrets.settings import default_settings

secrets = SecretsCollection()
with default_settings():
    secrets.scan_file('test_data/config.ini')


import json
print(json.dumps(secrets.json(), indent=2))

More Advanced Configuration:

from detect_secrets import SecretsCollection
from detect_secrets.settings import transient_settings

secrets = SecretsCollection()
with transient_settings({
    # Only run scans with only these plugins.
    # This format is the same as the one that is saved in the generated baseline.
    'plugins_used': [
        # Example of configuring a built-in plugin
        {
            'name': 'Base64HighEntropyString',
            'limit': 5.0,
        },

        # Example of using a custom plugin
        {
            'name': 'HippoDetector',
            'path': 'file:///Users/aaronloo/Documents/github/detect-secrets/testing/plugins.py',
        },
    ],

    # We can also specify whichever additional filters we want.
    # This is an example of using the function `is_identified_by_ML_model` within the
    # local file `./private-filters/example.py`.
    'filters_used': [
        {
            'path': 'file://private-filters/example.py::is_identified_by_ML_model',
        },
    ]
}) as settings:
    # If we want to make any further adjustments to the created settings object (e.g.
    # disabling default filters), we can do so as such.
    settings.disable_filters(
        'detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign',
        'detect_secrets.filters.heuristic.is_likely_id_string',
    )

    secrets.scan_file('test_data/config.ini')

Installation

$ pip install detect-secrets
✨🍰✨

Install via brew:

$ brew install detect-secrets

Usage

detect-secrets comes with three different tools, and there is often confusion around which one to use. Use this handy checklist to help you decide:

1. Do you want to add secrets to your baseline? If so, use detect-secrets scan.
2. Do you want to alert off new secrets not in the baseline? If so, use detect-secrets-hook.
3. Are you analyzing the baseline itself? If so, use detect-secrets audit.

Adding Secrets to Baseline

$ detect-secrets scan --help
usage: detect-secrets scan [-h] [--string [STRING]] [--only-allowlisted]
                           [--all-files] [--baseline FILENAME]
                           [--force-use-all-plugins] [--slim]
                           [--list-all-plugins] [-p PLUGIN]
                           [--base64-limit [BASE64_LIMIT]]
                           [--hex-limit [HEX_LIMIT]]
                           [--disable-plugin DISABLE_PLUGIN]
                           [-n | --only-verified]
                           [--exclude-lines EXCLUDE_LINES]
                           [--exclude-files EXCLUDE_FILES]
                           [--exclude-secrets EXCLUDE_SECRETS]
                           [--word-list WORD_LIST_FILE] [-f FILTER]
                           [--disable-filter DISABLE_FILTER]
                           [path [path ...]]