joshlarsen/aws-reconMulti-threaded AWS inventory collection tool with a focus on security-relevant resources and metadata.
A multi-threaded AWS security-focused inventory collection tool written in Ruby.
This tool was created to facilitate efficient collection of a large amount of AWS resource attributes and metadata. It aims to collect nearly everything that is relevant to the security configuration and posture of an AWS environment.
Existing tools (e.g. AWS Config) that do some form of resource collection lack the coverage and specificity to accurately measure security posture (e.g. detailed resource attribute data, fully parsed policy documents, and nested resource relationships).
AWS Recon handles collection from large accounts by taking advantage of automatic retries (either due to network reliability or API throttling), automatic paging of large responses (> 100 resources per API call), and multi-threading parallel requests to speed up collection.
** usage does not imply endorsement
AWS Recon needs an AWS account role or credentials with ReadOnlyAccess. Full AdministratorAccess is over-privileged, but will work as well. The SecurityAudit policy is not sufficient as it omits access to many services.
Use Docker version 19.x or above to run the pre-built image without having to install anything.
If you already have Ruby installed (2.6.x or 2.7.x), you may want to install the Ruby gem.
AWS Recon can be run locally via a Docker container or by installing the Ruby gem.
To run via a Docker a container, pass the necessary AWS credentials into the Docker run command. For example:
$ docker run -t --rm \
-e AWS_REGION \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-v $(pwd)/output.json:/recon/output.json \
darkbitio/aws_recon:latest \
aws_recon -v -s EC2 -r global,us-east-1,us-east-2
To run locally, first install the gem:
$ gem install aws_recon
Fetching aws_recon-0.5.17.gem
Fetching aws-sdk-3.0.1.gem
Fetching parallel-1.20.1.gem
...
Successfully installed aws-sdk-3.0.1
Successfully installed parallel-1.20.1
Successfully installed aws_recon-0.5.17
Or add it to your Gemfile using bundle:
$ bundle add aws_recon
Fetching gem metadata from https://rubygems.org/
Resolving dependencies...
...
Using aws-sdk 3.0.1
Using parallel-1.20.1
Using aws_recon 0.5.17
AWS Recon will leverage any AWS credentials (see requirements) currently available to the environment it runs in. If you are collecting from multiple accounts, you may want to leverage something like aws-vault to manage different credentials.
$ aws-vault exec profile -- aws_recon
Plain environment variables will work fine too.
$ AWS_PROFILE=<profile> aws_recon
To run from a Docker container using aws-vault managed credentials (output to stdout):
$ aws-vault exec <vault_profile> -- docker run -t --rm \
-e AWS_REGION \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
darkbitio/aws_recon:latest \
aws_recon -j -s EC2 -r global,us-east-1,us-east-2
To run from a Docker container using aws-vault managed credentials and output to a file, you will need to satisfy a couple of requirements. First, Docker needs access to bind mount the path you specify (or a parent path above). Second, you need to create an empty file to save the output into (e.g. output.json). This is because only that one file is mounted into the Docker container at run time. For example:
Create an empty file.
$ touch output.json
Run the aws_recon container, specifying the output file.
$ aws-vault exec <vault_profile> -- docker run -t --rm \
-e AWS_REGION \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-v $(pwd)/output.json:/recon/output.json \
darkbitio/aws_recon:latest \
aws_recon -s EC2 -v -r global,us-east-1,us-east-2
You may want to use the -v or --verbose flag initially to see status and activity while collection is running.
In verbose mode, the console output will show:
<thread>.<region>.<service>.<operation>
The t prefix indicates which thread a particular request is running under. Region, service, and operation indicate which request operation is currently in progress and where.
$ aws_recon -v
t0.global.EC2.describe_account_attributes
t2.global.S3.list_buckets
t3.global.Support.describe_trusted_advisor_checks
t2.global.S3.list_buckets.acl
t5.ap-southeast-1.WorkSpaces.describe_workspaces
t6.ap-northeast-1.Lightsail.get_instances
...
t2.us-west-2.WorkSpaces.describe_workspaces
t1.us-east-2.Lightsail.get_instances
t4.ap-southeast-1.Firehose.list_delivery_streams
t7.ap-southeast-1.Lightsail.get_instances
t0.ap-south-1.Lightsail.get_instances
t1.us-east-2.Lightsail.get_load_balancers
t7.ap-southeast-2.WorkSpaces.describe_workspaces
t2.eu-west-3.SageMaker.list_notebook_instances
t3.eu-west-2.SageMaker.list_notebook_instances
Finished in 46 seconds. Saving resources to output.json.
# collect S3 and EC2 global resources, as well as us-east-1 and us-east-2
$ AWS_PROFILE=<profile> aws_recon -s S3,EC2 -r global,us-east-1,us-east-2
# collect S3 and EC2 global resources, as well as us-east-1 and us-east-2
$ AWS_PROFILE=<profile> aws_recon --services S3,EC2 --regions global,us-east-1,us-east-2
# save output to S3 bucket
$ AWS_PROFILE=<profile> aws_recon \
--services S3,EC2 \
--regions global,us-east-1,us-east-2 \
--verbose \
--s3-bucket my-recon-bucket
# save output to S3 bucket with a home region other than us-east-1
$ AWS_PROFILE=<profile> aws_recon \
--services S3,EC2 \
--regions global,us-east-1,us-east-2 \
--verbose \
--s3-bucket my-recon-bucket:us-west-2
Example OpenCSPM formatted (NDJSON) output.
$ AWS_PROFILE=<profile> aws_recon -l \
-s S3,EC2 \
-r global,us-east-1,us-east-2 \
-f custom
or