jonrau1/syntheticsunServerless AWS security automation framework that ingests threat intelligence, applies ML-based anomaly detection (RCF, IP Insights), and enriches security telemetry in Kibana for automated threat prevention, detection, and response.
SyntheticSun is a defense-in-depth security automation and monitoring framework which utilizes threat intelligence, machine learning, managed AWS security services and, serverless technologies to continuously prevent, detect and respond to threats.
You sleep in fragmented glass
With reflections of you,
But are you feeling alive?
Yeah let me ask you,
Are you feeling alive?
- Norma Jean, 2016
SyntheticSun is built around the usage of the Malware Information Sharing Platform (MISP) and Anomali's LIMO, which are community driven threat intelligence platforms (TIPs) that provide various types of indicators of compromise (IoC). Normalized and de-duplicated threat intel is looked up against in near-real time to quickly identify known threats in various types of network traffic. To add dynamism to the identification of potential threats IP Insights models are deployed to find anoamlies (and potential threats therein) between the pairing of IP addresses and entities (such as IAM principal ID's, user-agents, etc.), native RCF detectors are also used in Elasticsearch to find anomalies in near real-time security telemetry as it is streamed into Kibana. To democratize the usage and fine-tuning of ML models within security teams, utilities to train IP Insights models are provided as an add-on to the core solution.
To perform the both the orchestration and automation as well as extraction, transformation, and loading (ETL) of security telemetry into Kibana, various AWS serverless technologies such as AWS Lambda, Amazon DynamoDB, and AWS CodeBuild are used. Serverless technologies such as these are used for their scalability, ease of use, relatively cheap costs versus heavy MapReduce or Glue ETL-based solutions. A majority of the solution is deployed via CloudFormation with helper scripts in Python and shell provided throughout the various Stages to promote adoption and the potential deployment in continuous integration pipelines.
To make the "guts" of the solution as lean as possible basic Python modules such as boto3, requests, json, ipaddress, socket and re perform most of the extraction, transformation, and loading (ETL) into downstream services. Because all geolocation information is provided by ip-api.com, it does not require an account or paid tiers and has a great API which includes throttling information in their response headers. A majority of the Elasticsearch and Kibana dependencies are also provided in code (indicies, mappings, visualizations, etc) to avoid heavy manual configuration.
SyntheticSun is spread across three Stages due to the size of solution and the required dependencies. All architecture and installation instructions (and FAQs where appropriate) live within their own Stage. Add-ons modules (called an Appendix) are also provided to extend the functionality, which have their own architecture and installation instructions localized.
SyntheticSun, by virtue of being something you found on GitHub, is a proof-of-concept and therefore I did not go the extra mile for the first release to absolutely harden everything. Provided you are reading this at a point in time where I have not made the necessary changes, consider the following before you deploy this solution into a production environment (or any environment with heightened security needs). I will put these items on a roadmap and update them as appropiate.
SyntheticSun is an easy way to start using cyber threat intelligence and machine learning for your edge protection security use cases on the AWS Cloud without having to invest in one or more commercial tools, or hiring a data scientist for your security team (though you should ideally do the latter). This solution, after initial configuration, is fully automated, allowing you to identify and respond to threats at machine speed. Finally, this solution provides basic visualizations for your incident response team to use for threat response, such as allowed inbound or outbound connections or DNS queries to and from IP addresses or domains deemed to be malicious. The core of the solution relies on very lightweight automation and data engineering pipelines, which theoretically, can be reused for other purposes where multi-stage normalization and enrichment or scheduled, fast-paced batch jobs are needed.
