gitlab-da/tutorials/security-and-governance/custom-scanner-integration/fern-pattern-scanner
Scans selected files for patterns stated in rules. This is used in order to find secrets you may have accidentally written to a file. This scanner is used to show how the GitLab [vulnerability report](https://docs.gitlab.com/ee/user/application_security/vulnerability_report/) can be populated by a custom scanner. You can see a demo of it in action be following the documentation in the [Secret List](https://gitlab.com/gitlab-de/tutorials/security-and-governance/custom-scanner-integration/secret-list) project.
Welcome to Fern's Pattern Scanner. This scanner scans through your suggested files and finds any patterns stated within the rules/rules.yaml using regex. Then a report is generated with all the findings. This project is used to showcase how a custom security scanner can be integrated in GitLab can be used to populate the Pipeline Security Tab, the vulnerability reports and the merge request widget
Note: Accessing vulnerability reports requires a GitLab Ultimate subscription
You can activate this security scanner by adding the following to your .gitlab-ci.yml:
include:
- remote: "https://gitlab.com/gitlab-de/tutorials/security-and-governance/custom-scanner-integration/fern-pattern-scanner/-/blob/main/jobs/.fern-pattern-scanner.gitlab-ci.yml"
You can see the scanner run on the Secret List project. Once the scanner completes, it generates a report named gl-secret-detection-report.json. That report is validated and loaded into the vulnerability report and merge request widget.
If you wish to ignore scanning certain paths in the project the scanner is running on, you can set the following variable with comma separated values:
variables:
FERN_ANALYZER_EXCLUDED_PATHS: "docs, tests"
If you wish to run your own rule set of regex patterns in GitLab, you must do the following:
patterns:
- name: 'PATTERN NAME 1'
pattern: 'REGEX 1'
- name: 'PATTERN NAME 2'
pattern: 'REGEX 2'
fern_secret_detection:
script:
- /app/fern-pattern-scanner scan --rules "/path/to/rules.yaml" --report
Check in your code and the scanner should begin running with the new rule set. This scanner does not scan commit diffs. It searches for patterns across the entire project
When the pipeline completes, you can click on the Security tab to view all the detected issues. If you have GitLab Ultimate, these issues can also be seen in the Vulnerability Report
You can use this scanner to detect patterns for selected file types on your local machine. In order to run this scanner on your local machine follow the steps below:
$ go install gitlab.com/gitlab-de/tutorials/security-and-governance/custom-scanner-integration/fern-pattern-scanner
Note: This will download the file to your $GOBIN. If you are new to Go, this playground goes over the GO installation
$ wget https://gitlab.com/gitlab-de/tutorials/security-and-governance/custom-scanner-integration/fern-pattern-scanner/-/raw/main/rules/rules.yaml
...
Resolving gitlab.com (gitlab.com)... 2606:4700:90:0:f22e:fbec:5bed:a9b9, 172.65.251.78
Connecting to gitlab.com (gitlab.com)|2606:4700:90:0:f22e:fbec:5bed:a9b9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 283 [text/plain]
Saving to: ‘rules.yaml’
rules.yaml 100%[===================>] 283 --.-KB/s in 0s
2023-05-20 16:02:46 (22.5 MB/s) - ‘rules.yaml’ saved [283/283]
Note: You can also download the file directly from the project
$ fern-pattern-scanner scan --rules path/to/rules.yaml
INFO[0000] The scanner is starting
INFO[0000] Skipping the following directories: []
INFO[0000] Possible Vulnerability Detected: {Id:139999 Name:password variation 1 Message:The pattern in rule: password variation 1 was found in a file Description:The pattern in rule: password variation 1 was found in a file. Examine the file, remove and rotate your secret. Location:/Users/fern/Desktop/secret-list/.git/index Line:3}
...
INFO[0000] The scanner completed successfully
Created and maintained by Fern🌿