Shift Left with AWS Codepipleine — Scanning every single code change

IN the agile world, where continuous iteration of development and testing happens throughout the SDLC (software development lifecycle), where there is a constant collaboration with stakeholders and continuous improvement and iteration happening at every stage, and also where development of features takes place at the rapid pace. This all makes the chances of potential security loopholes to be more and more real.

This is especially true in a startup world. Having a swift development where more code is written and more releases are made that always introduces even more unknown dependencies and vulnerabilities at each step. It eventually leads to consuming more time and efforts to fix it which is almost like revisiting the complete SDLC cycle.

Shift-left refers to the measures implemented to ensure that security concerns are taken into consideration during the whole application development, rather than at the end of the process. The goal of shift-left security is to identify and resolve potential vulnerabilities earlier in the development life cycle, when these issues are less expensive to fix.

Manual security processes, such as traditional VAPT, threat modeling that mostly relies on human operational knowledge and trust, and cannot scale to meet the needs of ever-evolving automated build and release infrastructure. This brought to the need for Integrating Security within every single code change — which is the essence of this article.

A fast-moving lean and agile culture makes it necessary to bring the testing of software support earlier in the development and release process. This brings to the quote — “Security shouldn’t be treated as an after-thought”, it should be brought as close to engineers and as early in SDLC. When we bring something close to the source, and in this context, if we bring Security closer to the source, we call it Shift Left Security.

Some of the direct benefit it provides :

1. It gives a much better opportunity to see improved security outcomes in products sooner, and include the requirements, suggestions, advice at an earlier stage.
2. It also saves time, effort, and overall cost of product delivery.
3. Further, it brings improved testing, monitoring, and response to support security drift detection.
4. It also helps to implement security adherence and adoption amongst our engineers.

But How…?

To do this well, the most logical place security can be checked is through code reviews. Now the series of questions arises—

• First and foremost - How it can be achieved?
• How can we make sure every release that goes to production has proper security sign-off?
• How can we scan and test every piece of code that is changed from DAST or SAST point of view and include wide and flexible security test cases?

Below we would be talking about how it can be achieved if your organization is using AWS Codepipeline for CI/CD and the architecture diagram at the last.

Shifting Left Security in the Pipeline

Talking specifically about AWS codepipeline, AWS provides a great build feature in terms of AWS codebuild. You can add as many build stages as you want during the CICD pipeline. Instead of waiting for security testing at the end of the development process, one can include security checks seamlessly within their developer’s workflow in the form of AWS build (a continuous integration service that compiles source code, runs tests and produces software packages that are ready to deploy).

Before this, to make sure you are listening to each code change made to your code and trigger the pipeline, you have to set up Code Commit.

Two ways you can detect any changes made to your source code either by real-time Github webhook or periodic AWS CodePipepline. Choosing a GitHub webhook will automatically add a webhook in your respective repository as below. One can also choose if they want to perform tests when any developer pushes changes to the dev, feature branch, or master branch.

Coming to CodeBuild, each CodeBuild project contains a build specification (build spec) file, which is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build. This is the file where you define the commands/script and steps to perform specific security checks.

I worked on writing buildspec for multiple security checks namely — Secret scanning in git commits, Soure code scanning, DAST, Dockerfile linting, and Docker trusted images check. Below is the screenshot of some security checks I have integrated using open-source projects to automate security testing into AWS CI/CD Codepipeline.

Capabilities at a glance:

1. SAST — Static Application Security Testing using SonarQube. I have used SonarQube for SAST which performs code analysis and results are parsed and posted to AWS Build output, Slack, and can also be sent over email using SNS.

2. DAST — Dynamic Application Security Testing. CodeBuild initiates the DAST scanning. When scanning is completed and if there are any vulnerabilities found, it invokes the Lambda function and parses and posts the results to either Slack or AWS Build Output.

3. Dependency Scanning (WIP)— Analyze external dependencies for known vulnerabilities. In this I am using OWASP ZAP dependency scanner.

4. Secret Detection — TruffleHog is used here to detect secrets in the code. Wrote a Build spec file to run TruffleHog with a custom list of regex to detect secrets.

A basic buildspec.yml looks like as follow having three stages namely -
Pre_Build →Setup dependencies, Pull scanners
Build →Run scanners and scripts, Parse result
Post_Build →Call lambda, Push results

Trufflehog buildspec.yml

5. Additional Capabilities — Dockerfile linter, Check whitelist of base images, etc. Hadolint is being used here for Dockerfile linting while checking whether the base image used in dockerfile to build container is whitelisted/allowed or not, I created an in-house tool that rejects the image build if the base image is not in whitelisting list.

Some of the improvement it brings —

• Real time Checks —Real time changes to the code trigger security checks and the result and status can be shared across different communication channels using SNS and slack webhooks in lambda.
• Highly Modular — The security checks are very modular. We can add more checks as we want and configure them to perform response-based action.
• Fully self-operating — Automation is the key here. Testing, Alerting, Notification, and reporting all are completely automated. This is mostly done using Lambda.
• Vulnerability Management — All the findings are managed in a single place. I have used AWS Security Hub for vulnerability management, parsing, and pushing all the findings found out during the pipeline. Lamda and boto3 libraries are being used here.

AWS DevSecOps Pipeline Architecture

1. As soon as any code changes are committed to the repository it triggers the AWS codepipeline
2. All the AWS codebuild start running which are basically security checks here.
3. Lambda is invoked when each codebuild is completed the scanning/testing.
4. Results are parsed and pushed to AWS Security Hub for vulnerability management
5. And also the findings are sent over mail using SNS and slack using slack webhooks.

What’s next

Shifting left is not just a technical concept but it’s a culture and mindset that takes time to nurture in the corporate world (but not in every). This article wanted to highlight the advantage it brings to the table and the technical implementation to achieve it.

If you know any other security checks that can be integrated into the code pipeline, would love to know them. Leave your comments below.

All the configuration and buildspec will be over my Github profile. I am currently working on terraforming it, just applying the terraform with custom values will spin up the complete framework for you and integrate security checks in your AWS CI/CD pipeline.

Thanks for reading!

~Logicbomb ( https://logicbomb.in/ )