A Practical Developer's Guide to Shift Left Security

While security has long been touted as a priority for organizations, the reality on the ground paints a different picture. According to a recent GitLab Developer Survey, a staggering 41% of engineers find it difficult to prioritize fixing security vulnerabilities [1]. Why? As one Site Reliability Engineer succinctly puts it👇️ 

The statistics back this sentiment—while 56% of organizations claim to implement security-centric methodology of DevSecOps, less than a quarter actually implement its crucial components [1]. This lack of comprehensive security practices isn't just concerning; it's a ticking time bomb for software vulnerabilities.

Enter Shift Left Security—the practice that aims to integrate security seamlessly into the software development process, right from the get-go.

In this Bug Driven Development guide, we'll dissect the key elements of DevSecOps and Shift Left Security. We'll offer actionable insights on how developers can implement these practices based on the size and demands of their engineering teams.

We’ve got a lot to cover, but before we do, just wanted to say a big thanks to everyone who helped with this guide. In addition to several industry guiding surveys and my own personal experience having led a ~50 engineer organization in the telehealth space, I was fortunate enough to bounce ideas around several engineers and engineering leaders to put together the recommendations in this guide. So let’s dive in 👇️ 

🔺The Security Testing Pyramid

Many of you may be familiar with the popular software testing metaphor, the Testing Pyramid, popularized by Mike Cohn from Mountain Goat Software and Ham Vocke’s post in Martin Fowler’s blog. The Security Testing Pyramid is an adaptation of the original concept, introduced by Andreas Falk at the 2022 Global AppSec Conference in the EU hosted by the OWASP Foundation. While Andreas’ Security Test Pyramid is a great start, I think it misses some areas like Secret & Credential Detection. In the chart below, the items in black text are from Andreas’ original Security Test Pyramid and the items in blue are ones I added.

The Security Test Pyramid has three layers:

1. Security Unit & Component Tests + Static Application Security Testing (SAST)

2. API / Service Security Tests

3. Dynamic Application Security Tests (DAST)

🏗️ Foundational Layer: Security Unit & Component Tests + Static Application Security Testing (SAST)

The foundational layer primarily comprises tests that can be integrated into a Continuous Integration (CI) pipeline.

These tests focus on:

• Identifying code vulnerabilities potentially exploitable by attackers

• Scanning for vulnerable dependencies (like if your code consumes an open source library that contains a vulnerability)

• Scanning for vulnerable dependencies (like if you store an API Key in your code instead of consuming it as an environment variable)

🥪 Middle Layer: API / Service Security Tests

This layer zeroes in on service-level tests, including:

• Authentication & Authorization Paths

• Injections (like a SQL Injection)

• File uploads vulnerabilities (like if a user can upload an executable file when your application is supposed to just store PDFs)

🪂 Top Layer: Dynamic Application Security Tests (DAST)

This layer operates in a deployed environment, focusing on vulnerabilities that can only be detected in a running application.

DAST tests aim to identify:

• Real-world attack vectors like Cross-Site Scripting (XSS)

• Vulnerabilities in third-party services and APIs

• Logical flaws that might escape static tests

⬅️ Practical Strategies for Shifting Security Left

Alright, let's get into the meat of the matter—how do you shift security left within your development process?

🏢 Foundation Layer

The cornerstone of the foundational layer in the Shift Left security model is running security checks continuously as part of the code check-in process. Ideally, this means that every time a developer pushes a commit, these checks are executed within the Continuous Integration (CI) pipeline.

While there are a variety of security tests that can be ran in your foundational layer of your security testing pyramid, I think the following three are the most practical in balancing the level security testing coverage while minimizing adding hurdles to your development process.

1. Code Security Scanning

2. Dependency Security Scanning

3. Secret / Credential Scanning

Code Security Scanning

When we talk about code security scanning in this context, we are specifically focusing on automated scans that are integrated into your Continuous Integration (CI) pipeline—not the traditional, manual scans that require sending off your source code to auditors for one-off analyses, such as Synopsis Black Duck audits. Those traditional methods are labor-intensive and not seamlessly incorporated into your workflow. I've been through a few of those, and trust me they're far from enjoyable 🤮.

In contrast, the code security scanning we advocate for is tightly integrated into your development process through a CI job that scans your code for vulnerabilities upon each commit. Not to point favorites in terms of tool selection (this is more for a reference so you know what is out there) you should be looking for something like a GitHub Code Scanning or a Snyk type of tool. This approach not only saves time but also fosters a culture of continuous security awareness among developers.

📦️ Dependency Security Scanning

I've seen two effective approaches to dependency scanning, and I recommend employing both. The first identifies new dependencies as you commit code and checks for known security issues within those dependencies. This is done as a job in your CI pipeline which reviews the newly added dependencies in your commit and flags them appropriately.

The second continuously monitors your existing dependencies and alerts you about vulnerabilities in the versions you're using. This helps you ensure your existing dependencies are always on a secure version and are doing frequent updates to dependencies. Both methods are equally useful and I recommend using them concurrently.

By the way, this is commonly known as a part of software composition analysis, which is typically abbreviated as SCA.

🕵️‍♂️ Secret / Credential Scanning

Sensitive authentication data, like API keys and passwords, should ideally be stored as secrets. Sure, there are some exceptions but even Google now recommends storing Firebase API keys as secrets (nice try Firebase devs 👀).

Secret scanning jobs in your CI pipeline scan your code for strings that look like these sensitive types of credentials. Typically there are three ways to resolve a scanned secret, the first is to remove the secret from your code and wipe the commit which contains the secret from your git history, the second is to flag it as a false positive when it finds something that looks like credentials, but is not, and the third is to confirm it's a credential that's safe to store in code by dismissing the scanned secret.

🛡️ Service Security Testing Layer

The Service Security Testing Layer is geared towards ensuring robust defenses at the service level. It is pivotal to focus on this layer, as APIs and services often act as the gateways to your application's core logic and data.

🔐 Authentication & Authorization Paths

When considering security at the service level, a critical area of focus is the integrity of your authentication and authorization paths. This involves not only basic username/password combinations but also more advanced schemes like OAuth and JWT (JSON Web Tokens).

Integrating OAuth or JWT testing into a CI pipeline might sound unconventional, but it's actually a strong practice. Utilize mock services that emulate your actual OAuth or JWT token generation and validation. These mock services, coupled with test scripts simulating various authentication scenarios, seamlessly fit into your CI/CD pipeline.

By incorporating these tests, you're achieving two key outcomes: ensuring the robustness of your authentication and authorization mechanisms and halting the merging of code that introduces potential vulnerabilities.

💉 Injections

Injections remain a notorious risk in API and service layers. They most commonly occur when inputs aren't properly sanitized or validated, allowing attackers to inject malicious code. At the service testing layer, you can implement static analysis tools that scan your application's code to detect vulnerabilities related to SQL injections. These tools can be seamlessly integrated into your CI pipeline, right alongside your existing unit and integration tests.

📁 File Upload Vulnerabilities

When it comes to file uploads, this layer ensures that only the intended file types (e.g., PDFs, JPEGs, etc) are uploaded and executed. Tests should verify that the application restricts file types to prevent the system from security threats like a user uploading an executable.

These types of tests should also be executed in your automation pipelines continuously.

🔭 Dynamic Application Security Testing (DAST)

Dynamic Application Security Testing (DAST) serves as your real-world security scout, focusing on vulnerabilities that only manifest in a running application. By integrating DAST into your CI/CD pipeline, these tests operate in a staging environment that mirrors your production setup, triggered automatically by changes to the codebase or infrastructure.

🔍 Real-world Attack Vectors

One of the core strengths of DAST is its ability to uncover real-world attack vectors, such as Cross-Site Scripting (XSS). Unlike SAST, DAST tests your application from an external perspective, simulating how an attacker might exploit vulnerabilities. You'll want to use automated scanning tools specifically designed for dynamic testing to identify these types of issues. These scans can be scheduled to run at regular intervals or triggered by specific events like major code merges.

🧠 Logical Flaws

Logical flaws, or vulnerabilities that stem from the application's business logic, are often hard to catch with static tests. DAST can help identify these by simulating various user roles and performing sequences of actions that may expose flaws in the application's logic.

🛠️ Integrating DAST into CI/CD

Incorporating DAST into your CI/CD pipeline typically involves running these tests in a staging environment after the application has been built but before it is deployed to production. While DAST tests can be more time-consuming than their static counterparts, the additional runtime context they provide is invaluable. To make the most of your CI/CD pipeline's efficiency, you might schedule DAST scans to run at other low-activity times or when a PR is close to being merged.

Thanks for diving into this Bug Driven Development post with us! We're mixing it up a bit by switching to two deep-dive articles a month instead of our usual weekly quick takes. If you are interested in finding out more about how to incorporate security into your developer workflow check out my session at last year’s GitHub Universe 👇️ 

Cheers,

Justin

Follow Justin on Twitter here

Additional Sources

1. 2023 GitLab Global DevSecOps Report Security Without Sacrifices

2. Shift Left Security with the Security Test Pyramid - Andreas Falk