Writing Semgrep Rules

This page describes the thought process behind writing a new Semgrep rule and gives a methodology for doing so effectively.

These docs focus on the process and solving example problems. For the authoritative, detailed docs on:

• The power of Semgrep patterns, see here.
• YAML configuration options like pattern-either, pattern-inside, and more, see here.

For walkthrough examples of writing rules, see these docs, or an interactive, example-based tutorial here: https://semgrep.dev/learn.

A Rule Writing Methodology

There's no one right way or process for writing a Semgrep rule, do what works for you.

The following outlines a process we've found effective. At a high level, it's:

1. Brainstorm what you want to find
2. Determine how what you want to find concretely looks like in code
3. Create a sample source file with example code snippets you'd like to find, and write an initial Semgrep rule that matches it
4. Test your Semgrep rule and refine it by first running it on one real code repository, then across many
5. After you're happy with your rule's performance, integrate it into your continuous integration (CI) systems so that it's run on every pull request to ensure your code maintains a high quality bar.

1. Brainstorm What You Want to Find

First, determine what to look for.

It could be something very concrete and demand-driven, like:

a. Finding other instances of known vulnerable code patterns ("variant analysis"):

Our pen test report / bug bounty program just reported an issue <like this>. I want to find all other similar code across all of my repos, because it might be vulnerable too.

b. Finding business logic bugs

In my company, I know that our internal API method foo() must always be called a certain way. For example, one of the arguments must or must not be a hardcoded string, a certain flag or option should be set, etc. I want to find all the code locations where this implicit calling pattern is not being followed.

In my applications, I know that one particular API call, bar() must always be called before another, baz(), or else it's a bug. I want to find all the places this calling convention isn't followed.

Or, your search could start a bit more abstract and exploratory, where you don't know exactly what you're looking for, like:

c. Auditing dangerous function use

I know eval() is a potentially dangerous function, I want to see where it's used, and if so, how.

d. Auditing API or technology uses

My company has a history of security issues with JWTs (or <technology>), let's find all of the code that touches JWT logic so we can audit it.

e. Reviewing authentication or authorization logic

My company uses the acme_corp_auth library for all authentication or authorization logic. I want to review everywhere it's called to see if how the library is used makes sense, ensure that trust boundaries are enforced, confirm that enforcement is done consistently, and locate any places that are missing authn/authz.

These are a few examples, but in summary, the first step in writing a new rule is determining:

What is an interesting aspect of my code that I want to find?

2. Determine What That Concretely Looks like in Code

After you have a rough idea of what you want to find, the next step is to identify what that concretely looks like in code. The more specific the better.

What does "good" or "safe" code look like?

For example, after reading some internal code and reading API docs, you find that for the Python subprocess module, call() and other methods are more dangerous when passed the argument shell=True, so you decide:

I want to find all calls to subprocess.call() in which one of the keyword arguments is shell=True.

Or perhaps you're auditing Java Spring applications and you want to find all routes that do not perform authorization checks. You review some example routes in your internal repos and see code like this:

@Controller
@RequestMapping("/api/")
public class AcmeController {

    @RequestMapping(method = RequestMethod.POST)
    @Authorize(Permissions.ADMIN)
    @ResponseBody
    public ResponseEntity<Map<String, Object>> createProfile() {
        return new ResponseEntity<>(result, HttpStatus.OK);
    }

    @RequestMapping(method = RequestMethod.GET)
    @ResponseBody
    public ResponseEntity<Map<String, Object>> showResults() {
        return new ResponseEntity<>(result, HttpStatus.OK);
    }
}

So you determine:

I want to find all routes, which are methods with the the @RequestMapping annotation, that do not also have an @Authorize annotation.

The key at this stage is to go from an idea to one or more concrete example code snippets that demonstrate secure and/or insecure examples of what you're looking for.

3. Create a Test File, Write an Initial Rule

Now the fun part begins!

Create a YAML file in your current working directory, for example, the project root of a repo you intend to scan. You can name the YAML file whatever you want, but Semgrep will look for rules defined in a file named .semgrep.yml by default.

rules:
  - id: my-pattern-name
    pattern: |
      TODO
    message: "Some message to display to the user"
    languages: [python]
    severity: ERROR

Create an example test file, containing snippets of code that should and should not match.

Fill out the TODO in the above YAML file with a Semgrep pattern, using various pattern features and additional pattern clauses as needed.

For additional help, see these docs for walkthrough rule writing examples, or an interactive, example-based tutorial here: https://semgrep.dev/learn.

Check that the rule you're writing matches the example file by running:

$ semgrep example_file.py
$ semgrep --config my-rule-file.yml

Break it down

If you're writing a complex, multi-part pattern, rather than writing the whole pattern and then testing it, similar to writing a complex chunk of code, try building it in pieces, and testing each step along the way.

pattern and pattern-not: If there are a number of cases for the code you're searching for that you want to filter out, first write an initial pattern (using pattern) that finds the general case, and ensure it works as you expect.

Then, add a series of pattern-not clauses to filter out the cases that you don't want to match, that would cause your Semgrep rule to return "false positives."

pattern-either: If there are a series of code snippets you'd like to find that can't all be matched by the same pattern, create a test file with each of these code snippets, and then write a pattern clause for each.

Then, combine all of these clauses under a pattern-either clause, which will cause Semgrep to match code for which any subclause matches. Your Semgrep rule will look something like this:

rules:
  - id: my-pattern-name
    patterns:
      pattern-either:
      - pattern: |
          <first pattern here>
      - pattern: |
          <second pattern here>
      - pattern: |
          <and however many more you want to match...>
    message: "Some message to display to the user"
    languages: [java]
    severity: ERROR

See here for an example of using pattern-either.

4. Iterate and Refine

After you have a Semgrep rule that's been vetted against some test examples you created, it's time to see how it performs on real code.

4a. Test on One Repo

Clone one repo locally, if you don't have one ready, and scan it:

$ semgrep --config path/to/my-rules.yml path/to/repo

Look through the results. Are you finding what you intended to find?

False Positives

False positives, in simplest terms, are results that your tool gives you that are not code instances you care about.

Review Semgrep's output: for the code locations that your pattern matched, are they "interesting" code snippets that you intended to match?

If not, what makes them not interesting?

• If there are multiple results that you'd like to filter out in the future, is there a common reason that makes them something you'd like to filter?
• e.g. "Whenever foo() is called before check_auth(), the result is generally something I don't care about, because of {business logic reasons}." or "If the second argument is a hardcoded string, the method call is safe."

False Negatives

False negatives are source code locations that you intended to find but that your pattern missed

In the general case, regardless of the tool you use, it's difficult to impossible to eliminate all false negatives. Provably finding every bug can be done, in some specialized systems, with a massive amount of work, using techniques like formal methods.

But in most contexts, in terms of "what can I practically do right now without person-years of effort," you can use the trusty ripgrep to find code your Semgrep rule may have missed.

If your pattern involves a specific method call or annotation, search the code base for all references to the string and manually audit them to determine if they are code locations your Semgrep rule should have matched.

# -C returns a file lines of code above
# and below the match
$ rg -C 5 "exec\(" .

$ rg -C 5 "@RequestMapping" .

4b. Test on Multiple Repos

In the previous step, you tested your Semgrep rule on one real code base and iterated on the rule to ensure it catches the code patterns you intended (decreasing false negatives) and limiting the cases where it matches code you don't want to match (false positives).

Now it's time to vet your rule on a bigger corpus.

Gather a number of additional repos to test on that contain the programming language and functionality that are relevant to the rule you've been writing.

Note that if you put the target repos into the same directory, you can scan all of them at once by running Semgrep in the parent directory:

# Scans all repos inside the current working directory
$ semgrep --config path/to/my-rules.yml .

As before, review the results, and iterate on the rule to make it more precise.

5. Roll Out to CI

Finding individual instances of bugs or antipatterns is nice, but ultimately, what's more impactful to continuously scanning your code for issues, and either alerting on or blocking bad code as soon as it's entered.

Staying on top of this manually would be a lot of work, so it's easier to have this done automatically by including Semgrep in your continuous integration (CI) so that your existing infrastructure handles scanning for you.

See the integration docs for details on how to integrate Semgrep into platforms including AppVeyor, CircleCI, TravisCI, GitHub Actions, and GitLab.

You may also want to spend some time considering how you plan to handle Semgrep results.

• Blocking the build: In some cases, you may want to fail the build if certain rules trigger, as you have high confidence that they've identified an impactful security issue that needs to be fixed.
• Alerting: In other cases, Semgrep may identify code patterns that are security-relevant, but are not necessarily vulnerabilities. For these, it likely doesn't make sense to fail the build.
  • Instead, the security team can be notified, for example, in an #appsec Slack channel, that they may want to do a code review or reach out to the corresponding developer for more context.

5a. Make data-driven improvements to rules based on user feedback

Despite your best efforts, there will likely be edge cases that occur in practice that trip up the rule you've worked so hard writing - either vulnerable code that it misses, or safe code that it incorrectly identifies.

The key is to build and maintain close relationships with engineering teams so that they feel comfortable giving you feedback when your rules aren't performing as well as you'd like, so that the rules can be improved.

If possible, collect metrics around your continuous code scanning, like:

• For each rule, how often does it fire? (in total, and per repo)
• For each rule, when it fires, how often do developers perceive it to be a real issue vs a false positive?
  • How often do developers fix the underlying code?

Making it easy for developers to provide feedback on the signal quality of a rule is quite valuable for building a continuous scanning system that both provides real security value and is appreciated by engineers. See Tricorder: Building a Program Analysis Ecosystem whitepaper by Google for more details on creating analysis feedback loops.