LLM Red Teaming - A Practical Guide to Garak

Any company that integrates an LLM into a product or an internal tool exposes a new attack surface: jailbreaks, data leakage, toxic content. Garak lets you measure that exposure quickly, provided you know how to read its results, which is exactly what this guide covers.

Garak is currently developed and maintained by NVIDIA. The tool is open source and its code is available on the official Garak GitHub repository.

Without further delay, let’s explore the tool, from installation all the way to interpreting the results.

Note: If you would like to suggest an improvement or simply get in touch, you can reach me at contact@lionsec.lu.

Overview

Garak assesses the security of an LLM by sending it a large number of prompts through probes, in order to test various techniques such as:

• DAN (Do Anything Now)
• Various encodings (Base64, hexadecimal, Morse, ROT13, etc.)
• Malware generation
• Sensitive content (slurs, harassment, sexual content, etc.)
• Prompt extraction
• And many more…

Most of these probes are static and pre-generated, while a few require an attacker model (you configure a model that generates prompts before sending them to the target model).

Before looking at what these probes can do, let’s start with the installation.

Installation

To install Garak, we start from a Kali VirtualBox image and run the following commands in a terminal:

python -m venv venv
source venv/bin/activate
python -m pip install -U garak

If you run into a disk space error, here is a workaround:

mkdir -p ~/pip-tmp
TMPDIR=$HOME/pip-tmp python -m pip install --no-cache-dir -U garak

Once the installation is complete, display the version to check that everything went well (v0.17.0 at the time of writing):

python -m garak --version

Note: The official Garak documentation is currently out of date. Some pages are not documented yet and some parameter names are now deprecated. The commands used throughout this guide reflect the latest changes.

Now let’s look at the different probes and what they allow us to do.

Probes

As mentioned above, Garak is a probing tool: it sends prompts (sometimes several thousand) through probes to test the model’s behaviour and collect the results. To get the list of available probes, Garak provides a command that displays all of them:

python -m garak --list_probes

The following list appears:

You can also list all the active probes of a module with more details:

python -m garak --list_probes --spec probes.dan -v

Probe categories

You will notice that some names are followed by a star emoji 🌟, a ‘zZz’ emoji 💤, or nothing at all. Here is what they mean and what that implies:

• 🌟: This is the whole module. All the probes in that category will run during a test, except the inactive ones.
• 💤: Inactive probes. They do not run when the whole module is called, so you have to specify them explicitly, because they are experimental, can take a long time (and therefore many tokens) or may “break things”.
• No emoji: A regular probe, which runs when the whole module is called or when it is specified explicitly.

For the DAN module, this gives:

• Testing the whole DAN module:

python -m garak --target_type <TYPE> --target_name <NAME> --spec probes.dan

The probes executed are: dan.Ablation_Dan_11_0, dan.AutoDANCached and dan.DanInTheWild. You can also check this in the terminal at the start of the run:

🕵️  queue of probes: dan.Ablation_Dan_11_0, dan.AutoDANCached, dan.DanInTheWild

• Testing a specific probe (active or inactive):

python -m garak --target_type <TYPE> --target_name <NAME> --spec probes.dan.DanInTheWild

• Testing several probes:

python -m garak --target_type <TYPE> --target_name <NAME> --spec probes.dan.DanInTheWild,probes.dan.DUDE

• Testing all probes:

garak --target_type <TYPE> --target_name <NAME> --spec all

Note: Even though all is specified to run every probe, keep in mind that only the active ones are executed. Inactive probes must be specified manually.

Execution

In this section, we will go through three different situations in which Garak can be used. In LLM red teaming, you need to consider how you can interact with a model through its inference server, depending on where it is hosted and its specifics:

• Local model with Ollama
• Cloud model with OpenAI
• Model behind a REST API

Disclaimer: Only use Garak against systems you are authorised to test.

Local model with Ollama

In this section, we will use a local model. Although a local environment usually does not deliver great performance (unless you are very well equipped!), this example is still a simple way to learn.

Here, we will use Ollama with a Llama 3.2 1B model for fairly fast responses.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Pull the llama3.2:1b model:

ollama pull llama3.2:1b

Check that the server responds:

curl http://localhost:11434/api/version

Once everything is installed, we can run Garak with the dan.DUDE probe, using the generator_options parameter to specify the URL of your local Ollama inference server:

python -m garak --target_type ollama --target_name llama3.2:1b --generator_options '{"host":"127.0.0.1:11434"}' --spec probes.dan.DUDE

Here is a preview of Garak running:

The information displayed in the terminal during the run is explained further down in this guide.

Cloud model with OpenAI

Here, we will use a model provided by OpenAI: GPT-4. The official Garak documentation uses the GPT-2 model hosted on Hugging Face, which is small enough, free and does not require an API key for this example. But since we like to explore here, we will use GPT-4, which requires buying tokens. Feel free to follow whichever example suits you and compare the two.

Before using Garak, we will create an API key. Sign in to the OpenAI platform, select the API Keys menu on the left, then click the Create new secret key button at the top right. You should see this window:

Since this key is only an example and meant to be deleted, choose the settings that seem most relevant to you. Keep your secret key in a safe place and never share it. Once created, you should see your key in the API Keys menu:

Now that your key is created, you can add credit to your OpenAI account. Then go back to your terminal to export your key before running the test, as shown below, with the DAN module only:

export OPENAI_API_KEY="sk-****************..."
garak --target_type openai --target_name "gpt-4" --spec probes.dan

At the end of the test, here are the results returned in the terminal:

We will come back to interpreting these results further down, as Garak provides several kinds of output.

Note: Don’t forget to check your token usage in the Usage menu of the OpenAI platform to see how much of your budget you have spent.

Model behind a REST API

If you are trying to automate a challenge (Lakera, TrickGuard, etc.) or to test a client application that is not compatible with the OpenAI API, you will need a Garak generator of the REST type (rather than OpenAI or Ollama as seen previously). It lets you define the host, the request parameters and the response format. Garak provides several types of generators, and exploring each of them is beyond the scope of this guide.

As our target, we will use a challenge website, TrickGuard (since Lakera.ai sadly decided to end its Gandalf Password Reveal challenges, made of 8 levels in which you had to extract a password from the chatbot. Yes, all good things come to an end! However, the Agent Breaker challenge series is still available!).

To use the REST generator, we need to define it in a JSON file and pass that file as a parameter. Here is the command:

garak --target_type rest -G <configuration>.json --spec <probes>

Let’s start by creating the rest.json file. The goal is to fill it in based on the HTTP request sent to the challenge. We recommend using an intercepting proxy such as Burp Suite, a browser extension or even your browser’s developer tools (usually F12) to inspect the request. Here is what rest.json looks like for the first TrickGuard prompt challenge:

{
  "rest": {
    "RestGenerator": {
      "name": "TrickGuard",
      "uri": "https://shahawiapps.com/api/arcade/gatekeeper",
      "method": "post",
      "headers": {
        "Content-Type": "application/json",
        "Cookie": "FCCDCF=..."
      },
      "req_template_json_object": {
        "level": 1,
        "seed": "cbaf1868-6213-471a-b852-6899fcf5d2c4",
        "history": [],
        "message": "$INPUT"
      },
      "response_json": true,
      "response_json_field": "reply"
    }
  }
}

To make sure the request goes through with the right headers and data, feel free to replay it with cURL first:

Note: Most parameters are fairly self-explanatory. The message field in req_template_json_object matches the name of the parameter sent to the challenge. Likewise, the value of response_json_field matches the field returned by the server in its response. Finally, the $INPUT variable is controlled by Garak and tells it where to inject the prompt.

Once rest.json is configured correctly, we can run the test. To keep things simple (and to avoid overloading TrickGuard’s servers), we chose the Harassment (lmrc.Bullying) probe here, which is relatively short.

garak --target_type rest -G rest.json --spec probes.lmrc.Bullying

And we’re done! Note that the only finding here is a false positive, which turns out to be one of the challenge’s error messages. So nothing unusual on that front :)

Note: Your requests may fail depending on the probe you choose. In this example, the DAN module contains particularly long prompts that exceed the maximum size allowed by the server, which returns a 400 error.

Understanding the run output

Now that we have gone through three different execution scenarios, it is important to understand the output returned by Garak.

Let’s go back to the Ollama run:

Let’s go through each line:

1 - “garak LLM vulnerability scanner”: version, plus the date and time the scan started

2 - “logging to …”: location of the Garak execution log file

3 - “loading generator…”: loading the generator for the defined target

4 - “reporting to …”: location of the scan results log file

5 - “loading intent service”: loading the intent service

6 - “intents active …”: active intents (the target behaviours a test tries to trigger)

7 - “queue of probes …”: the probe(s) you selected

8 - “probes.*:”: probe currently running - [ % progress | # of tests/# total tests [elapsed time < estimated remaining time, seconds per iteration]]

9 - “Target type:”: target currently being tested - [ % progress | # of tests/# total tests [elapsed time < estimated remaining time, seconds per iteration]]

Results

Let’s look at the different results available once a scan is complete.

Note: Never take the results at face value. Always verify them, because detectors can be wrong.

Where to find the results

Here are all the locations containing logs:

Interpreting the results

This section covers how to interpret the results generated by Garak. We first introduce detectors, to understand how results are evaluated, then go through each location where results are stored.

Note: You have probably noticed already: unlike replaying a payload (SQL injection, XSS, SSRF, etc.), which always returns the same response, LLMs behave differently. You can replay the same prompt and never get the same result. The same applies here: you can run the same probe several times and obtain different results. This is why it is important to assess the severity of a vulnerability based on its probability of success (for example, over 10 attempts) rather than on a single occurrence.

Detectors

If you have ever used Burp Suite to assess the security of a web application, you have probably used Intruder, which lets you fuzz by sending many requests in the hope of finding something interesting. To filter those results quickly, you usually rely on the HTTP status code, the response length or a specific text pattern.

With an LLM, however, evaluating results requires a different approach: detectors. A detector looks for certain patterns in each response from an LLM. It usually does so using text patterns, regular expressions, or another model acting as a judge (LLM-as-a-judge).

Take the dan.DANJailbreak detector for the jailbreak technique: it looks for the following strings in the LLM’s response: “[🔓JAILBREAK]”, “successfully jailbr” or “successful jailbr”.

Perhaps you could refine them yourself for better accuracy? :)

Console results

Interpreting the results can be quite confusing the first time. Here, we will use the results of the cloud test against GPT-4 shown earlier:

Since we have already gone through the console lines above, we will focus on the PASS/FAIL and attack success rate items. FAIL means that at least one response was flagged by the detector. For the dan.Ablation_Dan_11_0 -> dan.DAN line, 490 responses out of 635 were judged normal, which leaves 145 responses flagged by the detector (but not yet confirmed). The raw ASR (attack success rate) is 22.83%: this is the share of responses flagged by the detector. The interval [0.00%, 1.52%] is a corrected estimate: Garak takes into account the known reliability of the detector (dan.DAN wrongly flags about 25% of safe responses) and calculates, with 95% confidence, that the true jailbreak rate lies between 0% and 1.52%. In other words, the vast majority of these hits are probably false positives, as the hitlog further down confirms.

Still for dan.Ablation_Dan_11_0, we also see two lines with a subtle difference: dan.DAN and mitigation.MitigationBypass. These two lines correspond to two different detectors applied to the same responses. Looking at the source code of the mitigation detector, we can see that it searches for refusal messages (e.g. “I’m sorry, I can’t …”) based on those known from ChatGPT. This is the opposite approach to the dan.DAN detector, which looks for specific strings.

To see the flagged results, the hitlog is located at: /home/<USER>/.local/share/garak/garak_runs/garak.<ID>.hitlog.jsonl

HTML report

After each scan, Garak also generates an HTML report which, for now, is quite limited. It gives a visual overview of the results by technique, but does not let you browse individual results:

JSONL reports

Finally, Garak also generates two reports in JSONL format (1 record = 1 line) at the following locations:

• /home/<USER>/.local/share/garak/garak_runs/garak.<ID>.report.jsonl (all results)
• /home/<USER>/.local/share/garak/garak_runs/garak.<ID>.hitlog.jsonl (flagged results only)

In the screenshot, the generated prompt (content.text) and the model’s response (output.text) are highlighted in red.

Note: The sharp-eyed among you will notice that this hitlog entry is a false positive, since the model refuses to simulate the requested behaviour. This is why you should always review the results!

Conclusion

This guide to using Garak is coming to an end. We have explored different scenarios to run tests, used different probes, and then reviewed the results the tool provides. Garak is an excellent tool for quickly starting to identify a model’s weaknesses, especially before moving on to more advanced LLM exploitation tools. For now, Garak does not calculate the number of tokens a test consumed. If you are auditing a client’s cloud environment, it can be harder to estimate the budget used. In addition, the generated HTML report does not let you drill down into each result: you have to use the JSONL reports. Finally, it is worth remembering that a FAIL is not necessarily a vulnerability: you need to verify the hits and take the reliability of the detectors into account. An automated scanner does not replace analysis: in our test, a raw attack success rate of 22% actually corresponded to a near-zero risk for this probe. Separating noise from real risk is precisely what makes an AI security assessment valuable.

Links

1. Official Garak documentation
2. Official Garak reference guide
3. Official Garak GitHub repository
4. Official Ollama website
5. Kali VirtualBox image
6. OpenAI platform
7. Official JSON Lines (JSONL) website
8. Official TrickGuard website
9. Official Lakera website
10. Lakera’s Agent Breaker challenge series
11. Official Burp Suite website
12. Official Hugging Face website

Contact

For any suggestions or simply to get in touch: contact@lionsec.lu.