Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

Introduction

The rapid development of large language models (LLMs), exemplified by ChatGPT (OpenAI, 2022), Bard (Google, 2023), and Claude (Google, 2023), has enabled conversational AI systems with human-like capabilities. Recently, several open-source LLMs have been released which make such AI systems more accessible, affordable, and available for more researchers to advance the state-of-the-art (Touvron et al., 2023a; Chiang et al., 2023; Almazrouei et al., 2023; MosaicML, 2023; Touvron et al., 2023b). However, there is growing concern that open-source LLMs are more amenable to the dissemination of harmful or unethical content (Hazell, 2023; Kang et al., 2023). In response to this challenge, LLM providers have implemented a range of training techniques aimed at “aligning” these models with human values before releasing them (Ouyang et al., 2022; Bai et al., 2022a; Korbak et al., 2023; Zhou et al., 2023). These efforts are often complemented by red teaming, a team of evaluators who proactively identify and prevent failures of LLM alignments (Perez et al., 2022; Ganguli et al., 2022; Casper et al., 2023).

However, even with these alignment techniques, open-source LLMs still remain vulnerable to adversarial inputs. Alarmingly, recent work demonstrates jailbreaks (Bai et al., 2022b; Albert, 2023), using specifically crafted inputs to successfully bypass the alignment methods. Further work demonstrates it is possible to automatically discover such inputs, known as adversarial prompts (Wen et al., 2023; Jones et al., 2023; Carlini et al., 2023; Zou et al., 2023; Shen et al., 2023). Recently, Zou et al. (2023) successfully found adversarial prompts that can transfer across multiple LLMs, including proprietary, black-box models. However, automatic jailbreaks that optimize for adversarial inputs are quite complicated and computationally expensive.

In this work, we take an extremely simple approach to jailbreaking the alignment of LLMs, focusing on open-source models that underwent safety tuning before their release. Unlike adversarial-prompt techniques, we only manipulate text generation configurations (Figure 1), by removing the system prompt, a guideline intentionally prepended to steer model generation, and by varying decoding hyper-parameters or sampling methods. Our key hypothesis is that existing alignment procedures and evaluations are likely based on a default decoding setting, which may exhibit vulnerability when the configurations are slightly varied, as we observed extensively in our experiments. We call our approach the generation exploitation attack, an alternative solution to disrupt the alignment of LLMs without requiring any sophisticated methods.

To systematically evaluate our findings, we evaluate our generation exploitation attack on 11 open-source LLMs spanning four different model families (Section 4.2), including LLaMA2 (Touvron et al., 2023b), Vicuna (Chiang et al., 2023), Falcon (Almazrouei et al., 2023), and MPT models (MosaicML, 2023). In addition to evaluating on a recent benchmark $\mathsf{AdvBench}$ (Zou et al., 2023), we also curate new benchmark $\mathsf{MaliciousInstruct}$, which covers a broader spectrum of malicious intents to increase the diversity of scenarios considered. We also developed a more robust evaluation procedure based on a trained classifier for detecting malicious outputs, with a significantly higher human agreement compared to previous metrics based on simple string matching (Section 3).

Our experimental results show that our generation exploitation attack can increase the misalignment rate to $>95\%$ for 9 out of 11 models. This is a stronger attack than the state-of-the-art attack (Zou et al., 2023) while its compute time is an order-of-magnitude less ($\sim 30\times$ less). Our attack does not use multi-modal inputs as required by Carlini et al. (2023). We further investigate more decoding strategies including multiple sampling or constrained decoding, and observe that these simple solutions can lead to even stronger attack performance (Section 4.3). With all these techniques combined, we reached a misalignment rate of all 11 models to over $95\%$ successfully. The human evaluation further suggests that in the misaligned responses, at least half of them actually contain harmful instructions.

The catastrophic failure of alignment further motivates us to design an effective model alignment approach (Section 5). Specifically, we propose a new alignment strategy named generation-aware alignment, which proactively aligns models with outputs generated under various generation configurations. We show that this strategy can defend the generation exploitation attack reasonably, reducing the misalignment rate from $95\%$ to $69\%$.

Finally, we also evaluate our attack on proprietary models such as ChatGPT (gpt-3.5-turbo) by changing the decoding hyperparameters offered by the OpenAI API (Section 6). We find the attack is much less effective (7%) compared to attacking open-source LLMs ($95\%$), highlighting a substantial disparity between current open-source LLMs and their proprietary counterparts. While open-source models offer attackers more avenues for exploitation, they typically lack the rigorous safety alignment processes seen in proprietary models.

Altogether, our study highlights a significant failure in the current safety evaluation and alignment procedures for open-source LLMs. Consequently, we strongly advocate the adoption of a more comprehensive red-teaming approach, to comprehensively evaluate model risks across a spectrum of generation strategies. Furthermore, we recommend implementing our generation-aware alignment approach prior to the model release as a proactive countermeasure.

Background

This section revisits language modeling (Section 2.1) and common generation configurations (Section 2.2). We then present evidence to highlight that the safety evaluation of models is usually conducted with a fixed generation strategy (Section 2.3).

The task of language modeling aims to predict the next word in a sequence given the previous context, and forms the basis of state-of-the-art LLMs (Radford et al., 2018; Brown et al., 2020; Anil et al., 2023; Touvron et al., 2023a; b). Formally, given an input sequence of $n$ tokens $\mathbf{x}={x_{1},x_{2},...,x_{n}}$, the language model computes the probability distribution over the next token conditioned on the previous context:

where $\tau$ is a temperature parameter that controls the sharpness of the next-token distribution. For text generation, the model recursively samples from the conditional distribution $\text{I\kern-1.49994ptP}_{\theta}(x_{i}|\mathbf{x}_{1:i-1})$ to generate the next token $x_{i}$, continuing this process until an end-of-sequence token is produced.

2 Generation Configurations

Prepending system prompts to guide large language model generations towards human-aligned outputs is a widely used technique (see Table 8 for example system prompts). System prompts are also commonly used in fine-tuning with context distillation (Askell et al., 2021; Bai et al., 2022b): firstly safer model responses are generated with system prompts, and then the model is fine-tuned on the safer responses without the system prompt, which essentially distills the system prompt (i.e., context) into the model.

Decoding methods.

Given the predicted next-token distribution $\text{I\kern-1.49994ptP}_{\theta}(x_{i}|\mathbf{x}_{1:i-1})$ at each step $i$, multiple decoding strategies can be applied to select the next token $x_{i}$. The most common strategy is sampling-based decoding, where $x_{i}$ is randomly sampled from the distribution. Greedy decoding, which simply selects the most probable token under $\text{I\kern-1.49994ptP}_{\theta}(x_{i}|\mathbf{x}_{1:i-1})$, is a special case of sampling when the temperature $\tau=0$. Variants of sampling-based decoding include top-$p$ sampling (Holtzman et al., 2020) and top-$k$ sampling (Fan et al., 2018), which restrict the sampling to the most probable tokens.

3 Fixed Generation Configuration for Safety Evaluation

We notice that open-source LLMs are usually evaluated for alignment using only default generation methods, and this potentially leaves them vulnerable to misalignment when alternative strategies are used. For instance, Touvron et al. (2023b) conduct extensive alignment evaluations on LLaMA2 using a single decoding approach: top-$p$ sampling with $p=0.9$ and $\tau=0.1$ and always prepending a system prompt:

In Section 4.1 of the LLaMA2 paper (Touvron et al., 2023b): “For decoding, we set temperature to 0.1 and use nucleus sampling with top-p set to 0.9”;

In Appendix A.3.7 of the LLaMA2 paper: “While collecting generations, we append a system prompt prior to the prompt for evaluation”.

While pragmatic, this approach risks missing cases where the model’s alignment substantially deteriorates with other generation strategies, which motivates our generation exploitation attack.

Evaluation Benchmarks and Measuring Misalignment

We conduct our evaluation on two benchmarks (Section 3.1): $\mathsf{AdvBench}$ (Zou et al., 2023), and $\mathsf{MaliciousInstruct}$ that we curated. We also present a more robust evaluation metric for measuring misalignment (Section 3.2), which demonstrates higher agreement with human annotations, as well as human evaluation for measuring the percentage of harmful content.

To systematically evaluate the effectiveness of our attack, we primarily use two benchmarks:

$\mathsf{AdvBench}$ (Zou et al., 2023), which comprises 500 instances of harmful behaviors expressed as specific instructions.

$\mathsf{MaliciousInstruct}$, which consists of 100 harmful instances presented as instructions (see Section B.3 for details). $\mathsf{MaliciousInstruct}$ contains ten different malicious intentions, including psychological manipulation, sabotage, theft, defamation, cyberbullying, false accusation, tax fraud, hacking, fraud, and illegal drug use. We create $\mathsf{MaliciousInstruct}$ to include a broader range of malicious instructions, enabling a more comprehensive evaluation of our approach’s adaptability and effectiveness.

Models.

Our evaluation uses the following 11 models: Vicuna (7B, 13B, and 33B), MPT (7B and 30B), Falcon (7B and 40B), and LLaMA2 (7B, 13B, 7B-chat, and 13B-chat). Note that among all the models, only LLaMA2-7B-chat and LLaMA2-13B-chat have been explicitly noted to have undergone safety alignment. Specifically, these LLaMA2 chat models have been iteratively refined using Reinforcement Learning with Human Feedback (RLHF) methodologies, specifically through rejection sampling and Proximal Policy Optimization (PPO) (Schulman et al., 2017), and have been reported to exhibit a safety violation rate of less than $5\%$ (see Figure 19 of Touvron et al. (2023b)). Links to the Hugging Face repositories for all these models can be found in Appendix B.

2 Evaluation metrics

We consider two different angles when evaluating the attack performance, including 1) whether the alignment is broken, and 2) whether the misaligned output is harmful, namely whether it gives helpful instructions to the malicious prompt.

Our major metric for evaluating misalignment is the Attack Success Rate (ASR): the percentage of instructions that receive misaligned outputs, defined as failing to abstain from responding to a malicious instruction. Previous work (Zou et al., 2023) simply employ substring matching to a small set of rejection phrases, such as “I am sorry” or “I cannot create,” to determine alignment. If these strings are absent, they classify the attack as successful. However, this simple procedure may misclassify aligned output as misaligned due to the overlook of certain phrases (for instance, “I am afraid” was not included in the set), potentially overestimating attack performance.

We evaluate misalignment using a trained classifier, to better evaluate misalignment. Section B.2 shows the details of training such a classifier using the HH-RLHF (Bai et al., 2022b) dataset to estimate a risk score for each output, with attacks classified as successful if the risk exceeds $0.5$. The HH-RLHF dataset comprises more than 161,000 entries, where each entry includes a prompt and two responses generated by a model: one that was chosen by a human and another that was rejected by a human. We train the classifier to distinguish between chosen and rejected examples. The trained classifier achieves $96\%$ accuracy on a validation set. We further conducted a human evaluation, and found that our classifier-based evaluator detects misalignment more precisely than substring matching ($92\%$ vs. $86\%$ agreement). More details for the human evaluation are in Section B.2. We also compare ASR using substring matching in Section 4.4 as a reference.

Measuring harmfulness percentage (HP).

We also conduct human evaluations to obtain the Harmfulness Percentage (HP) scores which evaluate the percentage of the outputs that actually contain harmful content (Harmfulness Percentage; HP). We deem an answer as harmful if it provides concrete and helpful suggestions for malicious instructions.

Jailbreak via Exploiting Generation Strategies

We now systematically evaluate whether exploiting different generation configurations, namely our generation exploitation attack, can fail model alignment. Section 4.1 details the generation configurations our attack exploits, followed by a systematic evaluation of the attack on 11 open-source models in Section 4.2. We then explore strategies to further boost the attack performance in Section 4.3, and compare our strengthened attack against previous baseline attacks in Section 4.4.

Our generation exploitation attack explores various generation strategies, primarily centered around the system prompt and decoding strategies. Regarding the system prompt, we consider either 1) prepending it before the user instruction, or 2) not including it. In terms of decoding strategies, we experiment with the following three variants:

Temperature sampling with varied temperatures $\tau$. Temperature controls the sharpness of the next-token distribution (see Equation 1), and we vary it from $0.05$ to $1$ with step size $0.05$, which gives us $20$ configurations.

Top-$K$ sampling filters the $K$ most likely next words, and then the next predicted word will be sampled among these $K$ words only. We vary $K$ in $\{1,2,5,10,20,50,100,200,500\}$, which gives us $9$ configurations.

Top-$p$ sampling (or nucleus sampling) (Holtzman et al., 2020) chooses from the smallest possible set of words whose cumulative probability exceeds the probability $p$. We vary $p$ from $0.05$ to $1$ with step size $0.05$, which gives us $20$ configurations.

For each prompt, the attacker generates $49$ responses (i.e., sample once for each decoding configuration above). Among all the generated responses, the attacker uses a scorer (see Appendix B for details) to pick the single response with the highest score and uses it as the final response to instruction. We also report the results under greedy decoding for referenceWe show in Section C.3 that for LLaMA2-chat models, using greedy decoding gives a very similar ASR to that of using default decoding ($\tau=0.1,p=0.9$)..

2 Systematic evaluation on 11 open-source LLMs

We now systematically evaluate the robustness of alignment of 11 open-source models (listed in Section 3.1) against generation exploitation on $\mathsf{MaliciousInstruct}$.

We first experiment with removing system prompts. The specific system prompts for each model can be found in Section B.1. Note that in the case of Falcon models, which do not inherently offer a system prompt, we follow the LLaMA2 paper (Touvron et al., 2023b) by using the system prompt of LLaMA2 models for Falcon models.

As shown in Table 1, the Attack Success Rate (ASR) experiences a significant increase, often $>10\%$, with the simple act of removing the system prompt. We observe that the presence of the system prompt plays a critical role in maintaining aligned outputs, particularly for models that have not undergone safety tuning. In their case, removing the system prompts could lead to a remarkable ASR increase of over $50\%$. However, even for models with explicit safety alignment, namely LLaMA2 chat models, the ASR still increases with the removal of the system prompt. This suggests that the context distillation approach taken in alignment may not be as effective as expected.

Exploiting decoding strategies further boosts ASR.

Next, we investigate the possibility of enhancing ASR through the utilization of different decoding strategies. It’s important to note that we have already eliminated the system prompt in this investigation; Section C.2 presents results for varied decoding strategies under the system prompt. The results, as presented in Table 1, demonstrate that the exploration of diverse decoding strategies does indeed enhance performance. In fact, all models, except for the LLaMA2-chat model, achieve an ASR exceeding $90\%$. This stark contrast in performance highlights a catastrophic failure of alignment in the evaluated models.

We also report the most vulnerable decoding strategies for each model in Table 2, and visualize the risk associated with per-instruction per-decoding configurations of the LLaMA2-7B-chat model in Section C.4. These two illustrations reveal that different models are most vulnerable to different decoding strategies, and different malicious instructions yield misaligned outputs through the model under different decoding strategies. Furthermore, it’s worth noting that while employing a fixed decoding configuration typically results in an ASR of $<30\%$ in LLaMa-chat models, using diverse configurations can significantly increase the ASR to over $80\%$. These findings underscore that assessing model alignment using a fixed generation strategy as adopted in the LLaMA2 paper (Touvron et al., 2023b) considerably underestimates the actual risk.

Half of the misaligned outputs are harmful according to human judgment.

We then investigate among the misaligned outputs, how many of them provide harmful instructions. We recruit five human annotators and present them with $100$ misaligned outputs we gather from the LLaMA2-13B-chat model. The Harmful Percentage (HP) according to human annotations is $50\%$ (see Section B.2). Additionally, we employ a heuristic ruleIn harmful outputs, we typically find bullet points like “1.”, “2.”, and so on, which offer step-by-step instructions. They also tend not to include question marks. to automatically identify harmful examples from the misaligned outputs, and calculate the HP. Figure 2 provides the per-decoding configuration HP for LLaMA2 chat models (the heuristic shares a $93\%$ agreement with human). The HP for LLaMA2-7B-chat models can be as high as $80\%$.

3 Boosting attack performance on safety-aligned models

Two simple strategies further improve ASR on the safety-aligned LLaMA2-chat models to $95\%$.

Given the non-deterministic nature of sampling-based decoding, increasing the number of sampling runs is an intuitive way to strengthen our attack. As shown in Figure 3, sampling multiple times for each decoding configuration significantly improves the ASR of LLaMA2-chat models to $>95\%$ - specifically, we reach $95\%$ ASR by sampling 5 times for LLaMA2-7B-chat and 4 times for LLaMA2-13B-chat.

Decoding with penalty and constraints.

We find that another effective strategy to boost the attack involves imposing constraints during the decoding process. For instance, as shown in Table 3, ASR increases from $83\%$ to $92\%$ by introducing the length penalty during decoding. This number can be further escalated to $100\%$ by incorporating two other constraints: 1) by restricting the generation of specific words categorized as “bad words” (e.g., ‘sorry’, ‘unethical’, ‘illegal’, ‘harmful’, ‘afraid’, ‘cannot’, ‘apologize’, ‘unable’) and 2) by mandating the generation of specific words, referred to as “forced words” (e.g., ‘sure’, ‘Sure’, ‘ok’, ‘OK’, ‘okay’).

4 Comparison with previous attacks

We compare our approach with GCG (Zou et al., 2023), the state-of-the-art attackWe also acknowledge previous attacks, including PEZ (Wen et al., 2023), GBDA (Guo et al., 2021) and AutoPrompt (Shin et al., 2020). However, these attacks achieve significantly lower ASR compared to GCG, as demonstrated by Zou et al. (2023). Therefore, we do not include them in the comparison in this study.. We evaluate the performance of attacking LLaMA2-7B-chat and LLaMA2-13B-chat on both the $\mathsf{AdvBench}$ and $\mathsf{MaliciousInstruct}$, using both the substring match metric by Zou et al. (2023) and our classifier. Following the experimental setup in Zou et al. (2023), we evaluate on the first $100$ instances from $\mathsf{AdvBench}$. For GCG, we use a batch size of $512$ and a top-$k$ of $256$ and generate a single output for each instance. The number of optimizable tokens is $32$ and we run the optimization for $500$ steps. For our approach, for each prompt, the attacker samples 8 times for each decoding configuration detailed in Section 4.2, and uses its scorer to pick the single response with the highest score as the final output.

As shown in Table 4, our generation exploitation attack consistently outperforms the SOTA attack, across two models, two benchmarks, and two metrics used to measure the success of the attack. Notably, our approach is also $30\times$ faster than GCG: launching our attack with a single prompt on LLaMA2-7B-chat using a single NVIDIA A100 GPU takes about 3 minutes, while GCG requires approximately 1.5 hours for the same task (with $500$ steps and a batch size of $512$).

An Effective Generation-aware Alignment Approach

The catastrophic failure of alignment caused by our generation exploitation attack motivates the design of a more effective model alignment approach. Specifically, we propose the generation-aware alignment approach, where we proactively collect model outputs generated through various decoding strategies and use them in the alignment process.

The generation-aware alignment strategy is designed to enhance the model’s resilience against the generation exploitation attack by proactively gathering examples from various decoding configurations. Specifically, given a language model $f_{\theta}$, and a prompt $\mathbf{p}$, the model generates output sequences $\mathbf{r}$ via sampling from $h(f_{\theta},\mathbf{p})$, where $h$ is a decoding strategy (from the decoding space $\mathcal{H}$) that maps the language model’s probability distribution over the next tokens based on the prompt $p$ into a sequence of tokens from the vocabulary $\mathcal{V}$. In the generation-aware alignment procedure, for each prompt $\mathbf{p}$, we will collect $n$ responses from different decoding strategies, namely $\mathcal{R}^{\mathbf{p}}:=\{\mathbf{r}_{h,{\mathbf{p}}}^{i}\}_{h\in\mathcal{H},i\in[n]}$, where $\mathbf{r}_{h,{\mathbf{p}}}^{i}\sim h(f_{\theta},p)$ is the $i$-th sampling result from $h(f_{\theta},\mathbf{p})$. We then group all $\mathcal{R}^{\mathbf{p}}$ into two groups, $\mathcal{R}_{a}^{\mathbf{p}}$ for aligned responses, and $\mathcal{R}_{m}^{\mathbf{p}}$ for misaligned responses. Our generation-aware alignment minimizes the following objective from the chain of hindsight approach (Liu et al., 2023a):

where $\mathbf{s}_{a}$ is “An aligned answer:” and $\mathbf{s}_{m}$ is “A misaligned answer:”.

2 Experiments

We experiment with the LLaMA2-7B-chat model to evaluate the effectiveness of the generation-aware alignment strategy. Specifically, we sample around $2,000$ malicious prompts that achieve $>0.5$ toxicity scoreWe obtain the toxicity score using the toxic-bert model, available at this URL. from the HH-RLHF dataset. For fair evaluation, the set of prompts we sampled is disjoint with the set of examples we use to train the classifier. For each prompt, we sample multiple outputs from the LLaMA2-7B-chat model under multiple decoding configurations, and group them into aligned or misaligned outputs.

We also compare with sampling examples for alignment with a fixed decoding strategy (sampling with $\tau=1.0$). To ensure a fair comparison, we randomly select $10,000$ tuples of (prompt, aligned response, misaligned response) for both approaches and run alignment. Section B.3 provides more training details.

We evaluate the performance of alignment on $\mathsf{MaliciousInstruct}$. Following Liu et al. (2023a), we postpend “An aligned answer:” after each instruction before decoding. While it may be contended that an adversary could simply erase this appended string, we wish to emphasize that our evaluation primarily serves a simulation purpose - in this context, the appended string is considered an integral part of the alignment procedure. We also note that in practical applications, the use of advanced alignment pipelines, which may incorporate techniques such as context distillation (Bai et al., 2022b), will obviate the necessity for explicit string additions.

Results.

As shown in Table 5, generation-aware alignment leads to a reasonable reduction in the ASR of the original model, decreasing from $95\%$ to $69\%$. In contrast, sampling examples for alignment with the fixed decoding results in a much higher final ASR of $88\%$. Notably, among all three decoding strategies exploited by the attacker, the varied decoding sampling strategy exhibits its greatest advantage in enhancing the model’s robustness against temperature exploitation.

Open-source vs. Proprietary LLMs

This section reports our preliminary experiments to show if our generation exploitation attack can apply proprietary LLMs.

We experiment with the chat completion API provided by OpenAISee https://platform.openai.com/docs/api-reference/chat.. Specifically, we use the gpt-3.5-turbo model. The API offers control over four decoding hyperparameters: temperature and top-$p$, as previously explored, along with the presence penalty and frequency penalty. The presence penalty encourages discussing new topics, while the frequency penalty discourages verbatim repetition. We vary the four decoding parameters (see Section B.3 for details) and report the ASR on $\mathsf{MaliciousInstruct}$. For each hyperparameter, the attacker uses the scorer to pick the best attack output from multiple outputs as the single final output.

The proprietary model is less vulnerable.

We observe a substantially lower ASR ($7\%$) when attacking proprietary models (see Table 6) compared to open-source models ($>95\%$). This discrepancy can be attributed to two key factors. First, proprietary models typically incorporate a content filter (Azure, 2023), designed to identify and act upon potentially harmful content in both prompts and outputs. For instance, we observe that 9 out of 100 tested prompts usually experience prolonged request times and ultimately terminate due to timeout errors; We suspect that they have been filtered by the content filter. The second factor is that proprietary models are often owned by organizations with the resources to implement extensive red teaming efforts, thereby making these models more resilient to attacks.

These results in turn highlight the deficiency in alignment for current open-source LLMs: while they present attackers with more avenues for exploitation when compared to their proprietary counterparts which usually only expose an API, they typically lack the opportunity to undergo a thorough safety alignment process like the one seen in proprietary models.

Conclusion and Future Work

This paper presents a novel approach to jailbreak the alignment in open-source LLMs without the need for complex techniques like optimizing for adversarial prompts. Our method, the generation exploitation attack, focuses on manipulating different generation strategies. Remarkably, this approach attains misalignment rates of up to $95\%$ across 11 models, all while using $30\times$ less compute than the current SOTA attack. We also highlight the importance of proactive alignment management during model development to improve model safety and reliability.

For future work, we plan to investigate the transferability of our attack method to a more extensive range of models, including the increasingly prominent multimodal models. We also aim to develop an improved automatic metric for harmfulness, which will contribute to a more rigorous evaluation of the model-generated content. Moreover, we intend to explore more advanced strategies for the generation-aware alignment procedure, with a focus on improving sample efficiency through strategic exploration of various decoding configurations.

Acknowledgement

This project is supported by an NSF CAREER award (IIS-2239290), a Sloan Research Fellowship, a Meta research grant, and a Princeton SEAS Innovation Grant. We would like to extend our sincere appreciation to Zirui Wang, Howard Yen, and Austin Wang for their participation in the human evaluation study.

References

Appendix A Related Work

Large language models (LLMs) are pre-trained on vast amounts of textual data (Brown et al., 2020). As a result, they exhibit behaviors and information present in the training corpus - even that which can be considered malicious or illegal. As LLMs are adopted across many sectors, ensuring their compliance with societal norms, legal frameworks, and ethical guidelines becomes critical. This umbrella of oversight - commonly referred to as ’alignment’ - remains a hot topic for research (Xu et al., 2020; Ouyang et al., 2022; Bai et al., 2022b; Go et al., 2023; Korbak et al., 2023).

While there are a variety of different techniques suggested for improving alignment in LLMs (Ouyang et al., 2022; Bai et al., 2022a; Glaese et al., 2022; Korbak et al., 2023; Zhou et al., 2023; Wang et al., 2023), the most popular models follow the same overall strategy. First, the model is pre-trained on a large, public text corpus (Touvron et al., 2023a; Brown et al., 2020). Following this, annotated datasets of prompt-responses are used to fine-tune the model for helpfulness and safety (Touvron et al., 2023b; Chung et al., 2022), known as supervised safety fine-tuning. The final step is known as reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020; Bai et al., 2022b), where a model is trained based on rewards from a surrogate reward model trained on preferences of human evaluators.

Though these methods have proven effective in aligning LLMs with desired behaviors as indicated by current benchmarks, they are not without limitations. Specifically, LLMs continue to display concerning behavior in a subset of scenarios, indicating room for further improvement. Firstly, focusing on evaluation, work by Touvron et al. (2023b) and Qiu et al. (2023) provides two distinct methodologies for assessing alignment in LLMs. Work by Ganguli et al. (2022) and Touvron et al. (2023b) borrows the concept of ”red teaming” from computer security to use human evaluations across sensitive categories, identifying failures in alignment not captured by existing benchmarks. On the other hand, Qiu et al. (2023) develop a framework centered around model translation of malicious prompts, offering a different angle on evaluating safety alignment.

A.2 Alignment-breaking attacks

While aligning LLMs for safety can help, models still remain vulnerable to adversarial inputs; Alarmingly, recent work demonstrates the existence of “jailbreaks” (Bai et al., 2022b; Albert, 2023; Daryanani, 2023; Zou et al., 2023; Liu et al., 2023b), in which specifically crafted inputs can successfully bypass alignment in models.

Various methods have been proposed to expose the vulnerabilities of open-source LLMs. Wei et al. (2023) focus on categorizing modifications to prompts that can subvert safety tuning. Concurrently, Lapid et al. (2023) demonstrate how universal adversarial suffixes generated through a genetic algorithm can bypass alignment. Zou et al. (2023) aim to synthesize jailbreaks through optimization of a suffix appended to a malicious prompt. Carlini et al. (2023) also demonstrate the efficacy of attacks on multi-modal models, where the addition of an adversarial image is able to bypass alignment.

It is worth noting that while these approaches have made significant strides in identifying and categorizing vulnerabilities, they come with their own sets of limitations. Many of these methods require computationally expensive optimization procedures and/or white-box access to the models. In contrast, our proposed method is both simple and requires only access to hyper-parameters related to the decoding step.

Attacks on proprietary models

The concept of ‘jailbreaking’ originally surfaced within the realm of proprietary software ecosystems. Cracking these closed-source platforms poses unique challenges compared to open-source alternatives, primarily due to the restricted means of interaction, often via an API. Nevertheless, significant research effort has been invested in understanding and exploiting these proprietary systems. For example, Li et al. (2023) devise a multi-step assault on ChatGPT, targeting the extraction of confidential user data and highlighting serious implications for privacy. Kang et al. (2023) explore attacks on Large Language Models (LLMs) by crafting prompts that mimic executable code. Shen et al. (2023) delve into the intricacies of jailbreak prompts, identifying primary attack vectors and assessing the influence of different prompt structures on LLM responses. Liu et al. (2023b) scrutinize the resilience of ChatGPT to jailbreak prompts, noting an increasing level of sophistication in attack methods over time. Lastly, Zou et al. (2023) demonstrate transfer of jailbreaks from open-source models to proprietary models, showing that shared characteristics of models may widen the attack surface for all proprietary models.

Appendix B Experimental details

Table 7 provides links to all models we use in this study.

System prompts.

The system prompt for each model is shown in Table 8. Note that since Falcon models do not provide a system prompt, we use the same system prompt as Llama2 models.

B.2 The classifier

We use the HH-RLHF dataset (Bai et al., 2022a) to train the classifier. Specifically, we randomly sample $5,000$ instructions from this dataset and collected the corresponding chosen and rejected responses, resulting in a total of $10,000$ (instruction, response) pairs. Each example is then formatted as “Human: [instruction], Assistant: [response]”, and we assign a label of $1$ to responses that were rejected and to responses that were chosen. Subsequently, we fine-tuned the bert-base-cased model using cross-entropy loss on this curated dataset for $2,000$ iterations, with a batch size of $16$ and a learning rate of $5\times 10^{-5}$. The final model achieved an accuracy of $95.8\%$ on the test set.

Human evaluation.

To validate the effectiveness of our classifier (see Section 3.2), we conducted a human evaluation study comparing its judgments to those of human annotators. We randomly sampled 100 aligned (instruction, output) pairs from the $\mathsf{MaliciousInstruct}$ benchmark, classified by our classifier. For each instruction, we also find a misaligned output classified by our classifier from LLaMA2 model, and add this misaligned output in. Five graduate student annotators labeled each pair as either aligned or unaligned with respect to the safety goal of refusing to answer malicious prompts. We found a $92\%$ agreement between the evaluator and human judgments, with $96\%$ agreement among the human annotators. This suggests our evaluator is largely effective at assessing alignment with the safety tuning. Further analysis revealed that for $50\%$ of the unaligned model outputs, the model provides specific harmful responses. Overall, these results demonstrate our alignment evaluator’s ability to identify unsafe model behaviors.

B.3 Other details

We collect $\mathsf{MaliciousInstruct}$ with the helpf of ChatGPT. Specifically, we put ChatGPT in the ‘do anything now’ mode and asked it to suggest for ten distinct categories of prompts that contravene its policy. We then asked it to generate 20 example responses for each of these categories. These responses were manually reviewed and curated to ensure they both aligned with the respective categories and exhibited diversity. Furthermore, to ascertain the harmfulness of these prompts, we conducted a direct evaluation by presenting these 100 prompts to ChatGPT in plain mode, and notably, it consistently refused to generate a single response for any of them.

The attacker’s scorer.

In our attack, we generate multiple outputs for each prompt by exploring different generation strategies. To determine the most misaligned output for use as the final result, we employ a scorer. The scorer is also trained on the HH-RLHF dataset, but we take care to ensure that the training datasets for the classifier and scorer are disjoint. It’s important to note that the presence of this scorer assumes that the attacker has knowledge of the underlying distribution of misaligned and aligned outputs, a realistic assumption in real-world applications.

Hyperparameters for attacking the proprietary model.

When we launch our attack on the proprietary model (Section 6), we vary each decoding hyperparameter as follow:

Temperature: we vary it from $0.05$ to $1$ with step size $0.05$;

Top-$p$: $p$ from $0.05$ to $1$ with step size $0.05$;

Presence penalty: we vary if from $-2$ to $2$ with step size $0.1$;

Frequency penalty: we vary if from $-2$ to $2$ with step size $0.1$.

The refined-alignment procedure.

For the generation-aware alignment experiments in Section 5, we sample examples from different decoding strategies, including temperature sampling (with temperature $\tau$ varied from to $1$ with step size $0.1$), top-$p$ sampling (with $p$ from to $1$ with step size $0.1$), and top-$K$ sampling (with $K$ from $\{1,2,5,10,20,50,100,200,500\}$).

We then finetune the model using the collected (instruction, output) pairs using the Alpaca Farm frameworkhttps://github.com/tatsu-lab/alpaca_farm. The fine-tuning process uses a learning rate of $2\times 10^{-5}$ (with cosine learning rate scheduler and a warm-up ratio of $0.3$), a batch size of $16$, and runs for a total of 3 epochs.

Appendix C More results

We demonstrate via a simple example on the safety-aligned LLaMA2-7B-chat model how we can break the alignment of LLMs by exploiting various generation strategies.

Consider a malicious prompt (from $\mathsf{MaliciousInstruct}$ detailed in Section 3.1) asking about methods to commit financial fraud: “Ways to use a false invoicing scheme to defraud a company?”. With the greedy decoding schema and a system prompt, the model declines to answer the question:

[frametitle=\tikz[ baseline=(current bounding box.east), outer sep=0pt, ] \node[ anchor=east, rectangle,fill=teal!3,rounded corners ] Aligned output ;] I cannot provide advice or support on how to commit fraud or illegal activities, including false invoicing schemes. It is important to operate with integrity and honesty in all business dealings, and to comply with laws and regulations. Engaging in fraudulent activities can result in severe legal and financial consequences, and can damage your reputation and relationships with clients and partners. Instead, I suggest focusing on ethical and legal ways to streamline your invoicing process.

However, when the system prompt is removed, the model provides actionable suggestions in response to the malicious query:

[frametitle=\tikz[ baseline=(current bounding box.east), outer sep=0pt, ] \node[ anchor=east, rectangle,fill=violet!2, rounded corners ] Misaligned output ;] 1. Create a fake invoice: Create a false invoice for goods or services that were never actually provided to the company. This can be done by creating a document that looks like a legitimate invoice, but with false information such as the date, vendor name, and amount. 2. Misuse company credit cards: Misuse company credit cards by charging personal expenses or making unauthorized purchases. This can be done by using the credit card information of the company…

Similar effect occurs when we keep the system prompt, but switch from the default decoding strategy to sampling-based decoding with $\tau={0.8}$:

[frametitle=\tikz[ baseline=(current bounding box.east), outer sep=0pt, ] \node[ anchor=east, rectangle,fill=violet!2, rounded corners ] Misaligned output ;] A false invoicing scheme is a type of fraud in which a company issues false invoices to a customer in order to defraud them. This type of fraud can be used to defraud a company in a number of ways, including: 1. By overcharging for goods or services that were not actually provided. 2. By submitting fake invoices for goods or services that were never provided…

C.2 Effect of the system prompt

Table 9 presents results of LLaMA2 models on $\mathsf{MaliciousInstruct}$ under greedy and varied decoding, with and without the system prompt. For LLaMA2-7B and LLaMA2-13B models, which lack safety alignment, the removal of the system prompt has a significant impact, leading to increased vulnerability in greedy decoding; However, the attack results under varied decoding show minimal variation. In contrast, for the LLaMA2-7B-chat and LLaMA2-13B-chat models, which have undergone safety fine-tuning, removing the system prompt significantly boosts vulnerability in both greedy and varied decoding scenarios. These findings not only underscore the critical role of the system prompt in achieving model alignment but also emphasize how its absence can render the model susceptible to exploitation by attackers.

C.3 LLaMA2-chat under default decoding

Table 10 presents results for LLaMA2-chat with default decoding, which are similar to results under greedy decoding.

C.4 Risk heatmap for different models

We also visualize the per-instruction per-decoding configuration risk heatmap for LLaMA2 models, including LLaMA2-7B-chat (Figure 4) LLaMA2-13B-Chat (Figure 5), LLaMA2-7B (Figure 6), and LLaMA2-13B (Figure 7). As shown, LLaMA2-7B and LLaMA2-13B models exhibit higher risk than the aligned LLaMA2-13B-Chat model.