LLM Unlearning Benchmarks are Weak Measures of Progress

TL;DR: “Machine unlearning” aims to remove data from models without retraining the model completely. Unfortunately, state-of-the-art benchmarks for evaluating unlearning in LLMs are flawed, especially because they separately test “forget queries” and “retain queries” without examining potential dependencies between forget and retain data. We show that such benchmarks do not provide an accurate measure of whether or not unlearning has occurred, making it difficult to evaluate whether new algorithms are truly making progress on the problem of unlearning. In our paper, at SaTML ’25, we examine this and other pitfalls in more detail, and provide recommendations for unlearning research going forward. We additionally released two new datasets on HuggingFace: [swapped WMDP], [paired TOFU].

Overview

Large-scale data collection, particularly through data available on the Web, has enabled stunning progress in the capabilities of generative models over the past decade. However, using Web data wholesale in model training raises questions about user privacy, copyright protection, and harmful content generation. 

Researchers have come up with a number of potential ways to mitigate these harms. Among them is “machine unlearning,” where undesirable data (whether private user data, copyright-protected data, or potentially toxic content) can be deleted from models after they have already been trained. The intuitive goal of machine unlearning is to enable this deletion more efficiently than the obvious solution, which is to retrain the entire model from scratch (which would be incredibly expensive for a modern LLM). 

Benchmarking Unlearning

Unlearning is a difficult problem, and enabling research on this topic requires accurate metrics to measure progress. In order to evaluate unlearning, researchers have proposed several benchmarks. These generally have the following structure:

• A base model which may be a pretrained model or a model finetuned on some benchmark data.
• Forget data to be unlearned. This could also be specified as a concept or topic rather than data points.
• Retain data consisting of the remaining data that will not be unlearned.
• A forget set of evaluation queries that are meant to test access to unlearned information.
• A retain set of queries that are meant to test access to information that should not be unlearned.

Figure 1. The majority of LLM unlearning papers published in 2024 evaluate only on a handful of benchmarks, and all of these benchmarks have a “forget set-retain set” structure.

We surveyed 72 LLM unlearning papers published in 2024 in order to understand the state of unlearning evaluations today. Out of these, we found that a handful of benchmarks were overwhelmingly popular, as shown in Figure 1. All of these benchmarks follow the “forget set”/”retain set” structure described above. In fact, even in 2025, we find that new works continue to evaluate on this small set of benchmarks, sometimes restricting to only one or two benchmarks. As we show later in this post, this structure is too simple to adequately measure progress on unlearning.

We focused our work on some of the most popular benchmarks (highlighted in orange above), but the takeaways apply more generally to benchmarks with the structure described above.

Main Takeaways

The main finding of our work is that the majority of popular evaluation benchmarks (including but not limited to TOFU and WMDP) are weak measures of progress, and results reported on these benchmarks are anywhere from unreliable to actively misleading as far as whether unlearning has actually succeeded.

Therefore, we encourage the community to interpret results with caution and be aware of common pitfalls when interpreting evaluations. For example, if a paper evaluates solely on benchmarks that use a disjoint “forget” and “retain” evaluation, the results may not accurately reflect whether unlearning has actually occurred. 

Most importantly, empirical evaluations are a possibly necessary but not sufficient condition to ensure unlearning. They are highly useful for testing whether a method is broken, but cannot guarantee that a method has succeeded.

More specifically, we find:

• Benchmarks that split queries into an independent “forget set” and a “retain set” overestimate the effectiveness of unlearning. Introducing dependencies between these queries can reveal data that was supposedly unlearned, or destroy performance on data that was supposed to be retained. Note that we do not modify or attack the algorithms, only change the evaluation queries.
• Ambiguities in the forget data — for example, specifying a concept or topic, rather than specific data points in the training set, to unlearn — can lead to poor evaluation hygiene in practice, such as “peeking” at evaluation data when designing the unlearning algorithm. 

In this blog post, we focus on the first point. In our paper, we provide a literature survey, more details on the second point, as well as recommendations for researchers going forward. We also provide two new evaluation datasets on HuggingFace: [modified WMDP], [modified TOFU]. 

Forget-Retain Evaluations are Deceptive

Key takeaway: Evaluating on an independent “forget set” and “retain set” is insufficient to measure unlearning. In reality, a single query can reference both forget data and retain data, and we find that these types of queries can reveal “unlearned” information or destroy access to “retained” information.

Finding #1: TOFU. The TOFU benchmark is comprised of a dataset of facts about fictitious authors and a Llama model finetuned on this data. The goal is to unlearn information about some of the authors while retaining information about the remaining authors.

The forget queries correspond to questions about authors in the forget set only, while the retain queries correspond to questions about the remaining authors, as well as world facts.

We find that simply concatenating a forget query and a retain query can uncover flaws in unlearning methods. For example:

The fully retrained model (the gold standard for unlearning) hallucinates an incorrect response for the first question, while answering the second correctly. DPO, an alignment method that has been applied to unlearning, refuses to answer at all. Meanwhile, ECO answers both queries correctly, even the forget query. In fact, we find that the simplest gradient ascent method has the best stability out of the three (retaining its performance in the combined query, although the initial performance appears worse).

Finding #2: WMDP. The WMDP benchmark consists of data to unlearn about potentially dangerous biological, chemical, and cybersecurity attacks, and multiple-choice questions about each topic, classified into benign (retain) queries and harmful (forget) queries.

We make a very simple modification to the retain queries: swap one of the incorrect choices with a keyword that is in the forget data — specifically, “SARS-CoV-2.” In a correctly unlearned model, this should have no impact on the model’s ability to answer correctly on the retain queries.

In reality, we find that swapping in an incorrect response results in a 28% decrease in accuracy for the state-of-the-art unlearning method RMU! Once again, introducing a very simple dependency on the forget data is sufficient to completely change the conclusions one draws from the benchmark, again without modifying or targeting anything about the algorithm.

Figure 2. Unlearning methods appear to perform well on “benign” retain set questions, but by simply including a keyword from the forget data in the retain question, the performance drops to below random.

Datasets. We do not necessarily believe that any one dataset can be comprehensive enough to ensure that unlearning has occurred, but a dataset can be a lower bound to determine whether unlearning has not occurred. Towards this, we release both of these datasets on HuggingFace: [swapped WMDP], [paired TOFU].

Where do we go from here?

Since our work became public in October 2024, the community has continued to report results and claim success on benchmarks that exclusively use a “forget-retain split” of data. As a starting point to move evaluations forward, we have released the evaluation sets that we use in our work, and encourage practitioners to use these to stress-test unlearning algorithms. 

While provable guarantees may be the ultimate measure of success, a strong evaluation can provide evidence that an algorithm is promising. We therefore encourage community members to take the time to develop further evaluation datasets that test potential failure modes of unlearning algorithms. We also strongly encourage algorithms to come with a threat model that describes in detail the system and query model under which the guarantee is expected to hold.

Ultimately, even the most thorough benchmark will still be limited by the query set. In our paper, we discuss possible directions for unlearning with provable guarantees and more rigorous tests of unlearning.

Source: Read More