How can one prove that AI has forgotten?

Machine unlearning aims to remove the influence of specified data from a pre-trained model. Normal completion of processing does not indicate that the influence of the target has disappeared. Record input acceptance, processing execution, and verification pass as separate states.

TOFU presented a task for evaluating LLM forgetting using fictional author profiles and question-answering. It showed that existing methods fail to meet the simple baseline of retraining in terms of balancing forgetting and retention performance. [1]

Merely refusing to answer a specific name might be suppression in the prompt. Test whether it reappears via aliases, paraphrasing, related questions, mid-completion, or references in long texts, and distinguish between surface behavior and internal influence.

When information related to the deletion target is also public knowledge, what should be forgotten becomes ambiguous. Distinguish between memory of individual records, general knowledge, and evidence provided later, and define the deletion target and acceptable responses in advance.

The expression 'forgotten' comes with a level of guarantee. Do not summarize different levels—such as reduced target output, no leakage under known attacks, and difficulty distinguishing from a retrained model—into a single completion display.

• Reception: Identify target data and derivatives
• Processing: Record method, model version, and execution conditions
• Verification: Test forgetting, retention, and leakage separately
• Display: Show the achieved level of guarantee and remaining uncertainties

The ideal comparison target is a model retrained under the same conditions without including the deletion target from the start. Bourtoule et al. proposed SISA, which splits training to limit the scope of retraining, incorporating deletability into the learning design. [3]

Even if full retraining is expensive for large LLMs, its significance as a baseline remains. Measure the retraining difference using small models, limited data, and representative tasks, and confirm how closely approximate methods can follow.

Output agreement with the retrained model alone is insufficient. Two models may make the same incorrect answer. Compare dependency on target data, performance of the retention set, and attacker identifiability using multiple metrics.

Model differences arise from learning randomness and data order. Do not treat a single retrained model as absolute truth; distinguish the range of natural retraining variation from the difference after unlearning.

Do not omit verification on the grounds that retraining is impossible. In such cases, limit the guarantee and record compensatory controls such as access blocking, output monitoring, usage suspension, and exclusion from future models.

The simplest way to stop answering target information is to significantly damage the entire model. However, this is not useful unlearning. Measure the performance degradation on the deletion target and the maintenance of general abilities to be retained using the same change.

MUSE evaluated LLM unlearning from multiple perspectives, including forgetting, model utility, privacy leakage, scalability, and persistence. It reported the difficulty of stably meeting all criteria for the methods reviewed. [2]

The retention set includes not only general problems unrelated to the deletion target but also adjacent legitimate knowledge. Detect excessive forgetting that causes the loss of general rules of the same industry while deleting a specific customer record.

Also measure generalization to paraphrases and related concepts. Even if the target sentence is forgotten verbatim, the same fact might be reconstructed from different relationships. Conversely, related concepts might be deleted too broadly.

Acceptance requires both a maximum for target-information reproduction or leakage and a minimum for retained utility. Maximizing a forgetting score alone is insufficient, and performance on one criterion does not offset a breach of the other.

• Target Forgetting: Do not reproduce specified information and its paraphrases
• Adjacent Retention: Do not destroy adjacent legitimate general knowledge
• General Ability: Monitor regression in reasoning, language, safety, etc.
• Excessive Forgetting: Detect overly broad suppression or increased refusals

Even for a model that does not answer normal questions, the influence of target data might be inferred from membership inference, prompt completion, iterative questioning, or probabilistic information. Do not equate output non-display with privacy removal.

Attack evaluation is limited to known methods and does not provide safety proof against unknown attacks. Clarify the auxiliary information, API access, and number of trials available to the attacker, and record under which threat model it passed.

Look not only at the leakage difference before and after deletion, but also at the difference from a model retrained without the target. If leakage is reduced but remains higher than the retrained baseline, residual risk exists.

Target data may remain in model output logs, caches, search indexes, and evaluation sets. If only parameters are changed and peripheral systems are left alone, deletion from the user's perspective is not complete.

Ensure that target information is not redistributed to the verification data itself. Evaluate in an access-controlled environment and leave only the necessary minimum aggregation in the results. Ensure that deletion verification does not become a new source of leakage.

In actual operations, deletion requests do not end in one go. If approximate processing is repeated with each new request, degradation of retention ability or reappearance of previous forgetting may occur. Do not extrapolate single success to continuous operation.

MUSE showed issues with the persistence of existing methods in evaluations including continuous deletion. [2] Change the deletion order, number of items, and proximity of targets, and re-test previous targets and the retention set each time.

If additional learning or model editing is performed after deletion, check whether forgotten information is restored. Unlearning completion is not a permanent state but depends on subsequent changes.

If cumulative degradation exceeds the threshold, stop continuing approximate deletion and switch to retraining or model replacement. Determine the threshold including general ability, leakage, verification cost, and deletion deadline.

If results change depending on the order of deletion requests, a fair processing procedure is also necessary. Do not treat technically easy requests as completed first; clearly state the deadline and level of guarantee to the applicant.

Even if only the original data is specified, it may have derived into pre-processed copies, embeddings, synthetic examples, fine-tuning data, and evaluation examples. Do not define the deletion target as a single file path.

Without a lineage connecting identifiers at collection, conversion history, learning jobs, and model versions, it is impossible to confirm which models were affected. Unlearning capability depends on data management capability.

If target information is mixed into synthetic data, it may not be traceable by string matching alone. Keep records of the generation source and sampling history, and have humans verify semantic proximity to the deletion target.

If learning is delegated to an external model provider, confirm in the contract the deletion method, verification evidence, handling of downstream models, and backups. Do not interpret a deletion button on the API as complete guarantee.

If derivatives cannot be traced, communicate this uncertainty to the applicant and decision-makers. Do not describe it as complete deletion; separately report access suspension, model usage suspension, and exclusion from future learning.

For low-impact personalization removal, output suppression and search index deletion may suffice. For high-sensitivity data or strong deletion obligations, retraining or model disposal may be necessary in addition to approximate unlearning.

The choice sheet should include the request purpose, target scope, deadline, acceptable leakage, retention ability, and retraining cost. Do not choose the newest method, but the minimum safe means that meets the required guarantee.

The completion report should include the implemented processing, verification set, attack conditions, difference from retraining, remaining derivatives, and next verification time. Do not use a single successful screenshot as evidence.

Do not judge legal deletion obligations solely by technical evaluation. Legal and privacy officers interpret the request, and the technical side accurately explains the achieved guarantee and unmet parts.

Unlearning is not a function to create the appearance of 'forgetting.' It is an operation that demonstrates how close one has come to a world without the target, using multiple falsifiable tests including retention ability and leakage.

The verification record places deletion-request ID, target, and execution model version in identifier fields, with a maximum leakage result and minimum retained capability as separate acceptance criteria. A run that exceeds leakage or misses retained capability is not marked verified; operations switch only to a version confirmed not to contain the target data. If no such version exists, release is held until retraining without the target is complete, and verified status returns only after the privacy and model-risk owners approve an independent retest.