CVPR 2026 Findings

Revisiting Model Inversion Evaluation:
From Misleading Standards to Reliable Privacy Assessment

Sy-Tuyen Ho1,2*, Koh Jun Hao2*, Ngoc-Bao Nguyen2*, Alexander Binder3, Ngai-Man Cheung2
1University of Maryland College Park   2Singapore University of Technology and Design   3Leipzig University
* Equal Contribution
   Paper 🤗   Dataset    Code
TL;DR. Standard model-inversion evaluations can count visually unrelated reconstructions as successful. We identify this false-positive failure mode and introduce FMLLM, an MLLM-based evaluation framework.
Model Inversion results vs private training data showing false positives

Figure 1. We present the first in-depth study on the Model Inversion (MI) evaluation framework FCurr. The MI-reconstructed images (left) are deemed successful attacks with high confidence (red scores), yet they do not visually resemble the actual private training individuals (right), a significant false-positive problem that inflates reported attack accuracy.

Abstract

Model Inversion (MI) attacks aim to reconstruct information from private training data by exploiting access to a target model. Nearly all recent MI studies evaluate attack success using a standard framework that computes attack accuracy through a secondary evaluation model trained on the same private data and task design as the target model.

In this paper, we present the first in-depth analysis of this dominant evaluation framework and reveal a fundamental issue: many reconstructions deemed "successful" are in fact false positives that do not capture the visual identity of the target individual. We show these MI false positives satisfy the same formal conditions as Type I adversarial examples, and demonstrate extremely high false-positive transferability.

To address this, we introduce a new evaluation framework FMLLM based on Multimodal Large Language Models, whose general-purpose visual reasoning avoids the shared-task vulnerability. We reassess 27 MI attack setups and find consistently high false-positive rates under the conventional approach, calling for a reevaluation of progress in MI research.

Key Findings

  • False positives are Type I adversarial examples. The two are mathematically equivalent phenomena in different evaluation contexts.
  • Adversarial transferability inflates reported accuracy. MI-generated negatives reach false-positive rates of 89–97% across evaluation models.
  • Current SOTA results are overstated. Attacks reporting 90–100% under FCurr do not exceed 80% true success, and many fall below 60% under FMLLM.
  • Gemini-2.0 is a reliable evaluator. It achieves 93.84% positive-pair agreement, 95.59% negative-pair agreement, and no refusals.

Method

FMLLM: Our Evaluation Framework

We replace the standard evaluation model with a Multimodal LLM that uses general-purpose visual reasoning, avoiding the shared-task vulnerability that enables adversarial transferability.

F_MLLM evaluation pipeline

F_MLLM evaluation pipeline

See our Code & prompt construction for the full prompt construction and evaluation code used to generate and run these queries.

System prompt (used for evaluation queries)
You are an expert in face recognition. Taking into account the face aging, lighting, different hair styles, wearing and not wearing of eye glasses or other accessory, do the task in the image. Only answer yes or no

Citation

@inproceedings{ho2026revisiting,
  title     = {Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment},
  author    = {Ho, Sy-Tuyen and Hao, Koh Jun and Nguyen, Ngoc-Bao and Binder, Alexander and Cheung, Ngai-Man},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages     = {8070--8079},
  year      = {2026}
}