CVPR 2024

Model Inversion Robustness:
Can Transfer Learning Help?

Sy-Tuyen Ho1 Koh Jun Hao1 Keshigeyan Chandrasegaran2 Ngoc-Bao Nguyen1 Ngai-Man Cheung1

1Singapore University of Technology and Design    2Stanford University

   Paper    Code
TL;DR. Transfer learning limits the layers that encode sensitive private data, reducing model-inversion performance without conflicting with the training objective. TL-DMI achieves strong MI robustness with a simple fine-tuning procedure.
TL-DMI overview: transfer learning defense limits layers encoding private data, degrading MI attack performance

Figure 1. TL-DMI leverages transfer learning to limit the number of layers encoding sensitive information from the private training dataset. By freezing pre-trained layers and fine-tuning only a small portion on private data, TL-DMI reduces the information exploitable by MI attacks — as justified by Fisher Information analysis — while preserving strong model utility.

Abstract

Model Inversion (MI) attacks aim to reconstruct private training data by abusing access to machine learning models. Contemporary MI attacks have achieved impressive attack performance, posing serious threats to privacy. Meanwhile, all existing MI defense methods rely on regularization that is in direct conflict with the training objective, resulting in noticeable degradation in model utility.

In this work, we take a different perspective, and propose a novel and simple Transfer Learning-based Defense against Model Inversion (TL-DMI) to render MI-robust models. By leveraging transfer learning, we limit the number of layers encoding sensitive information from private training data, thereby degrading the performance of MI attacks. We conduct an analysis using Fisher Information to justify our method. Our defense is remarkably simple to implement — without bells and whistles, TL-DMI achieves state-of-the-art MI robustness across extensive experiments.

TL-DMI vs. Existing MI Defenses

Unlike all existing MI defenses, TL-DMI introduces no conflict with the training objective and requires no regularization hyperparameter search — while achieving SOTA privacy-utility trade-off.

Defense Defense Type Conflicts w/ Training? Hyperparam Search? MI Robustness
No Defense — — — Low
MID Mutual Info Reg. Yes Yes Medium
BiDO Bilateral Reg. Yes Yes (extensive) Medium–High
LS Label Smoothing Partial Yes Medium
TL-DMI (Ours) Training Paradigm No No SOTA

TL-DMI is the first MI defense to operate at the training paradigm level — achieving SOTA robustness with zero conflict with the task objective.

Citation

@inproceedings{ho2024model,
  title     = {Model Inversion Robustness: Can Transfer Learning Help?},
  author    = {Ho, Sy-Tuyen and Hao, Koh Jun and Chandrasegaran, Keshigeyan and Nguyen, Ngoc-Bao and Cheung, Ngai-Man},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages     = {12183--12193},
  year      = {2024}
}