Figure 1. (I) MI attack on a ResNet-like architecture exploits gradient backpropagation via skip connections. (II) Skip connections allow gradients to bypass residual modules, reinforcing MI attacks. (III) We propose MI-resilient architecture designs that reduce this vulnerability for both additive (ResNet) and concatenative (DenseNet) skip connections.
Abstract
Skip connections are fundamental architecture designs for modern deep neural networks (DNNs) such as CNNs and ViTs. While they help improve model performance significantly, we identify a vulnerability associated with skip connections to Model Inversion (MI) attacks, a type of privacy attack that aims to reconstruct private training data through abusive exploitation of a model.
In this paper, as a pioneer work to understand how DNN architectures affect MI, we study the impact of skip connections on MI. We discover that: (1) skip connections reinforce MI attacks and compromise data privacy; (2) skip connections in the last stage are the most critical; (3) RepVGG, an established approach to remove skip connections at inference time, cannot mitigate this vulnerability; and (4) based on our findings, we propose MI-resilient architecture designs for the first time. Without bells and whistles, our methods outperform SOTA defense methods in MI robustness and are complementary to existing defenses.
MI-Resilient Architecture Designs
Our findings directly motivate three simple, architecture-level defense strategies. All maintain the same training procedure as the original model, require no additional data, and are complementary to existing regularization-based defenses (e.g., BiDO).
RoLSS: Removal of Last Stage Skip-Connection
Remove skip connections only from the final stage of the target network (the most critical for MI). No additional hyperparameters. Works for both additive (ResNet) and concatenative (DenseNet) skip connections.
SSF: Skip-Connection Scaling Factor
Introduce a learnable scale factor 0 ≤ k ≤ 1 on the last-stage skip connection: zi+1 = gi(zi) + k·zi. At k=0 this recovers RoLSS, at k=1 it is the original network. We set k to 0.2 (additive) or 0.01 (concatenative) to achieve a good balance between accuracy and MI robustness.
TTS: Two-Stage Training Scheme
Stage 1: Train with full skip connections for M epochs to warm-start parameters. Stage 2: Remove last-stage skip connections (RoLSS) and continue for N epochs using Stage 1 weights as initialization (M=5, N=95).
Comparison Against SOTA MI Defenses
Results under PPA attack (Dpriv = FaceScrub, Dpub = FFHQ). Δ = ratio of attack accuracy drop to natural accuracy drop — higher is better. Our methods consistently outperform SOTA BiDO with no extensive hyperparameter grid search.
| Architecture | Defense | Acc ↑ | AttAcc ↓ | Δ ↑ |
|---|---|---|---|---|
| ResNet-34 | ||||
| ResNet-34 | No Defense | 94.69 | 90.78 | — |
| ResNet-34 | BiDO | 91.66 | 81.98 | 2.90 |
| ResNet-34 | RoLSS (Ours) | 91.38 | 71.86 | 5.72 |
| ResNet-34 | SSF (Ours) | 94.21 | 79.79 | 22.90 |
| ResNet-34 | TTS (Ours) | 94.40 | 81.65 | 31.48 |
| ResNet-101 | ||||
| ResNet-101 | No Defense | 94.86 | 83.00 | — |
| ResNet-101 | BiDO | 90.31 | 67.07 | 3.50 |
| ResNet-101 | RoLSS (Ours) | 92.40 | 58.68 | 9.89 |
| ResNet-101 | SSF (Ours) | 93.79 | 71.06 | 11.16 |
| DenseNet-121 | ||||
| DenseNet-121 | No Defense | 94.67 | 78.09 | — |
| DenseNet-121 | RoLSS (Ours) | 86.86 | 24.48 | 6.86 |
| DenseNet-121 | SSF (Ours) | 91.73 | 56.32 | 7.35 |
RoLSS+BiDO is complementary: BiDO+RoLSS on ResNet-101 achieves AttAcc 41.44% (vs. BiDO alone: 67.07%), with only a 1% additional accuracy drop.
Citation
@inproceedings{hao2024vulnerability,
title={On the Vulnerability of Skip Connections to Model Inversion Attacks},
author={Hao, Koh Jun and Ho, Sy-Tuyen and Nguyen, Ngoc-Bao and Cheung, Ngai-Man},
booktitle={European Conference on Computer Vision},
pages={140--157},
year={2024},
organization={Springer}
}