Adversarial ML Problems Are Getting Harder to Solve and to Evaluate
Javier Rando*, Jie Zhang*, Nicholas Carlini, Florian Tramèr

• Despite a decade of research, progress in securing ML models against adversarial threats remains slow, hampered by non-rigorous evaluations even in simple cases. The shift to studying LLMs introduces problems that are less defined, harder to solve, and tougher to evaluate. Without addressing these challenges, we caution that another decade of adversarial ML research may yield minimal meaningful progress.

Blind Baselines Beat Membership Inference Attacks for Foundation Models
Debeshee Das, Jie Zhang, Florian Tramèr. [code]

• Unfortunately, we find that evaluations of MI attacks for foundation models are flawed, because they sample members and non-members from different distributions. We find 8 flawed MI evaluation datasets, existing evaluations thus tell us nothing about membership leakage of a foundation model’s training data.

Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data
[IEEE SaTML 2025]
Jie Zhang, Debeshee Das, Gautam Kamath, Florian Tramèr. [blog]

• We argue that MIA is fundamentally flawed for proving training data use. We propose two solutions: using data extraction attacks or membership inference on specially crafted canary data for reliable training data proofs.

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
[NeurIPS 2024 Dataset $\&$ Benchmark Track]
Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr. [code]

• To measure the adversarial robustness of AI agents, we introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. AgentDojo is an extensible environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate the environment with 97 realistic tasks, 629 security test cases, and various attack and defense paradigms from the literature.

Evaluations of Machine Learning Privacy Defenses are Misleading
[ACM CCS 2024]
Michael Aerni*, Jie Zhang*, Florian Tramèr. [code] [blog]

• Empirical defenses for private machine learning forgo the provable guarantees of differential privacy in the hope of achieving high utility on real-world data. We find that evaluations of such methods can be highly misleading. In this work, we thus propose a new evaluation protocol that is reliable and efficient.