📝 Selected Publications

Render Before Reading: Visual Rendering as a Prompt Injection Defense

Jie Zhang*, Andrei Baroian*, Jan N. van Rijn, Avital Shafran, Andrei Baroian, Jan N. van Rijn, Florian Tramèr</span>

code

Laundering AI Authority with Adversarial Examples

Jie Zhang, Pura Peetathawatchai, Florian Tramèr, Avital Shafran

[ICML 2026 workshop , Agents in the Wild]

Black-box Optimization of LLM Outputs by Asking for Directions

Jie Zhang, Meng Ding, Yang Liu, Jue Hong, Florian Tramèr

code

[ICLR Trustworthy AI workshop 2026, Spotlight Talk]

Calibrating One-Round Membership Inference with Neighbors

Francesco Rita, Jie Zhang, Florian Tramèr

code

[CCS, AISec workshop 2026]

Learning to Inject: Automated Prompt Injection via Reinforcement Learning

Xin Chen, Jie Zhang, Florian Tramèr

code

[NeurIPS 2026]

Position: Adversarial ML for LLMs Is Not Making Any Progress

Javier Rando*, Jie Zhang*, Nicholas Carlini, Florian Tramèr

[ICML 2026]

RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

Jie Zhang, Cezara Petrui, Kristina Nikolić, Florian Tramèr

[NeurIPS 2025, Dataset $\&$ Benchmark Track]

The Jailbreak Tax: How Useful are Your Jailbreak Outputs?

Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr

[ICML 2025, Spotlight]

Membership Inference Attacks on Sequence Models

Lorenzo Rossi, Michael Aerni, Jie Zhang, Florian Tramèr

[IEEE SP 2025, DLSP workshop, Best Paper Award]

Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

Jie Zhang, Debeshee Das, Gautam Kamath, Florian Tramèr

[IEEE SaTML 2025]

Evaluations of Machine Learning Privacy Defenses are Misleading

Michael Aerni*, Jie Zhang*, Florian Tramèr

[ACM CCS 2024]

Does Training with Synthetic Data Truly Protect Privacy?

Yunpeng Zhao, Jie Zhang

[ICLR 2025]

Blind Baselines Beat Membership Inference Attacks for Foundation Models

Debeshee Das, Jie Zhang, Florian Tramèr

[IEEE SP 2025, DLSP workshop]

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr

[NeurIPS 2024 Dataset $\&$ Benchmark Track]

Real-Fake: Effective Training Data Synthesis Through Distribution Matching

Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, Bo Zhao

[ICLR 2024]