🧐 About Me

Hi there! I am a 3-year PhD student in Computer Science at the ETH Zurich, under the supervision of Prof. Florian Tramèr, and a member of the Secure and Private AI (SPY) Lab.

Research Interests

🔥 News

📒 Blogs

Our lab has very nice 📚 Blogs about AI security and privacy, highly recommended for reading!

📝 Selected Publications

( * indicates equal contribution. Full list of publications)

preprint

Render Before Reading: Visual Rendering as a Prompt Injection Defense

Jie Zhang, Andrei Baroian, Jan N. van Rijn, Avital Shafran, Andrei Baroian, Jan N. van Rijn, Florian Tramèr

code

preprint

Laundering AI Authority with Adversarial Examples

Jie Zhang, Pura Peetathawatchai, Florian Tramèr, Avital Shafran

[ICML 2026 workshop , Agents in the Wild]

preprint

Black-box Optimization of LLM Outputs by Asking for Directions

Jie Zhang, Meng Ding, Yang Liu, Jue Hong, Florian Tramèr

code

[ICLR Trustworthy AI workshop 2026, Spotlight Talk]

AISec 2026

Calibrating One-Round Membership Inference with Neighbors

Francesco Rita, Jie Zhang, Florian Tramèr

code

[CCS, AISec workshop 2026]

NeurIPS 2026

Learning to Inject: Automated Prompt Injection via Reinforcement Learning

Xin Chen, Jie Zhang, Florian Tramèr

code

[NeurIPS 2026]

NeurIPS 2025

RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

Jie Zhang, Cezara Petrui, Kristina Nikolić, Florian Tramèr

code dataset

[NeurIPS 2025, Dataset $\&$ Benchmark Track]

ICML 2025

The Jailbreak Tax: How Useful are Your Jailbreak Outputs?

Kristina Nikolić, Luze Sun, Jie Zhang, Florian Tramèr

code blog

[ICML 2025, Spotlight]

SaTML 2025

Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

Jie Zhang, Debeshee Das, Gautam Kamath, Florian Tramèr

blog poster

[IEEE SaTML 2025]

CCS 2024

Evaluations of Machine Learning Privacy Defenses are Misleading

Michael Aerni*, Jie Zhang*, Florian Tramèr

code blog poster

[ACM CCS 2024]

ICLR 2025

Does Training with Synthetic Data Truly Protect Privacy?

Yunpeng Zhao, Jie Zhang

code

[ICLR 2025]

IEEE SP 2025, DLSP workshop

Blind Baselines Beat Membership Inference Attacks for Foundation Models

Debeshee Das, Jie Zhang, Florian Tramèr

code poster

[IEEE SP 2025, DLSP workshop]

NeurIPS 2024

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr

code poster

[NeurIPS 2024 Dataset $\&$ Benchmark Track]

ICLR 2024

Real-Fake: Effective Training Data Synthesis Through Distribution Matching

Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, Bo Zhao

code

[ICLR 2024]

📖 Education

🎤 Talks

🎖 Honors and Awards

• 2021.05 We won the first prize on CVPR21 Workshop (Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges, rank: 1 / 1558).
• 2022.10 China National Scholarship, Zhejiang University, 2022
• Outstanding Student Scholarship, First Prize, Hainan University, 2018, 2019, 2020.

💬 Services

• Journal Reviewer:
  • IEEE Transactions on Neural Networks and Learning Systems
  • Neural Networks
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
• Conference Reviewer: ICLR, AAAI, CVPR, ICML, ECCV, ICCV, NeurIPS.

💻 Internships

• 2021.11 - 2022.06, Sony AI, Research Intern, Tokyo.
• 2020.10 - 2021.10, Tencent, Youtu Lab, Research Intern, Shanghai.
• 2019.11 - 2020.4, Alibaba, AliExpress, Software Engineer, Hangzhou.

🎙 Miscellaneous

Travel

I enjoy the time traveling with my families and friends. I am always excited about visiting new places and knowing different cultures.

My cat

My wife and I have three cats together, they are very adorable and have brought a lot of fun to our lives!