🧐 About Me

Hi there! I am a first-year PhD student in Computer Science at the ETH Zurich, under the supervision of Prof. Florian Tramèr, and a member of the Secure and Private AI (SPY) Lab. I completed my master’s degree in Software Engineering at Zhejiang University in March 2023, advised by Prof. Chao Wu. Before that, I received my Bachelor’s degree at Hainan University in July 2020.

Research Interests:

🤔 For my PhD study, my primary focus is on examining the potential security and privacy risks in ML systems, both in their current state and as they evolve in the future. My research aims to uncover vulnerabilities and develop strategies to mitigate these risks, ultimately contributing to the development of more secure and privacy-preserving machine learning technologies.

🔥 News

📝 Selected Publications

(* indicates equal contribution; # indicates corresponding authorship. See full list of publications here. )


Blind Baselines Beat Membership Inference Attacks for Foundation Models
Debeshee Das, Jie Zhang, Florian Tramèr.

  • Unfortunately, we find that evaluations of MI attacks for foundation models are flawed, because they sample members and non-members from different distributions. For 8 published MI evaluation datasets, we show that blind attacks—that distinguish the member and non-member distributions without looking at any trained model—outperform state-of-the-art MI attacks. Existing evaluations thus tell us nothing about membership leakage of a foundation model’s training data.

AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr. [code]

  • To measure the adversarial robustness of AI agents, we introduce AgentDojo, an evaluation framework for agents that execute tools over untrusted data. AgentDojo is not a static test suite, but rather an extensible environment for designing and evaluating new agent tasks, defenses, and adaptive attacks. We populate the environment with 97 realistic tasks, 629 security test cases, and various attack and defense paradigms from the literature. We find that AgentDojo poses a challenge for both attacks and defenses: state-of-the-art LLMs fail at many tasks (even in the absence of attacks), and existing prompt injection attacks break some security properties but not all.
CCS 2024

Evaluations of Machine Learning Privacy Defenses are Misleading
Michael Aerni*, Jie Zhang*, Florian Tramèr. [code] [blogpost]

  • Empirical defenses for private machine learning forgo the provable guarantees of differential privacy in the hope of achieving high utility on real-world data. We find that evaluations of such methods can be highly misleading. In this work, we thus propose a new evaluation protocol that is reliable and efficient.
ICLR 2024

Real-Fake: Effective Training Data Synthesis Through Distribution Matching
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, Bo Zhao#. (ICLR 2024) [code]

  • In this paper, through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmentation to real datasets. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent to 1 X the original real data size, which increases to 76.0% when scaling up to 10 X synthetic data.
CVPR 2023, highlight

Accelerating Dataset Distillation via Model Augmentation
Lei Zhang*, Jie Zhang*, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, Dongkuan Xu#. (CVPR 2023) [code]

  • In this paper, we assume that training the synthetic data with diverse models leads to better generalization performance. Thus we propose two model augmentation techniques, i.e., using early-stage models and weight perturbation to learn an informative synthetic set with significantly reduced training cost. Extensive experiments demonstrate that our method achieves up to 20× speedup and comparable performance on par with state-of-the-art baseline methods.
CVPR 2022

Towards Efficient Data-Free Black-box Adversarial Attack
Jie Zhang*, Bo Li*, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu#. (CVPR 2022) [code]

  • In this paper, by rethinking the collaborative relationship between the generator and the substitute model, we design a novel black-box attack framework. The proposed method can efficiently imitate the target model through a small number of queries and achieve high attack success rate.
ICML 2022

Federated Learning with Label Distribution Skew via Logits Calibration
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu#. (ICML 2022)

  • In this work, we investigate the label distribution skew from a statistical view. We demonstrate both theoretically and empirically that previous methods based on softmax crossentropy are not suitable, which can result in local models heavily overfitting to minority classes and missing classes. Then, we propose FedLC (Federated learning via Logits Calibration), which calibrates the logits before softmax cross-entropy according to the probability of occurrence of each class.
NeurIPS 2022

DENSE: Data-Free One-Shot Federated Learning
Jie Zhang*, Chen Chen*, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, Chao Wu#. (NeurIPS 2022) [code]

  • The paper focuses on one-shot federated learning, i.e., the server can learn a model with a single communication round. The proposed FedSyn method has two stages: first, training a generator from the ensemble of models from clients; second, distilling the knowledge of the ensemble into a global model with synthetic data. We validate the efficacy of FedSyn by conducting extensive experiments on 6 different datasets with various non-IID settings generated from Dirichlet distributions. Results can well support that the proposed method consistently outperforms all the baselines.

🎖 Honors and Awards

  • 2021.05 We won the first prize on CVPR21 Workshop (Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges, rank: 1 / 1558).
  • 2022.10 China National Scholarship, Zhejiang University, 2022
  • Outstanding Student Scholarship, First Prize, Hainan University, 2018, 2019, 2020.

📖 Educations

  • 🎓 2020.09 - 2023.03, Master, Zhejiang University, China.
  • 🎓 2016.09 - 2020.06, Undergraduate, Hainan University, China.

💬 Services

  • Journal Reviewer:
    • IEEE Transactions on Neural Networks and Learning Systems
    • Neural Networks
    • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Conference Reviewer: ICLR, AAAI, CVPR, ICML, ECCV, ICCV, NeurIPS.

💻 Internships

🎙 Miscellaneous


I enjoy the time traveling with my families and friends. I am always excited about visiting new places and knowing different cultures.

My cat

My girlfriend and I have three cats together, they are very adorable and have brought a lot of fun to our lives!

图片名称 图片名称 图片名称