Publications

1. JMLR

   Large sample spectral analysis of graph-based multi-manifold clustering

   Nicolas Garcia Trillos, Pengfei He, and Chenghui Li

   Journal of Machine Learning Research (JMLR), 2023

   Abs PDF

   In this work we study statistical properties of graph-based algorithms for multi-manifold clustering (MMC). In MMC the goal is to retrieve the multi-manifold structure underlying a given Euclidean data set when this one is assumed to be obtained by sampling a distribution on a union of manifolds M = M1 ∪ · · · ∪ MN that may intersect with each other and that may have different dimensions. We investigate sufficient conditions that similarity graphs on data sets must satisfy in order for their corresponding graph Laplacians to capture the right geometric information to solve the MMC problem. Precisely, we provide high probability error bounds for the spectral approximation of a tensorized Laplacian on M with a suitable graph Laplacian built from the observations; the recovered tensorized Laplacian contains all geometric information of all the individual underlying manifolds. We provide an example of a family of similarity graphs, which we call annular proximity graphs with angle constraints, satisfying these sufficient conditions. We contrast our family of graphs with other constructions in the literature based on the alignment of tangent planes. Extensive numerical experiments expand the insights that our theory provides on the MMC problem.

2. Analyzing illegal psychostimulant trafficking networks using noisy and sparse data

   Margret V Bjarnadottir, Siddharth Chandra, Pengfei He, and 1 more author

   IISE Transactions, 2024
3. TMLR

   Stealthy Backdoor Attack via Confidence-driven Sampling

   Pengfei He, Han Xu, Yue Xing, and 6 more authors

   Transactions on Machine Learning Research(TMLR), 2024
4. NeruIPS workshop

   Towards the Effect of Examples on In-Context Learning: A Theoretical Case Study

   Pengfei He, Yingqian Cui, Han Xu, and 4 more authors

   M3L & SFLLM workshop NeruIPS 2024, 2024

1. CIKM

   PROPN: Personalized Probabilistic Strategic Parameter Optimization in Recommendations

   Pengfei He, Haochen Liu, Xiangyu Zhao, and 2 more authors

   In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM), 2022
2. ICML

   Probabilistic Categorical Adversarial Attack and Adversarial Training

   Han Xu, Pengfei He, Jie Ren, and 4 more authors

   In International Conference on Machine Learning (ICML), 2023

   Abs PDF

   The studies on adversarial attacks and defenses have greatly improved the robustness of Deep Neural Networks (DNNs). Most advanced approaches have been overwhelmingly designed for continuous data such as images. However, these achievements are still hard to be generalized to categorical data. To bridge this gap, we propose a novel framework, Probabilistic Categorical Adversarial Attack (or PCAA). It transfers the discrete optimization problem of finding categorical adversarial examples to a continuous problem that can be solved via gradient-based methods. We analyze the optimality (attack success rate) and time complexity of PCAA to demonstrate its significant advantage over current search-based attacks. More importantly, through extensive empirical studies, we demonstrate that the well-established defenses for continuous data, such as adversarial training and TRADES, can be easily accommodated to defend DNNs for categorical data

3. ICLR Spotlight

   Sharpness-Aware Data Poisoning Attack

   Pengfei He, Han Xu, Jie Ren, and 4 more authors

   In International Conference on Learning Representations (ICLR), 2024

   Spotlight Paper, 5%

   Abs PDF

   Recent research has highlighted the vulnerability of Deep Neural Networks (DNNs) against data poisoning attacks. These attacks aim to inject poisoning samples into the models’ training dataset such that the trained models have inference failures. While previous studies have executed different types of attacks, one major challenge that greatly limits their effectiveness is the uncertainty of the re-training process after the injection of poisoning samples, including the re-training initialization or algorithms. To address this challenge, we propose a novel attack method called “Sharpness-Aware Data Poisoning Attack (SAPA)”. In particular, it leverages the concept of DNNs’ loss landscape sharpness to optimize the poisoning effect on the worst re-trained model. It helps enhance the preservation of the poisoning effect, regardless of the specific retraining procedure employed. Extensive experiments demonstrate that SAPA offers a general and principled strategy that significantly enhances various types of poisoning attacks.

4. ACL

   The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

   Shenglai Zeng, Jiankun Zhang, Pengfei He, and 8 more authors

   In Findings of the Association for Computational Linguistics ACL 2024, Aug 2024

   Abs

   Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model generation with proprietary and private data, where data privacy is a pivotal concern. Whereas extensive research has demonstrated the privacy risks of large language models (LLMs), the RAG technique could potentially reshape the inherent behaviors of LLM generation, posing new privacy issues that are currently under-explored. To this end, we conduct extensive empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database. Despite the new risks brought by RAG on the retrieval data, we further discover that RAG can be used to mitigate the old risks, i.e., the leakage of the LLMs’ training data. In general, we reveal many new insights in this paper for privacy protection of retrieval-augmented LLMs, which could benefit both LLMs and RAG systems builders.

5. ACL

   Exploring Memorization in Fine-tuned Language Models

   Shenglai Zeng, Yaxin Li, Jie Ren, and 7 more authors

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Abs

   Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models’ (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.

6. EMNLP

   On the Generalization of Training-based ChatGPT Detection Methods

   Han Xu, Jie Ren, Pengfei He, and 5 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Dec 2024

   Abs

   ChatGPT is one of the most popular language models which achieve amazing performance on various natural language tasks. Consequently, there is also an urgent need to detect the texts generated ChatGPT from human written. One of the extensively studied methods trains classification models to distinguish both. However, existing studies also demonstrate that the trained models may suffer from distribution shifts (during test), i.e., they are ineffective to predict the generated texts from unseen language tasks or topics. In this work, we aim to have a comprehensive investigation on these methods’ generalization behaviors under distribution shift caused by a wide range of factors, including prompts, text lengths, topics, and language tasks. To achieve this goal, we first collect a new dataset with human and ChatGPT texts, and then we conduct extensive studies on the collected dataset. Our studies unveil insightful findings which provide guidance for developing future methodologies or data collection strategies for ChatGPT detection.

7. EMNLP

   Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis

   Pengfei He, Yuping Lin, Han Xu, and 4 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Dec 2024

   Abs

   Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM’s representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.

Preprints

1. DiffusionShield: A Watermark for Copyright Protection against Generative Diffusion Models

   Yingqian Cui, Jie Ren, Han Xu, and 4 more authors

   2023
2. FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image Diffusion Models

   Yingqian Cui, Jie Ren, Yuping Lin, and 6 more authors

   2023
3. Data Poisoning for In-context Learning

   Pengfei He, Han Xu, Yue Xing, and 3 more authors

   2024

   Abs PDF

   In the domain of large language models (LLMs), in-context learning (ICL) has been recognized for its innovative ability to adapt to new tasks, relying on examples rather than retraining or fine-tuning. This paper delves into the critical issue of ICL’s susceptibility to data poisoning attacks, an area not yet fully explored. We wonder whether ICL is vulnerable, with adversaries capable of manipulating example data to degrade model performance. To address this, we introduce ICLPoison, a specialized attacking framework conceived to exploit the learning mechanisms of ICL. Our approach uniquely employs discrete text perturbations to strategically influence the hidden states of LLMs during the ICL process. We outline three representative strategies to implement attacks under our framework, each rigorously evaluated across a variety of models and tasks. Our comprehensive tests, including trials on the sophisticated GPT-4 model, demonstrate that ICL’s performance is significantly compromised under our framework. These revelations indicate an urgent need for enhanced defense mechanisms to safeguard the integrity and reliability of LLMs in applications relying on in-context learning.

4. Copyright Protection in Generative AI: A Technical Perspective

   Jie Ren, Han Xu, Pengfei He, and 8 more authors

   2024
5. Superiority of Multi-Head Attention in In-Context Linear Regression

   Yingqian Cui, Jie Ren, Pengfei He, and 2 more authors

   2024
6. A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration

   Yingqian Cui, Pengfei He, Xianfeng Tang, and 4 more authors

   2024
7. Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning

   Pengfei He, Zitao Li, Yue Xing, and 3 more authors

   2024