Related work: document similarity, membership inference, backdoor attack, and watermark
최근 연구된 XML에 대해서 내용을 정리하고, SIP 논문에 적을 내용을 소개한다.
As the training of source identification is not scalable with CE loss, the alternative loss is binary cross entropy loss (BCE) which is widely used in one-versus-all training framework [1,2,3] and in the field of extreme classification (EC) or extreme multi-label classification (XML). Deep learning-based XML methods have three properties in common : feature learning, negative label shortlisting, and classifier training [3]. We follow the XML training framework to train the source identification problem with integer labels. We do not utilize the label features, for example, the book name is not used.
Although recent works utilizes the label features to handle the lack of training samples [5,6,7], source identification could not directly use label features. As the input embedding GPT hidden representations are not sentence embeddings and the similarity between the labels and inputs are not guaranteed. Therefore, SIP problem is EC problem without label features.
AttentionXML forms a tree for label features [6]. SiameseXML enhanced zero-shot Siamese architecture to few-shot classification problem [7]. Renee propose optimization loss to stablize the training and increase the size of training to the fart more extreme number of labels (1B) [2]. DEXA [4] use additional parameters to complement the gap of pure label feature embeddings.
[1] Rifkin, Ryan, and Aldebaro Klautau. “In defense of one-vs-all classification.” The Journal of Machine Learning Research 5 (2004): 101-141.
[2] Jain, Vidit, et al. “Renee: END-TO-END TRAINING OF EXTREME CLASSIFICATION MODELS.” Proceedings of Machine Learning and Systems 5 (2023). (Renee)
[3] Dahiya, Kunal, et al. “Deepxml: A deep extreme multi-label learning framework applied to short text documents.” Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021. (DeepXML)
[4] Dahiya, Kunal, et al. “Deep Encoders with Auxiliary Parameters for Extreme Classification.” Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023. DEXA
[5] Jiang, Ting, et al. “Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021. (LightXML)
[6] You, Ronghui, et al. “Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification.” Advances in Neural Information Processing Systems 32 (2019). (AttentionXML)
[7] Dahiya, Kunal, et al. “Siamesexml: Siamese networks meet extreme classifiers with 100m labels.” International Conference on Machine Learning. PMLR, 2021. (SiameseXML)
[8] @Misc{Bhatia16, author = {Bhatia, K. and Dahiya, K. and Jain, H. and Kar, P. and Mittal, A. and Prabhu, Y. and Varma, M.}, title = {The extreme classification repository: Multi-label datasets and code}, url = {http://manikvarma.org/downloads/XC/XMLRepository.html}, year = {2016} }