Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials

Nolano G, Blum M, Ell B, Cimiano P (2024)
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

Konferenzbeitrag | Englisch
 
Download
Es wurden keine Dateien hochgeladen. Nur Publikationsnachweis!
Abstract / Bemerkung
In recent years, large language models have achieved state-of-the-art performance across various NLP tasks. However, investigations have shown that these models tend to rely on shortcut features, leading to inaccurate predictions and causing the models to be unreliable at generalization to out-of-distribution (OOD) samples. For instance, in the context of relation extraction (RE), we would expect a model to identify the same relation independently of the entities involved in it. For example, consider the sentence "Leonardo da Vinci painted the Mona Lisa" expressing the created(Leonardo_da_Vinci, Mona_Lisa) relation. If we substiute "Leonardo da Vinci" with "Barack Obama", then the sentence still expresses the created relation. A robust model is supposed to detect the same relation in both cases. In this work, we describe several semantically-motivated strategies to generate adversarial examples by replacing entity mentions and investigate how state-of-the-art RE models perform under pressure. Our analyses show that the performance of these models significantly deteriorates on the modified datasets (avg. of -48.5% in F1), which indicates that these models rely to a great extent on shortcuts, such as surface forms (or patterns therein) of entities, without making full use of the information present in the sentences.
Erscheinungsjahr
2024
Serien- oder Zeitschriftentitel
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Konferenz
Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)
Konferenzort
Turin
Konferenzdatum
2024-05-20 – 2024-05-25
Page URI
https://pub.uni-bielefeld.de/record/2988689

Zitieren

Nolano G, Blum M, Ell B, Cimiano P. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024.
Nolano, G., Blum, M., Ell, B., & Cimiano, P. (2024). Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Nolano, Gennaro, Blum, Moritz, Ell, Basil, and Cimiano, Philipp. 2024. “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), .
Nolano, G., Blum, M., Ell, B., and Cimiano, P. (2024). Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).
Nolano, G., et al., 2024. Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).
G. Nolano, et al., “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024.
Nolano, G., Blum, M., Ell, B., Cimiano, P.: Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). (2024).
Nolano, Gennaro, Blum, Moritz, Ell, Basil, and Cimiano, Philipp. “Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials”. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (2024).

Link(s) zu Volltext(en)
Access Level
OA Open Access

Export

Markieren/ Markierung löschen
Markierte Publikationen

Open Data PUB

Suchen in

Google Scholar