Transfer Learning from Pre-trained BERT for Pronoun Resolution

The paper describes the submission of the team “We used bert!” to the shared task Gendered Pronoun Resolution (Pair pronouns to their correct entities). Our final submission model based on the fine-tuned BERT (Bidirectional Encoder Representations from Transformers) ranks 14th among 838 teams with a multi-class logarithmic loss of 0.208. In this work, contribution of transfer learning technique to pronoun resolution systems is investigated and the gender bias contained in classification models is evaluated.

Computer Science Linguistics

View graph Source

Paper

Full text

Transfer Learning from Pre-trained BERT for Pronoun Resolution

Semantic Scholar · Computer Science · 2019

Abstract

The paper describes the submission of the team “We used bert!” to the shared task Gendered Pronoun Resolution (Pair pronouns to their correct entities). Our final submission model based on the fine-tuned BERT (Bidirectional Encoder Representations from Transformers) ranks 14th among 838 teams with a multi-class logarithmic loss of 0.208. In this work, contribution of transfer learning technique to pronoun resolution systems is investigated and the gender bias contained in classification models is evaluated.

Source of record · Semantic Scholar

References (20)

01On Measuring Social Biases in Sentence EncodersChandler May, Alex Wang, Shikha Bordia et al.2019 · North American Chapter of the Association for Computational Linguistics · 732 citations In Library

02Deep Contextualized Word RepresentationsMatthew E. Peters, Mark Neumann, Mohit Iyyer et al.2018 · North American Chapter of the Association for Computational Linguistics · 12k citations In Library

03Simple and Effective Dimensionality Reduction for Word EmbeddingsVikas Raunak2017 · 27 citations

04End-to-end Neural Coreference ResolutionKenton Lee, Luheng He, M. Lewis et al.2017 · Conference on Empirical Methods in Natural Language Processing · 971 citations In Library

05Bidirectional Attention Flow for Machine ComprehensionMinjoon Seo, Aniruddha Kembhavi, Ali Farhadi et al.2016 · International Conference on Learning Representations · 2.1k citations In Library

06Google's Neural Machine Translation System: Bridging the Gap between Human and Machine TranslationYonghui Wu, M. Schuster, Z. Chen et al.2016 · arXiv.org · 7.3k citations In Library

07Semantics derived automatically from language corpora contain human-like biasesAylin Caliskan, J. Bryson, Arvind Narayanan2016 · Science · 3.2k citations In Library

08Improving Coreference Resolution by Learning Entity-Level Distributed RepresentationsKevin Clark, Christopher D. Manning2016 · Annual Meeting of the Association for Computational Linguistics · 349 citations In Library

09Learning Global Features for Coreference ResolutionSam Wiseman, Alexander M. Rush, Stuart M. Shieber2016 · North American Chapter of the Association for Computational Linguistics · 201 citations In Library

10Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter2015 · International Conference on Learning Representations · 6.2k citations In Library

11GloVe: Global Vectors for Word RepresentationJeffrey Pennington, R. Socher, Christopher D. Manning2014 · Conference on Empirical Methods in Natural Language Processing · 35k citations

12CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotesSameer Pradhan, Alessandro Moschitti, Nianwen Xue et al.2012 · EMNLP-CoNLL Shared Task · 871 citations

Scroll for more · 8 remaining

Similar papers

Fill the GAP: Exploiting BERT for Pronoun ResolutionKai-Chou Yang, Timothy Niven, Tzu-Hsuan Chou et al.2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 4 citations In Library

Resolving Gendered Ambiguous Pronouns with BERTKellie Webster, Marta Recasens, Vera Axelrod et al.2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 50 citations In Library

Gendered Pronoun Resolution using BERT and an extractive question answering formulationRakesh Chada2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 10 citations In Library

MSnet: A BERT-based Network for Gendered Pronoun ResolutionZili Wang2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 4 citations In Library

Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution ChallengeBo Liu2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 8 citations In Library

Resolving Gendered Ambiguous Pronouns with Gender-Fair Modeling Based on BERT Word EmbeddingsZhi Ling2023 · International Conference on Computing and Artificial Intelligence · 0 citations In Library

Repairing Pronouns in Translation with BERT-Based Post-EditingReid Pryzant2021 · arXiv (Cornell University) · 0 citations In Library

BERT Masked Language Modeling for Co-reference ResolutionFelipe Alfaro, M. Costa-jussà, José A. R. Fonollosa2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 11 citations In Library

Coreference Resolution for Ambiguous Pronoun with BERT and MLPRohan Nair, Vadla Niranjan Vara Prasad, A Sreenadh et al.2021 In Library

Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun ResolutionYinchuan Xu, Junlin Yang2019 · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · 22 citations In Library

Source of record

Semantic Scholar

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal