Valentina Pyatkin
Postdoctoral Researcher
Allen Institute for AI
University of Washington
News
- Dec. 2024: Co-organizing the SoLaR workshop on Socially Responsible Language Modelling Research at NeurIPS 2024.
- I'm attending COLM and will also be a mentor at the MLR at Penn workshop!
- 2 paper awards at ACL 2024!
- Jun. 2024: Serving as Internal Communication Chair for ACL 2024
- Apr. 2024: Invited talk at the UMass NLP Seminar.
- Mar. 2024: Invited talk at the University of Edinburgh.
- Mar. 2024: Co-organized the UnImplicit workshop at EACL-2024.
- Mar. 2024: Invited talk at the Harvard Efficient ML Seminar.
- Mar. 2024: DAAD sponsored visit to the University of Saarbrücken, the Max Planck Institute for Software Systems and the University of Stuttgart.
- Feb. 2024: Invited talk at the UBC NLP group.
- Dec. 2023: Invited talk at Brown/Tübingen.
- Sep. 2023: Gave an invited talk at the KR 2023 workshop on Computational Machine Ethics.
- Jan. 2023: Invited talk at the UT Austin Seminar on "Social Implications and Impact of NLP".
- Jul. 2022: Co-organized the UnImplicit workshop at NAACL-2022.
Bio
I am a postdoctoral researcher (and Young Investigator) at the Allen Institute for AI and the University of Washington, advised by Prof. Yejin Choi. I completed my PhD in Natural Language Processing at the NLP lab of Bar Ilan University, supervised by Prof. Ido Dagan and Prof. Reut Tsarfaty. I also was a visiting PhD student at UW NLP and had the pleasure of interning twice at the Allen Institute for AI. My work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award. I am also very honored to have received the AI2 Outstanding Intern of the Year Award. Previously I did a research internship at Google, obtained an MSc from the University of Edinburgh and a BA from the University of Zurich.
Research
My research focuses on Post-Training and the Adaptation of Language Models, in order to make them better semantic and pragmatic reasoners. In the past, I worked on question generation, natural language representations, and discourse. I am also interested in underspecified, ambiguous and implicit language and in teaching language models how to better deal with such phenomena. More specifically, my research is centered around:
- Post-Training and LM Adaptation: Finding optimal recipes for LM post-training - from analyzing preference data to algorithms.
- Natural Language Understanding: Deeper semantic reasoning across a broader discourse and better handling of ambiguity and underspecification for LLMs.
- Critical Evaluation: Testing LMs' abilities to perform pragmatic inferences, extracting their implicit values, and evaluating Reward Models.
Awards
- Aug. 2024: ACL Outstanding Paper Award!
- Aug. 2024: ACL Theme Paper Award for OLMo!
- Oct. 2023: Was selected as a DAAD AInet fellow
- Feb. 2023: Was awarded a postdoctoral scholarship from the Eric and Wendy Schmidt Foundation.
- Jan. 2023: Was awarded the AI2 Outstanding Intern of the Year Award
- Jan. 2021: Awarded the Nadav Award for Excellence in Research.
Publications
Below is a selection of my recent publications; for my full publication record, please see my Google Scholar page.
2024
Diverging Preferences: When do Annotators Disagree and do Models Know?
Michael J.Q. Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, Valentina Pyatkin
ArXiV
SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation
Jing-Jing Li, Valentina Pyatkin, Max Kleiman-Weiner, Liwei Jiang, Nouha Dziri, Anne G. E. Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, Sydney Levine
ArXiV
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
Lester James V. Miranda*, Yizhong Wang*, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hanna Hajishirzi, Pradeep Dasigi
ArXiV
Superlatives in Context: Modeling the Implicit Semantics of Superlatives
Valentina Pyatkin, Bonnie Webber, Ido Dagan, Reut Tsarfaty
ArXiV
Explicating the Implicit: Argument Detection Beyond Sentence Boundaries
Paul Roit, Aviv Slobodkin, Eran Hirsch, Arie Cattan, Ayal Klein, Valentina Pyatkin, Ido Dagan
In ACL 2024
ArXiV
Self-Directed Synthetic Dialogues and Revisions Technical Report
Nathan Lambert, Hailey Schoelkopf, Aaron Gokaslan, Luca Soldaini, Valentina Pyatkin, Louis Castricato
ArXiV
The Art of Saying No: Contextual Noncompliance in Language Models
Faeze Brahman*, Sachin Kumar*, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
ArXiV
Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, Hannaneh Hajishirzi
In NeurIPS 2024
ArXiV
WILDBENCH: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi
ArXiV
RewardBench: Evaluating Reward Models for Language Modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
ArXiV
Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models
Paul Röttger*, Valentin Hofmann*, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, Dirk Hovy
In ACL 2024
⭐Outstanding Paper Award⭐
ArXiV
OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A Smith, Hannaneh Hajishirzi
In ACL 2024
⭐Best Theme Paper Award⭐
ArXiV
Promptly Predicting Structures: The Return of Inference
Maitrey Mehta, Valentina Pyatkin, Vivek Srikumar
In NAACL 2024
ArXiV
2023
Camels in a Changing Climate: Enhancing LM Adaptation with TÜLU 2
Hamish Ivison*, Yizhong Wang*, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi
ArXiV
” You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation
Allyson Ettinger, Jena D Hwang, Valentina Pyatkin, Chandra Bhagavatula, Yejin Choi
In EMNLP Findings
PDF
What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations
Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, Yejin Choi
In EMNLP Findings
PDF
Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, Xiang Ren
In ICLR
PDF
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, Yejin Choi
In AAAI
PDF
PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning
Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, Yejin Choi
In ICLR
PDF
Retrieving Texts based on Abstract Descriptions
Shauli Ravfogel, Valentina Pyatkin, Amir DN Cohen, Avshalom Manevich, Yoav Goldberg
In COLM
ArXiV
Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design
Valentina Pyatkin, Frances Yung, Merel C.J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg
In TACL
PDF
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi and Chandra Bhagavatula
In ACL
PDF
Revisiting Sentence Union Generation as a Testbed for Text Consolidation
Eran Hirsch, Valentina Pyatkin, Ruben Wolhandler, Avi Caciularu, Asi Shefer, Ido Dagan
In ACL Findings
PDF
2022
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE
Gu, Yuling, Yao Fu, Valentina Pyatkin, Ian H. Magnusson, Bhavana Dalvi and Peter Clark
In Proceedings of the Workshop on Figurative Language Processing at EMNLP 2022
PDF
QASem Parsing: Text-to-text Modeling of QA-based Semantics
Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, Ido Dagan
In EMNLP 2022
PDF
Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training
Merel C.J. Schoman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, Vera Demberg
In LREC 2022
PDF
Draw Me a Flower: Grounding Formal Abstract Structures Stated in Informal Natural Language
Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, Reut Tsarfaty
In TACL
PDF
2021
Asking It All: Generating Contextualized Questions for any Semantic Role
Valentina Pyatkin*, Paul Roit*, Julian Michael, Reut Tsarfaty, Yoav Goldberg, Ido Dagan
In EMNLP 2021
PDF
The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processing
Valentina Pyatkin*, Shoval Sadde*, Aynat Rubinstein, Paul Portner, Reut Tsarfaty
In ACL 2021
PDF
2020
QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines
Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, Ido Dagan
In EMNLP 2020
PDF
QA-Nom: Question-Answer driven SRL for Nominalizations
Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan
In COLING 2020
PDF
2017
Discourse Relations and Conjoined VPs: Automated Sense Recognition
Valentina Pyatkin, Bonnie Webber
In EACL SRW 2017
PDF
* : Equal contribution.
Misc
Besides this I love rowing (currently at Lake Washington Rowing Club) and going to the “cinemathèque”. My Erdős number is 3 (Paul Erdős → Noga Alon → Ido Dagan → Me) and my Kevin Knight number is 2 (Kevin Knight → Yejin Choi → Me).