metaxa

Valentina Pyatkin

Postdoctoral Researcher
Allen Institute for AI
University of Washington


News

I am on the academic job market for faculty positions! Feel free to reach out if you have an opening in your department.

Bio

I am a postdoctoral researcher (and Young Investigator) at the Allen Institute for AI and the University of Washington, advised by Prof. Yejin Choi. I completed my PhD in Natural Language Processing at the NLP lab of Bar Ilan University, supervised by Prof. Ido Dagan and Prof. Reut Tsarfaty. I also was a visiting PhD student at UW NLP and had the pleasure of interning twice at the Allen Institute for AI. My work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award. I am also very honored to have received the AI2 Outstanding Intern of the Year Award. Previously I did a research internship at Google, obtained an MSc from the University of Edinburgh and a BA from the University of Zurich.


Research

My research focuses on Post-Training and the Adaptation of Language Models, in order to make them better semantic and pragmatic reasoners. In the past, I worked on question generation, natural language representations, and discourse. I am also interested in underspecified, ambiguous and implicit language and in teaching language models how to better deal with such phenomena. More specifically, my research is centered around:

  • Post-Training and LM Adaptation: Finding optimal recipes for LM post-training - from analyzing preference data to algorithms.
  • Natural Language Understanding: Deeper semantic reasoning across a broader discourse and better handling of ambiguity and underspecification for LLMs.
  • Critical Evaluation: Testing LMs' abilities to perform pragmatic inferences, extracting their implicit values, and evaluating Reward Models.

Awards


Publications

Below is a selection of my recent publications; for my full publication record, please see my Google Scholar page.

2024


Diverging Preferences: When do Annotators Disagree and do Models Know?

Michael J.Q. Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, Valentina Pyatkin
ArXiV

SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation

Jing-Jing Li, Valentina Pyatkin, Max Kleiman-Weiner, Liwei Jiang, Nouha Dziri, Anne G. E. Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, Sydney Levine
ArXiV

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

Lester James V. Miranda*, Yizhong Wang*, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hanna Hajishirzi, Pradeep Dasigi
ArXiV

Superlatives in Context: Modeling the Implicit Semantics of Superlatives

Valentina Pyatkin, Bonnie Webber, Ido Dagan, Reut Tsarfaty
ArXiV

Explicating the Implicit: Argument Detection Beyond Sentence Boundaries

Paul Roit, Aviv Slobodkin, Eran Hirsch, Arie Cattan, Ayal Klein, Valentina Pyatkin, Ido Dagan In ACL 2024
ArXiV

Self-Directed Synthetic Dialogues and Revisions Technical Report

Nathan Lambert, Hailey Schoelkopf, Aaron Gokaslan, Luca Soldaini, Valentina Pyatkin, Louis Castricato
ArXiV

The Art of Saying No: Contextual Noncompliance in Language Models

Faeze Brahman*, Sachin Kumar*, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
ArXiV

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, Hannaneh Hajishirzi In NeurIPS 2024
ArXiV

WILDBENCH: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi
ArXiV

RewardBench: Evaluating Reward Models for Language Modeling

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
ArXiV

Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models

Paul Röttger*, Valentin Hofmann*, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, Dirk Hovy
In ACL 2024
Outstanding Paper Award
ArXiV

OLMo: Accelerating the Science of Language Models

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A Smith, Hannaneh Hajishirzi In ACL 2024
Best Theme Paper Award
ArXiV

Promptly Predicting Structures: The Return of Inference

Maitrey Mehta, Valentina Pyatkin, Vivek Srikumar In NAACL 2024
ArXiV

2023


Camels in a Changing Climate: Enhancing LM Adaptation with TÜLU 2

Hamish Ivison*, Yizhong Wang*, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi
ArXiV

” You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation

Allyson Ettinger, Jena D Hwang, Valentina Pyatkin, Chandra Bhagavatula, Yejin Choi In EMNLP Findings
PDF

What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations

Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, Yejin Choi In EMNLP Findings
PDF

Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement

Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, Xiang Ren In ICLR
PDF

Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties

Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, Yejin Choi In AAAI
PDF

PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning

Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, Yejin Choi In ICLR
PDF

Retrieving Texts based on Abstract Descriptions

Shauli Ravfogel, Valentina Pyatkin, Amir DN Cohen, Avshalom Manevich, Yoav Goldberg In COLM
ArXiV

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

Valentina Pyatkin, Frances Yung, Merel C.J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg In TACL
PDF

ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations

Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi and Chandra Bhagavatula In ACL
PDF

Revisiting Sentence Union Generation as a Testbed for Text Consolidation

Eran Hirsch, Valentina Pyatkin, Ruben Wolhandler, Avi Caciularu, Asi Shefer, Ido Dagan In ACL Findings
PDF

2022


Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE

Gu, Yuling, Yao Fu, Valentina Pyatkin, Ian H. Magnusson, Bhavana Dalvi and Peter Clark In Proceedings of the Workshop on Figurative Language Processing at EMNLP 2022
PDF

QASem Parsing: Text-to-text Modeling of QA-based Semantics

Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, Ido Dagan In EMNLP 2022
PDF

Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training

Merel C.J. Schoman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, Vera Demberg In LREC 2022
PDF

Draw Me a Flower: Grounding Formal Abstract Structures Stated in Informal Natural Language

Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, Reut Tsarfaty In TACL
PDF

2021


Asking It All: Generating Contextualized Questions for any Semantic Role

Valentina Pyatkin*, Paul Roit*, Julian Michael, Reut Tsarfaty, Yoav Goldberg, Ido Dagan In EMNLP 2021
PDF

The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processing

Valentina Pyatkin*, Shoval Sadde*, Aynat Rubinstein, Paul Portner, Reut Tsarfaty In ACL 2021
PDF

2020


QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines

Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, Ido Dagan In EMNLP 2020
PDF

QA-Nom: Question-Answer driven SRL for Nominalizations

Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan In COLING 2020
PDF

2017


Discourse Relations and Conjoined VPs: Automated Sense Recognition

Valentina Pyatkin, Bonnie Webber In EACL SRW 2017
PDF

* : Equal contribution.


Misc

Besides this I love rowing (currently at Lake Washington Rowing Club) and going to the “cinemathèque”. My Erdős number is 3 (Paul Erdős → Noga Alon → Ido Dagan → Me) and my Kevin Knight number is 2 (Kevin Knight → Yejin Choi → Me).