LUNAR Lab

Jun Tao, Ellie Pavlick, Amaris Grondin, Josue D. Bustamante, Harrison Martin, Hannah Parent, Natalie Fenn, Alexi Almonte, Amanda Maguire-Wilkerson, Mofan Gu, Jack Rusley, Bryce K. Perler, Tyler Wray, Amy S. Nunn, Philip A. Chan

Journal of Medical Internet Research

LLMs Model How Humans Induce Logically Structured Rules

Alyssa Loo, Ellie Pavlick, Roman Feiman

Journal of Memory and Language

2025

Video Finetuning Improves Reasoning Between Frames

Ruiqi Yang, Tian Yun, Zihan Wang, Ellie Pavlick

NeurIPS CogInterp Workshop 2025

Born a Transformer – Always a Transformer? On the Effect of Pretraining on Architectural Abilities

Mayank Jobanputra, Yana Veitsman, Yash Sarrof, Aleksandra Bakalova, Vera Demberg, Ellie Pavlick, Michael Hahn

NeurIPS 2025

Transferring Linear Features Across Language Models With Model Stitching

Alan Chen, Jack Merullo, Alessandro Stolfo, Ellie Pavlick

NeurIPS 2025

Can LLMs Subtract Numbers?

Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg

EMNLP MathNLP Workshop 2025

A Knapsack by Any Other Name: Presentation Impacts LLM Performance on NP-hard Problems

Alex Duchnowski, Ellie Pavlick, Alexander Koller

Findings of EMNLP 2025

Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

Meng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie Pavlick

EMNLP 2025

From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?

Thomas Serre, Ellie Pavlick

Neuron

$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources

Apoorv Khandelwal, Tian Yun, Nihal V. Nayak, Jack Merullo, Stephen H. Bach, Chen Sun, Ellie Pavlick

COLM 2025

Parallel Trade-offs in Human Cognition and Neural Networks: The Dynamic Interplay Between In-Context and In-Weight Learning

Jacob Russin, Ellie Pavlick, Michael J. Frank

PNAS

Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations

Alexa R. Tartaglini, Sheridan Feucht, Michael A. Lepori, Wai Keen Vong, Charles Lovering, Brenden M. Lake, Ellie Pavlick

CCN 2025

LLMs as Models for Analogical Reasoning

Sam Musker, Alex Duchnowski, Raphaël Millière, Ellie Pavlick

Journal of Memory and Language

What is an “Abstract Reasoner”? Revisiting Experiments and Arguments about Large Language Models

Tian Yun, Chen Sun, Ellie Pavlick

CoNLL 2025

Cognitively Inspired Interpretability in Large Neural Networks

Anna Leshinskaya, Taylor Webb, Ellie Pavlick, Jiahai Feng, Gustaw Opielka, Claire Stevenson, Idan A. Blank

CogSci 2025

Learning Imposes a Bottleneck Beyond Anatomical Constraints: A Computational Investigation into the Nature of WM Capacity Limits

Aalok Sathe, Ellie Pavlick, Michael J. Frank

CogSci 2025

Step-by-Step Analogical Reasoning in Humans and Neural Networks

Jacob Russin, Joonhwa Kim, Ellie Pavlick, Michael J. Frank

CogSci 2025

How Do Vision-Language Models Process Conflicting Information Across Modalities?

Tianze Hua, Tian Yun, Ellie Pavlick

Preprint

Does Training on Synthetic Data Make Models Less Robust?

Lingze Zhang, Ellie Pavlick

NAACL Insights from Negative Results in NLP Workshop 2025

Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting

Suraj Anand, Michael A. Lepori, Jack Merullo, Ellie Pavlick

ICLR 2025

The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling

Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, Ellie Pavlick

ICLR 2025

The Promise and Pitfalls of Generative AI

Monojit Choudhury, Zohar Elyoseph, Nathanael J. Fast, Desmond C. Ong, Elaine O. Nsoesie, Ellie Pavlick

Nature Reviews Psychology

2024

Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

Michael A. Lepori, Alexa R. Tartaglini, Wai Keen Vong, Thomas Serre, Brenden M. Lake, Ellie Pavlick

NeurIPS 2024

Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

NeurIPS 2024

Adaptable Artificial Intelligence

Azish Filabi, Nick Masi, Ellie Pavlick, Rico Picone

Journal on AI Policy and Complex Systems

Re-Evaluating Evaluation for Multilingual Summarization

Jessica Zosa Forde, Ruochen Zhang, Lintang Sutawika, Alham Fikri Aji, Samuel Cahyawijaya, Genta Indra Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick

EMNLP 2024

Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers

Enyan Zhang, Zewei Wang, Michael A. Lepori, Ellie Pavlick, Helena Aparicio

Preprint

Uncovering Intermediate Variables in Transformers using Circuit Probing

Michael A. Lepori, Thomas Serre, Ellie Pavlick

COLM 2024

Is Human Compositionality Meta-Learned?

Jacob Russin, Sam Whitman McGrath, Ellie Pavlick, Michael J. Frank

Behavioral and Brain Sciences

How Can Deep Neural Networks Inform Theory in Psychological Science?

Sam Whitman McGrath, Jacob Russin, Ellie Pavlick, Roman Feiman

Current Directions in Psychological Science

Human Curriculum Effects Emerge with In-Context Learning in Neural Networks

Jacob Russin, Ellie Pavlick, Michael J. Frank

CogSci 2024

Testing Causal Models of Word Meaning in LLMs

Sam Musker, Ellie Pavlick

CogSci 2024

Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Tasks

Aaron Traylor, Jack Merullo, Michael J. Frank, Ellie Pavlick

CogSci 2024

Language Models Implement Simple Word2Vec-style Vector Arithmetic

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

NAACL 2024

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?

Tianze Hua, Tian Yun, Ellie Pavlick

Findings of NAACL 2024

Lessons from the Trenches on Reproducible Evaluation of Language Models

Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Baber Abbasi, Jessica Zosa Forde, Leo Gao, Jonathan Tow, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, Andy Zou

Preprint

Circuit Component Reuse Across Tasks in Transformer Language Models

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

ICLR 2024

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Martha Lewis, Nihal V. Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H. Bach, Ellie Pavlick

Findings of EACL 2024

Bayesian Preference Elicitation with Language Models

Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, Belinda Z. Li

Preprint

Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction

Senjuti Dutta, Sherol Chen, Sunny Mak, Amnah Ahmad, Katherine Collins, Alena Butryna, Deepak Ramachandran, Krishnamurthy Dvijotham, Ellie Pavlick, Ravi Rajakumar

Preprint

2023

Break It Down: Evidence for Structural Compositionality in Neural Networks

Michael A. Lepori, Thomas Serre, Ellie Pavlick

NeurIPS 2023

NeuroSurgeon: A Toolkit for Subnetwork Analysis

Michael A. Lepori, Ellie Pavlick, Thomas Serre

EMNLP BlackboxNLP Workshop 2023

Analyzing Modular Approaches for Visual Question Decomposition

Apoorv Khandelwal, Ellie Pavlick, Chen Sun

EMNLP 2023

Are Language Models Worse than Humans at Following Prompts? It's Complicated

Albert Webson, Alyssa Loo, Qinan Yu, Ellie Pavlick

Findings of EMNLP 2023

Characterizing Mechanisms for Factual Recall in Language Models

Qinan Yu, Jack Merullo, Ellie Pavlick

EMNLP 2023

Emergence of Abstract State Representations in Embodied Sequence Modeling

Tian Yun, Zilai Zeng, Kunal Handa, Ashish V. Thapliyal, Bo Pang, Ellie Pavlick, Chen Sun

EMNLP 2023

Instilling Inductive Biases with Subnetworks

Enyan Zhang, Michael A. Lepori, Ellie Pavlick

Preprint

Properties of LoTs: The Footprints or the Bear Itself?

Sam Whitman McGrath, Jacob Russin, Ellie Pavlick, Roman Feiman

Behavioral and Brain Sciences

Symbols and Grounding in Large Language Models

Ellie Pavlick

Philosophical Transactions of the Royal Society A

Training Priors Predict Text-To-Image Model Performance

Charles Lovering, Ellie Pavlick

Preprint

Can Neural Networks Learn Implicit Logic from Physical Reasoning?

Aaron Traylor, Roman Feiman, Ellie Pavlick

ICLR 2023

Linearly Mapping from Image to Text Space

Jack Merullo, Louis Castricato, Carsten Eickhoff, Ellie Pavlick

ICLR 2023

Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?

Tian Yun, Usha Bhalla, Ellie Pavlick, Chen Sun

TMLR

Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs

Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, Tolga Bolukbasi

Preprint

Comparing Trajectory and Vision Modalities for Verb Representation

Dylan Ebert, Chen Sun, Ellie Pavlick

Preprint

2022

Evaluation beyond Task Performance: Analyzing Concepts in AlphaZero in Hex

Charles Lovering, Jessica Zosa Forde, George Konidaris, Ellie Pavlick, Michael L. Littman

NeurIPS 2022

Unit Testing for Concepts in Neural Networks

Charles Lovering, Ellie Pavlick

TACL

Pretraining on Interactions for Learning Grounded Affordance Representations

Jack Merullo, Dylan Ebert, Carsten Eickhoff, Ellie Pavlick

*SEM 2022

Do Prompt-Based Models Really Understand the Meaning of Their Prompts?

Albert Webson, Ellie Pavlick

NAACL 2022

Do Trajectories Encode Verb Meaning?

Dylan Ebert, Chen Sun, Ellie Pavlick

NAACL 2022

Hierarchical Planning with State Abstractions for Temporal Task Specifications

Yoonseon Oh, Roma Patel, Thao Nguyen, Baichuan Huang, Matthew Berg, Ellie Pavlick, Stefanie Tellex

Autonomous Robots

Mapping Language Models to Grounded Conceptual Spaces

Roma Patel, Ellie Pavlick

ICLR 2022

The MultiBERTs: BERT Reproductions for Robustness Analysis

Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ellie Pavlick

ICLR 2022

Where, When & Which Concepts Does AlphaZero Learn? Lessons from the Game of Hex

Jessica Zosa Forde, Charles Lovering, George Konidaris, Ellie Pavlick, Michael L. Littman

AAAI Reinforcement Learning in Games Workshop 2022

Semantic Structure in Deep Learning

Ellie Pavlick

Annual Review of Linguistics

Affordance-based Robot Object Retrieval

Thao Nguyen, Nakul Gopalan, Roma Patel, Matt Corsaro, Ellie Pavlick, Stefanie Tellex

Autonomous Robots

2021

A Novel Corpus of Discourse Structure in Humans and Computers

Babak Hemmatian, Sheridan Feucht, Rachel Avram, Alexander Wey, Muskaan Garg, Kate Spitalnic, Carsten Eickhoff, Ellie Pavlick, Björn Sandstede, Steven Sloman

EMNLP CODI Workshop 2021

Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, Anders Søgaard

CoNLL 2021

Does Vision-and-Language Pretraining Improve Lexical Grounding?

Tian Yun, Chen Sun, Ellie Pavlick

Findings of EMNLP 2021

Frequency Effects on Syntactic Rule Learning in Transformers

Jason Wei, Dan Garrette, Tal Linzen, Ellie Pavlick

EMNLP 2021

Was It “Stated” or Was It “Claimed”?: How Linguistic Bias Affects Generative Language Models

Roma Patel, Ellie Pavlick

EMNLP 2021

Spatial Language Understanding for Object Search in Partially Observed City-scale Environments

Kaiyu Zheng, Deniz Bayazit, Rebecca Mathew, Ellie Pavlick, Stefanie Tellex

RO-MAN 2021

AND Does Not Mean OR: Using Formal Languages to Study Language Models' Representations

Aaron Traylor, Roman Feiman, Ellie Pavlick

ACL 2021

Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions

Nam Do, Ellie Pavlick

Findings of ACL 2021

Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, Deepak Ramachandran

ACL 2021

Can Computers Tell a Story? Discourse Structure in Computer-Generated Text and Humans

Alex Wey, Babak Hemmatian, Rachel Avram, Sheridan Feucht, Kate Spitalnic, Muskaan Garg, Carsten Eickhoff, Ellie Pavlick, Björn Sandstede, Steven Sloman

CogSci 2021

The Anatomy of Discourse: Linguistic Predictors of Narrative and Argument Quality

Sheridan Feucht, Babak Hemmatian, Rachel Avram, Alex Wey, Kate Spitalnic, Muskaan Garg, Carsten Eickhoff, Ellie Pavlick, Björn Sandstede, Steven Sloman

CogSci 2021

Predicting Inductive Biases of Pre-Trained Models

Charles Lovering, Rohan Jha, Tal Linzen, Ellie Pavlick

ICLR 2021

2020

A Visuospatial Dataset for Naturalistic Verb Learning

Dylan Ebert, Ellie Pavlick

*SEM 2020

What Happens To BERT Embeddings During Fine-tuning?

Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, Ian Tenney

EMNLP BlackboxNLP Workshop 2020

Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces

Albert Webson, Zhizhong Chen, Carsten Eickhoff, Ellie Pavlick

EMNLP 2020

Sochiatrist: Signals of Affect in Messaging Data

Talie Massachi, Grant Fong, Varun Mathur, Sachin R. Pendse, Gabriela Hoefer, Jessica J. Fu, Chong Wang, Nikita Ramoji, Nicole R. Nugent, Megan L. Ranney, Daniel P. Dickstein, Michael F. Armey, Ellie Pavlick, Jeff Huang

CSCW 2020