Tal Linzen
Tal has been a research scientist at Google Research since 2021. He is also an Associate Professor of Linguistics and Data Science at New York University. At Google, his research focuses on evaluating and analyzing language models, in particular using approaches from cognitive science and linguistics.
Research Areas
Authored Publications
Sort By
Preview abstract
Large language models are typically pretrained on a corpus of natural language text. In recent years, the desire to create language models which can interpret and generate code in different programming languages has led to the inclusion of non-linguistic code in the pretraining corpora for language models. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks: To study this, we pretrain suites of language models on parameterized `code mixture' datasets which interleave natural language and code in two different settings: competitive, in which the total volume of data seen during pretraining is held constant; and additive, in which the volume of language data is held constant. We study how the pretraining mixture affects (a) general reasoning on BigBench tasks, and (b) compositionality, measured by generalization accuracy on finetuned compositional benchmarks. We find that increased code mixtures cause higher performance on compositional and reasoning tasks involving structured formal outputs (like semantic parsing and arithmetic) and, conversely that code harms performance on purely-linguistic or world knowledge tasks.
View details
The MultiBERTs: BERT Reproductions for Robustness Analysis
Steve Yadlowsky
Jason Wei
Naomi Saphra
Iulia Raluca Turc
2022
Preview abstract
Experiments with pretrained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure (which includes the model architecture, training data, initialization scheme, and loss function). Recent work has shown that re-running pretraining can lead to substantially different conclusions about performance, suggesting that alternative evaluations are needed to make principled statements about procedures. To address this question, we introduce MultiBERTs: a set of 25 BERT-base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random initialization and data shuffling. The aim is to enable researchers to draw robust and statistically justified conclusions about pretraining procedures. The full release includes 25 fully trained checkpoints, as well as statistical guidelines and a code library implementing our recommended hypothesis testing methods. Finally, for five of these models we release a set of 28 intermediate checkpoints in order to support research on learning dynamics.
View details
Frequency Effects on Syntactic Rule Learning in Transformers
Jason Wei
Conference on Empirical Methods in Natural Language Processing (2021)
Preview abstract
Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. We investigate this question using the case study of BERT's performance on English subject-verb agreement. Unlike prior work, we train multiple instances of BERT from scratch, allowing us to perform a series of controlled interventions at pre-training time. We show that BERT often generalizes well to subject-verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. We also find, however, that performance is heavily influenced by word frequency, with experiments showing that both the absolute frequency of a verb form, as well as the frequency relative to the alternate inflection, are causally implicated in the predictions BERT makes at inference time. Closer analysis of these frequency effects reveals that BERT's behavior is consistent with a system that correctly applies the SVA rule in general but struggles to overcome strong training priors and to estimate agreement features (singular vs. plural) on infrequent lexical items.
View details
