[Day 1 · Day 2 · References]
A two-day online workshop on compositionality and artificial intelligence organized by Gary Marcus and Raphaël Millière.
Day 1: Why Compositionality Matters for AI
Gary Marcus (New York University, Emeritus)
“Compositionality and Natural Language Understanding” [Slides]
Allyson Ettinger (University of Chicago)
“Shades of Meaning Composition: Defining Compositionality Goals in NLU” [Slides]
Paul Smolensky (Johns Hopkins University/Microsoft Research Redmond)
“Human-Level Intelligence Requires Continuous, Robustly Compositional Representations: Neurocompositional Computing for NECST-Generation AI” [Slides]
Raphaël Millière (Columbia University)
“Compositionality Without Classical Constituency” [Slides]
Day 2: Can Language Models Handle Compositionality?
Dieuwke Hupkes (European Laboratory for Learning and Intelligent Systems / Meta AI)
“Are Neural Networks Compositional, and How Do We Even Know?” [Slides]
Tal Linzen (New York University / Google AI)
“Successes and Failures of Compositionality in Neural Networks for Language” [Slides]
Stephanie Chan (DeepMind)
“Data Distributions Drive Emergent In-Context Learning in Transformers” [Slides]
Ellie Pavlick (Brown University / Google AI)
“No One Metric is Enough! Combining Evaluation Techniques to Uncover Latent Structure” [Slides]
Brenden Lake (New York University / Meta AI)
“Human-Like Compositional Generalization Through Meta-Learning” [Slides]
References
We have listed some relevant papers discussed by each speaker below.
Gary Marcus
- Marcus, G. (2022). Horse rides astronaut. Blog post
- Marcus, G. (2022). The New Science of Alt Intelligence. Blog post
- Marcus, G. (2020). The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (arXiv:2002.06177). PDF
- Marcus, G. (2001). The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press. Chapter 5.
Allyson Ettinger
- Pandia, L., Ettinger, A. (2021). Sorting through the noise: Testing robustness of information processing in pre-trained language models. Proceedings of The 2021 Conference on Empirical Methods in Natural Language Processing. PDF
- Yu, L., Ettinger, A. (2020). Assessing Phrasal Representation and Composition in Transformers. Proceedings of The 2020 Conference on Empirical Methods in Natural Language Processing. PDF
- Ettinger, A., Elgohary, A., Phillips, C., Resnik, P. (2018). Assessing Composition in Sentence Vector Representations. Proceedings of the 27th International Conference on Computational Linguistics. PDF
Paul Smolensky
- Paul Smolensky, R. Thomas McCoy, Roland Fernandez, Matthew Goldrick, Jianfeng Gao. In press. Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems. AI Magazine. PDF
- Paul Smolensky, R. Thomas McCoy, Roland Fernandez, Matthew Goldrick, Jianfeng Gao. 2022. Neurocompositional computing in human and machine intelligence: A tutorial. Microsoft Technical Report MSR-TR-2022-5. PDF
- R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky. RNNs Implicitly Implement Tensor Product Representations. PDF
- Paul Soulos, Tom McCoy, Tal Linzen, Paul Smolensky. Discovering the Compositional Structure of Vector Representations with Role Learning Networks. PDF
- Paul Smolensky. On the proper treatment of connectionism. 1988. The Behavioral & Brain Sciences 11:1, 1–74. PDF
- March 1988 debate at MIT: Fodor & Pylyshyn vs. Paul Smolensky
Raphaël Millière
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., … Wu, Z. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (arXiv:2206.04615). PDF
- Ontanon, S., Ainslie, J., Fisher, Z., & Cvicek, V. (2022). Making Transformers Solve Compositional Tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3591–3607. PDF
- Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., … Olah, C. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread. Online
- Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., … Olah, C. (2022). In-context learning and induction heads. Transformer Circuits Thread. Online
Dieuwke Hupkes
- Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2020). Compositionality Decomposed: How do Neural Networks Generalise? Journal of Artificial Intelligence Research, 67, 757–795. PDF
- Dankers, V., Bruni, E., & Hupkes, D. (2022). The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 4154–4175. PDF
Tal Linzen
- Najoung Kim & Tal Linzen (2020). COGS: A compositional generalization challenge based on semantic interpretation. EMNLP. PDF
- Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni (2018). Colorless green recurrent networks dream hierarchically. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1195–1205. PDF
- R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao & Asli Celikyilmaz. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. PDF
- Linlu Qiu, Peter Shaw, Panupong Pasupat, Paweł Krzysztof Nowak, Tal Linzen, Fei Sha, Kristina Toutanova. Improving Compositional Generalization with Latent Structure and Data Augmentation. NAACL. PDF
- R. Thomas McCoy, Ellie Pavlick & Tal Linzen (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. ACL. PDF
Stephanie Chan
- Santoro, Adam, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. “Meta-Learning with Memory-Augmented Neural Networks,” 2016, 9. PDF
- Vinyals, Oriol, Charles Blundell, and Timothy Lillicrap. “Matching Networks for One Shot Learning,” 2016, 9. PDF
- Wang, Jane X., Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. “Learning to Reinforcement Learn.” ArXiv:1611.05763 [Cs, Stat], November 17, 2016. PDF
- Alayrac, Jean-Baptiste, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, et al. “🦩 Flamingo: A Visual Language Model for Few-Shot Learning,” n.d., 66. PDF
- Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. “Language Models Are Few-Shot Learners.” ArXiv:2005.14165 [Cs], July 22, 2020. PDF
- Akyürek, Ekin, and Jacob Andreas. Compositionality as Lexical Symmetry. arXiv, January 30, 2022. PDF
- Andreas, Jacob. Good-Enough Compositional Data Augmentation. ArXiv:1904.09545 [Cs], May 19, 2020. PDF
- Hill, Felix, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro. “Environmental Drivers of Systematicity and Generalisation in a Situated Agent,” 2020, 15. PDF
- Chan, Stephanie C. Y., Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. “Data Distributional Properties Drive Emergent In-Context Learning in Transformers.” arXiv, May 30, 2022. PDF
Ellie Pavlick
- Pavlick, E. (2022). Semantic Structure in Deep Learning. Annual Review of Linguistics, 8(1), 447–471. PDF
- Unit Testing for Concepts in Neural Networks. TACL 2022 (to appear).
Brenden Lake
- Lake, B. M. and Murphy, G. L. (2021). Word meaning in minds and machines. Psychological Review. PDF
- Lake, B. M. (2019). Compositional generalization through meta sequence-to-sequence learning. Advances in Neural Information Processing Systems 32. PDF
- Lake, B. M., Linzen, T., and Baroni, M. (2019). Human few-shot learning of compositional instructions. In Proceedings of the 41st Annual Conference of the Cognitive Science Society. PDF