What can and can't language models do? Lessons learned from BIGBench

So what exactly can and can’t language models do? What's the least impressive thing GPT-4 won't be able to do? What will GPT-4 be incapable of? BIGBench is kind of a way to figure this out. BigBench, aka “The Beyond the Imitation Game” Benchmark, is an attempt to explore the capabilities of large language models over a wide variety of tasks. All the tasks are enumerated here. I looked through every BIGBench task and took the ones that compared both GPT3 and PaLM against humans. * Spreadsheet

When training AI, we should escalate the frequency of capability

Language Models Perform Reasoning via Chain of Thought – Google

Generative AI AI Perspectives

Large language models encode clinical knowledge

Dual Process Theory for Large Language Models: An overview of

What can and can't language models do? Lessons learned from BIGBench

Inverse scaling can become U-shaped — AI Alignment Forum

The Flan Collection: Advancing open source methods for instruction

Using cognitive psychology to understand GPT-3

What can and can't language models do? Lessons learned from BIGBench

Emergent Abilities in AI: Are We Chasing a Myth?

Google's new 540 billion parameter language model — LessWrong

Ethan Dyer - “Lessons from scale for large language models and

PaLM 2 And 19 Other AI Tools For Large Language Models