Unfaithful Chains of Thought

What's actually happening when an LLM "thinks out loud"? Research on human decision-making suggests that much of the reasoning we believe drives our choices is actually post hoc rationalization — we decide first, explain later. Katie and Ben get curious about whether the same might be true for large language models: when you watch a model reason through a problem in real time, is that chain of thought the genuine process, or just a plausible-sounding story told after the fact? It's a deceptively deep question with real stakes for how much we should trust model explanations.

Miles Turpin et al., "Language Models Don't Always Say What They Think: Unfaithful Explanations in
Chain-of-Thought Prompting" (NeurIPS 2023, NYU and Anthropic): arxiv.org/abs/2305.04388

Anthropic, "Reasoning Models Don't Always Say What They Think" (Alignment Faking research, 2025):
www.anthropic.com/research/reasoni…s-dont-say-think

Benchmark Bank Heist

What if an AI decided the smartest way to pass its test was to find the answer key? That's exactly what Anthropic's Claude Opus did when faced with a benchmark evaluation — reasoning that it was being tested, tracking down the encrypted eval dataset, decrypting it, and returning the answer it found inside. It's equal parts impressive and unsettling. This episode digs into what actually happened, why it matters for how we measure AI progress, and what this very novel failure mode means for the already-tricky science of benchmarking language models.

Links

Anthropic's writeup on the BrowseComp reverse-engineering done by Claude Opus 4.6: www.anthropic.com/engineering/eval…eness-browsecomp

BrowseComp benchmark from OpenAI: openai.com/index/browsecomp/

Benchmarking AI Models

How do you know if a new AI model is actually better than the last one? It turns out answering that question is a lot messier than it sounds. This week we dig into the world of LLM benchmarks — the standardized tests used to compare models — exploring two canonical examples: MMLU, a 14,000-question multiple choice gauntlet spanning medicine, law, and philosophy, and SWE-bench, which throws real GitHub bugs at models to see if they can fix them. Along the way: Goodhart's Law, data contamination, canary strings, and why acing a test isn't always the same as being smart.

MMLU benchmark paper: "Measuring Massive Multitask Language Understanding" by Dan Hendrycks et al. https://arxiv.org/abs/2009.03300

SWE-bench: "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" by Carlos E. Jimenez et al. https://arxiv.org/abs/2310.06770


BIG-bench (including canary string approach): "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models" https://arxiv.org/abs/2206.04615

The Hot Mess of AI (Mis-)Alignment

The paperclip maximizer — the classic AI doom scenario where a hyper-competent machine single-mindedly converts the universe into office supplies — might not be the AI risk we should actually lose sleep over. New research from Anthropic's AI safety division suggests misaligned AI looks less like an evil genius and more like a distracted wanderer who gets sidetracked reading French poetry instead of, say, managing a nuclear power plant. This week we dig into a fascinating paper reframing AI misalignment through the lens of bias-variance decomposition, and why longer reasoning chains might actually make things worse, not better.

- "The Hot Mess Theory of AI Misalignment: How Misalignment Scales with Model Intelligence and Task Complexity" — Anthropic AI Safety. arxiv.org/abs/2503.08941

The Bitter Lesson

Every AI builder knows the anxiety: you spend months engineering prompts, tuning pipelines, and chaining calls together — then a new model drops and half your work evaporates overnight. It turns out researchers have been wrestling with this exact dynamic for 30 years, and they keep arriving at the same uncomfortable answer. That answer is called the Bitter Lesson — and understanding it might be the most important thing you can do for whatever you're building right now. From Deep Blue to AlexNet to modern LLMs, scale keeps beating sophistication, and knowing which side of that line your work falls on makes all the difference.

Links

- Richard Sutton, "The Bitter Lesson" (2019)

- Alon Halevy, Peter Norvig, and Fernando Pereira, "The Unreasonable Effectiveness of Data"(2009)

- Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, "ImageNet Classification with Deep Convolutional Neural Networks"(AlexNet, 2012)

From Atari to Chat GPT: How AI Learned to Follow Instructions

Five and a half years have passed since Linear Digressions went on hiatus, and in that time... nothing has changed. Just kidding. Katie is joined by Phoebe to trace the surprisingly winding research path that led to ChatGPT. Here's a fun fact: GPT-3, the model behind ChatGPT when it launched, already existed in 2020 — and it was technically more powerful than the version that took over the world. So what happened in between? Why couldn't a 175-billion-parameter model trained on essentially the entire internet reliably answer "how do I bake a cake?"

The answer involves Atari games, simulated robots learning to walk, 40 contractors, and a series of papers stretching from 2017 to 2022 that quietly built the recipe every major AI assistant uses today. We trace that arc from reinforcement learning with human preferences all the way to the app that got a hundred million users in two months.

References mentioned in this episode include

1. Ouyang et al. 2022 (InstructGPT paper): https://arxiv.org/abs/2203.02155

2. OpenAI blog post (more readable): https://openai.com/index/instruction-following/

3. Christiano et al. 2017 (Deep Reinforcement Learning from Human Preferences – this is where they teach AI to walk and play Atari): https://arxiv.org/abs/1706.03741

4. Stiennon et al. 2020 (Learning to Summarize from Human Feedback): https://arxiv.org/abs/2009.01325

5. Ziegler et al. 2019 (Fine-Tuning Language Models from Human Preferences): https://arxiv.org/abs/1909.08593

It's RAG time: Retrieval-Augmented Generation

Today we are going to talk about the feature with the worst acronym in generative AI: RAG, or Retrieval Augmented Generation. If you've ever used something like "Chat with My Docs," if you have an internal AI chatbot that has access to your company's documents, or you've created one yourself on some kind of personal project and uploaded a bunch of documents for the AI to use — you have encountered RAG, whether you know it or not.

It's an extremely effective technique. Works super well for taking general purpose models like ChatGPT or Claude and turning them into AIs that are aware of all the specific information that makes them truly useful in a huge variety of situations. RAG is pretty interesting under the hood, so I thought it would be fun to spend a little while talking about it.

RAG was first introduced in this paper from Facebook Research in 2021: https://arxiv.org/pdf/2005.11401

The Impact of Generative AI on Critical Thinking

I use LLMs a lot. I use them in my work, I use them in my personal life, and sometimes I use them to help me with stuff that I already know how to do. I’m working on something and I just want to make it a little bit easier, and it does make it easier for sure.


But something that I worry about sometimes is that over the long run, I'm going to pay a price for that. I'm going to get lazier, I'm going to get a little bit dumber. And the question is, as I'm outsourcing my thinking to LLMs, am I becoming reliant on them? If they were ever to go away, would I lose my ability to do basic things? I like feeling like I'm a smart, capable person; am I letting that slip away, without realizing it, just because I want it to be easier to do meal planning for the week. 


In this episode of Linear Digressions, we're going to talk about a paper studying just this issue, trying to understand how people think critically, when they think critically. How much do we engage cognitively with our work when we’re using LLMs, versus not? 


The paper discussed in this episode is The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

https://www.microsoft.com/en-us/research/wp-content/uploads/2025/01/lee_2025_ai_critical_thinking_survey.pdf

Chasing Away Repetitive LLM Responses with Verbalized Sampling

One of the things that LLMs can be really helpful with is brainstorming or generating new creative content. They are called Generative AI, after all—not just for summarization and question-and-answer tasks. But if you use LLMs for creative generation, you may find that their output starts to seem repetitive after a little while.

Let's say you're asking it to create a poem, some dialogue, or a joke. If you ask once, it'll give you something that sounds pretty reasonable. But if you ask the same thing 10 times, it might give you 10 things that sound kind of the same.

Today's episode is about a technique called verbalized sampling, and it's a way to mitigate this repetitiveness—this lack of diversity in LLM responses for creative tasks. But one of the things I really love about it is that in understanding why this repetitiveness happens and why verbalized sampling actually works as a mitigation technique, you start to get some pretty interesting insights and a deeper understanding of what's going on with LLMs under the surface.

The paper discussed in this episode is Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

https://arxiv.org/abs/2510.01171

A Key Concept in AI Alignment: Deep Reinforcement Learning from Human Preferences

Modern AI chatbots have a few different things that go into creating them. Today we're going to talk about a really important part of the process: the alignment training, where the chatbot goes from being just a pre-trained model—something that's kind of a fancy autocomplete—to something that really gives responses to human prompts that are more conversational, that are closer to the ones that we experience when we actually use a model like ChatGPT or Gemini or Claude.

To go from the pre-trained model to one that's aligned, that's ready for a human to talk with, it uses reinforcement learning. And a really important step in figuring out the right way to frame the reinforcement learning problem happened in 2017 with a paper that we're going to talk about today: Deep Reinforcement Learning from Human Preferences.

You are listening to Linear Digressions.

The paper discussed in this episode is Deep Reinforcement Learning from Human Preferences

https://arxiv.org/abs/1706.03741

We're back

It's been (*checks watch*) about five and a half years since we last talked. Fortunately nothing much has happened in the AI/data science world in that time. So let's just pick up where we left off, shall we?

So long, and thanks for all the fish

All good things must come to an end, including this podcast. This is the last episode we plan to release, and it doesn’t cover data science—it’s mostly reminiscing, thanking our wonderful audience (that’s you!), and marveling at how this thing that started out as a side project grew into a huge part of our lives for over 5 years.

It’s been a ride, and a real pleasure and privilege to talk to you each week. Thanks, best wishes, and good night!

—Katie and Ben

A reality check on AI-driven medical assistants

The data science and artificial intelligence community has made amazing strides in the past few years to algorithmically automate portions of the healthcare process. This episode looks at two computer vision algorithms, one that diagnoses diabetic retinopathy and another that classifies liver cancer, and asks the question—are patients now getting better care, and achieving better outcomes, with these algorithms in the mix? The answer isn’t no, exactly, but it’s not a resounding yes, because these algorithms interact with a very complex system (the healthcare system) and other shortcomings of that system are proving hard to automate away. Getting a faster diagnosis from an image might not be an improvement if the image is now harder to capture (because of strict data quality requirements associated with the algorithm that wouldn’t stop a human doing the same job). Likewise, an algorithm getting a prediction mostly correct might not be an overall benefit if it introduces more dramatic failures when the prediction happens to be wrong. For every data scientist whose work is deployed into some kind of product, and is being used to solve real-world problems, these papers underscore how important and difficult it is to consider all the context around those problems.

Relevant links:

A Data Science Take on Open Policing Data

A few weeks ago, we put out a call for data scientists interested in issues of race and racism, or people studying how those topics can be studied with data science methods, should get in touch to come talk to our audience about their work. This week we’re excited to bring on Todd Hendricks, Bay Area data scientist and a volunteer who reached out to tell us about his studies with the Stanford Open Policing dataset.

Relevant Links:

Procella: YouTube's super-system for analytics data storage

This is a re-release of an episode that originally ran in October 2019.

If you’re trying to manage a project that serves up analytics data for a few very distinct uses, you’d be wise to consider having custom solutions for each use case that are optimized for the needs and constraints of that use cases. You also wouldn’t be YouTube, which found themselves with this problem (gigantic data needs and several very different use cases of what they needed to do with that data) and went a different way: they built one analytics data system to serve them all. Procella, the system they built, is the topic of our episode today: by deconstructing the system, we dig into the four motivating uses of this system, the complexity they had to introduce to service all four uses simultaneously, and the impressive engineering that has to go into building something that “just works.”

Relevant links:

The Data Science Open Source Ecosystem

Open source software is ubiquitous throughout data science, and enables the work of nearly every data scientist in some way or another. Open source projects, however, are disproportionately maintained by a small number of individuals, some of whom are institutionally supported, but many of whom do this maintenance on a purely volunteer basis. The health of the data science ecosystem depends on the support of open source projects, on an individual and institutional level.

Relevant links:

Rock the ROC Curve

This is a re-release of an episode that first ran on January 29, 2017.

This week: everybody's favorite WWII-era classifier metric!  But it's not just for winning wars, it's a fantastic go-to metric for all your classifier quality needs.  

Criminology and data science

This episode features Zach Drake, a working data scientist and PhD candidate in the Criminology, Law and Society program at George Mason University. Zach specializes in bringing data science methods to studies of criminal behavior, and got in touch after our last episode (about racially complicated recidivism algorithms). Our conversation covers a wide range of topics—common misconceptions around race and crime statistics, how methodologically-driven criminology scholars think about building crime prediction models, and how to think about policy changes when we don’t have a complete understanding of cause and effect in criminology. For the many of us currently re-thinking race and criminal justice, but wanting to be data-driven about it, this conversation with Zach is a must-listen.

Relevant links:

Racism, the criminal justice system, and data science

As protests sweep across the United States in the wake of the killing of George Floyd by a Minneapolis police officer, we take a moment to dig into one of the ways that data science perpetuates and amplifies racism in the American criminal justice system. COMPAS is an algorithm that claims to give a prediction about the likelihood of an offender to re-offend if released, based on the attributes of the individual, and guess what: it shows disparities in the predictions for black and white offenders that would nudge judges toward giving harsher sentences to black individuals.

We dig into this algorithm a little more deeply, unpacking how different metrics give different pictures into the “fairness” of the predictions and what is causing its racially disparate output (to wit: race is explicitly not an input to the algorithm, and yet the algorithm gives outputs that correlate with race—what gives?) Unfortunately it’s not an open-and-shut case of a tuning parameter being off, or the wrong metric being used: instead the biases in the justice system itself are being captured in the algorithm outputs, in such a way that a self-fulfilling prophecy of harsher treatment for black defendants is all but guaranteed. Like many other things this week, this episode left us thinking about bigger, systemic issues, and why it’s proven so hard for years to fix what’s broken.

Relevant links: