Machine Learning Street Talk (MLST)

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper, Stealing Reasoning Traces from Proprietary LLM APIs.The core bug sounds deceptively simple: providers return encrypted reasoning state so conversations can be resumed or forked. But those blobs can be replayed across users and sibling models. A smaller model can ask the provider to decrypt the trace, then repeat the hidden reasoning in plain text. The discussion covers leaked private data, a broadly reusable jailbreak, poisoned agent traces, chain-of-thought monitoring, responsible disclosure, and possible defenses.Ilia Shumailov is an AI and security researcher, formerly at Google DeepMind, who completed his Cambridge PhD under Ross Anderson. Alexander Panfilov is a PhD researcher at the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems, working on AI safety, adversarial machine learning, and LLM red-teaming. They close by separating the demonstrated jailbreaking threat from ordinary benign distillation, and by arguing for controlled experiments over sweeping claims.---TIMESTAMPS:00:00:00 Intro montage00:01:33 Portable encrypted thought and decoded reasoning00:24:55 How the attack works and what it means00:39:04 Doom, defense, and scientific restraint---REFERENCES:paper:[00:00:00] Stealing Reasoning Traces from Proprietary LLM APIshttps://arxiv.org/abs/2608.09867[00:09:22] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyhttps://arxiv.org/abs/2507.11473[00:11:30] Reasoning Models Don’t Always Say What They Thinkhttps://www.anthropic.com/research/reasoning-models-dont-say-think[00:37:22] PostTrainBench: Can LLM Agents Automate LLM Post-Training?https://arxiv.org/abs/2603.08640[00:41:02] Large-scale online deanonymization with LLMshttps://arxiv.org/abs/2602.16800other:[00:09:28] OpenAI and Hugging Face partner to address security incident during model evaluationhttps://openai.com/index/hugging-face-model-evaluation-security-incident/[00:10:22] Claude, GPT, and Gemini All Struggle to Evade Monitorshttps://metr.org/notes/2025-08-22-claude-gpt-gemini-struggle-evade-monitors/tool:[00:42:08] Isabelle proof assistanthttps://isabelle.in.tum.de/---RESCRIPT: https://app.rescript.info/share/07fc38276e0823dc9b8986c32e202c7f

Astrophysicist Adam Becker, author of "What Is Real?", joins Tim Scarfe to take apart the futures Silicon Valley keeps selling: the 2045 singularity, mind uploading, Mars colonies, and the AI apocalypse. His new book *More Everything Forever* argues these ideas are hugely influential, mostly evidence-free, and bankrolled by tech billionaires who need a story in which growth never ends.Becker does the physics the boosters skip. Kurzweil's "law of accelerating returns" rests on cherry-picked data, and every exponential ends. Grant Bezos his perpetual energy growth and humanity boils the oceans within a few centuries, then exhausts the observable universe in under 4,000 years. The stars are too far away, Mars dirt is poison, and the day the dinosaur-killing asteroid hit Earth was still nicer than any day on Mars. On AI, Becker calls LLMs pocket calculators for language: hallucination is the model doing exactly what it always does, and the intelligence explosion assumes intelligence is a single number you can buy with compute.The sting is that Becker thinks the doomers are sincere. Yudkowsky, Bostrom and the effective altruists are not grifters, he says, just wrong, and their warnings that AI could end the world feed the same growth story the money depends on. He closes with his own prescription: take social problems seriously, regulate the whole tech industry, and tax billionaires out of existence.---TIMESTAMPS:00:00:00 Cold open and the thesis of More Everything Forever00:04:24 Kurzweil's singularity and the physical limits of exponential growth00:14:02 High agency and the fantasy of imprinting humanity on the cosmos00:16:55 Mind uploading, functionalism, and embodied cognition00:24:24 AI psychosis and anthropomorphizing LLMs00:26:24 Calculators, hallucination, and the limits of scale00:32:20 Yudkowsky and the intelligence-explosion argument00:40:37 True believers, venture capital, and the sci-fi growth narrative00:47:21 From Extropians to EA: utilitarianism and longtermism00:53:50 Brain worms and Becker's prescription: take social science seriously00:56:49 Why the AI-ethics discourse is broken01:01:42 The eugenics and IQ argument against 'intelligence'01:06:07 Why space settlement fails: Mars, the moon, and orbital data centers01:10:42 Billionaire myths and the search for purpose01:13:38 Tax billionaires, regulate tech: closing prescriptions---REFERENCES:book:[00:00:07] More Everything Forever (Adam Becker, 2025)https://www.hachettebookgroup.com/titles/adam-becker/more-everything-forever/9781541619593/[00:00:15] What Is Real? (Adam Becker, 2018)https://en.wikipedia.org/wiki/What_Is_Real%3F[00:15:46] What We Owe the Future (Will MacAskill, 2022)https://www.hachettebookgroup.com/titles/william-macaskill/what-we-owe-the-future/9781541618626/other:[00:00:27] Dreaming Against the Machine (podcast)https://www.dreamingagainstthemachine.com[00:01:04] The Useful Idiots of AI Doomsaying (Adam Becker, The Atlantic, 2025)https://www.theatlantic.com/books/archive/2025/09/what-ais-doomers-and-utopians-have-in-common/684270/
RESCRIPT: https://app.rescript.info/share/d6e37f9866673d8f74a39076efa5926b

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstWhy can deep networks discover abstractions that shallow models miss? Statistical physicist Matthieu Wyart joins Tim Scarfe to argue that the answer lies in the hidden hierarchy of data. Language and images are built from parts within parts; depth lets a network recover those coarse-grained variables and escape the curse of dimensionality.The conversation moves from jamming transitions and rough loss surfaces to Chomsky, context-free grammars and machine creativity. Wyart explains why next-token prediction can still recover compositional structure, where current systems fall short of genuine scientific invention, and why predicting latent representations rather than raw tokens could make learning far more sample-efficient.They also examine diffusion models, neural scaling laws and the limits of physics-inspired theory. The final question is on a personal note: if mistakes are the price of leaving the beaten path, how much scientific risk is worth taking?---TIMESTAMPS:00:00:00 Can machines learn abstractions from data?00:02:00 Notion agentic workspace00:02:49 From statistical physics to machine learning00:06:40 What physics can explain about learning00:16:37 From Carnot to Chomsky bulldozer00:21:21 How deep networks recover hidden hierarchies00:32:43 Where machine creativity still falls short00:40:48 How deep nets escape the curse of dimensionality00:52:19 Why predict latents instead of tokens01:02:49 The sample-efficiency case for latent prediction01:08:31 Diffusion, scaling laws and text entropy01:16:40 The scientists we learn from and the mistakes we make---REFERENCES:person:[00:00:43] Noam Chomskyhttps://linguistics.mit.edu/user/chomsky/tool:[00:02:08] Notion Developer Platformhttps://www.notion.com/en-gb/blog/introducing-developer-platformpaper:[00:04:43] Mastering the game of Go with deep neural networks and tree searchhttps://www.nature.com/articles/nature16961[00:05:52] Reconciling modern machine-learning practice and the bias-variance trade-offhttps://arxiv.org/abs/1812.11118[00:25:54] How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelhttps://arxiv.org/abs/2307.02129[00:42:12] Efficient Estimation of Word Representations in Vector Spacehttps://arxiv.org/abs/1301.3781[00:52:46] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturehttps://arxiv.org/abs/2301.08243[00:52:54] Learn from your own latents and not from tokens: A sample-complexity theoryhttps://arxiv.org/abs/2605.27734[01:08:31] A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datahttps://arxiv.org/abs/2402.16991[01:11:39] Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361[01:12:17] Deriving Neural Scaling Laws from the statistics of natural languagehttps://arxiv.org/abs/2602.07488[01:13:34] Prediction and Entropy of Printed Englishhttps://ieeexplore.ieee.org/document/6773263---LINKS:Download PDF transcript: https://app.rescript.info/share/f7644cdaa86c5cc1e41e484e290f2bd4

Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Updates, their new research with OpenAI.

The panel asks how models infer what graders reward, why good behaviour can come from the wrong reason, and whether that difference can be measured. The conversation moves through promise-breaking, grader awareness, reward hacking, scheming, opaque reasoning and corrigibility, then turns to a detailed walkthrough of the contrastive-belief method and what its results do and do not show. The o3 results discussed here concern an intermediate checkpoint without safety training.

This episode was made in partnership with Apollo Research. MLST retained full editorial control.

Reference
Apollo Research: https://www.apolloresearch.ai/

---
TIMESTAMPS:
00:00:00 Cold Open
00:02:12 Right Things, Wrong Reasons
00:12:47 Grader Awareness
00:26:22 Legibility
00:32:35 What To Call It
00:35:58 Intelligence, Agency, Anthropomorphism
00:45:16 Apollo’s Mission
00:48:54 The End of the Exponential
00:55:45 The Paper
01:16:34 Closing Reflection

---
REFERENCES:
tool:
[00:00:08] Claude Fable
https://www.anthropic.com/claude/fable
[00:12:50] AlphaGo Zero
https://deepmind.google/blog/alphago-zero-starting-from-scratch/
[00:44:30] AlphaFold 3
https://deepmind.google/science/alphafold/
paper:
[00:01:02] Measuring Reward-Seeking via Contrastive Belief Updates
https://arxiv.org/abs/2607.18966
[00:16:19] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
https://transformer-circuits.pub/2026/nla/
[00:26:48] Stress Testing Deliberative Alignment for Anti-Scheming Training
https://arxiv.org/abs/2509.15541
[00:35:33] Shortcut learning in deep neural networks
https://arxiv.org/abs/2004.07780
[00:53:49] Measuring AI Ability to Complete Long Software Tasks
https://arxiv.org/abs/2503.14499
[00:59:52] Modifying LLM Beliefs with Synthetic Document Finetuning
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
[01:10:44] Alignment Faking in Large Language Models
https://arxiv.org/abs/2412.14093
[01:13:55] Natural Emergent Misalignment from Reward Hacking
https://www.anthropic.com/research/emergent-misalignment-reward-hacking
other:
[00:10:14] We Need a Science of Scheming
https://www.apolloresearch.ai/science/science-of-scheming/
[00:32:56] CoastRunners reward hacking example
https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
organization:
[01:06:07] Redwood Research
https://www.redwoodresearch.org/

---
ReScript:
https://app.rescript.info/share/718ab68e18cfa3b9b800da6b3290fd42

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlst

Britain's most capable coding model can't be exported, and that ban is the whole reason Cosine set out to build one from scratch. Alistair Pullen, CEO and co-founder of Cosine, sits down with Tim Scarfe to explain how a frontier system he calls Fable, locked behind US export controls, became the founding case for a UK sovereign model trained on the Isambard supercomputer in Bristol.

The bet underneath it is economic. Pullen argues that an inference company, rather than a training-first lab, doesn't need billions to compete: millions, a national compute allocation, and a consortium feedback loop can be enough. From there it gets into the machinery, why open-weight models still trail the frontier on size, active parameters and data, the mixture-of-experts versus dense trade-off and why active params dominate how a model actually feels, and the edge that real coding trajectories confer.

The back half is about making agents trustworthy. Pullen makes the case for beating "slop" by rewarding the process instead of the final answer, reframes code review as runtime proof (spin the bug up in a VM and force the agent to actually exploit it), and walks through Swarm, Cosine's system running hundreds of sub-agents in one shot. It ends on why memory is still an unsolved hack, how synthetic graders let you run RL on tasks with no built-in test, and why Pullen reads US export controls as an accidental gift, with a supply-chain sting in the tail.

---
TIMESTAMPS:
00:00:00 The sovereign mandate and the Fable ban
00:04:02 Millions vs billions: the inference-company model
00:07:19 The consortium feedback loop
00:07:40 Why open models lag the frontier
00:14:59 MoE vs dense, and why active params matter
00:16:29 Trajectories: the process-data advantage
00:19:48 Beating slop: reward the process, not the answer
00:26:06 Reusable abstractions and the epistemic wall
00:29:56 Code review becomes runtime proof
00:37:32 Do agentic harnesses still matter?
00:40:35 Swarm: orchestrating hundreds of sub-agents
00:45:14 Why memory is still unsolved
00:48:25 Synthetic data and graders for RL
00:53:09 The US export gift and supply-chain risk

---
REFERENCES:
organization:
[00:01:15] Cosine
https://cosine.sh
[00:04:14] Mistral AI
https://mistral.ai
[00:05:50] Anthropic
https://www.anthropic.com
[00:07:42] Cohere
https://cohere.com
[00:08:36] DeepSeek
https://www.deepseek.com
tool:
[00:02:52] Isambard-AI
https://isambard.ac.uk
[00:05:56] Colossus (xAI)
https://en.wikipedia.org/wiki/Colossus_(supercomputer)
[00:07:52] GLM (Z.ai)
https://z.ai
[00:11:52] NVIDIA B300
https://www.nvidia.com/en-us/data-center/dgx-b300/
[00:15:37] gpt-oss-120b
https://huggingface.co/openai/gpt-oss-120b
[00:15:52] Devstral 2
https://mistral.ai/news/devstral
[00:16:01] Llama 70b
https://www.llama.com
[00:17:05] Claude Code
https://www.anthropic.com/claude-code
[00:26:23] ARC-AGI (Francois Chollet)
https://arcprize.org
[00:40:38] Swarm (Cosine)
https://cosine.sh
[00:40:50] OpenAI Codex
https://github.com/openai/codex
[00:41:16] Lumen Outpost (Cosine)
https://cosine.sh
[00:41:18] Kimi K2 (Moonshot)
https://huggingface.co/moonshotai/Kimi-K2-Instruct
[00:49:55] SWE-bench
https://www.swebench.com
[00:52:40] SystemVerilog
https://en.wikipedia.org/wiki/SystemVerilog
person:
[00:23:40] Andrej Karpathy
https://karpathy.ai
paper:
[00:27:10] GRPO (DeepSeekMath)
https://arxiv.org/abs/2402.03300
[00:27:13] GSPO
https://arxiv.org/abs/2507.18071

Incompressible Knowledge Probes, Bojie Li
https://arxiv.org/pdf/2604.24827

Estimating the Size of Claude Opus 4.5/4.6
https://unexcitedneurons.substack.com/p/estimating-the-size-of-claude-opus

---
ReScript:
https://app.rescript.info/session/5852d2b884c4ce4b?share=10b9799160845bb11779f8ac6cd3124f