Will Marshall: What Happens When AI Becomes Too Powerful to Control?
Will Marshall's essay quotes Demis Hassabis saying the more AI becomes a race, the harder it is to keep it safe, and Mark tests that against motor racing. So the race to get faster Formula One cars made the cars safer.
In this episode · 21 chapters
- Who is Will Marshall?
- What the essay proposes
- Sam Altman on unimaginable power
- The nuclear parallels
- Why would they say these things?
- What the CEOs put the risk at
- Mira Murati
- The Spotify takedown
- The hinge moment
- What happens when we reach AGI
- The nuclear model
- Trust, but verify
- What is AI's Cuban missile crisis?
- Three categories of concern
- Three companies control the compute
- An IAEA for AI
- Chips that watch themselves
- Limits on autonomous replication
- What happens when someone does not buy in
- It's an advert for Planet
- The Fermi paradox
I think that question needs a caveat because I don’t think that the more something becomes a race, the harder it is to keep it safe. I think it depends who’s in the race.
Mark and Jeremy read The Hard Question of AI by Will Marshall, CEO of Planet. The essay argues that the US and China should agree limits on AI, and borrows its model from the way the world came to manage nuclear weapons.
They work through trust but verify: chips that monitor themselves, inspection of frontier labs, and red lines on self-improving systems. They ask what AI's Cuban missile crisis would look like, and Mark suggests one section reads like an advert for Planet.
The Hard Question of AI: An Urgent Ask to Steward Superintelligence, an essay by Will Marshall, CEO of the satellite company Planet, argues that AI governance needs a verification regime modelled on nuclear arms control. On the Thinking On Paper podcast, Mark Fielding and Jeremy Gilbertson test the nuclear parallel and ask what AI's Cuban missile crisis would be.
Why does Will Marshall call today the hinge moment for AI?
Marshall calls the present “the hinge moment”, because AI may soon reach closed loop recursive self-improvement, which Mark reads as “the capacity to rewrite its own code to become more capable without human intervention”. The essay quotes Alan Turing in 1951: once machine thinking starts, “it would not take long to outstrip our feeble powers.”
Why does Marshall think AI can be monitored?
The essay argues AI can be monitored because its supply chain is narrow: Jeremy reads that the chips depend on “tools from ASML in the Netherlands, and manufacturing from TSMC in Taiwan”. Mark adds that the electricity data centres draw is another thing inspectors could track.
Could AI labs be inspected like nuclear sites?
Marshall cites Demis Hassabis's call for an AI equivalent of the International Atomic Energy Agency (IAEA), the body that inspects nuclear sites. He argues that inspecting a frontier AI lab “involves company intellectual property rather than state military secrets”, which should make it easier to agree. Mark disagrees: a nuclear bomb is binary, while AI's risks are abstract, so building political agreement is harder.
What AI verification work already exists?
The essay points to existing verification work. The RAND Corporation has proposed a six-layer verification architecture, which Jeremy explains as chips that monitor themselves, outside devices that monitor the chips, and people who monitor the people. The International Dialogues on AI Safety (IDAIS), with researchers from the US, Europe and China, has agreed red lines, including no systems capable of recursive self-improvement without human oversight.
How does Marshall explain the Fermi paradox?
Marshall ends on the Fermi paradox, the puzzle of why we see no sign of alien civilisations. Jeremy reads his answer: “one disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it.” Mark offers a plainer explanation: space is very big and nothing travels faster than light.
Where they disagree
Mark challenges the essay's quote from Demis Hassabis that the more AI becomes a race, the harder it is to keep safe. “So the race to get faster Formula One cars made the cars safer.” He says it depends who is in the race, doubts the US and China can reach real trust, and calls the essay's proposal to invest in planetary intelligence “an advert for planet”.
Links from this episode
Questions
Who is Will Marshall?
Mark and Jeremy read The Hard Question of AI by Will Marshall, CEO of Planet.
What is The Hard Question of AI about?
The Hard Question of AI: An Urgent Ask to Steward Superintelligence, an essay by Will Marshall, CEO of the satellite company Planet, argues that AI governance needs a verification regime modelled on nuclear arms control. The essay argues that the US and China should agree limits on AI, and borrows its model from the way the world came to manage nuclear weapons.
What did Alan Turing predict about machine thinking?
The essay quotes Alan Turing in 1951: once machine thinking starts, “it would not take long to outstrip our feeble powers.”
Transcript
Jeremy Gilbertson Disruptors and curious minds. Welcome to Thinking on Paper. My name is Jeremy. This is Mark. We talk about technology. We talk about humans. We talk about systems. You guys know who we are. we're on it again. This is a pocket edition. We like to read. We like to unpack what we read. No guests today. Mark, you're the guest. I'm the guest. We're both the guests. we are unpacking today a an essay from Will Marshall,
Who is Will Marshall?
Mark Fielding Who who's Will Marshall, Jeremy?
Jeremy Gilbertson That's a great question, my friend.
So Will Marshall was actually in this book, Space to Grow. That's the first place that I heard about Will and heard about his company Planet. Planet has I guess their first generation was was doves, these satellites that live in lower earth orbit that kind of monitor the earth and the systems on the earth and all of these interesting things to tell.
Mark Fielding In in real time they had like hundreds of these things small because they their whole according to space to grow, they were former Amazon employees, I think, and they satellites were really expensive and they kind of homemade satellites in their garage almost and sent hundreds of them up to yeah, to to scan the earth meter by meter, second by second, yeah.
Jeremy Gilbertson Yeah, and actually I think yesterday or or a a day or so ago they had a big big event, Planet Did and Will Marshall with the launch of Pelican Two, Talk to me about that.
Mark Fielding Pelican two is they just I think they l they announced that they had connected, successfully communicated with Pelican two, a part of their new system of satellites, new technologies and capabilities building on generation one pelican's ability ability to capture up to 50 cm class resolution imagery. Generation two are designed to provide up to 30 centimeters class imagery in alignment with.
planet's approach to agile aerospace. This tech demonstration satellite is expected to allow rapid hardware testing and calibration in the space and environment. So it's about more more satellites, more scanning the Earth's surface.
Jeremy Gilbertson Mm.
Very cool. Yeah. Seeing seeing the pale blue dot, right? Letting letting the Carl Sagan thing happen in in real time. I like it. well, we'll stay in touch with what Planet's doing in our space series, among other things. But this is where the worlds intersect, Mark. This is why thinking on paper exists, is to find these connections, these through lines. Dare I say forces and currents.
Mark Fielding Yes.
You should, you could, you may.
Jeremy Gilbertson Within within this. We'll be using that language a lot, guys. hang tight and that'll make sense down the road. But we're working on something big that we're gonna release very soon. But stay tuned. So here's what we're doing. Will Marshall wrote an essay called The Hard Question of AI, an urgent ask to steward superintelligence. Subtitle Why AI Governance Should Top President Trump's Agenda.
Mark Fielding And Xi's agenda.
Jeremy Gilbertson Xi's
What the essay proposes
Mark Fielding But we're not, are we? Because we'll get into the essay, but he proposes that maybe we shouldn't be in a race with China. Maybe the US and China should be in collaboration. Imagine that.
Jeremy Gilbertson Imagine that. Imagine that. What's really well, it's really interesting to think about this because his perspective is probably pretty interesting, right? He's designing or planet is designing this system to monitor the rock we're sitting on in real time. What does that do? It takes us out of us and or takes us out of like me standing here on the planet in my own little vacuum tube echo chamber versus like the image of the world,
Creating a global, global look at things is what Planet actually is doing. And I think this essay is really interesting because he's proposing a global look at things. so you read it, I read it. Give me your first impressions, give me some, give me where you wanna, where you wanna dive into. There's tons, I think.
Mark Fielding It's my favorite subject, Jeremy. Existential risk to humanity. Artificial intelligence and the demise of the human species. Artificial intelligence, AGI, super intelligence, and some other layers of intelligence in between. And what happens if it all goes wrong? And Will Marshall's essay is about possible solutions, possible frameworks, ideas.
collaborations and experiments so that AI and humans are aligned and we survive into the future to kill ourselves in other ways that don't involve AI.
Jeremy Gilbertson Yes, for sure. So so yeah, there yeah, everything you just said. and we're gonna talk about a little Fermi paradox rabbit hole down the end that will make some sense into what into what Mark just said. So let's start with the big three. open AI, anthropic, Google Deep Mind, I'm gonna read some quotes here just to kind of set the tone of where things are and why why this essay is important and timely.
Mark Fielding Yeah.
Sam Altman on unimaginable power
Jeremy Gilbertson The first one is from Sam Altman, OpenAI. I think the good case around AI is just so unbelievably good that you sound like a really crazy person to start talking about it. The bad case, and I think this is important to say, is the lights out for all of us. Wow. Boom. All right. I'm going right into quote number two. Dario Amodei chief executive of Anthropic, who we've unpacked his essay Machines of Love and Grace. If you want to listen to that, boop, boop, boop.
Mark Fielding Okay.
Jeremy Gilbertson his quote humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems can possess the maturity to wield it. Tech moving faster than our ability to deal with it. 100% agree there. Last one. Demis Hassabis chief executive, Google DeepMind. The risk of a catastrophic scenario is not zero.
So we must dedicate significant resources to mitigating it. A little more of a tempered quote there, right? But the risk of catastrophic scenario that pours right in. And we're just going right into this essay because it it it's it sets it sets a nice stage, I think. as he's talking through this, as as Will Marshall is talking through this. So the acceptable risk for nuclear reactors.
Mark Fielding Yeah.
The nuclear parallels
Jeremy Gilbertson And a lot of there's a lot of nuclear parallels in this essay. The acceptable risk for nuclear reactor scenarios, right? The risk of catastrophic
Mark Fielding Yeah.
Jeremy Gilbertson Catast catastrophic meltdown.
Jeremy Gilbertson So we're talking the Three Mile Island, we're we're talking Fukushima, Chernobyl, all of that stuff, right? One in one million. One in one million is kind of that acceptable risk. Not acceptable risk, but that's kind of the
Mark Fielding Chernobyl.
What's my what's my the odds of me getting hit by lightning tomorrow? Is that more or less than that? It must it must or bit by bitten by a shark must be longer odds than that. There must be more chance of a nuclear meltdown than me being bitten by a shark. He says.
Jeremy Gilbertson Or Or bit by a shark.
Well we'll look that look that up while I'm while I'm transitioning to this to that's the acceptable risk for catastrophic melt meltdown in a nuclear power plant, according to Will's research and essay. the AI translation, I'm just gonna read a quote from this from his essay. AI experts estimate the risk of an AI-caused catastrophic event at 10 to 50 percent.
Mark Fielding Whip that up, yeah. I will. You transition.
Jeremy Gilbertson Quite a big jump. Quite a big jump in that. And according to what he says in this essay, the risk is coming directly from the people building it. So that's something we'll get into. Mark, do you have any numbers on the lightning or shark attacks?
Mark Fielding I I some very very rough numbers, but your lifetime odds of being killed by a shark are roughly one in three point seven million. So three catastrophic nuclear meltdowns in a lifetime. Can I ask you a question? So we have we we had Sam Altman, we had Dario Amodei we had Demis Hasabis. Okay, three of the biggest, most wealthiest, important people in artificial intelligence.
Jeremy Gilbertson Better than a meltdown, okay.
Why would they say these things?
Mark Fielding What we talk about incentives all the time.
Why are they saying these things?
if I had a company that made AI and I told everyone that AI was gonna kill us all, that would be bad for business, wouldn't it?
Jeremy Gilbertson Well, if you remember this, th this is a call back to our review of Karen Hao's book, Empire of AI. There was something in there that was really interesting that that immediately jumps to mind referring to Sam Altman and his advocacy for not just the good things about AI, but the potential harms of AI is like by coming out and saying this really could be bad for people.
Governments have to kind of rely on the guy saying that, right? And if you rely on the guy saying that, then you get to be part of how that thing is regulated. So I think there's a little bit of that that sticks out. And I'm not an expert in economic policy, gov government policy, all any of that. But enough to reference that Empire of AI, Karen Hao positioning that you know, reflected Sam Altman's, I think desire to be at the be at the table when the governance
Talks were happening.
What the CEOs put the risk at
Mark Fielding AI experts estimate the estimate the risk of an AI caused catastrophic event at between ten and fifty percent. Well, we check out our episode on prophecy, the book, the excellent book by Carissa Veliz on why you should take those this huge range of percentages with perhaps a pinch of salt. But let's go on because the so we we've laid we've painted the picture. This is what's at stake.
This is what the CEOs and founders think we should weigh up. They're not saying it's going to happen, but we should be thinking about it. And Will Marshall proposes a solution that. But he begins with another scary prospect, what he calls the hinge moment. Basically, we don't have much time. Within months, artificial intelligence may achieve what researchers call closed loop recursive self-improvement, RSI, not repetitive strain injury.
Closed loop recursive self-improvement, which should be C RSI, shouldn't it? The acronym, not RSI, but anyway, the capacity to rewrite its own code to become more capable without human intervention. This is no longer a theoretical proposition. Anthropic has already declosed that roughly 90% of the code for its company's latest model was written by its predecessor, RSI.
a system that can improve itself without human direction, Jeremy. What do you what does that make you think, feel think and feel?
Jeremy Gilbertson A little HAL Let's let's let's read so so in the right, right. So let's let's go back at the end of the essay. There are a couple more quotes in here that are that are really interesting Elon Musk. I think AI is one of the biggest threats to humans. Okay.
Mark Fielding I will not open the bay doors. Yeah.
Mm.
Mira Murati
Jeremy Gilbertson Mira Marathi, we remember Mira from Empire of AI, right? Former
CTO of OpenAI Thinking Machines Lab. So there are a lot of hard problems to figure out. How do you get the model to do the thing you want it to do? How do you make sure it's aligned with human intention and ultimately in service of humanity? There are also a ton of questions around societal impact, and I think there are a lot of ethical and philosophical questions we need to consider. It's important we bring in different voices like philosophers, social scientists, artists, people from humanities, and Mark and Jeremy from Thinking on Paper.
Mira, wow, that's amazing. I'm really feel very
Mark Fielding That is a compliment. Thank you, Mira.
Jeremy Gilbertson special. feel very special for that.
Mark Fielding She didn't say that just to clarify.
Jeremy Gilbertson You're busting the dream.
The Spotify takedown
Mark Fielding Well, we've just been kicked off Spotify for breaking copyright rules, Jeremy. So we gotta, you know, maybe we should sp tell tell the truth so we don't get ousted. One of our episodes has been taken down, an old episode from over a year ago now, because there was an eight second snippet of Ghostbusters, the music from Ghostbusters in the background when we were talking about virtual reality, and Spotify a year later flagged it and globally banned that episode.
Jeremy Gilbertson Have we really?
Mark Fielding For breaking copyright.
Jeremy Gilbertson Was was that how was that th music played? Was it played in the context of a discussion around Ghostbusters or was it
Mark Fielding It was so it was about the history of augmented reality. And one of the first use cases of augmented reality were these backpacks on a I think it was Stanford University, guy at Stanford University, and they it they looked like the Ghostbuster backpacks. So I put a little bit of the Ghostbusters music behind that as we were explaining that story. And so and and lit yeah, but as we're talking about AI, and I'm just I'm just I'm happy.
Jeremy Gilbertson Not allowed.
Mark Fielding That Spotify have banned that episode globally because we did break the copyright law. And imagine if if these big technology companies were just going around stealing people's work and music and stories without their permission, what kind of world will we be in? So, you know, I I think they're setting a precedent that thou shalt not steal.
Jeremy Gilbertson I'd like to I'd like to on behalf of Thinking on Paper, I'd like to publicly apologize to the Ray Parker Jr. family. because I grew up with that song and man, I danced around to that song and I appreciate the value of it. move
Mark Fielding So can it wasn't right. So according to the the cease and desist notice, it w it wasn't Ray Parker. So they they've said the the artist was George Doherty. Does that mean anything to you? So I d I don't act no. It what I it might not even have been the real Ghostbusters song because I didn't want to put the real Ghostbusters song. So that I think that is actually George Doherty made
Jeremy Gilbertson man, this is going down a rabbit hole, but let's keep it up.
Mark Fielding No. So he actually he actually wrote the song.
Mark Fielding his own version of the Ghostbusters song. Actually
Jeremy Gilbertson okay. Well hopefully he got permission to do it.
Mark Fielding Well the the meta irony goes down so many layers. Right, sorry.
The hinge moment
Jeremy Gilbertson All right, so so we're on this we're on this hinge moment and basically what what Will Marshall is is talking about here is, the existential threats aren't fully defined or realized, but they're coming quickly. We can see them. We can see this like this seeing this cloud firing at you very quickly. And you don't know what the cloud's going to eventually be when it gets here, but it damn sure is moving fast. And even Alan Turing back in 1951, as Will writes, here's a quote from Alan Turing: it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers.
At some stage, therefore, we should have we should have to expect the machines to take control.
Okay. So acknowledging, yeah, he was acknowledging it in fifty-one. I think here's the challenge though, man. as long as there's this race to the top, it's hard to look at the repercussions.
of the end result of the race while the race is still going because the race is powered by money, the pay the race is powered by the drive to need and control a market. And there are a lot of things that kind of fall to the wayside while you're racing.
Mark Fielding Yes. They many things fall to the wayside. ethics, responsibility, accountability, truth, for example. that long ago, people were already
Thinking of the worst case scenario. So, okay, it's always good to think of the worst case scenarios because then you can work to avoid the worst case scenarios. Reverse engineer what you want by describing what you don't want. It's interesting though that that long ago, obviously the one of the the the name at the beginning of computing said this. How many of his students took on that?
Thought process. Like it kind of almost like this negativity, this AI is going to kill us all and take over the world, has been passed on from one great computing mind to the next. And now you have
Jeremy Gilbertson I wouldn't call it a negativity though, I don't think. I would call it a I mean, I I I don't I didn't know Alan Turing, right, of course, but like reading about him, reading things he wrote, he doesn't seem like a very like I'm gonna be unobjective about this thing. You know, be very objective about here's a thing, here's a capability. Hey, we made some shit work, but you know, you pour jet fuel on this stuff down the road. He's just looking and seeing, you know, the the the evidence of what could happen, I don't see it as a negative thing. I think it's a great responsibility of a of a someone like that to to say that early on.
Mark Fielding Yeah, night
Yeah, it was a it was the wrong choice of words, but it reminded me of another of the godfathers of AI, Sir Geoffrey Hinton. And he's come out and he's doing the rounds because he's said similar things that if we don't get a hold of this, that it the the the fallout could be existential for humanity. And
It reminds me of Frankenstein a little bit. That th these the people have created this monster. Obviously Frankenstein was actually an it was a good person. It was society that made him, turned him. So like it just there's echoes of that kind of Frankenstein where the the people create it and then after they've created it they start saying, what if it all goes wrong? I mean that there was afterthought, not forethought.
Jeremy Gilbertson Well, yeah, I mean, it goes back to we're really good at creating new technology. We're terrible at making sure it is for the good of our species versus the bad, or not just the good of a small segment of our species, which I think is what we're probably gonna witness in the coming years, if some of these safeguards aren't aren't put in place.
Mark Fielding Well, let's go, Utopia. So anyone who's been to the new Thinking on Paper website will see that every episode features on the doom to utopia scale. We're all about the good and the bad. We want to both sides of the story. So there is no question, there is no doubt that today's large language models are well, let's quote, quote, generating substantial economic and scientific value, compressing research timelines, expanding access to expertise, enabling new forms of environmental modelling.
monitoring the reliability and security challenges at this level are significant yeah and and new forms of environmental monitoring. So there's a lot of positive, there's a lot of good. How do we make sure that we continue along that path and not the other path,
What happens when we reach AGI
Jeremy Gilbertson well, what happens when we get to AGI? And then we're like people like loosely understand what that is. And then this article references ASI. And that was the first time I actually heard.
ASI as a as a term that was written here. Artificial superintelligence, which is quote, I'm quoting Will Marshall here, systems whose capabilities exceed human performance by margins that are difficult to conceptualize. shit getting so good that we don't even know.
Mark Fielding Yeah.
Jeremy Gilbertson what it can do with how good it is. Like we can't even we can't even explain what that might look like. 'Cause we haven't seen it. We can't imagine it. Like we can't like in our minds imagine it even.
Mark Fielding Yeah. Yeah.
It it's like the the AI equivalent of what's on the edge of the universe, like what's beyond the universe. You can't conceptualize, you can't imagine that. Like what is beyond soup AGI is this intelligence which we can't even conceptualize in our heads. Our feeble minds.
Jeremy Gilbertson You know what's really funny? What's really funny here, like with and this is this is such a weird comparison, but I saw this little little video today that kind of made me chuckle a little bit. So club sports, meaning like travel sports for youth, like not just the re recreational level. These are the folks that are a little better. You put them on teams, you take all over the country and stuff like that. And
There was this one, there was this one silly video with like a parent to parent kind of thing. And it's like, hey, you know, Jimmy, do you you should probably join this Cobra travel team? And they're like, nah. And like, well, you should join this Cobra Elite Superstar Travel Team. They're like, nah. And then like the third one, you you should you should join this global superstar, superhuman, intergalactic elite squad. And they're like, Yeah, that sounds awesome, right?
So here's my why do I say that? And why is it so I'm keep thinking like here's AGI and here's ASI and then what's after ASI? Like we can't even like think and imagine, but but it's coming and it's always coming and something's always coming. we gotta worry about today. ASI plus, right? So so with all of this stuff, we keep pushing it out to the future. But you know, what's what the author of this essay talks about doing is advocating for, hey, let's
Mark Fielding A A A S A S I plus.
Jeremy Gilbertson Let's freaking grab it by the shirt and sit it down in a chair and make it behave at least a little bit, right? which which sounds really silly, but y you get where I'm going.
Mark Fielding Yeah.
Jeremy Gilbertson I'm just kind of moving down this this S Will's essay here. There's a section called the competitive trap, which is really interesting. It falls back into the race dynamic, right? So, you know, this me having to outdo you and you having to outdo me, and I'm getting more clusters and more network and more power and more utility grid. And hey, I built my own power station, and hey, my network is this, and it's just rise to the top. All the meanwhile, like capital is just like flooding into this, you know, so it's fueling, and so no one's gonna stop because the money's rolling in.
Mark Fielding Yeah.
Jeremy Gilbertson I'm gonna build and I'm gonna do and all of that. So here's what's interesting. Well, Hasabis, you know, back to Google Deep Mind, the more AI becomes a race, the harder it is to keep the powerful technology from becoming unsafe. Absolutely. 100% agree with that. But how do we go, yo?
Mark Fielding Is that true though? Is that true though? Because okay, let's just use my simple non-ASA mind for a comparison. Let's talk about race cars, Formula One cars. So the race to get faster Formula One cars made the cars safer. So is that like what why why does the race make it become more difficult to keep things safe?
Jeremy Gilbertson 'Cause 'cause this this race isn't doing all left turns. This race is going all directions all the time. no, it's an interesting analogy to think about.
Mark Fielding well, let's think about another technology. Okay, aeroplanes, that would be the same thing. So it's similar to cars, we can't use that. What about what about well you're an American, you're an expert, you tell me about that.
Jeremy Gilbertson Safety's on guns. Safety's on guns.
man, low blow. Low blow. I don't have a gun. I actually don't have one. I don't have a Second Amendment right sticker on my car, any of that stuff. but
Mark Fielding I think that question needs a caveat because I don't think that the more something becomes a race, the harder it is to keep it safe. I think it depends who's in the race.
Jeremy Gilbertson Well, this is tricky because you have countries racing countries, you have companies racing companies, you have providers providing things to the companies in the countries that are racing. it's complicated, man. This is this is the most complex of systems.
Mark Fielding Yeah.
as Will alludes to, if we could take something from our history of human our our our our collective history, something which was and has the potential to extinguish us all. And if we could take that, there must be something we must have created something in the past that we could use as a model for this. Is there anything can you think of anything?
Jeremy Gilbertson Yeah, maybe. We may have even talked about this.
We may have even talked about this on Thinking on Paper at some point years ago, didn't we?
The nuclear model
Mark Fielding We did we d what what was it like? Nuclear weapons. Yes. The nuclear model. Can I ju yeah, Jeremy, Will, if you're listening, Jeremy brought this up.
Jeremy Gilbertson yeah.
If you're listening, they're listening if they're listening, right?
Mark Fielding Two years ago, so maybe you and you should come on and think on paper we can discuss this. We can kick the can around the nuclear field.
Jeremy Gilbertson Yes. No, I it it it is a very interesting model. We've talked about it before. We've also talked about the Martins Clause, which that we did a whole episode on the Martin's Clause. If you don't know about that, boop boop, wherever the pop-up is, you can go listen to that too. But the nuclear model is interesting, right? Because the the technology
Mark Fielding Yeah.
Jeremy Gilbertson was moving fast. the technology has or had and has the ability to basically extinguish the whole of the globe, right? So at some point there had to be a, wait, this is starting to get out of control. How do we how do we get this going? And there were a couple things that happened. one, a race, right? So the US was out in the lead, then the Soviet Union kind of came back in pretty quickly. And then the Cold War started and there was this back and forth. So
Mark Fielding Yeah.
Jeremy Gilbertson When you have that kind of thing going, and you know, you're the you're the Soviet Union, I'm the US, I'm building my stuff, you're building your stuff. I have to kind of trust you that you're not gonna freak out and push a button. You're gonna have to trust me to not freak out and push a button. You're also gonna have to trust me when I tell you I'm not enriching any more uranium, you know, based on beyond these quantities, you're gonna have to believe and trust that. But
Trust, but verify
Jeremy Gilbertson What the essay talks about is trust, but verify, right? So we want to go into verification, which I think is really interesting as he lays it out. Because there are systems already in place that that could be potentially used. But one question I have for you, Mark, is there has to be, I think for people to take anything seriously, there's got to be some kind of near-miss event for people to go, holy shit, we got to do something about this.
I think the Cuban Missile Crisis, was an interesting near miss. whether or not I know that that actually was a catalyst to to figure out the nuclear arms race, what is a what could AI's what is AI's near miss look like in your mind?
What is AI's Cuban missile crisis?
Mark Fielding What is
AI's Cuban missile crisis? I I dread to think. I like the parallel in the essay. So, okay, there wasn't much nuclear control before the Cuban missile crisis escalated. And after that, it would they started to think about the the restraint and the controls that were necessary to stop the proliferation to and to stop this. What is the AI Cuban missile crisis near miss? What is that?
W well it could be anything. So let's let our imaginations go wild. So perhaps the the next iteration of fable takes it upon itself to break into the North Korean computer system, find the the the codes and North Korea kind of shoot themselves in the foot, or North Korea shoot South Korea, something somebody gets involved and it could escalate via that, or the same could happen.
China and Taiwan. There's different scenarios that could play out. know. What do you think? What's the AI Cuban missile crisis?
Three categories of concern
Jeremy Gilbertson let's point to a couple of categories that I thought were really interesting that that Will put in his essay that Dario Amodei had had outlined according to Will and there's nothing cited in here, but we're just gonna take it take it at, you know, what we're reading from Will.
Three categories of concern biological weapons design, enabling mass casualties. That's a quote from the from the line here. quote, large-scale cyber attacks on critical infrastructure. So kind of stuff you're talking about, right? Breaking in, shutting down power plants, making things go off that shouldn't go off. The third is really interesting that I hadn't thought about until I read this is the entrenchment of political control at a scale previously impossible.
you could take over countries it with with a keystroke,
Mark Fielding Yeah, that that could happen probably next election cycle. But may may maybe that he's referring to a more well some kind of in artificial intelligence taking on that on itself and if it's iterating on the process and maybe it wants to rig an election, it could do that bizarrely via Facebook. But listeners, tell us in the comments wherever you are listening to this, Spotify, Apple, YouTube, wherever you are listening.
what could be the AI Cuban missile crisis equivalent? What could be the event that pushes humanity to the brink and makes it come together to realize we gotta work together to harness this? What is the AI Cuban missile crisis equivalent?
Jeremy Gilbertson So back to the back to the nuclear analogy.
Comparing it comparing it to AI, it's a it's a little it's little more difficult because you know, monitoring nuclear is kind of easier than monitoring AI in a way. And we can go go down into this because, you for nuclear you need enrichment facilities, weapons grade material, and you can't really do that.
Mark Fielding Yep.
Jeremy Gilbertson under the radar, There there are ways to kind of track and watch that kind of thing. even from space as it as alluded into here. there are ways to get it
Mark Fielding Well we're seeing it play out in real time how difficult it is that the Iranians are prove showing us how difficult it is to actually make something destructive in this way. So yeah, it it's not readily available to everybody. Whereas a well, as Will points out well, you you tell us, Jeremy, like AI kind of follows some of those rules, but not totally.
Jeremy Gilbertson this is where the essay turns from, you trust to verification. I think he uses the language verification, which is really cool and and interesting. It's it's a methodical trust, A methodical trust, a framework based.
Trust. So there are a couple of things that make it a little bit more difficult to to monitor. But he breaks down, think about this, that the supply chain. This is a really interesting quote here, talking about all the different things that go into creating AI and defining what those things are and and creating mechanisms to track those things.
Three companies control the compute
Jeremy Gilbertson Quote from the essay, just three companies control over 90% of advanced AI chip design overwhelming overwhelmingly in California, with the vast majority of the minerals coming from China, tools from ASML in the Netherlands, and manufacturing from TSMC in Taiwan. Now, there are other groups doing things in that world, but that that really kind of explains.
super simply and there's it there's way more depth to the entire system but this the supply chain, you know, that is yeah, that's the thing that you gotta you gotta kind of kind of wrangle,
Mark Fielding It's tight.
So obviously with all of this you need huge amounts of electricity as well, which is we bring in Dennis Hassabis, who calls for the equivalent of the the International Atomic Energy Agency, which has had inspectors for decades. Maybe there could be something with AI that monitors either the supply chain or the electricity usage.
An IAEA for AI
Jeremy Gilbertson Yeah, and the argument here too is, you if the know IAEA, International Atomic Energy Agency, you know, can inspect military properties, can, get involved in, state military stuff to, figure out if people are doing the right thing, what's stopping us from doing that related to intellectual property of a company, right? If you can do it
From a military standpoint, you'd you'd you'd tend to argue, or I think that's this where Will's kinda tending to argue that it's easier to do from a company perspective.
Mark Fielding Quote, inspecting a frontier AI lab involves company intellectual property rather than state military secrets, a political economy of compliance that is considerably more tract tractable than the inspection of military installations. you spoke about trust. I think it's easier to get everybody on board when the the downside is so obvious.
The nuclear bomb goes boom, the nuclear bomb doesn't go boom. It's binary. AI, when there are so many scenarios and we don't know any of them, and we don't we can't imagine others. It's a lot that's a different political pay hot potato to wrangle. You can go to any country in the world, that's why it was that's why it's it worked, because this is what happens if we don't get this right. Whereas the AI, this is what happens if we don't get it right, is much more.
abstract, it's much more imaginative. We don't know the answer so readily. So it's politically much more harder to get people cross party agreement, let alone international agreement.
Jeremy Gilbertson it's really interesting that, you know, the the CEO of a company called Planet that's actually helping to create a real-time global state of the rock we're standing on, trying to push forward an agenda of we have to think globally related to this. So
He references some advances, some things that are happening in verification that are that are pretty interesting. Some of them are, quite simple and we've we've heard about Other ones are a little bit more interesting that wanna dive into further. export controls, we've we've heard a lot about this in the news, right? US and Netherlands having controls on the the advanced AI chips,
The other ones, I'm jumping across this list. the UK has got some stuff rolling, inspection is actually happening.
The UK, UK-ASI, UK's AI Safety Institute. So these are voluntary evaluations of frontier models, of which, according to Will here, Google and Anthropic have voluntarily submitted to that. So not sure what that means or or how aggressive those peaks under the curtain are. But hey, congrats to the UK.
Mark Fielding don't mind if I don't get too excited about voluntary evaluations of frontier models coming from the UK AI Safety Institute. I would like to see the results before I get too excited about that. That was number four. You forgot number two, verification architectures. The RAND Corporation, which helped to develop nuclear arms control, has proposed a six-layer verification architecture.
AI constraints, monitoring devices embedded in AI training chips, software inspection protocols capable of detecting capability concealment behaviors. I like the sounds of that, systematic tracking of data center construction and energy consumption, which we alluded to earlier, and whistleblower programs with legal protections, hardware export controls, and formal international inspection regimes. that sounds very much like the
I A E A.
Chips that watch themselves
Jeremy Gilbertson let's stay with that one because I think to me that's the most interesting because it's the most tangible. It's the most Rand has done this before with nuclear, So they're creating that creating that framework. So I did a very basic research on this. I think we should do a whole nother show on the actual framework. But according to, my quick research on this, the the verification falls into a few different layers, layer one being on chip.
Kind of meaning the chip watches itself, Built directly into the chips. and let's see. Layers two and three is off chip, So external devices are watching the chips, network taps, analog sensors, that sort of thing. And layers four through six are personnel based. Basically, people watch the people, which is humans watching humans that are watching these systems, So whistleblower programs, interviews, natural national intelligence activities, that sort of thing. But
The chip piece I think is a little interesting. I'm I'm using that word because I don't have a better one at the moment, but
All of these chips will have this little thing on them that allows a third party to get into the chip and figure out what the chip's doing. Again, who is who decides on who's gonna be the peaker into the information on the chip? And it goes back to humans being the biggest challenge and problem in a lot of this stuff. So.
Limits on autonomous replication
Mark Fielding But I also think that links nicely into the third one, the limits, because who decides? Well, what if AI decides that nobody's going to have a peak? So the international dialogues on AI safety, the IDAIS, involving senior researchers from the United States, Europe and China has produced a set of consensus red lines for such a regime. Prohibitions on systems capable of recursive self-improvement without human oversight.
on systems that can autonomously replicate and acquire resources and on AI-enabled assistance in the development of biological, chemical, or nuclear weapons. So I think there's long four ways that we could get a handle on this. And as Will says, yes.
It's hard, but it hasn't been attempted yet, and nothing good ever comes easy, so
Gotta do something. Let's do it. Come on, countries of the world, unite!
Jeremy Gilbertson Yeah, it's and and that's exactly what he's advocating. It you know, he he actually puts it in here like a lot of people just pontificate about, you know, hey, this is what's happening and this is the world and this is where it's going. And but this is like, no, here's what you should do. Step one. Step one, US, China, put an agreement in place.
Mark Fielding Yeah.
Jeremy Gilbertson that will focus on some of the core principles that are of major concern with AI safety. Easier said than done. Couple of egos involved in that in that conversation on a one to one ratio and then also the race agenda that is that is powering that and has powered the relationship between those two countries for for such a long time.
Mark Fielding D does the the Cuban Missile Crisis, the does the AI Cuban missile crisis equivalent lead to this sentence here? The first priority should be an agreement between the United States and China. Okay. Yes, you can get some surface-level political agreement, but you could push to that. But you sure as shit aren't getting
A level of trust and a level of agreement and a level of what is necessary right now between China and America. And I don't care what you say. I've I've been watching global politics for too long to think that that's gonna happen. So do do we need that Cuban missile crisis to make this happen?
Jeremy Gilbertson Yeah, it's the way I think about it, We do a great job of coming together when shit hits the fan.
Mark Fielding Yeah.
Jeremy Gilbertson Like in a really special way, but the the strength of those connections, they're they're a little bit fleeting. we come together, there's something that's unfair, someone was done wrong. We come together, we protest, we sh we share our voices, we we advocate within organizations for the right thing. But then that quickly those that connection quickly kind of dissipates.
And we're kind of back in the swing of, hey, I've got to do my thing. I've got to, keep my head down to to to handle my business. this might be a this might be one of those problems that creates sustainability of that connection. And here's here's the issue. And I I'm all over the place on this, Mark, but trying to trying to processes it, process it in in real time. There's also we talk about the disconnect between the use of the technology from an individual human perspective. Like I'm, I'm on my phone using Claude, and trying to, don't know, book a vac have it book a vacation for me or, you know, whatever, you know, the simplest way to use that is. And there's just very little understanding of like what all the behind the scenes stuff after you push enter, where that kind of query goes back and and the clusters are talking and things are coming back, like the understanding of the capability behind hitting that button is being masked by the convenience of how that button is super helpful.
the separation between the tools that we use to make our lives easier, better, faster, more dumb in some ways. we don't we don't have the visibility into the extensions of that.
Mark Fielding Easier.
you wanna turn the the the the dial back to the the the good side. But before we get there, Jeremy, I wanna ask you a question.
Very high level. Let's summarize. AI is good and bad. The next iterations of AI, AGI, ASI, could be good, but there is a small percent that they could lead to existential risk. So here in this essay, they're providing okay, well, well, yeah, I I don't buy into those percentages. I'm an optimist. But okay, so there's a a percent chance that it leads to the end.
Jeremy Gilbertson Not a small percent, ten to fifty percent. That's a big percent.
What happens when someone does not buy in
Mark Fielding So Will Marshall here is proposing something similar to the the way that we safeguard humanity and the earth from nuclear weapons.
And he uses he uses reference to the the Cuban Missile Crisis. We go to war over that. We are at war right now because of that. The Cuban Missile Crisis almost led to a world war. At some point, we're probably going to go to war with North Korea because of that.
If you draw the similarities there.
What happens when somebody doesn't buy in to this framework? What what hap what the AI equivalent of Iran, the AI equivalent of North Korea? What happens then? Do we go to war? If we're using this framework, that's the natural that's how we defend it.
Jeremy Gilbertson And who do you go to war with? You know, this this
Mark Fielding Whoever whoever the rogue state is, whoever the rogue player is.
Jeremy Gilbertson it could be not even nation states. It could be it could be two people, it could be one person, right? Like
Mark Fielding No.
It'd be very that'd be a very short war.
Jeremy Gilbertson No, I mean like just the just the ability to not everyone can spin up nuclear capability. People can spin up AI capability. And that's gonna be that that turns the whack-a-mole into like an infinite many worlds whack-a-mole. how do you manage that?
Mark Fielding get somebody with more a little bit more than thinking on paper to solve it. Green shoots.
I like this. Rather than focusing on what could go wrong, we pour money into what could go right. We should actively invest into AI benefits, especially into areas where AI can have disproportionate benefits. I like that disproportionate benefits. This might include planetary intelligence. Ooh, don't know where we got that one from. Planetary security, adaptive collective intelligence, enabling organisations across corporations to countries to become vastly more efficient.
Jeremy Gilbertson like this but when I first read it I was just like geez okay here we go like invest in planet invest in planet I mean but you know like it it also goes back to this global vision right having a real time global vision and and that's what this stuff kind of does and and it
It's an advert for Planet
Mark Fielding It's really a market it's an advert for planet.
Jeremy Gilbertson You're right.
Jeremy Gilbertson advocating for us to think more like that is is is interesting. I think even if you overly invest in the AI benefits, the other ones aren't gonna go away and they're gonna be just as scary, yes, you're gonna have, know, AI doing good things, but you also have this looming, unaddressed thing that keeps getting swept under the rug and before you know it, it's freaking super dangerous and no one's thought about how to deal with it. And then more
Yeah, how does the world how does the world if there's no coordination between nation states organizations, companies around this, when shit really hits the fan, how does it all we all just like get in a fucking conference room and just be like, yo, all right, all right, Jimmy, what do you think we should do? we're not gonna have that much time to react, I think, is the big thing. it's gonna be like, shit, there it goes and it's done.
Mark Fielding Yeah.
It it all goes back to me to one of the the words of thinking on paper, we have trust, but I think this is all about incentives. a lot of AI is about incentives. And if you talk about the the difficult problems that AI can help us with, conflict, food security, water security, climate change, displacement, economic inequality, they are the hardest challenges that we have.
It's really fucking easy to get AI to make your spreadsheet look better. And it's there's a lot of money in that because there's a lot of spreadsheets. It's really easy. It's really quick. The incentives are there. And the incentives to do something about this.
Fucking weather heat bombs like that. The incentives aren't there because the the incentives are long term and even though those incentives are the survival of us.
Jeremy Gilbertson You talking about global warming with the the fucking weather heat bomb? Is that what is that what you meant by that?
Mark Fielding I believe it's called climate change, global warming. It's not the nineties anymore, Jeremy Next should be talking about there's a hole in the ozone layer.
Jeremy Gilbertson Well it my god, dude. No, it y you talk a lot about incentives, Mark. You do. And I think a lot of action gets taken once incentives are put in place. And if you have incentives, you can in you incent the behavior that you want to have happen. this goes from like simple as coaching, right? Coaching a team. You put incentives pla in place.
to have the behavior happen the way you want to have happen. In organizations you do the same things, bonus structures, this and that, KPIs, all of that. but this becomes global KPIs. This becomes like this this becomes this becomes like humanity KPIs. This becomes like
Mark Fielding Google AI KPIs. This is what global AI KPIs.
Jeremy Gilbertson bigger the the the highest of architectures that you could have in human systems. This is where this is the root folder where this shit should live in, right? Like
Mark Fielding Yeah.
Jeremy Gilbertson But who's making who's making room for the root folder and who's standing up and saying like, hey, you know, all of this stuff, all of this stuff has has significant repercussions, though we can't immediately identify those yet, but they are. They're there. They're documented. Researchers have science shown it. Yeah, researchers show it. Alan Turing said it in fifty one, for Pete's sake. who's gonna advocate for it?
Mark Fielding Luckily, there are some very, very incredible people. And we speak to them a lot on thinking on paper. And they are drawn like a moth to the flame, to the hard shit, to getting the hard stuff done. Last week we spoke to somebody, we're gonna go mine asteroids. Why? It's really hard, but I wanna do it, we're gonna do it. Like people have been drawn to these almost impossible tasks, and that
And I always talk about incentives, but I always come back to the the human spirit, the human endeavour, the human resilience to get hard stuff done and there are people out there doing it. And good luck to 'Cause yeah.
Jeremy Gilbertson And we want to talk to more of
Talk to more of them. All right, let's land the plane on the fur no, on the Fermi paradox. Let's let's land the plane on how Will Marshall lands the plane in this essay about the Fermi paradox.
Mark Fielding Cuba
Hello.
Go on then.
Jeremy Gilbertson you want me to go?
The Fermi paradox
Mark Fielding What's the Fermi paradox, Jeremy?
Jeremy Gilbertson Well, we we we talk i let me set the stage a little bit by in throughout this whole episode and a lot of the threads on thinking on paper, the forces and the currents that we talk about on Thinking on Paper talk about tech moving faster than our ability to wrangle it, right? Wrangle it is a loose word. It means govern it, incentivize the right behavior within it, all of that stuff. So the Fermi paradox talks about there's infinite amount of planets with similar, you know, structure to our planet, infinite stars, infinite suns all across the universe, why haven't we seen anybody yet? And it's really interesting that what he I'm gonna r let's just read the quote and make it easy. Quote, one disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it.
destroying itself or returning permanently to something like the Iron Age. Maybe that's why we haven't seen other species out there because they've gotten to an inflection point where they've created a technology so freaking powerful that they don't know what to do with it and it totally implodes on them.
Mark Fielding Yeah.
Jeremy Gilbertson Wow. So that's that's the that's the doom side. Let's swing it to Utopia before we wrap it up, Mark.
Mark Fielding Well, I I just the th the Fermi paradox is space is really, really, really, really, really, really, really, really big and you can't go faster than the speed of light.
And you just can't travel across the cosmos. And that's why we haven't seen them, because they just haven't got here yet.
Jeremy Gilbertson Well, I'm gonna challenge you on that, Mark. According to some of the research that we've done, you can travel faster than this information can travel faster than the speed of light in quantum systems, but it's not intelligible information. It's
Mark Fielding Go on then.
no but no no because you it's it's yeah, it's not information, is it? It's it's it's not it's and unless the aliens are actually made of then they they can't
Jeremy Gilbertson my gosh. Rabbit hole achieved.
There you have it. Will Marshall, thanks for writing this piece. Thanks for getting us thinking about it. If you want to read the piece itself, The Hard Question of AI, an urgent ask to steward superintelligence, Will Marshall, CEO of Planet, who will hopefully be joining us on Thinking on Paper to talk about pelicans.
Mark Fielding www.thinkingonpaper.xyz for the convergence, the technology convergence, all our shows on quantum AI, robotics, bioengineering, biotech, society, music, love, culture, movies, cinema, and the umbrella of technology. They're all there. Go pull a thread and listen to the episodes that you want to listen to. Follow your curiosity. Like, subscribe and leave a comment.
And tell us, remember, what is the AI Cuban missile crisis equivalent?
Jeremy Gilbertson Mark is being super humble. His new version of thinkingonpaper.xyz is a choose your own adventure that allows you to bounce between categories of technology, explore the threads between them, do exactly what we're doing on a daily basis on this show, but you can pull the threads. You can see the map. You can see the knowledge graph that he's built. It's the most interactive website I've seen on podcasting platforms.
Dude, check it out. Do yourself a favor thinking on paper.xyz. Until then, stay curious. Be disruptive.
Mark Fielding Keep thinking on paper.




