What If We Get AI Right? Dario Amodei’s Machines of Loving Grace

Dario Amodei guesses that powerful AI could give us the next 50 to 100 years of biological progress in five to 10 years. How long did it take us to assimilate the car?

Clips and outtakes on Instagram ›

In this episode · 20 chapters

  1. Who Dario Amodei is
  2. Where the title comes from
  3. Defining powerful AI
  4. The factors that complicate things
  5. The need for data
  6. Intrinsic complexity
  7. Engineered serendipity
  8. Compressing a century into ten years
  9. Neuroscience and mind
  10. Doubling the lifespan
  11. What winning looks like in neuroscience
  12. The flywheel of corruption
  13. Peace and governance
  14. Atoms for Peace, and who leads
  15. If AI solves everything, what is left
  16. Meaning, and the economic problem
  17. A capitalist economy of AI systems
  18. How long did it take us to assimilate the car?
  19. What is meaningful work?
  20. Closing thoughts
You don’t stop doing something just because you’re not the best in the world. Whatever your hobbies or interests are, you do them because they bring you some kind of meaning. They bring you some kind of pleasure regardless.
Mark Fielding

Mark Fielding and Jeremy Gilbertson read Machines of Loving Grace, Anthropic CEO Dario Amodei's essay on what a world with powerful AI looks like if everything goes right. He sums up powerful AI as a country of geniuses in a data centre.

The hosts test his claims for biology, neuroscience, poverty and governance against limits he names himself, from the need for data to intrinsic complexity. Then they ask how people find meaning, and who pays for it, once AI does the work.

Machines of Loving Grace, the October 2024 essay by Anthropic CEO Dario Amodei, asks what powerful AI could do if everything goes right, including 50 to 100 years of biological progress in five to 10 years. On the Thinking On Paper podcast, Mark Fielding and Jeremy Gilbertson accept parts of the science case but doubt that people and institutions can change that fast.

How does Dario Amodei define powerful AI?

Amodei avoids the term AGI and defines “powerful AI” instead: a model smarter than a Nobel Prize winner across most relevant fields, one that can work on its own for hours, days or weeks and run as millions of copies. He sums it up as “a country of geniuses in a data center”.

What are the marginal returns to intelligence?

The essay's framework is the marginal returns to intelligence, meaning how much faster a problem gets solved when you add more intelligence. Amodei lists what limits those returns: the speed of the outside world, the need for data, intrinsic complexity, human constraints and physical laws. Mark's example: AI can help find a cancer drug, but “The tests still have to be run.”

What is the compressed 21st century?

Amodei calls this a “compressed 21st century”, where AI multiplies the rate of the few big discoveries that drive biology. Jeremy notes the essay's CRISPR example, an idea known since the 1980s that took 25 years to apply. Mark reads Amodei's list of what could follow if all goes well, including the elimination of most cancers, prevention of Alzheimer's and a doubling of the human lifespan.

What does Amodei say about AI and the developing world?

Amodei writes that if AI makes rich countries richer while doing little for the developing world, “we should view that as a terrible moral failure”. Mark argues that history gives no example of the powerful sharing this kind of progress, and calls universal access “a pipe dream”.

Will AI advance democracy, according to Amodei?

Amodei is least optimistic about peace and governance: “I see no strong reason to believe AI will preferentially or structurally advance democracy”. Mark reads his entente strategy, in which a coalition of democracies tries to gain a clear advantage on powerful AI by securing its supply chain, scaling quickly and blocking or delaying adversaries. Jeremy compares the moment to Eisenhower's Atoms for Peace and asks who would run global oversight of AI.

Where Mark and Amodei disagree

Mark accepts that technology may move 10 times faster but says society cannot. “But on the other hand, we're talking five to 10 years. It doesn't compute.” He asks how long it took us to assimilate the car, and calls Amodei's prediction of curing most mental illness in five to 10 years “a very ambitious, very, very ambitious claim”.

Questions

What is Machines of Loving Grace about?

Machines of Loving Grace, the October 2024 essay by Anthropic CEO Dario Amodei, asks what powerful AI could do if everything goes right, including 50 to 100 years of biological progress in five to 10 years.

Who wrote Machines of Loving Grace?

Mark Fielding and Jeremy Gilbertson read Machines of Loving Grace, Anthropic CEO Dario Amodei's essay on what a world with powerful AI looks like if everything goes right.

What do Mark and Jeremy make of Machines of Loving Grace?

On the Thinking On Paper podcast, Mark Fielding and Jeremy Gilbertson accept parts of the science case but doubt that people and institutions can change that fast.

Transcript

Mark Fielding disruptors and curious minds, CEOs, founders. Welcome to Thinking on Paper. I'm Mark, this is Jeremy, and this is Pocket Edition, a smaller, more concise, more succinct exploration of one particular article, essay, news story, controversy. Today, we're going back to October 2004, and Dario Amodei, the CEO of Anthropic. Jeremy.

What do Anthropic, why are they important? Why is what Dario and Moody says important just before we explain what we're going to be doing?

Who Dario Amodei is

Jeremy Gilbertson Yeah, so he's been at OpenAI. He was the head of research, think, at OpenAI early, VP of research. So him and one other guy were kind leading the research charge over there. He left to start Anthropic. Anthropic is Claude, right, for those messing around with the models and stuff. And what caught my attention with him recently is just the things that he says about AI are really interesting. And he tends to lean, at least what he's saying outwardly, leaning more toward the humanity side of things. And one thing that caught my ear during an interview that I listened to was that he thinks like the number one skill for people moving into the age of AI is open-mindedness. I think that's pretty powerful. think AI, if there wasn't even AI, that would be like the number one thing. But that caught my eye and then kind of there lived the rabbit hole. We found this essay, Machines of Love and Grace. Again, a great title.

We've been exploring the difference between silicon and carbon consciousness and machines. And I was like, man, this could be a really interesting thing to unpack. So that's what sparked my curiosity.

Where the title comes from

Mark Fielding think it's a good one. think that, well, it's an essay called, as you said, Machines of Loving Grace by the CEO of Anthropic. It's the premise of the essay is the upside of AI. It's like, would a world with powerful AI look like if everything goes right? And that's the key of the essay is everything goes right. It's a long essay. I'm gonna read the poem on which it's based, which is by, it's based on a poem by Richard.

Brautigan, a key figure in the counterculture movement, belongs in the same lineage as the beatbixers, Carowack and Burroughs, Ginsburg, that kind of era. And he wrote a poem called All Watched Over by Machines of Loving Grace back in the 60s, I guess. I'm gonna read it. I like to think.

Jeremy Gilbertson Wait, wait, wait, are the Dharma Bums back?

Mark Fielding The Bahama-Dhama-Bhums are back. The Dhama-Bhums are back again, yes. Nothing is new. Everything is repeated, isn't it? It's just echoes and echoes everywhere. There is nothing new. Ideas just repeated. Anyway, all watched over by machines of love and grace. I like to think, and the sooner the better, of a cybernetic meadow where mammals and computers live together in mutually programming harmony.

like pure water touching clear sky. I like to think right now, please, of a cybernetic forest filled with pines and electronics where deer stroll peacefully past computers as if they were flowers with spinning blossoms. I like to think it has to be of a cybernetic ecology where we are free of our labours and joined back to nature, returned to our mammal brothers and sisters and all watched over.

by machines of loving grace. That's where the title from the essay came from. The premise, what will a world of powerful A look like if we get everything right? Number one, what's the first thing he gets into?

Defining powerful AI

Jeremy Gilbertson Well, the thing I want to start with on this real quick is, you know, a lot of people refer to what he's calling powerful AI as this idea of AGI or singularity kind of thing, right? And he calls it, he's very distinct about calling it powerful AI and he talks about defining intelligence. And as we kind of walk through it, he thinks about intelligence as problem solving.

across domains, right? So this ability to transverse different domains to solve problems. And he talks about the idea of this powerful AI being smarter than a Nobel Prize winner in all fields. Being this place where it's essentially, this quote was interesting, a country of geniuses in a data center.

Mark Fielding Okay, well, let's read the whole, like his actual definition of powerful AI from the essay. So we know where we're basing this. So by powerful AI, I have in mind an AI model likely similar to today's LLMs in form, though it might be based on a different architecture, might involve several interacting models and might be trained differently, but it has the following properties.

In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields, biology, programming, math, engineering, writing. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult code bases from scratch, etc. In addition to just being a smart thing you talk to, has all the interfaces available to a human working virtually, including text, audio, video, mouse and keyboard control and Internet.

access, it can engage in any actions, communications or remote operations enabled by this interface. It does not just passively answer questions, instead it can be given tasks that take hours, days or weeks to complete and then goes off and does those tasks autonomously in the way a smart employee would. It does not have a physical embodiment other than living on a computer screen, but it can control existing physical tools, robots or laboratory equipment.

and the resources used to train the model can be repurposed to run millions of instances of it. Each of these million copies can act independently on unrelated tasks, or if needed, can all work together in the same way humans would collaborate, perhaps with different subpopulations, fine-tuned to be especially good at particular tasks. We could summarize this as a country of geniuses in a data center, to get to your point. I think it's important to clarify what he means so we know.

what the results of that are, which he gets into. And I'll just say those from this essay. He wants to use that powerful AI to get to a world, a great world. And he's going to be focusing on the positive applications of AI across five domains, biology and physical health, neuroscience and mental health, economic development and poverty, peace and governments and work and meaning. Scene set.

Jeremy Gilbertson There you go. So this is a quick, you can edit this out. How do you, so I was just thinking like, when we unpack this stuff, it's important to read some of it, but are people gonna get, are they gonna push the pause button if we're just reading like a whole giant chunk of stuff?

Mark Fielding Sometimes, but I think if we're going to talk about an essay, need to follow the essay, we need to kind of...

Reading that essay, I thought it was important to say what the CEO of Anthropic says is powerful AI. And I think it's important for me to go through each of the five domains he speaks about and say what's going to happen. I don't think we can challenge him or speak about it if the audience doesn't know what we're talking about.

Jeremy Gilbertson Fair enough, fair enough. Just a thought is.

Mark Fielding But you're right as well that like reading too much of it is definitely going to be boring.

Jeremy Gilbertson Because a lot of people are trying to get to the easy button of things with it. But no, I'm with you. I think we can balance it. I think we can balance it. Just something I was thinking as you were reading the passage.

Mark Fielding Okay, well, since I've read that passage, don't, like, so I say, and Jeremy, you can summarise that, what's the easy button? And then it's a country full of geniuses or whatever.

The factors that complicate things

Jeremy Gilbertson Yeah, no, I think we're good. I think we're good. All right, so he casts out this vision, this real definition of what he calls powerful AI, right? And I think it's cool to think about it that way. He talks about two extremes of where people think this whole thing is heading. And I think what I gathered from what he was saying, it's not.

as with many things, it's not this or that. There's various stages of in-between stuff, right? So he talks about the one extreme being the singularity, which is humans are kind of out of the loop. The machines are smarter. They're doing everything, and we're doing our best to kind of hang on, right? And then the other piece of it is, hey, tech's not going to progress that quickly. There are significant limitations and that sort of thing. So...

Between that, he suggests some new frameworks, I think, that are really interesting. And I thought the marginal returns to intelligence was pretty cool. It's almost like ROI, not return on investment, but return on intelligence, where sometimes the human side works really good, and sometimes the human side could be complemented by an AI to generate even more returns.

Let's talk about the factors that are complimentary to intelligence that he listed. Did you run through these like the speed of the outside world, need for data, intrinsic complexity, human constraints, physical laws? You want to start through that?

Mark Fielding Yeah, I did. Yeah, I got all I got those.

Jeremy Gilbertson Speed of the outside world is interesting.

Mark Fielding Okay, so I think we need to be clear. So now I'm just thinking on the show, like, so we've gone, so we go into this, how do we frame the marginal returns of intelligence? I think it's really interesting. We should focus on it. How do we like...

Jeremy Gilbertson I, in my mind, it was a, it was an addition of a new framework to, to, to re-envision these two extremes of where AI is headed. You have the singularity extreme, have the, yeah, tech's not gonna get their extreme good luck, right? And this is a framework to look at all the in-between phases of that. That's how I saw it, but if you're seeing it different, we can approach it differently too.

Mark Fielding I agree with you. think that it's essentially things we get wrong or things we don't take into account when we're talking about the future of AI. Is that what the marginal returns of intelligence are?

Jeremy Gilbertson Well, it's basically like how much is AI helping us generate the return on intelligence? How much is AI helping this human process? Because some things it can be very helpful with, some things it isn't helpful with.

Mark Fielding So kind of constraints. he said he speaks about the speed of it, like you can't the speed of the outside world. So AI for whatever it can be capable of doing, it can't break the laws of physics. It can't bend the laws of time, however long it takes to test drugs for cancer, for example, you still have to test them. AI can't warp the

Jeremy Gilbertson Right, right. So maybe, so maybe, maybe, maybe when we cut back and edit this, we look at those as, as, like you said, more the, kind of the limiting factors, the constraints.

Mark Fielding I mean, it's got to work in a framework of reality.

The need for data

Mark Fielding limiting factors of what... so like need for data...

Jeremy Gilbertson Okay, let's, let's refer, let's, let's recast that then. So, so as we talk about the marginal returns to intelligence, there are definitely some constraints that he outlines that are helpful to think about as, as AI continues to improve and become applied and that sort of thing. And he lists these out. Let's, let's go through them kind of one by one really quickly and kind of talk through those things. And then we can hit on all of these five areas that he thinks.

would will be impactful in at the intersection of AI and humans. So speed of the outside world. So

These are things that the intelligent, any applied intelligence needs to rely on to act and do their thing, right? So you have things like, you know, you have slower systems, like human systems that are involved that will slow down what AI can do super speedy because we're the bottleneck, right? So think about current systems, think about hardware.

Think about gaining consensus, how hard it is to gain consensus in a giant group of people. Social systems are a bottleneck, right? So no matter how fast we get, when we apply AI to solutions to social issues and social, like societal issues, we're still gonna be the bottleneck, right? That's what I read when he talked about speed of the outside world.

Mark Fielding Yeah. And the same thing applies when you're using this, the AI to solve biological problems. For example, you want to find a cure for cancer. AI is going to help us find a cure for cancer. AI can't change the laws of physics. The tests still have to be run. The experiments still have to be run. The results still have to be found. They're still, and what he calls like an irreducible time for certain things. And AI can't change the laws of of that timing. on the, as you said, on the cultural front, have us people, we're like oil tankers, you can't move us on mass, it takes time. And when you're doing using AI to solve medical issues, there are two entire limits that it cannot supersede. And I think it's very important in the conversation when everyone goes, AI is going to save the world and AI is going to save climate change and AI is going to do this and do that. It has to work within the limits of what can happen. And as we've spoken about a lot on this show, Jeremy, you can't.

Jeremy Gilbertson the invented construct. So let's talk about data, right? What's the fuel for all of these models, right? It's data, right? And not all data is created equal. There's good data, there's bad data, there's wrong data, right? And parsing through that to make it a good input for the model is pretty challenging. So is there gonna be a limit to data? Like I read through this, he's like, seems like there's gonna be like a limit.

Mark Fielding Need for data, yes.

Jeremy Gilbertson to data. What did you think about that?

Intrinsic complexity

Mark Fielding Yeah. And I think that the example he uses, he uses for physicists, particle physicists and particle accelerator data is limited. And I think that ties back to speed of the outside world. There's only so many times you can bash atoms around a particle accelerator and get the results from that. So data is limited and that connects nicely to the speed of the outside world.

intrinsic complexity. I know that this is your favourite specialised subject, Jeremy, complexity.

Jeremy Gilbertson Well, is this complexity as defined by Julio Ottino, complicated versus complex? I think it aligns with what he thinks about that, right? So yeah, what I wrote here is AI is marginally better at working on chaotic systems like the three body problem. So this goes back to return on intelligence. If we point AI at problems that it's currently not better than we are at solving

Jeremy Gilbertson That's you're not getting the best return on the use of that AI, right? defining, so humans in the loop, humans in the loop again, pointing AI to the right problems that it can be successful at solving quickly, efficiently, all of that kind of stuff, right? Is that where you landed on this?

Mark Fielding It's funny, I was thinking of Nexus as well with Julio and I remember one of the book club episodes thinking on paper.xyz to watch that. We spoke about complex systems and it was this idea of, was, I was a shoal of fish in a very small area of seawater off the coast of Canada and the interactions between these fish and the food chain and how quickly it just exploded exponentially into mind bending numbers in like, it was literally a few square meters of water and AI just simply cannot process this equals that equals that to a ridiculous amount needed. Although I do bring into that this the intrinsic complexity returns to intelligence, which he mentions as a hamper to what could possibly be achieved with AI. I have two words for him. Quantum computers.

Jeremy Gilbertson There you have it. Yeah, so the other factors we kind of, there's a little repetitive nature between human constraints and physical laws and speed of the outside world. They all kind of relate, but I think that the need for data and intrinsic complexity kind of stand on their own a little bit. Let's dive into the first of these five areas, right? So biology and health, we've always heard about the idea of this, right? Like making new drugs is a very brute force process.

Jeremy Gilbertson You know, it's, test this thing, push it through, see what happens, test this thing, push it through, see what happens. And then there's also the bureaucracy on the other side of it, you know, that, that helps, I guess, helps make sure the right stuff gets pushed through in theory. Right. so there's this interesting, interesting balance of that, but, he goes into the idea of like AI really helping these structures with, you know, alpha fold and alpha proteo, right.

So basically doing this very complex mapping and understanding of proteins and then even creating new proteins. Is that what Alpha, Alpha Proteo does?

Mark Fielding I think that's what they're looking for. I didn't look into that, but I assume it can. We get the obvious in biology and health, and we get AI is gonna cure cancer. It's gonna make us all live longer. I think perhaps he's been drinking the Brian Johnson Kool-Aid and the Silicon Valley obsession with longevity. Maybe it gets more importance here than it perhaps would elsewhere, maybe not.

What I found really interesting is how explains why the AI will be so net positive and very quickly. we probably should have mentioned this earlier, but in Machines of Loving Grace, he's painting a very short window of like five to 10 years of what, if this goes right, AI will produce. So we're not talking 100 years out, we're talking five, 10 years out for this. And it was very interesting to think of it like this. He says that essentially, historically, it's a very small number of discoveries which have created the biggest impact. Like you have one small discovery and it just, it's a tidal wave of impact everywhere else. And so he mentions some of those discoveries. I just meant, I just, no.

Engineered serendipity

Jeremy Gilbertson Well, yeah, these discoveries, major discoveries each year drive, I think you said 50 % of the progress in biology, which is really interesting, right? So examples of these could be, you know, CRISPR, which, which has been out there. This was interesting. I didn't know that, that, that the idea of CRISPR was known since the 1980s, but it took 25 years to apply it. It's like this, this idea kind of has to sit and have its aha moment, but you know, what, what

Mark Fielding Maybe it doesn't with AI, that's the thing. Maybe there's lots of discoveries like that, which are sitting under our noses and we just don't have the, haven't clicked yet. We haven't found the little missing ingredient. AI will help find that missing ingredient. He talks about speeding up the discovery of the big discovery. So we have like five or 10 discoveries that have changed the world. Genomics, for example, DNA sequencing, and that changes all these other thing. AI can, you know, speed up 10X that and...

get those aha moments, those eureka moments more easily.

Jeremy Gilbertson engineered serendipity is a really interesting thought. he's, yeah, he's thinking like, this could be 10x faster with AI, some of these things, right? And the returns to intelligence are high for things like this. And we have this, the idea of serial dependence, this was really, really cool. You mentioned parallel, right? So in science, a lot of it is as he outlines in here, and I'm not a scientist, but I'm just kind of.

Jeremy Gilbertson taking what he's writing and with what I know about science, but you need to make discovery A in order to have tools or the knowledge to create discovery B. And what he's saying potentially is there could be a lot of discovery A's happening at the same time to push discovery B's along and basically automating the ahas or automating the potential of ahas by increasing the potential.

Compressing a century into ten years

Mark Fielding Yeah, but yeah. Quote, it's my guess that powerful AI could at least 10x the rate of these discoveries giving us the next 50 to 100 years of biological progress in five to 10 years. And then why not 100x?

Jeremy Gilbertson of those aha moments. Yeah, I thought that was pretty interesting.

Jeremy Gilbertson Yeah. And there's, he says that I'm loosely paraphrasing. This isn't a definite quote, but AI can improve it, finding the connections between the available knowledge that we have in biology. know a lot of things about biology, but we haven't smashed all of the things around at scale, I think. And, and I think that's a, that's an interesting concept. And you know, the idea of the compressed 21st century is what he refers to in here. And it's what you're saying instead of something happening in 50 to a hundred years, we're, creating a scale where that could possibly happen in five to 10 years.

Neuroscience and mind

Mark Fielding Just quickly before we move on to neuroscience and mind for anybody wondering exactly what he thinks we got in the next five to 10 years, if all goes well, reliable prevention and treatment of nearly all natural infectious disease, elimination of most cancers, very effective prevention and effective cures for genetic disease, prevention of Alzheimer's, that's a big one, improved treatment of most other ailments, biological freedom, doubling of the human lifespan. So yeah, good news.

Jeremy Gilbertson Biological freedom was a little,

Face melting maybe a little bit. It's just the idea of like, man, are we all gonna be able to become Marvel superheroes down the road? That's where my head immediately went with that. But I thought that was the one that was like, ugh, that's kinda.

Mark Fielding Is that, I know he gets into the culture layer, but it's about augmenting humans. Like he doesn't say you can choose what you want to be. So I want to be bigger, faster, runner, smarter, you know, okay. Upgrade, it's a bit, bit matrix-y.

Jeremy Gilbertson Yeah, a little bit, a little bit. And it's funny because he talks about the idea of we need to, and we'll get to this later. He talks about the idea of needing to steer away from these science fiction narratives. That sounds like very science fiction narrative-y a little bit. But we talk about the doubling of human lifespan. It's actually interesting. He casts out that like this doubling has happened. Like it's happened in the past. It's, you we've already doubled our lifespan from 1900 to 2021, according to what he says in here, which which is pretty interesting, right? So why does it have to stop there? Who's saying that we've hit the limit of that? But then again, it turns up like if we push our lives out to 200 years, we still have the problem of meaning and purpose that people haven't hit yet.

Doubling the lifespan

Mark Fielding So we need an expert on that. We've doubled the lifespan because I think like we've doubled the lifespan in the last hundred years is dishonest to what actually has happened. basically people don't, it's not that they live longer. They just don't die as early on average. So obviously the hand washing.

Jeremy Gilbertson What's the difference between that? Live longer versus dying early.

Mark Fielding Well, statistics, isn't it? You can make a statistic tell you what you want. And statistics says like the average lifespan has increased twofold because people don't children don't die in childbirth at the numbers they used to. Disease doesn't kill people like it used to. People don't die on average as early as they once did, which makes the average lifespan appear much longer on average than it was. Like if you were born a hundred years ago and you didn't die in childbirth obviously and you didn't catch any diseases and you didn't injure yourself and like is your life expectancy well like I'm not an exponent so I'll delete that bit out but yeah.

Jeremy Gilbertson Isn't that what he's saying though? Isn't that like solving the things that are making our lives shorter now? Like the things that are in the way of us living longer. It's the same thing in my book.

Mark Fielding Let me just... Where has it gone?

Yeah, probably right. I'll delete it out.

Jeremy Gilbertson What did you think about the ratio of working to retired people will change dramatically?

Mark Fielding What was the ratio?

Jeremy Gilbertson He didn't list a ratio, I just wrote down, made a little footnote here, he references the ratio of working to retired people will change dramatically. Is he saying there'd be more people retired as we move into the future or more people working?

Mark Fielding Well, as our age increases, people will be working longer. So there'll be less retired people because they'll be able to work more healthily into their 70s and 80s and 90s.

Jeremy Gilbertson Yeah, or the nature of work will change. You know, does the word retire go away? It's just, does it go from work to meaningful work?

Mark Fielding Well, we get into that, don't we?

Jeremy Gilbertson my goodness. All right. Let's, let's, let's move from all of biology to neuroscience and the mind. basically the same framework he talks about in biology, the, the expediency, the, the, the 10 Xing of progress, applied to neuroscience, right? It really is pretty, pretty similar in there.

Mark Fielding It's like the 80-20 principle on steroids. just kind of that 20 % that creates 80 % of results is just amplified where AI can just notch that up and 10x that. you know, how quickly can find that 20 % is increased. And yeah, the same principles for neuroscience and mind apply as they did.

What winning looks like in neuroscience

Jeremy Gilbertson So what does it look like to win with AI in this realm, right? So he talks about, you know, being able to work with the fine-grained, he calls it, neural measurement, like the things that happen in the brain, and the intervention, being able to interact with that information at scale. think, you know, talk about complexity, talk about intrinsic complexity. know, mood disorders, psychosis, like what happens in the brain when that goes on?

Like we still don't really understand that as humans. I think the mental health aspects and benefits here are going to be tremendous if AI can help solve that complexity. is our mood disorders as complex as the three body problem? And is AI going to be good for that?

Mark Fielding I'm not gonna answer that, no idea. But he says, quote, it's my guess that these four routes of progress working together would, as with physical disease, be on track to lead to the cure or prevention of most mental illness in the next 100 years, even if AI was not involved, and thus might reasonably complete it in five to 10 AI accelerated years. And then, yeah, I mean, it's...

Mark Fielding I'm not qualified to speak about it, but it's a very ambitious, very, very ambitious claim.

Jeremy Gilbertson I thought about this like with a lot of things that happen neurologically, you know, whether it's, whether it's mood disorders or whether it's, you know, just general, you know, focus issues or whatever it is. When our, when we're younger, our brains are more pliable, right? As we get older, these connections become less pliable. think that

I don't know how, where I got this, the note I wrote was like, what if AI helps us.

point our brains back to the more pliable and childlike version of that to reconstruct certain things, right? To be able to reverse engineer and go backwards, time machine in brains. I thought that was something that could be interesting.

Mark Fielding Do want me to leave that in, Jeremy?

Jeremy Gilbertson I feel like we're a mess today, Mark. Do you feel like we're a mess? I feel like we're a mess.

Mark Fielding I think we're going good. know. I think we're getting there. I think that I think we have to be careful with claiming that mental illness that we can flick the easy button on that and just rewind. And I think that that's

Mark Fielding I mean, we've been speaking about quantum consciousness. if, if consciousness is quantum, there ain't no way that you're, you're, you're, breaking, you're solving mental illness in five to 10 years. I don't care what he says about AI. I think that, we've got to be careful with that one, but

Jeremy Gilbertson Okay, well let's go ahead and edit that one out. So, neuroscience, the ability to scale things, make things happen quicker. Human baseline improves is the idea. The general human baseline neurologically, biologically can improve because these aha moments are happening quicker.

Mark Fielding Happy days, Jeremy, so surely this will be available to everybody.

Jeremy Gilbertson Well, there goes the next piece of the puzzle. Access to everything. Will everyone have access to the technologies that let us do this, right?

Mark Fielding Can I hazard a guess?

Jeremy Gilbertson So, man, I think that would be tremendous and great, but like.

those wielding and controlling the powers of a new resource, a new asset, we've seen it over the years, control it and can limit access to it, right? To certain areas, to certain economies and that sort of thing. I mean, he sets this stuff out there that, back to the human problems, right? Back to the human problems and he says we gotta put the work in to fix the human problems that will let AI do.

do what it does and have the returns on intelligence that we're hoping to have and take us to a spot where we can thrive, right? But he uses the GDP example, right? And this is always a great refresher just to understand and know, but like the GDP per capita in the US, 75,000 bucks, know, sub-Saharan Africa is still at 2000 bucks, right? So his idea is like, can AI help close the gap? Well, theoretically it can, but humans get in the way still, right?

The flywheel of corruption

Jeremy Gilbertson So have you ever read a book called the economic hit man? Man, it was a really, really interesting book at how, um, how these developing countries and nations are controlled by larger nations, by those larger nations, inserting leaders and dictators and that sort of thing that are, that are corrupt and can be manipulated. it's, you always, you always think about that stuff happening, but that book really

Mark Fielding No, sounds good.

Jeremy Gilbertson outline was a guy who did it, you know, and he maintains, you know, he doesn't push his identity out there. But this stuff's kind of real, right? And social problems have high complexity. Can AI help the social problems, right? the social solutions to the social problems will let AI do the best things it can do for us. But are we at risk more?

Mark Fielding AI can't slow down the flywheel of corruption, can it? I mean, he says it himself.

If AI further increases economic growth and quality of life in the developed world while doing little to help the developing world, we should view that as a terrible moral failure and a blemish on the genuine humanitarian victories of the previous two. If we get to where he says with biology and mental health.

Jeremy Gilbertson You don't really hear Sam Altman saying things like that, do you?

Mark Fielding Well, I think for this, we should point people to Nexus. And I think we speak about peace and governments next. And we did like a 10 part chapter by chapter breakdown of Nexus by Yuval Noah Harari. And he speaks a lot of these things. And I think in short, too long didn't read this idea that the benefits of this technology will be available to all is a pipe dream based on a narrative that is not going to come to fruition because history has never, ever, ever shown us that that is possible. And when the powers that be, when it's in their interest not to share the seeds of progress with other nations, when it's in their political, financial, economic and cultural interest not to do it, they won't do it. And it's all good and well saying, AI, is path to freedom for everybody. But if the powers that be are putting that path up in space and you need a spaceship to go and get on the path to start walking it, you're just not having it. And I think that this is for all of the positivity and optimism of this essay, like everything is actually down to the human temperament at the end.

Jeremy Gilbertson Well, he references like there's, hard work to be done on that aspect of it. And he's the one, and I haven't done a deep dive in into, you know, Sam Altman's position and maybe some other people developing AI technologies position on this, but his seems to be at least with the awareness of the hard problems that have the hard problems are not, you know, how to make GPUs connect faster. The hard problems aren't how to, how to, you know,

Jeremy Gilbertson clean up the data to make the training the best. The hard problems are us. We're the bottom, we're the hard problem and the issue. and we're going to be here no matter what happens. So that's the, all right, here's the thing. Here's my hot take for you today. So we, we, we get with, the family of Alan Watts and we basically create a model that helps us remind us of our

Jeremy Gilbertson humanity focusing on empathy focusing on open mindedness focusing on gratitude and having models like that that can nudge us in that right direction or have pieces of these larger models what if there was a requirement for a piece of that in a larger model right because anyone can build a model based on anything someone can build a model to be an asshole right have this AI be an asshole we're gonna it's just like a kid you train the model you train it to be an asshole it's gonna be an asshole but if you instill and still the right stuff, not just how quickly you can calculate, know, logarithmic math or whatever the heck it is, right? But it's how much empathy and how much open-mindedness can you instill in these models? And I think that's a bigger thing, but a harder thing. It's messy. That's messy.

Mark Fielding I think it'd actually be easier to have our AIs have all of those facets, to be compassionate, to be empathetic, to be culturally aware, to have patience and empathy on a level that actually is really, really hard to put into the powers that be. I think that...

power corrupts ultimate power corrupts ultimately or power corrupts and ultimate power corrupts completely.

is true. And I think that maybe we asked this a lot on thinking on paper, like can AIs really be empathetic or is that just a mirage? Is that just there mimicking us? And

If they're mimicking us, if they're mimicking those behaviors, but those behaviors are actually acted on, maybe that's better.

Jeremy Gilbertson Well, maybe so all these models, every interaction is captured, right? By the model, like everyone that interacts with that model affects the future output of the model, right? So we all have people in our lives that affected us in good ways and affected us in bad ways. And sometimes great ways. Sometimes you have someone that comes into your life for a week that really changes your mind, really opens you up to something.

Jeremy Gilbertson even more powerful. So maybe the hope is at least some of those interactions are influencing the model unless for some reason the way the model is trained, someone can block those things from having consideration on the overall model, which again is like super scary.

Mark Fielding But this is all optimism. So I think you're right. And we spoke about the flywheel of corporate greed and power crazy people running in silent, but you also have the flywheel of economic growth. And it's been proven that economic growth increases all levels of empathy and compassion. And that's the flywheel. So as AI helps increase, well, I mean, yeah, it should fuel that.

Peace and governance

Jeremy Gilbertson Totally. So as we move into peace and governance, which is the next section, this like Noah Yuval Harari nexus thing comes up quite a bit. The tail end of this section, he references AI finance ministers and central bankers. To me, it goes back to trust, right? Number one is the country that is getting ready to employ an AI central banker.

Is it going to trust the model that someone's convincing it to employ for, for the AI? So like if I'm a smaller country, you're a bigger country. I've had, I have this great resource that, that I have access natural resource, right. And you want my natural resource. You might position an AI central banker or an AI finance minister that convinces me to give all my shit to you. So it's like that, that, that is what kind of goes through my head, but let's.

Let's move to peace and governance. think it's all in the same line. What did you get out of this section?

Mark Fielding that this well, I thought this is the the glue that sticks that keeps all the others together. So in in in machines of love and grace, we get in 510 shrunken years, we get to this place where AI has everything goes right. And everyone lives longer, there is no disease, there is economic freedom for everybody, there is biological freedom, whatever that means for everybody. And that all hinges on the governance. He says so himself. He also says that he is not as optimistic about this one. He says, I see no strong reason to believe AI will preferentially or structurally advance democracy in peace in the same way that I think it will structurally advance human health and alleviate poverty. Then he speaks about his Attente strategy in which a coalition of democracies seeks to gain a clear advantage, even just a temporary one, on powerful AI by securing its supply chain, scaling quickly and blocking or delaying adversaries. What if our enemies are more advanced than us? What if the non-democratic world is more advanced than we are? Then what happens?

Atoms for Peace, and who leads

Jeremy Gilbertson Yeah, I enjoyed the callback to Atoms for Peace and Eisenhower when like the idea of atomic weapons first came up and it was like, shit, like things are gonna, things are different and we gotta really think about this stuff. think this is very similar to that in a lot of ways and even more impactful, right? So I immediately thought like now there's like a nuclear.

coalition or commission like I'm not sure what the global org is for Making you know doing checks like UN has something there's a there's another org that has that checks on You know what's going on from a nuclear capacity or capability across the world? Do we are we gonna have like a central monitoring agency for that and who is it and who? Who leads it into what benefit?

Mark Fielding Well, I'll turn that question back on you then. Who does lead it?

Jeremy Gilbertson Oh, that's an easy, that's an easy volley mark. it's difficult, it's difficult, right? It's difficult. So let's talk about democracies. You referenced democracies in this Entente strategy is like, all the democracies get together. We, we try to control the, the, the, the access and scaling of these systems through availability of chip sets and hardware to, um, you know, to other things that you can try and control stuff. then

You know, you yourself are using AI for militaristic stuff, right? Everyone else kind of is doing that. and then it becomes this carrot stick thing that he talked about, right? So, so the, so the stick is all of these democratic nations pulled together are using it to develop the stick to wield whatever kind of militaristic advantage AI gives us. And then there's the carrot. Hey, we figured out how to, how to, you know, get our arms around this technology and scale it, but now we're gonna give you access to it only if you jump on board with how we do our thing, right? So it all sounds fine and good, but like who are the people doing that? And did they need oversight? And a lot of people, especially in the States, whenever you say oversight, people freak out and they're like, no, free market, it's all of this stuff. But dude, something's as big as this.

Some things as powerful as, this is more powerful, I think, than nuclear weapons were back in the 30s, 40s, 50s when that was all coming up. And we've determined that those need global oversight and control.

Mark Fielding Let's assume for the sake of this essay that we overcome that hurdle, government, and peace AI solves that little riddle that's plagued us since we first stepped foot out of the lake and evolved into some kind of humanoid thing.

If AI solves everything, what is left

Mark Fielding It raises a question that in this world where AI has solved all of our problems, cured all of our diseases, what the hell do we do?

Jeremy Gilbertson Yeah, so the way I thought about that is like AI is helping us take the elevator up globally. Maslow's hierarchy of needs, right? So we're figuring out food, shelter, keeping us safe, healthy, all of that stuff. When you have those basic needs figured out, then you start thinking about meaning and purpose. But guess what? Meaning and purpose, there's no easy button to meaning and purpose, right? So that's like...

Jeremy Gilbertson That's the interesting work, that's the hard work we can do as individuals, as humans, as all of that stuff. But that would be really cool if all the other things were generally solved and we're able to point towards meaning and purpose. And I use meaning and purpose as this just general collective of like figuring out the stuff beyond, hey, I've got to work today to make sure I have a sandwich at night kind of thing.

Meaning, and the economic problem

Mark Fielding Yeah, the economic piece is much more difficult than the meaning piece. mean, what do we do? Like people can...

find meaning. could... People like doing stuff. People don't care if like there's people who are better than them at something. You don't stop doing something just because you're not the best in the world. Whatever your hobbies or interests are, you do them because they bring you some kind of meaning. They bring you some kind of pleasure regardless. And it would be a very strange world if there was only the best do anything. So people find enjoyment, but...

UBI has been floated around, UBI has been, depending on who you read, it's a cure all or it will be a complete failure. Like how do you fix the economic side of meaning? I want someone, how do I pay to do all my hobbies all the time? I don't have to work anymore. So how do I pay for that?

Jeremy Gilbertson It's really interesting. read it. read a book a long time ago. it was, it was definitely a communist kind of system idea. think it was called on the conquest of bread. And actually I might have people knock on my door right now, just because I read that book and tell me I'm in trouble. But you know, it's the idea of like collective farming back in the day where the efficiency of that system.

pulled together actually creates, if it's done the right way, actually creates time for leisure, time to invest in things that you wouldn't normally have time to do because your basic stuff is taken care of. But the way collective farming works, you still have to work. You still gotta put your energy in in the farm, but then you have more time outside of that, because collectively everything is pulled together. Again.

Mark Fielding Collective farming didn't work in Russia did it Nexus chapter 7 check it out. Yeah the flip side of that

Jeremy Gilbertson That's because of people maybe. Theory when applied through people. I don't know. It's funky, man.

A capitalist economy of AI systems

Mark Fielding I liked his solution for the economic or one potential solution for the economic side of this.

It could be a capitalist economy of AI systems which then give out resources, huge amounts of them, since the overall economic pie will be gigantic. They'll give this out to humans based on some secondary economy of what the AI systems think makes sense to reward in humans based on some judgment ultimately derived from human values. I think you should have a question mark after that.

Jeremy Gilbertson Is this when we go into like the wiffle points or the wiffy points?

Mark Fielding Wow, that's really, yeah, reputation. Reputation points on an internet-based society is a load of rubbish for me. Yeah, it's bullshit.

Jeremy Gilbertson social scoring and all that crazy stuff. Yeah, that was a book he referenced, I think like a sci-fi book about Disneyland or something. It was interesting. we talked to Ram in our episode about kind of post-labor economics. And I think he's kind of, that's where this is kind of going, right? Which would require an entire reinvention of how society is laid out. And when you think about it, like, how the hell is that gonna happen? But then look, dude, over the years, we've gone from hunter-gatherers to farmers. We've gone from, as he lists out farmers, to like a feudalistic economy, and then from feudalism to industrialism as he outlays, you know, what's next? So we have evolved in the past.

Mark Fielding Yeah, but how many centuries did that take? How many millennia did that shift to farming take? How many centuries did it take? And so on the one hand, he's saying, yes, we can do that. But on the other hand, we're talking five to 10 years. It doesn't compute. It doesn't add up.

Jeremy Gilbertson Well, he talks about scaling, right? Things happening 10x faster, not happening in 100 years, but happening in 10 years. So maybe that's the idea there.

Mark Fielding Yeah, but he's talking about the outward, not the inward. He's not talking about like the, like you mentioned it earlier, didn't you? Like the societal change, the cultural change, the changes that are needed within us. And maybe this is the slack in the system. The problem with all this is that with AIs, we create this world and we're simply not ready for it. And so you have however much time it takes for us to be ready. I mean, how many centuries was industrial revolution? How short can we make it? 10 years, 50 years, 100 years, can we do it in 100 years? So if what he promises takes 20 years, can we as a society prepare ourselves in 100 years? is that, what's the ratio here? Collateral damage, what's the collateral AI damage of all of this?

Jeremy Gilbertson Yeah, I think what I'm through all of this is tech's moving, the technology for this is moving way quicker than our ability to understand the effects of it and how it's gonna change the world. It's going to. And it's like one of those things, I wanna talk to the people that are thinking about those things as in depth as the people that are building the technology.

And if you are one of those people, we would love to talk to you or you know one of those people that's been working on that stuff, you know, that we'd love to kind of chat with you about it. But as we quick.

How long did it take us to assimilate the car?

Mark Fielding How long did it take us to assimilate the car?

There's a technology that changed everything. How long did it take us to assimilate that? 40 years? I don't know.

Jeremy Gilbertson That's a good question.

Mark Fielding I mean, we've assimilated the internet, I guess, so that took, what, 20, 25?

Jeremy Gilbertson Yep. So when you talk about assimilate is more the adoption of it and the widespread use live with it.

Mark Fielding I'd to live with it as part of this new paradigm of humans plus cars, humans plus internet, humans plus the systems AIs create. How long does that take?

Jeremy Gilbertson We're about to find out, man. Hang tight, buckle your seat belts, friends and neighbors. So let's move into work and meaning, but let's finish this last little bit on peace and governance. And I really think if you're interested in learning his perspective on that, I would listen to our Nexus book club, No Yuval Harari. Listen to those episodes. It's pretty much aligned with what he says in here with, you know, the repressive governments thrive on lack of information and what can AI do to to to to counter that but what what could also AI do to help the dictators help the repressive government stay repressive so that that's the thing and then in law and interpretation this was the one quick quote that I that I pulled out of this that I thought was that was pretty interesting he said AI is the first technology to make broad fuzzy judgments in a repeatable and mechanical way and why

I thought that was interesting is is as it applies to laws and judgments upon laws, right? Because laws were created by people right consensus of people laws are interpreted by people which has ton of subjectivity which which helps right because you know people acting under the laws It's not always a yes or no. Maybe there's an in-between state but sometimes it is and he talks about the idea of like AI can potentially alert to biases in decision making, which could be interesting how you formulate the ability to, now we're getting into like freaking like minority report, pre-cog kind of stuff a little bit, but it definitely is interesting because right now social media will monitor for when we see human injustices happen, we monitor for that.

And now the attention to that is at scale. But again, there's not much that people can do other than look at and go, damn, that sucks. I can't believe this is happening. what if there's a technology to not only just throw it out there, but to kind of automate calling it out and kind of excising it from the experience.

Mark Fielding Excising bias? That would be ironic, wouldn't it? If AI helped us exercise bias.

Jeremy Gilbertson And he teased the idea of constitutional AI, I think I want to get into. And that's a whole nother episode, I think. And if we can get, there's a very talented woman at Anthropic that I listened to on another show. Amanda, I think is her name, I want to say. But she actually is the, she's a philosopher that is actually training and interacting with Claude specifically on a lot of these things and talking about.

what constitutional AI could be. So let's leave that at that, unless you have anything else on the governance piece.

Mark Fielding No, I think that the alignment Russian style, I think we covered a lot of the good stuff in Nexus. So yeah, I'm all right.

What is meaningful work?

Jeremy Gilbertson Yeah, work and meaning. What's meaningful, Mark?

Mark Fielding Well, horses for courses, it depends.

Mark Fielding Personally, I could spend the rest of my life quite easily in pursuit of personal and family experiences and have great fun doing it. But we already spoke about this. Who pays for this? The economics versus the meaning. How do you find meaning? How do you find meaning when...

your work is taken away from you. think that's a bit, that's the big question here. How do you find meaning when your work has been taken away from you? So you have to change how you think about your meaning. Like, like I think.

the human science of it says that your real meaning when you're on your deathbed and you're looking back, and I think that's how you measure what meaning really is. It's your connections, it's the human contact, it's your friends, it's your family, it's how you spent your time, who you spent your time with rather than what you spent your time doing to a certain degree. So that's meaning.

Jeremy Gilbertson So that's really interesting what you said there, the idea of what do you have?

with like, you're, let me back up here. So you talked about meaning, you talked about work and the things beyond work that are meaningful, Which are your relationships, your connection, maybe a sense of accomplishment, that sort of thing. But I think the older generation, I noticed this with my dad particularly because he worked for one company for 38 years, did a hell of a job doing that, put a lot of time and energy into that to.

you know, to give us what we had grown up. But you think about how much of his identity was, was tied to that, right? That when he retired, like that was it, that's probably a good example of it. It's like when you retired and you're so much of you was identified with this work, you, where is the rest and where is the purpose and the meaning and that sort of thing. And it becomes tricky, I think for the generation before us, because

There was more of that, I think, than there is now.

Mark Fielding We can't measure happiness, would he have been happier than a 25 year old who will never know what it's like to work for one company for all your life because his links were so deep, because his relationships were so deep, because the people that he knew, he'd known for 40 years because they shared moments, they shared the good times, they shared the bad times and it transcends the workplace. I'm sure that...

when your dad looked looks back on that. It's not really the job that he'll remember fondly is it?

Jeremy Gilbertson It's the networks, it's the relationships, it's the thing, the interactions, the things that he was able to accomplish. it's maybe it's the same thing, but when you retire that piece, sometimes it's taken from you, right? Maybe it's a, maybe it's a forced retirement thing in certain cases or, whatever it is. But I just, I don't know. I thought about that story.

Mark Fielding We don't have to retire anymore. We don't have a we won't have a job to retire from. But if we do are fortunate enough to have a job to retire from in 40 years, we won't need to retire because we'll be living to 120 like marathon runners. So retirement will retire that word.

Jeremy Gilbertson So there was a question in here that popped up, or the idea of like, how many things do you do each day with zero economic value? And I started thinking, I've always thought about this, because there's value in things that, value in these interactions that aren't necessarily tied to monetary return. But they have impact, like this show, right? Like this show. You have an impact on

Mark Fielding like this show.

Jeremy Gilbertson Yeah, someone's understanding of technology. Someone's, you we, you know, go in three, four or five levels deep in meaning and discussion and application and that sort of thing. But the quantification of non economic value is really interesting. And what could that look like in a post powerful AI world? That to me is the interesting piece.

Jeremy Gilbertson and what things won't be economically rewarded. What things do we do now that we get rewarded economically from that we won't be able to in the future?

Jeremy Gilbertson In general, the whole, of it, all of it.

Mark Fielding All work. I mean, I think that machines of loving grace...

many of the conversations we have of late. It's incredibly...

This is where the tech companies want to go on the whole, this picture that he paints with machines of Louis and Grace is where he mentions the culture, the E banks, sci-fi at the end, like that kind of, that's where we wanna go. And if they wanna go there, if they're gonna build their own future and this is where they wanna go and essays like this will help, hopefully.

It just always comes back to like the human in the loop.

Mark Fielding is the weakest link and yet the human in the loop at some point has to make a decision.

philosophically for where this goes. And I don't know how we solve that.

Jeremy Gilbertson It is a funky cycle.

I wrote this question down, like what happens when, and I think this was a quote, but I asked the question, what happens when most humans can contribute, what, all right, what happens when, quote, most humans can contribute meaningfully in a sufficiently advanced AI society, unquote, end quote.

Mark Fielding then we all sail off into the future happy, content, secure, knowledgeable, the complete augmented human.

Closing thoughts

Jeremy Gilbertson As we move up the high level of abstraction, Packy McCormick. So, so much for the pocket edition, Mark. We're, we're, we're, this is, this is the giant sack of books edition. Um, but let's, let's try to, let's try to land this plane. Here's another quote for you. It's the experience of watching a long held set of beliefs. I'm sorry, a long, let me re quote this guy. So a quote to.

Jeremy Gilbertson to wrap us up here in this just from a closing thought perspective as he landed the plane of the essay, quote, the experience of watching a long held set of ideals materialize in front of us all at once. It's pretty mind blowing right there. we're, the popcorn out. Like we're gonna watch and see a lot of things that have been.

a long part of our life and world and society and that sort of thing shift in very new ways. And I think a lot of people don't realize how big the shift is going to be, but it could be the biggest that humans have ever seen.

Mark Fielding Yeah, agreed. He starts the essay with saying that most people underestimate the potential upside of AI and the same people probably underestimate the potential risk of AI, but whichever side the coin lands. it's... You ain't seen nothing yet.

Jeremy Gilbertson And Dario Amodei, I absolutely agree with you that we have to work harder, fiercely at the human side of this thing to make sure the tech side has the effects that we want it to have. Because if we don't spend time on the human side, if we don't figure out the implications of this stuff, we are gonna be in a bad way. So kudos to you for putting these things out into the world. Come on the show and unpack them in real time.

That's all I have on this one. This has been a review pocket edition of Machines of Loving Grace by Dario Amodei. Mark, closing thoughts on your side.

Mark Fielding Closing thoughts on my side, I'll leave you with the verse from the original, all watched over by machines of loving grace. I like to think, and the sooner the better, of a cybernetic meadow where mammals and computers live together in mutually programming harmony. Let's hope we get there. Sounds like the future that we want to build, not the opposite.

Jeremy Gilbertson Off we go. Be curious. Stay disruptive.

Mark Fielding keep thinking on paper.